NFTest cases fail due to schema validation errors

[nwiltsie]

Describe the bug
None of the NFTest cases make it through configuration:

2024-01-03 22:24:46,830 - NFTest - INFO - Beginning tests...
2024-01-03 22:24:46,830 - NFTest - INFO - a_mini_n2-spark: test development branch with a-mini-n2 and Spark's mark duplicates
...
2024-01-03 22:24:51,546 - NextFlow.syslog - ERROR - Unable to parse config file: '/hot/code/nwiltsie/pipelines/pipeline-align-DNA/./test/a_mini_n2-spark.config'

  Config file invalid. Required parameter read_group_identifier is missing.
...
2024-01-03 22:24:52,583 - NFTest - ERROR -  [ failed ]
2024-01-03 22:24:53,097 - NFTest - INFO - a_mini_n2-picard: test development branch with a-mini-n2 and Picard's mark duplicates
...
2024-01-03 22:24:57,267 - NextFlow.syslog - ERROR - Unable to parse config file: '/hot/code/nwiltsie/pipelines/pipeline-align-DNA/./test/a_mini_n2-picard.config'

  Config file invalid. Required parameter read_group_identifier is missing.
...
2024-01-03 22:24:58,299 - NFTest - ERROR -  [ failed ]
2024-01-03 22:24:58,512 - NFTest - INFO - a_mini_n2-picard_yaml: test development branch with a-mini-n2 and Picard's mark duplicates with a YAML config file
...
2024-01-03 22:25:02,524 - NextFlow.syslog - ERROR - Unable to parse config file: '/hot/code/nwiltsie/pipelines/pipeline-align-DNA/./test/a_mini_n2-picard_yaml.config'

     ### ERROR ###   The input directory params.work_dir: /scratch is not writeable. Please verify and try again.
...
2024-01-03 22:25:03,539 - NFTest - ERROR -  [ failed ]

(Don't worry about the Nextflow.syslog module, I'm using a development version of NFTest to work around uclahs-cds/tool-NFTest#56. It'll show up in a PR soon).

The input directory params.work_dir: /scratch is not writeable error for a_mini_n2-picard_yaml is simple and fixable by removing the local copy of methods.set_env(), but the Config file invalid. Required parameter read_group_identifier is missing. errors seem spurious.

If I remove these two lines from the config (and patch the /scratch thing) then a_mini_n2-picard passes (a_mini_n2-spark still fails for some other reason):

pipeline-align-DNA/config/methods.config

Lines 331 to 332 in 88ab6e0

	schema.load_custom_types("${projectDir}/config/custom_schema_types.config")
	schema.validate()

All of these tests should work out-of-the-box. I'm not sure if the fix is to add additional parameters to the config files or to tweak the schema.

• Pipeline release version: Current main (88ab6e0)
• Cluster you are using: Slurm
• Node type: Any F-series
• Submission method: nftest run and submit_nftest.py
• Actual submission script: /hot/code/nwiltsie/pipelines/pipeline-align-DNA/dotests.sh
• Sbatch or qsub command and logs if applicable:

sbatch \
    --exclusive \
    --partition='F16' \
    -J 'sbatch-nftest-20240103T222442Z' \
    -e 'sbatch-nftest-20240103T222442Z.error' \
    -o 'sbatch-nftest-20240103T222442Z.log' \
    --wrap="if [ ! -d /hot/software/pipeline/$(basename -s .git `git -C /hot/code/nwiltsie/pipelines/pipeline-align-DNA config --get remote.origin.url`)/Nextflow/development/unreleased/$(git -C /hot/code/nwiltsie/pipelines/pipeline-align-DNA branch --s
how-current) ]; then mkdir -p /hot/software/pipeline/$(basename -s .git `git -C /hot/code/nwiltsie/pipelines/pipeline-align-DNA config --get remote.origin.url`)/Nextflow/development/unreleased/$(git -C /hot/code/nwiltsie/pipelines/pipeline-align-DNA branch --show-current); fi && cd /hot/code/nwiltsie/pipelines/pipeline-align-DNA && export NFT_OUTPUT=/hot/software/pipeline/$(basename -s .git `git -C /hot/code/nwiltsie/pipelines/pipeline-align-DNA config --get remote.origin.url`)/Nextflow/development/unreleased/$(git -C /hot/code/nwiltsie/pipelines/pipeline-align-DNA branch --show-current) && nftest run " \
    --mail-user '[email protected]' \
    --mail-type=ALL

/hot/software/pipeline/pipeline-align-DNA/Nextflow/development/unreleased/dependabot/submodules/external/nextflow-modules-b0b2798/log-nftest-20240103T222446Z.log

• Config files: The standard NFTest ones
• Path to the working directory: (It didn't get this far)

[yashpatel6]

It seems like the input CSV file has BOM characters, causing the param validation to fail since the first column name now includes those characters:

Not sure how they ended up there now and haven't shown up before when the tests were initially implemented though

[yashpatel6]

I've updated the CSV files and the tests should pass now, I ran the test past the schema.validate() call