MultiQC is not consolidating QUAST reports for assemblies

[amizeranschi]

Hi,

As the title mentions, with more recent versions of the pipeline (tested with 3.2.1 and 3.3.0), it seems that MultiQC is not consolidating QUAST reports for assemblies, which used to be the case with older pipeline versions like 3.0.2.

Is this expected behavior, or is it a bug?

To reproduce:

nextflow run nf-core/mag -r 3.0.2 \
-profile docker,test \
--skip_spades \
--skip_prodigal \
--skip_prokka \
--skip_metaeuk \
--skip_binning \
--skip_gtdbtk \
--outdir test-mag-3.0.2

ls -1 test-mag-3.0.2/multiqc/multiqc_data/multiqc_quast.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_quast.yaml

ls -1 test-mag-3.0.2/multiqc/multiqc_data/
# test-mag-3.0.2/multiqc/multiqc_data/multiqc.log
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_Kraken2_Kraken2.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_bowtie2.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_citations.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_data.json
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_fastp.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_fastqc.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_fastqc_1.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_general_stats.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_quast.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_software_versions.yaml
# test-mag-3.0.2/multiqc/multiqc_data/multiqc_sources.yaml


nextflow run nf-core/mag -r 3.3.0 \
-profile docker,test \
--skip_spades \
--skip_prodigal \
--skip_prokka \
--skip_metaeuk \
--skip_binning \
--skip_gtdbtk \
--outdir test-mag-3.3.0

ls -1 test-mag-3.3.0/multiqc/multiqc_data/multiqc_quast.yaml
# ls: cannot access 'test-mag-3.3.0/multiqc/multiqc_data/multiqc_quast.yaml': No such file or directory

ls -1 test-mag-3.3.0/multiqc/multiqc_data/
# test-mag-3.3.0/multiqc/multiqc_data/multiqc.log
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_Kraken2_Kraken2.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_bowtie2.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_citations.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_data.json
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_fastp.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_fastqc.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_fastqc_1.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_general_stats.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_software_versions.yaml
# test-mag-3.3.0/multiqc/multiqc_data/multiqc_sources.yaml

[jfy133]

What do you mean by consolidating @amizeranschi ? As in there is no multiQC output sections for QUAST?

[amizeranschi]

Yes, that is indeed what I mean. The multiqc_quast.yaml doesn't get generated, and the corresponding section is missing from the MultiQC HTML report.

[jfy133]

I was able to reproduce the error - interestingly the QUAST sections are in the MultiQC report if you don't skip things

[jfy133]

I think it might be --skip-binning that is the issue - however the QUAST files are in the channel going into multiQC

[jfy133]

Update, not even - the QUAST folders Are in the MultiQC directory in this case... so question is why MultiQC isn't rendering them

[jfy133]

.
├── 1
│   └── methods_description_mqc.yaml -> /home/james/git/nf-core/mag/testing/work/tmp/69/36ed8b9b5117dcaf90b5269c2e4177/methods_description_mqc.yaml
├── 10
│   └── test_minigut_sample2_run0_trimmed_1_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/c3/7dcce45f4c51b5c1c6558369f77e69/test_minigut_sample2_run0_trimmed_1_fastqc.zip
├── 11
│   └── test_minigut_sample2_run0_trimmed_2_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/c3/7dcce45f4c51b5c1c6558369f77e69/test_minigut_sample2_run0_trimmed_2_fastqc.zip
├── 12
│   └── test_minigut_run0_phix_removed.bowtie2.log -> /home/james/git/nf-core/mag/testing/work/d7/a8daf47f717db708cd552c94f28d0a/test_minigut_run0_phix_removed.bowtie2.log
├── 13
│   └── test_minigut_run0_trimmed_1_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/41/3bfdb90991d6585e17f1d6e8d4e700/test_minigut_run0_trimmed_1_fastqc.zip
├── 14
│   └── test_minigut_run0_trimmed_2_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/41/3bfdb90991d6585e17f1d6e8d4e700/test_minigut_run0_trimmed_2_fastqc.zip
├── 15
│   └── test_minigut.kraken2_report.txt -> /home/james/git/nf-core/mag/testing/work/65/f80b8da12bf93508868aa9c02c03dc/test_minigut.kraken2_report.txt
├── 16
│   └── test_minigut_sample2.kraken2_report.txt -> /home/james/git/nf-core/mag/testing/work/37/89fc31a778825cedbdb72418798dd6/test_minigut_sample2.kraken2_report.txt
├── 17
│   └── test_minigut_kreport.txt -> /home/james/git/nf-core/mag/testing/work/c0/0d965e829362993b5b10e30659e2f0/test_minigut_kreport.txt
├── 18
│   └── test_minigut_sample2_kreport.txt -> /home/james/git/nf-core/mag/testing/work/82/cec84f3f24215d99453bd37bd2b81a/test_minigut_sample2_kreport.txt
├── 19
│   └── nf_core_mag_software_mqc_versions.yml -> /home/james/git/nf-core/mag/testing/test-^Cg-3.3.0-skip/pipeline_info/nf_core_mag_software_mqc_versions.yml
├── 2
│   └── workflow_summary_mqc.yaml -> /home/james/git/nf-core/mag/testing/work/tmp/e3/50251862ac734ab6debdb80b124090/workflow_summary_mqc.yaml
├── 20
│   └── report_rawassemblies.tsv -> /home/james/git/nf-core/mag/testing/work/34/37fa1256f33f6afbd933bc01e928b2/QUAST/report_rawassemblies.tsv
├── 21
│   └── report_rawassemblies.tsv -> /home/james/git/nf-core/mag/testing/work/62/2b0473ba0ef07b1c33c2d43847444c/QUAST/report_rawassemblies.tsv
├── 22
│   └── report_rawassemblies.tsv -> /home/james/git/nf-core/mag/testing/work/6e/0109fcc63da605041b71d09dea44eb/QUAST/report_rawassemblies.tsv
├── 23
│   └── report_rawassemblies.tsv -> /home/james/git/nf-core/mag/testing/work/97/e4fc546a4306e813fba54694557901/QUAST/report_rawassemblies.tsv
├── 3
│   └── test_minigut_sample2_run0_fastp.fastp.json -> /home/james/git/nf-core/mag/testing/work/03/d24db41af1878bb99ddeb5ef216c87/test_minigut_sample2_run0_fastp.fastp.json
├── 4
│   └── test_minigut_run0_fastp.fastp.json -> /home/james/git/nf-core/mag/testing/work/72/2c7858c1f6aee10f49081801a1a583/test_minigut_run0_fastp.fastp.json
├── 5
│   └── test_minigut_sample2_run0_raw_1_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/4d/21c3cb956e24740e84a47659956c79/test_minigut_sample2_run0_raw_1_fastqc.zip
├── 6
│   └── test_minigut_sample2_run0_raw_2_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/4d/21c3cb956e24740e84a47659956c79/test_minigut_sample2_run0_raw_2_fastqc.zip
├── 7
│   └── test_minigut_run0_raw_1_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/3e/5c8a958e22c3393ab9a59414e7a99e/test_minigut_run0_raw_1_fastqc.zip
├── 8
│   └── test_minigut_run0_raw_2_fastqc.zip -> /home/james/git/nf-core/mag/testing/work/3e/5c8a958e22c3393ab9a59414e7a99e/test_minigut_run0_raw_2_fastqc.zip
├── 9
│   └── test_minigut_sample2_run0_phix_removed.bowtie2.log -> /home/james/git/nf-core/mag/testing/work/4b/d822d7e2502cd8e1fe49679a9498a7/test_minigut_sample2_run0_phix_removed.bowtie2.log
├── mag_logo_mascot_light.png -> /home/james/git/nf-core/mag/docs/images/mag_logo_mascot_light.png
├── multiqc_config.yml -> /home/james/git/nf-core/mag/assets/multiqc_config.yml
├── multiqc_data
│   ├── bowtie2_pe_plot.yaml
│   ├── fastp_filtered_reads_plot.yaml
│   ├── fastp-insert-size-plot.yaml
│   ├── fastp-seq-content-gc-plot_Read_1_After_filtering.yaml
│   ├── fastp-seq-content-gc-plot_Read_1_Before_filtering.yaml
│   ├── fastp-seq-content-gc-plot_Read_2_After_filtering.yaml
│   ├── fastp-seq-content-gc-plot_Read_2_Before_filtering.yaml
│   ├── fastp-seq-content-n-plot_Read_1_After_filtering.yaml
│   ├── fastp-seq-content-n-plot_Read_1_Before_filtering.yaml
│   ├── fastp-seq-content-n-plot_Read_2_After_filtering.yaml
│   ├── fastp-seq-content-n-plot_Read_2_Before_filtering.yaml
│   ├── fastp-seq-quality-plot_Read_1_After_filtering.yaml
│   ├── fastp-seq-quality-plot_Read_1_Before_filtering.yaml
│   ├── fastp-seq-quality-plot_Read_2_After_filtering.yaml
│   ├── fastp-seq-quality-plot_Read_2_Before_filtering.yaml
│   ├── fastqc-1_per_base_n_content_plot.yaml
│   ├── fastqc-1_per_base_sequence_quality_plot.yaml
│   ├── fastqc-1_per_sequence_gc_content_plot_Counts.yaml
│   ├── fastqc-1_per_sequence_gc_content_plot_Percentages.yaml
│   ├── fastqc-1_per_sequence_quality_scores_plot.yaml
│   ├── fastqc-1_sequence_counts_plot.yaml
│   ├── fastqc-1_sequence_duplication_levels_plot.yaml
│   ├── fastqc-1-status-check-heatmap.yaml
│   ├── fastqc-1_top_overrepresented_sequences_table.yaml
│   ├── fastqc_per_base_n_content_plot.yaml
│   ├── fastqc_per_base_sequence_quality_plot.yaml
│   ├── fastqc_per_sequence_gc_content_plot_Counts.yaml
│   ├── fastqc_per_sequence_gc_content_plot_Percentages.yaml
│   ├── fastqc_per_sequence_quality_scores_plot.yaml
│   ├── fastqc_sequence_counts_plot.yaml
│   ├── fastqc_sequence_duplication_levels_plot.yaml
│   ├── fastqc_sequence_length_distribution_plot.yaml
│   ├── fastqc-status-check-heatmap.yaml
│   ├── fastqc_top_overrepresented_sequences_table.yaml
│   ├── Kraken2-top-n-plot.yaml
│   ├── multiqc_bowtie2.yaml
│   ├── multiqc_citations.yaml
│   ├── multiqc_data.json
│   ├── multiqc_fastp.yaml
│   ├── multiqc_fastqc_1.yaml
│   ├── multiqc_fastqc.yaml
│   ├── multiqc_general_stats.yaml
│   ├── multiqc_Kraken2_Kraken2.yaml
│   ├── multiqc.log
│   ├── multiqc_software_versions.yaml
│   └── multiqc_sources.yaml
├── multiqc_plots
│   ├── pdf
│   │   ├── bowtie2_pe_plot-cnt.pdf
│   │   ├── bowtie2_pe_plot-pct.pdf
│   │   ├── fastp_filtered_reads_plot-cnt.pdf
│   │   ├── fastp_filtered_reads_plot-pct.pdf
│   │   ├── fastp-insert-size-plot.pdf
│   │   ├── fastp-seq-content-gc-plot_Read_1_After_filtering.pdf
│   │   ├── fastp-seq-content-gc-plot_Read_1_Before_filtering.pdf
│   │   ├── fastp-seq-content-gc-plot_Read_2_After_filtering.pdf
│   │   ├── fastp-seq-content-gc-plot_Read_2_Before_filtering.pdf
│   │   ├── fastp-seq-content-n-plot_Read_1_After_filtering.pdf
│   │   ├── fastp-seq-content-n-plot_Read_1_Before_filtering.pdf
│   │   ├── fastp-seq-content-n-plot_Read_2_After_filtering.pdf
│   │   ├── fastp-seq-content-n-plot_Read_2_Before_filtering.pdf
│   │   ├── fastp-seq-quality-plot_Read_1_After_filtering.pdf
│   │   ├── fastp-seq-quality-plot_Read_1_Before_filtering.pdf
│   │   ├── fastp-seq-quality-plot_Read_2_After_filtering.pdf
│   │   ├── fastp-seq-quality-plot_Read_2_Before_filtering.pdf
│   │   ├── fastqc-1_per_base_n_content_plot.pdf
│   │   ├── fastqc-1_per_base_sequence_quality_plot.pdf
│   │   ├── fastqc-1_per_sequence_gc_content_plot_Counts.pdf
│   │   ├── fastqc-1_per_sequence_gc_content_plot_Percentages.pdf
│   │   ├── fastqc-1_per_sequence_quality_scores_plot.pdf
│   │   ├── fastqc-1_sequence_counts_plot-cnt.pdf
│   │   ├── fastqc-1_sequence_counts_plot-pct.pdf
│   │   ├── fastqc-1_sequence_duplication_levels_plot.pdf
│   │   ├── fastqc-1-status-check-heatmap.pdf
│   │   ├── fastqc-1_top_overrepresented_sequences_table.pdf
│   │   ├── fastqc_per_base_n_content_plot.pdf
│   │   ├── fastqc_per_base_sequence_quality_plot.pdf
│   │   ├── fastqc_per_sequence_gc_content_plot_Counts.pdf
│   │   ├── fastqc_per_sequence_gc_content_plot_Percentages.pdf
│   │   ├── fastqc_per_sequence_quality_scores_plot.pdf
│   │   ├── fastqc_sequence_counts_plot-cnt.pdf
│   │   ├── fastqc_sequence_counts_plot-pct.pdf
│   │   ├── fastqc_sequence_duplication_levels_plot.pdf
│   │   ├── fastqc_sequence_length_distribution_plot.pdf
│   │   ├── fastqc-status-check-heatmap.pdf
│   │   ├── fastqc_top_overrepresented_sequences_table.pdf
│   │   ├── general_stats_table.pdf
│   │   ├── Kraken2-top-n-plot-cnt.pdf
│   │   └── Kraken2-top-n-plot-pct.pdf
│   ├── png
│   │   ├── bowtie2_pe_plot-cnt.png
│   │   ├── bowtie2_pe_plot-pct.png
│   │   ├── fastp_filtered_reads_plot-cnt.png
│   │   ├── fastp_filtered_reads_plot-pct.png
│   │   ├── fastp-insert-size-plot.png
│   │   ├── fastp-seq-content-gc-plot_Read_1_After_filtering.png
│   │   ├── fastp-seq-content-gc-plot_Read_1_Before_filtering.png
│   │   ├── fastp-seq-content-gc-plot_Read_2_After_filtering.png
│   │   ├── fastp-seq-content-gc-plot_Read_2_Before_filtering.png
│   │   ├── fastp-seq-content-n-plot_Read_1_After_filtering.png
│   │   ├── fastp-seq-content-n-plot_Read_1_Before_filtering.png
│   │   ├── fastp-seq-content-n-plot_Read_2_After_filtering.png
│   │   ├── fastp-seq-content-n-plot_Read_2_Before_filtering.png
│   │   ├── fastp-seq-quality-plot_Read_1_After_filtering.png
│   │   ├── fastp-seq-quality-plot_Read_1_Before_filtering.png
│   │   ├── fastp-seq-quality-plot_Read_2_After_filtering.png
│   │   ├── fastp-seq-quality-plot_Read_2_Before_filtering.png
│   │   ├── fastqc-1_per_base_n_content_plot.png
│   │   ├── fastqc-1_per_base_sequence_quality_plot.png
│   │   ├── fastqc-1_per_sequence_gc_content_plot_Counts.png
│   │   ├── fastqc-1_per_sequence_gc_content_plot_Percentages.png
│   │   ├── fastqc-1_per_sequence_quality_scores_plot.png
│   │   ├── fastqc-1_sequence_counts_plot-cnt.png
│   │   ├── fastqc-1_sequence_counts_plot-pct.png
│   │   ├── fastqc-1_sequence_duplication_levels_plot.png
│   │   ├── fastqc-1-status-check-heatmap.png
│   │   ├── fastqc-1_top_overrepresented_sequences_table.png
│   │   ├── fastqc_per_base_n_content_plot.png
│   │   ├── fastqc_per_base_sequence_quality_plot.png
│   │   ├── fastqc_per_sequence_gc_content_plot_Counts.png
│   │   ├── fastqc_per_sequence_gc_content_plot_Percentages.png
│   │   ├── fastqc_per_sequence_quality_scores_plot.png
│   │   ├── fastqc_sequence_counts_plot-cnt.png
│   │   ├── fastqc_sequence_counts_plot-pct.png
│   │   ├── fastqc_sequence_duplication_levels_plot.png
│   │   ├── fastqc_sequence_length_distribution_plot.png
│   │   ├── fastqc-status-check-heatmap.png
│   │   ├── fastqc_top_overrepresented_sequences_table.png
│   │   ├── general_stats_table.png
│   │   ├── Kraken2-top-n-plot-cnt.png
│   │   └── Kraken2-top-n-plot-pct.png
│   └── svg
│       ├── bowtie2_pe_plot-cnt.svg
│       ├── bowtie2_pe_plot-pct.svg
│       ├── fastp_filtered_reads_plot-cnt.svg
│       ├── fastp_filtered_reads_plot-pct.svg
│       ├── fastp-insert-size-plot.svg
│       ├── fastp-seq-content-gc-plot_Read_1_After_filtering.svg
│       ├── fastp-seq-content-gc-plot_Read_1_Before_filtering.svg
│       ├── fastp-seq-content-gc-plot_Read_2_After_filtering.svg
│       ├── fastp-seq-content-gc-plot_Read_2_Before_filtering.svg
│       ├── fastp-seq-content-n-plot_Read_1_After_filtering.svg
│       ├── fastp-seq-content-n-plot_Read_1_Before_filtering.svg
│       ├── fastp-seq-content-n-plot_Read_2_After_filtering.svg
│       ├── fastp-seq-content-n-plot_Read_2_Before_filtering.svg
│       ├── fastp-seq-quality-plot_Read_1_After_filtering.svg
│       ├── fastp-seq-quality-plot_Read_1_Before_filtering.svg
│       ├── fastp-seq-quality-plot_Read_2_After_filtering.svg
│       ├── fastp-seq-quality-plot_Read_2_Before_filtering.svg
│       ├── fastqc-1_per_base_n_content_plot.svg
│       ├── fastqc-1_per_base_sequence_quality_plot.svg
│       ├── fastqc-1_per_sequence_gc_content_plot_Counts.svg
│       ├── fastqc-1_per_sequence_gc_content_plot_Percentages.svg
│       ├── fastqc-1_per_sequence_quality_scores_plot.svg
│       ├── fastqc-1_sequence_counts_plot-cnt.svg
│       ├── fastqc-1_sequence_counts_plot-pct.svg
│       ├── fastqc-1_sequence_duplication_levels_plot.svg
│       ├── fastqc-1-status-check-heatmap.svg
│       ├── fastqc-1_top_overrepresented_sequences_table.svg
│       ├── fastqc_per_base_n_content_plot.svg
│       ├── fastqc_per_base_sequence_quality_plot.svg
│       ├── fastqc_per_sequence_gc_content_plot_Counts.svg
│       ├── fastqc_per_sequence_gc_content_plot_Percentages.svg
│       ├── fastqc_per_sequence_quality_scores_plot.svg
│       ├── fastqc_sequence_counts_plot-cnt.svg
│       ├── fastqc_sequence_counts_plot-pct.svg
│       ├── fastqc_sequence_duplication_levels_plot.svg
│       ├── fastqc_sequence_length_distribution_plot.svg
│       ├── fastqc-status-check-heatmap.svg
│       ├── fastqc_top_overrepresented_sequences_table.svg
│       ├── general_stats_table.svg
│       ├── Kraken2-top-n-plot-cnt.svg
│       └── Kraken2-top-n-plot-pct.svg
├── multiqc_report.html
└── versions.yml

[jfy133]

MultiQC does not recognise the QUAST outputs when skipping binning

[2025-01-22 09:34:59,087] multiqc.core.file_search                           [INFO   ]  Only using modules: fastqc, fastp, porechop, filtlong, fastqc, bowtie2, bowtie2, kraken, kraken, quast, bowtie2, custom_content, quast, busco, prokka
[2025-01-22 09:34:59,141] multiqc.modules.fastqc.fastqc                      [INFO   ]  Found 4 reports
[2025-01-22 09:34:59,279] multiqc.modules.fastp.fastp                        [INFO   ]  Found 2 reports
[2025-01-22 09:34:59,565] multiqc.modules.fastqc.fastqc                      [INFO   ]  Found 4 reports
[2025-01-22 09:34:59,612] multiqc.modules.bowtie2.bowtie2                    [INFO   ]  Found 2 reports
[2025-01-22 09:34:59,622] multiqc.modules.kraken.kraken                      [INFO   ]  Found 2 reports
[2025-01-22 09:34:59,636] multiqc.modules.custom_content.custom_content      [WARNING]  HTML plot type found with more than one dataset in nf-core-mag-summary
[2025-01-22 09:34:59,637] multiqc.modules.custom_content.custom_content      [INFO   ]  nf-core-mag-summary: Found 1 sample (html)
[2025-01-22 09:34:59,637] multiqc.modules.custom_content.custom_content      [WARNING]  HTML plot type found with more than one dataset in nf-core-mag-methods-description
[2025-01-22 09:34:59,637] multiqc.modules.custom_content.custom_content      [INFO   ]  nf-core-mag-methods-description: Found 1 sample (html)

[amizeranschi]

There was a MultiQC module update 3 months ago in October, could it have something to do with that?

[jfy133]

Will have to investigate

[jfy133]

This is what it's looking for: https://docs.seqera.io/multiqc/modules/quast#metaquast

[jfy133]

But we tweak it, and according to the verbose multiQC log:

While we have,

/home/james/git/nf-core/mag/testing/work/34/37fa1256f33f6afbd933bc01e928b2/QUAST/report_rawassemblies.tsv

It's actually looking for

[2025-01-22 09:50:40,050] multiqc.config                                     [DEBUG  ]  Added to filename patterns: [{'host_removal': {'fn': 'host_removal_metrics.tsv'}, 'adapterremoval': {'fn': '*_ar2.settings'}, 'kraken': {'fn_re': '.*[kraken2|centrifuge].*report.txt'}, 'quast': {'fn_re': 'report.*.tsv'}, 'filtlong': {'num_lines': 20, 'fn_re': '.*_filtlong.log'}}]

[jfy133]

Fix implemented now @amizeranschi !

[jfy133]

Just needs a review and merge