Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bin-contigs-metabat fails if no MAGs are formed for a sample #27

Open
gregcaporaso opened this issue Apr 4, 2023 · 0 comments
Open
Labels
bug Something isn't working

Comments

@gregcaporaso
Copy link
Contributor

gregcaporaso commented Apr 4, 2023

If no MAGs are formed for a sample, bin-contigs-metabat fails with a validation error:

[bam_sort_core] merging from 1 files and 1 in-memory blocks...
Output depth matrix to /tmp/tmpgg2uftbn/KS_depth.txt
jgi_summarize_bam_contig_depths 2.15 (Bioconda) 2020-01-04T21:10:40
Output matrix to /tmp/tmpgg2uftbn/KS_depth.txt
0: Opening bam: /tmp/tmpgg2uftbn/KS_alignment_sorted.bam
Processing bam files
Thread 0 finished: KS_alignment_sorted.bam with 3838648 reads and 219961 readsWellMapped
Creating depth matrix file: /tmp/tmpgg2uftbn/KS_depth.txt
Closing most bam files
Closing last bam file
Finished
MetaBAT 2 (2.15 (Bioconda)) using minContig 2500, minCV 1.0, minCVSum 1.0, maxP 95%, minS 60, maxEdges 200 and minClsSize 200000. with random seed=1680628081
0 bins (0 bases in total) formed.
Traceback (most recent call last):
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/q2cli/commands.py", line 352, in __call__
    results = action(**arguments)
  File "<decorator-gen-40>", line 2, in bin_contigs_metabat
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
    outputs = self._callable_executor_(scope, callable_args,
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/sdk/action.py", line 408, in _callable_executor_
    artifact = qiime2.sdk.Artifact._from_view(
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/sdk/result.py", line 349, in _from_view
    result = transformation(view, validate_level)
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/core/transform.py", line 68, in transformation
    self.validate(view, level=validate_level)
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/core/transform.py", line 143, in validate
    view.validate(level)
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/plugin/model/directory_format.py", line 177, in validate
    getattr(self, field)._validate_members(collected_paths, level)
  File "/home/gcaporaso/mambaforge/envs/q2-shotgun/lib/python3.8/site-packages/qiime2/plugin/model/directory_format.py", line 109, in _validate_members
    raise ValidationError(
qiime2.core.exceptions.ValidationError: Missing one or more files for MultiFASTADirectoryFormat: '.+\\.(fa|fasta)$'

Plugin error from moshpit:

  Missing one or more files for MultiFASTADirectoryFormat: '.+\\.(fa|fasta)$'

See above for debug info.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: samtools sort /tmp/qiime2/gcaporaso/data/8a63aaeb-3bfc-4a2c-b3f3-6cfedfcf6a7a/data/KS_KS_All13-C0500000_alignment.bam -o /tmp/tmpgg2uftbn/KS_alignment_sorted.bam

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: jgi_summarize_bam_contig_depths --outputDepth /tmp/tmpgg2uftbn/KS_depth.txt /tmp/tmpgg2uftbn/KS_alignment_sorted.bam

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: metabat2 -i /tmp/qiime2/gcaporaso/data/c227e281-0838-4bfd-ac40-2d78f2e471e9/data/KS_KS_All13-C0500000_contigs.fa -a /tmp/tmpgg2uftbn/KS_depth.txt -o /tmp/tmpgg2uftbn/KS/bin --numThreads 40

I'm not sure what the right way to handle this internally is - e.g., the whole command fails, or no MAGs are provided for that sample - but we should catch and handle this case with a more informative error message.

UPDATE: This looks to actually be an issue with how the sample ids are obtained from the filenames, similar to what we discovered on bokulich-lab/q2-assembly#37. I am testing this now and will have a PR if it works.

gregcaporaso added a commit to gregcaporaso/q2-moshpit that referenced this issue Apr 4, 2023
misialq pushed a commit that referenced this issue Jun 16, 2023
* FEAT: started work on eggnog annotation stuff.

* IMP: linting....next to move dependencies

* IMP: just adding eggnog stuff

* IMP: just getting started on adding mapper.

* FEAT: initial pass at reference db downloader.

* IMP: more work on ancillary database downloader

* IMP: getting closer.

TODO: binarytextfile format to store reference database. Rewire.
Possibly multifile directory format and chose between name based on the
type of reference being created. Change subprocess stdout and stderr to
PIPE so that errors from there are captured by QIIME 2's logging
machinery.

* IMP: clean up trying to get tests passing.

TODO next time, figure out why not downloader not raising errror when
given mixed taxa types

* BUG: downloader functional/tests passing

* TEST: Tests for taxa checker utility.

* TEST: fixing failure from incorrect setup....

* TEST: skipping tests with actual downloads

* BUG: fixing linting

* BUG: fixing linting

* IMP: cleaning somethings up....

* YEP: getting there

* IMP: Formats & Types updated and sorted, started combining the
downloaders.

* FEAT: diamond_search method

* IMP: implementing functionality for search_diamond

* FEAT: eggnog_diamond_search now working?

* FEAT: diamond seed ortholog search for eggnogmapper

* LINT: cleaning up first draft

* FEAT: add eggnog_annotate_seed_orthologs

* FEAT: eggnog annotation working!

* FEAT: starting to add usage examples

* FEAT: added multi-cpu utilization

* IMP: linting

* FEAT: add read eggnog database into memory.

* GETTING q2-types-genomics and moshpit on same page

* reorganizing for just eggnog stuff

* IMP: dependency specification update.

* FEAT: Generate FT on eggnog diamond search

* EOD MONDAY

* BACKUP before cleanup

* IMP: fixing linting issues

* BUG: remove artifacts from merge

* TEST: added test data/reference artifacts and a basic test for eggnog
diamond mapper.

* LINT: cleanup test commit

* ONLY LOCAL FAILING are not eggnog related

* lint setup

* added dependency on q2_types_genomics

* make types genomics available?

* TEST: added general test to eggnog annotater.

* add --dbmem parameter to address issue with very long runtime

* TEST: revert imports and fix test_small_good_hits

* fixes incomplete extraction of sample ids from filenames

addresses #27

* IMP: preparing for merge, lint, etc

* BUG: linting errors in metabat2

* TEST: Updating metabat2 tests to be compatible w/ @greg-caporaso 's
suffix based sample name extraction pr

* linting and fixing more test bugs, maybe one left 🤷‍♂️

* more test fixes

* IMP: adding `.DS_Store` to .gitignore

* IMP: removing `.DS_Store` files

* addresses @ebolyen's comments, removed qza test data in favor of raw inputs

* lint

* small test file issue

* squash

* squash

* run tests on ubuntu only

* sta gitus

* Update ci/recipe/meta.yaml

* test OS X again

* Update ci/recipe/meta.yaml

* Update ci/recipe/meta.yaml

---------

Co-authored-by: Greg Caporaso <[email protected]>
Co-authored-by: Greg Caporaso <[email protected]>
Co-authored-by: Colin Vickers Wood <[email protected]>
Co-authored-by: colinvwood <[email protected]>
Co-authored-by: Evan Bolyen <[email protected]>
Co-authored-by: Evan Bolyen <[email protected]>
@misialq misialq added the bug Something isn't working label Aug 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants