Replace input CSV with YAML and parse sample ID from BAM #106

Faizal-Eeman · 2023-07-25T08:25:16Z

Description

Replace CSV with YAML input
Parse YAML input and organize input channels
Parse sample ID from tumor BAM and use it to name pipeline output directory

Closes

The following related issues,
#103
#104
#105

Testing Results

DNA A-mini TWGSAMIN000001-T003-S03-F
- internal id: TWGSAMIN000001-N003-S03-F, TWGSAMIN000001-T003-S03-F.bam
- sample: HG002.N-0, S2.T-0 (sample names in BAM file)
- input YAML: /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/input/yaml/call-sSV-input-TWGSAMIN000001-T003-S03-F.yaml
- config: /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/TWGSAMIN000001-T003-S03-F.config
- output: /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/TWGSAMIN000001-T003-S03-F/call-sSV-5.0.0/S2_v1.1.5/

This is an edge case where the sample ID in the BAM is different from the internal sample ID. However in production, the sample ID in the BAM would be the same as the internal sample ID.

DNA real sample ILHNLNEV000009-T002-L01-F
- sample: ILHNLNEV000009-N001-B01-F, ILHNLNEV000009-T002-L01-F
- input YAML: /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/ILHNLNEV000009-T002-L01-F_F32.yaml
- config: /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/ILHNLNEV000009-T002-L01-F_F32.config
- output: /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/ILHNLNEV000009-T002-L01-F/

Checklist

I have read the code review guidelines and the code review best practice on GitHub check-list.
I have reviewed the Nextflow pipeline standards.
The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
I have set up or verified the branch protection rule following the [github standards](https://confluence.mednet.ucla.edu/pages/viewpage.action?spaceKey=BOUTROSLAB&title=GitHub+Standards#GitHubStanda
rds-Branchprotectionrule) before opening this pull request.
I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request, am listed
already, or do not wish to be listed. (This acknowledgement is optional.)
I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.
I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)
I have tested the pipeline on at least one A-mini sample with algorithm = ['delly', 'manta']. The paths to the test config files and output directories are attached above.

Alfredo-Enrique

Lots of good stuff Faizal!

Made some comments/questions!

config/methods.config

main.nf

config/schema.yaml

main.nf

Alfredo-Enrique · 2023-07-25T20:02:35Z

main.nf

+tumor_id_bam_bai = input_ch_samples_with_index
+    .filter{ it.sample_type == 'tumor' }
+    .map{ it -> [it.id, it.path, it.index] }
+normal_bam_bai = input_ch_samples_with_index
+    .filter{ it.sample_type == 'normal' }
+    .map{ it -> [it.path, it.index] }

+input_paired_bams_ch = tumor_id_bam_bai.combine(normal_bam_bai)


Mmm this main input_paired_bams_ch has 4 items correct? I thought the first input used by the SV caller modules intakes a tuple with 5 arguments? From Delly:

input: tuple(val(tumor_id), path(tumor_bam), path(tumor_bai), path(normal_bam), path(normal_bai))

Did you run into any issues there?

No issues here. The input_paired_bams_ch channel has 5 items, 3 from tumor_id_bam_bai and 2 from normal_bam_bai which are combined in L99.

Alfredo-Enrique · 2023-07-25T20:16:15Z

Oh! Noticed this only parrtially addresses #103 . The output directory automatically created is named after the tumor "SM" tag I believe as opposed to the sample-id. In the case where these two are the same, this issue is hidden, but that's not the case for many datasets. Here for example, the sample ID is TWGSAMIN000001-N003-S03-F but you can see from the tree that the output dir created is named S2_v1.1.5

For context, the provided config file has the output directory path:

// Inputs/parameters of the pipeline
params {
....
    output_dir = "/hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/TWGSAMIN000001-T003-S03-F/"

...

But the final output directory structure is detached from the provided sample-id in the yaml:

(base) [alfgonzalez@ip-0A125248 TWGSAMIN000001-T003-S03-F]$ tree -L 5
.
└── call-sSV-5.0.0
    └── S2_v1.1.5
        ├── DELLY-1.1.3
        │   └── output
        │       ├── DELLY-1.1.3_TWGSAMIN000001_S2-v1.1.5.bcf
        │       ├── DELLY-1.1.3_TWGSAMIN000001_S2-v1.1.5.bcf.csi
        │       ├── DELLY-1.1.3_TWGSAMIN000001_S2-v1.1.5.bcf.csi.sha512
        │       └── DELLY-1.1.3_TWGSAMIN000001_S2-v1.1.5.bcf.sha512
        ├── log-call-sSV-5.0.0-20230725T072204Z
        │   └── nextflow-log
        │       ├── report.html
        │       ├── timeline.html
        │       └── trace.txt
        ├── log-call-sSV-5.0.0-20230725T072639Z
        │   ├── nextflow-log
        │   │   ├── report.html
        │   │   ├── timeline.html
        │   │   └── trace.txt
        │   └── process-log
        │       ├── call_sSV_Delly
        │       ├── call_sSV_Manta
        │       ├── filter_BCF_BCFtools
        │       ├── filter_sSV_Delly
        │       ├── generate_sha512_BCFtools
        │       ├── generate_sha512_Manta
        │       ├── PipeVal-4.0.0-rc.2
        │       └── query_SampleName_BCFtools
        ├── Manta-1.6.0
        │   ├── output
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSmallIndels.vcf.gz
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSmallIndels.vcf.gz.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSmallIndels.vcf.gz.tbi
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSmallIndels.vcf.gz.tbi.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSV.vcf.gz
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSV.vcf.gz.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSV.vcf.gz.tbi
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_candidateSV.vcf.gz.tbi.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_diploidSV.vcf.gz
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_diploidSV.vcf.gz.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_diploidSV.vcf.gz.tbi
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_diploidSV.vcf.gz.tbi.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_somaticSV.vcf.gz
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_somaticSV.vcf.gz.sha512
        │   │   ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_somaticSV.vcf.gz.tbi
        │   │   └── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_somaticSV.vcf.gz.tbi.sha512
        │   └── QC
        │       ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_alignmentStatsSummary.txt
        │       ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_svCandidateGenerationStats.tsv
        │       ├── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_svCandidateGenerationStats.xml
        │       └── Manta-1.6.0_TWGSAMIN000001_S2-v1.1.5_svLocusGraphStats.tsv
        └── validation
            └── run_validate_PipeVal
                └── input_validation.txt

yashpatel6

Added some comments:

config/custom_schema_types.config

config/methods.config

input/call-sSV-input.yaml

main.nf

Faizal-Eeman · 2023-07-25T23:16:59Z

Oh! Noticed this only parrtially addresses #103 . The output directory automatically created is named after the tumor "SM" tag I believe as opposed to the sample-id. In the case where these two are the same, this issue is hidden, but that's not the case for many datasets. Here for example, the sample ID is TWGSAMIN000001-N003-S03-F but you can see from the tree that the output dir created is named S2_v1.1.5

This is intended as the SM tag removes any chance of user caused mislabelling of sample_id.
This test run is an edge case where the Internal ID wasn't used during the alignment process, hence the original S2_v1.1.5 in the BAM SM tag. For production runs, the internal sample ID and the SM tag would match.

Faizal-Eeman · 2023-07-25T23:53:25Z

Testing the updated PR on a real sample ILHNLNEV000009-T002-L01-F from one of the registered projects.

/hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/yaml_input_F32_ILHNLNEV000009-T002-L01-F.log

Faizal-Eeman · 2023-07-26T20:39:54Z

While the test run for a real sample is still in progress, we can see the sample directory is set up as expected.

/hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/ILHNLNEV000009-T002-L01-F\
/call-sSV-5.0.0/ILHNLNEV000009-T002-L01-F/

any other comments here? @tyamaguchi-ucla @yashpatel6 @Alfredo-Enrique

config/methods.config

Faizal-Eeman · 2023-07-28T19:09:48Z

Test runs completed successfully.

/hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/TWGSAMIN000001-T003-S03-F/call-sSV-5.0.0/S2_v1.1.5/

/hot/software/pipeline/pipeline-call-sSV/Nextflow/development/unreleased/mmootor-replace-input-csv-with-yaml/ILHNLNEV000009-T002-L01-F/call-sSV-5.0.0/ILHNLNEV000009-T002-L01-F/

(Updated paths) @tyamaguchi-ucla @yashpatel6

Faizal-Eeman · 2023-07-28T19:11:20Z

@tyamaguchi-ucla @yashpatel6 I think this PR is ready to be merged and we can proceed with the next PR to release call-sSV v6.0.0?

yashpatel6

Looks good!

For the development inputs, /hot/software/pipeline/pipeline-call-sSV/Nextflow/development/input/yaml/call-sSV-input.yaml - I would suggest naming the file to indicate the sample rather than the generic name currently given; we'll likely have more YAMLs there in the future and it's clearer to name base on sample.

Anything else to add @tyamaguchi-ucla ?

config/methods.config

tyamaguchi-ucla

Looks good!

Mootor added 9 commits July 24, 2023 15:29

update submodules

9cc0f73

remove CSV and add YAML input

97723bc

add custom schema

f539666

udate ouput dir structure set up

177e035

change patient_id to sample_id

84ed7de

update pipeval

677736f

remove empty line

d4735ab

update channels to parse input from YAML

6d11b7b

Update CHANGELOG.md

f04f118

Faizal-Eeman requested review from Alfredo-Enrique, tyamaguchi-ucla and yashpatel6 July 25, 2023 08:26

Mootor added 2 commits July 25, 2023 02:16

fix YAML linting

d3553ba

fix YAML linting

847eba2

tyamaguchi-ucla assigned yashpatel6 Jul 25, 2023

Alfredo-Enrique reviewed Jul 25, 2023

View reviewed changes

config/methods.config Show resolved Hide resolved

main.nf Outdated Show resolved Hide resolved

config/schema.yaml Outdated Show resolved Hide resolved

config/schema.yaml Outdated Show resolved Hide resolved

main.nf Outdated Show resolved Hide resolved

Alfredo-Enrique reviewed Jul 25, 2023

View reviewed changes

yashpatel6 reviewed Jul 25, 2023

View reviewed changes

Mootor added 2 commits July 25, 2023 15:49

update help comments

63e9e2c

remove redundant code

c8e016e

Mootor added 3 commits July 25, 2023 16:43

update comment in config

41e8a0b

update logging

c852722

use params.samples_to_process to parse sample id

a2e86de

Mootor added 2 commits July 25, 2023 22:18

set sample ID from tumor ID extracted using params.samples_to_process

13059c4

remove redundant code

5854a75

Faizal-Eeman requested a review from yashpatel6 July 26, 2023 05:40

yashpatel6 reviewed Jul 26, 2023

View reviewed changes

config/methods.config Outdated Show resolved Hide resolved

check for multiple tumor samples before setting sample param

f3747be

Faizal-Eeman requested a review from yashpatel6 July 27, 2023 23:48

yashpatel6 approved these changes Aug 1, 2023

View reviewed changes

config/methods.config Outdated Show resolved Hide resolved

tyamaguchi-ucla approved these changes Aug 1, 2023

View reviewed changes

fix typo in error

06a0c67

Faizal-Eeman merged commit 9917c17 into main Aug 1, 2023
1 check passed

Faizal-Eeman deleted the replace-input-csv-with-yaml branch August 1, 2023 20:34

Faizal-Eeman mentioned this pull request Aug 1, 2023

Release 6.0.0-rc.1 #110

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace input CSV with YAML and parse sample ID from BAM #106

Replace input CSV with YAML and parse sample ID from BAM #106

Faizal-Eeman commented Jul 25, 2023 •

edited

Loading

Alfredo-Enrique left a comment •

edited

Loading

Alfredo-Enrique Jul 25, 2023

Faizal-Eeman Jul 25, 2023

Alfredo-Enrique commented Jul 25, 2023 •

edited

Loading

yashpatel6 left a comment

Faizal-Eeman commented Jul 25, 2023

Faizal-Eeman commented Jul 25, 2023

Faizal-Eeman commented Jul 26, 2023 •

edited

Loading

Faizal-Eeman commented Jul 28, 2023 •

edited

Loading

Faizal-Eeman commented Jul 28, 2023

yashpatel6 left a comment

tyamaguchi-ucla left a comment

Replace input CSV with YAML and parse sample ID from BAM #106

Replace input CSV with YAML and parse sample ID from BAM #106

Conversation

Faizal-Eeman commented Jul 25, 2023 • edited Loading

Description

Closes

Testing Results

Checklist

Alfredo-Enrique left a comment • edited Loading

Choose a reason for hiding this comment

Alfredo-Enrique Jul 25, 2023

Choose a reason for hiding this comment

Faizal-Eeman Jul 25, 2023

Choose a reason for hiding this comment

Alfredo-Enrique commented Jul 25, 2023 • edited Loading

yashpatel6 left a comment

Choose a reason for hiding this comment

Faizal-Eeman commented Jul 25, 2023

Faizal-Eeman commented Jul 25, 2023

Faizal-Eeman commented Jul 26, 2023 • edited Loading

Faizal-Eeman commented Jul 28, 2023 • edited Loading

Faizal-Eeman commented Jul 28, 2023

yashpatel6 left a comment

Choose a reason for hiding this comment

tyamaguchi-ucla left a comment

Choose a reason for hiding this comment

Faizal-Eeman commented Jul 25, 2023 •

edited

Loading

Alfredo-Enrique left a comment •

edited

Loading

Alfredo-Enrique commented Jul 25, 2023 •

edited

Loading

Faizal-Eeman commented Jul 26, 2023 •

edited

Loading

Faizal-Eeman commented Jul 28, 2023 •

edited

Loading