Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TASK-6722 - Variant Walker to enable user defined variant analysis #2522

Open
wants to merge 65 commits into
base: release-3.x.x
Choose a base branch
from
Open

TASK-6722 - Variant Walker to enable user defined variant analysis #2522

wants to merge 65 commits into from

Conversation

j-coll
Copy link
Member

@j-coll j-coll commented Oct 21, 2024

No description provided.

@halender
Copy link
Contributor

Task linked: TASK-6722 Variant Walker to enable user defined variant analysis

j-coll added 23 commits October 24, 2024 18:58
@j-coll
storage: Ensure walker output is sorted. #TASK-6722
c5375ea
@j-coll
storage: Extract walker STDERR file from MR execution. #TASK-6722
663c03a
@j-coll
storage: Do not write multiple headers. #TASK-6722
154befa
@j-coll
storage: Fix NoSuchMethodError creating StopWatch. #TASK-6722
85aac6d
@j-coll
storage: Ensure stderr file is moved from scratch dir. #TASK-6722
697b08b
@j-coll
storage: Fix stderr sorting. #TASK-6722
356567e
@j-coll
storage: Write \n after the json header #TASK-6722
6253da3
@j-coll
storage: Do not interrupt header with empty records. #TASK-6722
5789628
@j-coll
storage: Add a custom Partitioner to ensure sorted data with multiple…
4ff0655 
… reducers #TASK-6722
@j-coll
storage: Fix partitioner. #TASK-6722
8268266
@j-coll
storage: Restart process when changing chromosome to ensure correct s…
4147d01 
…orting. #TASK-6722
@j-coll
storage: Fix GenomeHellper generateBootPreSplits. #TASK-6722
7fd439a
@j-coll
storage: Do not interrupt header with empty lines while concat. #TASK…
e6128b0 
…-6722
@j-coll
storage: Replace ImmutableBytesWritable with VariantLocusKey as map o…
100fecf 
…utput key. #TASK-6722
@j-coll
storage: Use VariantLocusKey and VariantLocusPartitioner in VariantEx…
0df69dc 
…portDirver. #TASK-6722
@j-coll
storage: Fix VariantLocusKey serialization. #TASK-6722
f6fd3d4
@j-coll
storage: Fix "Request body si too large" #TASK-6722
fa3c9f2
@j-coll
analysis: Do not try to close twice the same ERM. #TASK-6722
b528c03
@j-coll
storage: Do not use flush on outputstream. HADOOP-16548 #TASK-6722
96e5679
@j-coll
storage: Add VariantExporterDirectMultipleOutputsMapper to ensure sor…
bcd8185 
…ted export without reduce step. #TASK-6722
@j-coll
storage: Do not use reduce step on variant-walker. #TASK-6722
c4c3d3b
@j-coll
storage: Fix VariantRecordWriter bytes_written counter. #TASK-6722
0100097
@j-coll
storage: Reduce number of intermediate mapper files. #TASK-6722
b52ca27
j-coll added 29 commits November 25, 2024 13:53
@j-coll
storage: Improve MapReduceOutputFile concatMrOutputToLocal. #TASK-6722
cd50a3c
@j-coll
storage: Increase mapreduce.task.timeout to 30min #TASK-6722
d430391
@j-coll
storage: Fix temporary mapreduce outdir. #TASK-6722
e35ee83
@j-coll
storage: Do not double copy hdfs files #TASK-6722
0c48603
@j-coll
storage: Use reducer to concat binary files #TASK-6722
ccf7438
@j-coll
storage: Do not fail vairant-walker if no output is produced. #TASK-6722
f87686e
@j-coll
storage: Split PhoenixInputSplits into smaller splits. #TASK-6722
a389e10
@j-coll
storage: Improve log message. #TASK-6722
f453090
@j-coll
storage: Add HadoopVariantWalkerTest. #TASK-6722
47535c1
@j-coll
storage: Rename some variant-walker params. Add descriptions #TASK-6722
003e467
@j-coll
storage: Fix NPE running SampleVariantStats #TASK-6722
48e1592
@j-coll
storage: Fix CustomPhoenixInputFormat generateSplit for first and las…
1d86756 
…t splits. #TASK-6722
@j-coll
analysis: Fix NPE at relatedness tool. #TASK-6722
5141031
@j-coll
Merge branch 'release-3.x.x' into TASK-6722
c48ce0a
@j-coll
cicd: Upload tests logs as artifacts. Reduce action log size. #TASK-6722
f2bc782
@j-coll
storage: Fix NPE at CohortVariantStatsDriver. #TASK-6722
dd684aa
@j-coll
cicd: Fix NPE. #TASK-6722
9795c6a
@j-coll
storage: Fix AIOOBE SampleVariantStatsDriver #TASK-6722
923651c
@j-coll
storage: Do not produce a .crc checksum file copying from hdfs. #TASK…
90010ac 
…-6722
@j-coll
storage: Improve docker process failure. Do not close the stdin twice…
9f326d9 
…. #TASK-6722
@j-coll
storage: Fix AIOOBE SampleVariantStatsDriver #TASK-6722
627e56a
@j-coll
storage: Do not produce a .crc checksum file copying from hdfs. #TASK…
98ce6f8 
…-6722
@j-coll
analysis: Do not use the scratchDir as intermediate folder for export…
14c07d9 
… and walk. #TASK-6722
@j-coll
storage: Improve collections usage in SampleVariantStatsDriver. #TASK…
050c1ee 
…-6722
@j-coll
analysis: Fix VariantAnalysisTest. #TASK-6722
a0c2a5f
@j-coll
app: Regenerate cli. #TASK-6722
3853c63
@j-coll
storage: Fix junit tests. #TASK-6722
eb61609
@j-coll
cicd: Increase "Publish Test Report on GitHub" memory #TASK-6722
54acc28
@j-coll
core: Fix NumberFormatException from IOUtils. #TASK-6722
4e96492
@j-coll j-coll requested a review from pfurio December 10, 2024 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pfurio pfurio Awaiting requested review from pfurio

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@j-coll @halender