Add --report
flag to save JSON test summary
#71
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This adds a
--report
flag that writes out a JSON test summary file alongside the NFTest log file. As an example, see/hot/software/pipeline/pipeline-convert-BAM2FASTQ/Nextflow/development/unreleased/main/log-nftest-20240909T225839Z.json
:This is related to but not quite the same thing as #7 - this saves the summary to a file for easier downstream parsing without printing anything new to the console.
I implemented this with a new
NFTestReport
class. I tried to keep it as modular and distinct as possible from the existing logic, so the main structural changes are:TestResult
enumeration defined incommon.py
.NFTestCase
s now have astatus
attribute that is initialized toTestResult.PENDING
.NFTestCase.test()
sets that toPASSED
,FAILED
,ERRORED
, orSKIPPED
as appropriate.NFTestRunner.main()
creates anNFTestReport
instance, and usesreport.track_case(case)
as a context manager around each test case. If--report
was passed it writes out that report after all the tests have completed.Along the way I found a hidden bug where
common.resolve_single_path()
raisedValueError
s, which were not being caught byNFTestRunner.main()
. That meant that NFTest would crash on the firstNFTestAssert
with 0 or 2+ matching files. I fixed that by:resolve_single_path
intoNFTestAssert
NFTestAssert
to throw custom exception classes derived fromNFTestAssertionError
, rather thanValueError
andAssertionError
.NFTestRunner.main()
to catchNFTestAssertionError
rather thanAssertionError
.Checklist
This PR does NOT contain Protected Health Information (PHI). A repo may need to be deleted if such data is uploaded.
Disclosing PHI is a major problem1 - Even a small leak can be costly2.
This PR does NOT contain germline genetic data3, RNA-Seq, DNA methylation, microbiome or other molecular data4.
.png
, .jpeg
),.pdf
,.RData
,.xlsx
,.doc
,.ppt
, or other output files.To automatically exclude such files using a .gitignore file, see here for example.
I have read the code review guidelines and the code review best practice on GitHub check-list.
I have set up or verified the
main
branch protection rule following the github standards before opening this pull request.The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
I have added the major changes included in this pull request to the
CHANGELOG.md
under the next release version or unreleased, and updated the date.Footnotes
UCLA Health reaches $7.5m settlement over 2015 breach of 4.5m patient records ↩
The average healthcare data breach costs $2.2 million, despite the majority of breaches releasing fewer than 500 records. ↩
Genetic information is considered PHI.
Forensic assays can identify patients with as few as 21 SNPs ↩
RNA-Seq, DNA methylation, microbiome, or other molecular data can be used to predict genotypes (PHI) and reveal a patient's identity. ↩