Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --report flag to save JSON test summary #71

Merged
merged 8 commits into from
Sep 12, 2024
Merged

Conversation

nwiltsie
Copy link
Member

@nwiltsie nwiltsie commented Sep 11, 2024

Description

This adds a --report flag that writes out a JSON test summary file alongside the NFTest log file. As an example, see /hot/software/pipeline/pipeline-convert-BAM2FASTQ/Nextflow/development/unreleased/main/log-nftest-20240909T225839Z.json:

{
  "start": "2024-09-09T22:58:39.724080+00:00",
  "passed_tests": {
    "test_bam2fastq": 112.80969
  },
  "skipped_tests": {},
  "errored_tests": {},
  "failed_tests": {},
  "cpus": 2,
  "end": "2024-09-09T23:00:32.534372+00:00",
  "success": true
}

This is related to but not quite the same thing as #7 - this saves the summary to a file for easier downstream parsing without printing anything new to the console.


I implemented this with a new NFTestReport class. I tried to keep it as modular and distinct as possible from the existing logic, so the main structural changes are:

  • There is a new TestResult enumeration defined in common.py.
  • NFTestCases now have a status attribute that is initialized to TestResult.PENDING. NFTestCase.test() sets that to PASSED, FAILED, ERRORED, or SKIPPED as appropriate.
  • NFTestRunner.main() creates an NFTestReport instance, and uses report.track_case(case) as a context manager around each test case. If --report was passed it writes out that report after all the tests have completed.

Along the way I found a hidden bug where common.resolve_single_path() raised ValueErrors, which were not being caught by NFTestRunner.main(). That meant that NFTest would crash on the first NFTestAssert with 0 or 2+ matching files. I fixed that by:

  • Moving resolve_single_path into NFTestAssert
  • Updating NFTestAssert to throw custom exception classes derived from NFTestAssertionError, rather than ValueError and AssertionError.
  • Updating NFTestRunner.main() to catch NFTestAssertionError rather than AssertionError.

Checklist

  • This PR does NOT contain Protected Health Information (PHI). A repo may need to be deleted if such data is uploaded.
    Disclosing PHI is a major problem1 - Even a small leak can be costly2.

  • This PR does NOT contain germline genetic data3, RNA-Seq, DNA methylation, microbiome or other molecular data4.

  • This PR does NOT contain other non-plain text files, such as: compressed files, images (e.g. .png, .jpeg), .pdf, .RData, .xlsx, .doc, .ppt, or other output files.

  To automatically exclude such files using a .gitignore file, see here for example.

  • I have read the code review guidelines and the code review best practice on GitHub check-list.

  • I have set up or verified the main branch protection rule following the github standards before opening this pull request.

  • The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

  • I have added the major changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

Footnotes

  1. UCLA Health reaches $7.5m settlement over 2015 breach of 4.5m patient records

  2. The average healthcare data breach costs $2.2 million, despite the majority of breaches releasing fewer than 500 records.

  3. Genetic information is considered PHI.
    Forensic assays can identify patients with as few as 21 SNPs

  4. RNA-Seq, DNA methylation, microbiome, or other molecular data can be used to predict genotypes (PHI) and reveal a patient's identity.

@yashpatel6 yashpatel6 self-assigned this Sep 11, 2024
@yashpatel6
Copy link
Contributor

I'll get to this tomorrow!

Copy link
Contributor

@yashpatel6 yashpatel6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good! Not in this PR but the report should facilitate #7

@nwiltsie nwiltsie merged commit 390700d into main Sep 12, 2024
6 checks passed
@nwiltsie nwiltsie deleted the nwiltsie-report branch September 12, 2024 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants