Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds flaky test info to build_message #14

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

Adds flaky test info to build_message #14

wants to merge 12 commits into from

Conversation

joseph-sentry
Copy link
Contributor

depends on #11

@joseph-sentry
feat: add failure message manipulation functions
6b46b85 
This commit ports over two functions for manipulating failure messages:
- escape_failure_message:

  escapes a failure message to prepare it to be displayed in the comment
  so that the content of the failure message does not interfere with
  the formatting of the PR comment.

- shorten_file_paths:

  truncates the beginning of long file paths in the failure message in
  order to reduce the horizontal space a failure message takes up.

Signed-off-by: joseph-sentry <[email protected]>
@joseph-sentry
fix: address feedback
a5663e7 
- fix ? to & for escaping
- make rust_string and pystring variants of functions so that we can
  compare the performance of using the Rust standard types (String) vs.
  the pyo3 types (PyString)
- Add some more granular test cases

Signed-off-by: joseph-sentry <[email protected]>
@joseph-sentry
build: only upload unit.junit.xml to codecov
da5fe36 
Signed-off-by: joseph-sentry <[email protected]>
@joseph-sentry
build: fix by disabling search
05eca38 
Signed-off-by: joseph-sentry <[email protected]>
@joseph-sentry
feat: add build_message function
bd5676d 
this function uses the shorten_file_paths and escape_failure_message
functions as well as the newly introduced generate_failure_info and
generate_test_description functions

We want to use this function to generate the contents of the test
results PR comment in both the CLI and the worker

Signed-off-by: joseph-sentry <[email protected]>
@joseph-sentry
fix: define parameter in build_message as MessagePayload
46ae30f
@joseph-sentry
fix: remove redundant attributes
1138b03
@joseph-sentry
fix: make message payload and failure public
79f21a9
@joseph-sentry
feat: add flaky tests to the test results PR comment
aa97639 
The Failure object passed in the MessagePayload now contains 2 extra
fields: flags and flake.

the flags field are the flags associated with the upload of the failure.
the field is a list of strings

the flake field represents the flake on the Test this failure is on.
this field is a list of Flake objects.

A Flake object consists of 2 fields: symptoms (a list of enums that
represent what kind of flake this test is a part of) and is_new_flake
(signifies whether the flake was a previously know one, or if it was
just detected)

This commit also moves all the sample test result files to the samples
directory in the tests directory.

Signed-off-by: joseph-sentry <[email protected]>
@joseph-sentry
feat: accept objects that may not have flags or flake attr as failure
3f05e74
@hootener
Copy link

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 15 tests with 1 failed, 14 passed and 0 skipped.

View the full list of failed tests
Test Description Failure message 
Testsuite:
pytest

Test name:
tests.test_failure_message::test_build_message

Envs:
- default
 
def test_build_message():
        class FlakeSymptomType(Enum):
            FAILED_IN_DEFAULT_BRANCH = "failed_in_default_branch"
            CONSECUTIVE_DIFF_OUTCOMES = "consecutive_diff_outcomes"
            UNRELATED_MATCHING_FAILURES = "unrelated_matching_failures"
    
        @dataclass
        class Flake:
            symptoms = []
            is_new_flake = True
    
        @dataclass
        class Run:
            name = ""
            testsuite = ""
            failure_message = ""
            flags = None
            flake = None
    
        @dataclass
        class Thing:
            failed = 0
            passed = 0
            skipped = 0
            failures = []
    
        run1 = Run()
        run2 = Run()
    
        run1.name = "hello"
        run1.testsuite = "world"
        run1.failure_message = "I failed"
        run1.flags = ["hello", "world"]
        run1.flake = Flake()
        run1.flake.is_new_flake = False
        run1.flake.symptoms = [FlakeSymptomType.FAILED_IN_DEFAULT_BRANCH, FlakeSymptomType.CONSECUTIVE_DIFF_OUTCOMES]
    
    
        run2.name = "hello"
        run2.testsuite = "again"
        run2.failure_message = None
    
        payload = Thing()
        payload.passed = 1
        payload.failed = 2
        payload.skipped = 3
        payload.failures = [run1, run2]
    
        res = build_message(payload)
    
>       assert res == """### :x: Failed Test Results:
    Completed 6 tests with 2 failed, 1 passed and 3 skipped.
    <details><summary>View the full list of failed tests</summary>

@codecov Codecov
Copy link

codecov bot commented Apr 17, 2024

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 15 tests with 1 failed, 14 passed and 0 skipped.

View the full list of failed tests
Test Description Failure message 
Testsuite:
pytest
Test name:
tests.test_failure_message::test_build_message
Envs:
- default
 
def test_build_message():
        class FlakeSymptomType(Enum):
            FAILED_IN_DEFAULT_BRANCH = "failed_in_default_branch"
            CONSECUTIVE_DIFF_OUTCOMES = "consecutive_diff_outcomes"
            UNRELATED_MATCHING_FAILURES = "unrelated_matching_failures"
    
        @dataclass
        class Flake:
            symptoms = []
            is_new_flake = True
    
        @dataclass
        class Run:
            name = ""
            testsuite = ""
            failure_message = ""
            flags = None
            flake = None
    
        @dataclass
        class Thing:
            failed = 0
            passed = 0
            skipped = 0
            failures = []
    
        run1 = Run()
        run2 = Run()
    
        run1.name = "hello"
        run1.testsuite = "world"
        run1.failure_message = "I failed"
        run1.flags = ["hello", "world"]
        run1.flake = Flake()
        run1.flake.is_new_flake = False
        run1.flake.symptoms = [FlakeSymptomType.FAILED_IN_DEFAULT_BRANCH, FlakeSymptomType.CONSECUTIVE_DIFF_OUTCOMES]
    
    
        run2.name = "hello"
        run2.testsuite = "again"
        run2.failure_message = None
    
        payload = Thing()
        payload.passed = 1
        payload.failed = 2
        payload.skipped = 3
        payload.failures = [run1, run2]
    
        res = build_message(payload)
    
>       assert res == """### :x: Failed Test Results:
    Completed 6 tests with 2 failed, 1 passed and 3 skipped.
    <details><summary>View the full list of failed tests</summary>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@joseph-sentry @hootener