Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix incorrect scan results with retry feature and -test-repetition-relaunch-enabled #21989

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tigger-tian
Copy link

Checklist

  • I've run bundle exec rspec from the root directory to see all new and existing tests pass
  • I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
  • I see several green ci/circleci builds in the "All checks have passed" section of my PR (connect CircleCI to GitHub if not)
  • I've read the Contribution Guidelines
  • I've updated the documentation if necessary.
  • I've added or updated relevant unit tests.

Motivation and Context

When the 'test-repetition-relaunch-enabled' option is enabled, the scan output includes results from all test retries. The following is an example of a scan run with this option enabled.

    scan(
      package_path: '.',
      workspace: '.', 
      device: 'iPhone 14',
      output_directory: test_output_path,
      code_coverage: true,
      result_bundle: true,
      xcargs: "-test-repetition-relaunch-enabled YES",
      number_of_retries: 3
    )

From the log, tests run twice and succeeded at the end.

Executed 1724 tests, with 2 failures (0 unexpected) in 9.243 (9.687) seconds
Executed 1724 tests, with 0 failures (0 unexpected) in 12.912 (13.143) seconds

But the scan results combined them together.

 +--------------------+------+
 |       Test Results        |
 +--------------------+------+
 | Number of tests    | 3448 |
 | Number of failures | 2    |
 +--------------------+------+

As scan results showed 2 failures, fastlane returned with failure

 fastlane finished with errors

Description

This change removes retry attempts and selects the best result from all retries with the minimum test failures as the final scan result, ensuring accurate scan output.

Executed 1724 tests, with 1 failures (0 unexpected) in 10.121 (10.674) seconds
Executed 1724 tests, with 0 failures (0 unexpected) in 13.318 (14.871) seconds

 +--------------------+------+
 |       Test Results        |
 +--------------------+------+
 | Number of tests    | 1724 |
 | Number of failures | 0    |
 +--------------------+------+

Testing Steps

@tigger-tian tigger-tian marked this pull request as ready for review April 23, 2024 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant