New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test results are missing when using parallel simulators on Xcode 9 #295
Comments
Any ETA on this issue? Is this being worked on? |
Seems like just need to update the regex in diff --git a/lib/xcpretty/parser.rb b/lib/xcpretty/parser.rb
index 103e36f..f298010 100644
--- a/lib/xcpretty/parser.rb
+++ b/lib/xcpretty/parser.rb
@@ -123,24 +123,24 @@ module XCPretty
# $1 = suite
# $2 = test_case
# $3 = time
- TEST_CASE_PASSED_MATCHER = /^\s*Test Case\s'-\[(.*)\s(.*)\]'\spassed\s\((\d*\.\d{3})\sseconds\)/
+ TEST_CASE_PASSED_MATCHER = /^\s*Test [Cc]ase\s'-\[(\w*)\s(\w*)\]'\spassed\so?n?\s?'?\w*\s?\w*'?\s?\((\d*\.\d{3})\sseconds\)/
# @regex Captured groups
# $1 = suite
# $2 = test_case
- TEST_CASE_STARTED_MATCHER = /^Test Case '-\[(.*) (.*)\]' started.$/
+ TEST_CASE_STARTED_MATCHER = /^Test [Cc]ase '-\[(.*) (.*)\]' started.$/
# @regex Captured groups
# $1 = suite
# $2 = test_case
- TEST_CASE_PENDING_MATCHER = /^Test Case\s'-\[(.*)\s(.*)PENDING\]'\spassed/
+ TEST_CASE_PENDING_MATCHER = /^Test [Cc]ase\s'-\[(.*)\s(.*)PENDING\]'\spassed/
# @regex Captured groups
# $1 = suite
# $2 = test_case
# $3 = time
- TEST_CASE_MEASURED_MATCHER = /^[^:]*:[^:]*:\sTest Case\s'-\[(.*)\s(.*)\]'\smeasured\s\[Time,\sseconds\]\saverage:\s(\d*\.\d{3}),/
+ TEST_CASE_MEASURED_MATCHER = /^[^:]*:[^:]*:\sTest [Cc]ase\s'-\[(.*)\s(.*)\]'\smeasured\s\[Time,\sseconds\]\saverage:\s(\d*\.\d{3}),/
PHASE_SUCCESS_MATCHER = /^\*\*\s(.*)\sSUCCEEDED\s\*\*/
@@ -176,7 +176,7 @@ module XCPretty
# @regex Captured groups
# $1 = suite
# $2 = time
- TEST_SUITE_STARTED_MATCHER = /^\s*Test Suite '(?:.*\/)?(.*[ox]ctest.*)' started at(.*)/
+ TEST_SUITE_STARTED_MATCHER = /^\s*Test [Ss]uite '(?:.*\/)?(.*[ox]ctest.*)' started at(.*)/ I'd love to send PR but I'm not familiar with Ruby testing (Cucumber, RSpec). |
@nicnocquee this does not work for me locally. I will take a look at making this work and open a PR. Update, I am working on this and it should be ready for review in the next day (or 2 at the most). |
@marcelofabri @nicnocquee @eneko I am using logs that have been pre-processed by |
Please see this PR |
Testing started on 'iPhone 7 Plus'
Test suite 'Selected tests' started on 'iPhone 7 Plus'
Test suite 'UangkuTests.xctest' started on 'iPhone 7 Plus'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone 7 Plus'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone 7 Plus' (2.343 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone 7 Plus' (4.197 seconds)
Testing passed on 'iPhone 7 Plus'
Testing started on 'iPhone 7'
Test suite 'Selected tests' started on 'iPhone 7'
Test suite 'UangkuTests.xctest' started on 'iPhone 7'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone 7'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone 7' (2.213 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone 7' (4.133 seconds)
Testing passed on 'iPhone 7'
Testing started on 'iPhone 5s'
Test suite 'Selected tests' started on 'iPhone 5s'
Test suite 'UangkuTests.xctest' started on 'iPhone 5s'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone 5s'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone 5s' (2.224 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone 5s' (4.096 seconds)
Testing passed on 'iPhone 5s'
Testing started on 'iPhone X'
Test suite 'Selected tests' started on 'iPhone X'
Test suite 'UangkuTests.xctest' started on 'iPhone X'
Test suite 'TopupMethodViewControllerTests' started on 'iPhone X'
Test case '-[TopupMethodViewControllerTests testTopUpCashView]' passed on 'iPhone X' (2.305 seconds)
Test case '-[TopupMethodViewControllerTests testTopUpTransferView]' passed on 'iPhone X' (4.143 seconds)
Testing passed on 'iPhone X'
** TEST SUCCEEDED ** |
@nicnocquee Those tests look pretty sequential to me, how are you specifying the multiple destinations? They should start at the same time and run in parallel. |
What's interesting is that running on a single destination vs. using Xcode 9's new support of multiple destination (parallel testing) seems to format logs completely differently; I'm not seeing the same logs as @nicnocquee. With a single destination, logs are formatted in Objective-C, but when targeting multiple destinations, logs are formatted in Swift and less verbose (no performance metrics, no test summary, etc.). One Destination
Multiple Destinations
I was able to stitch some of the pieces back together to match previous behavior by updating the few of the matchers, though they're far more convoluted than the ones mentioned above; for example, for passing tests: # @regex Captured groups
# $1 = suite
# $2 = test_case
# $3 = device (added)
# $4 = time
TEST_CASE_PASSED_MATCHER = /^\s*Test [Cc]ase\s'(?:-\[(?:.*)\.)?(.*)(?:\s|\.)(.*)(?:\]|\(\))'\spassed\s(?:on\s'(.*)'\s)?\((\d*\.\d{3})\sseconds\)/ I'd be willing to keep testing/moving forward if this still hasn't gotten much traction, but behavior will inevitably be different for parallel testing. |
@eneko sorry forgot to mention I added xcodebuild -workspace Uangku.xcworkspace
-scheme UangkuDemo
-configuration Debug
-sdk 'iphonesimulator11.0'
-destination 'platform=iOS Simulator,id=33ACF59B-E864-42F5-B05E-90E2EF139B36'
-destination 'platform=iOS Simulator,id=5C71F407-1930-4119-87AD-038388F1F02E'
-destination 'platform=iOS Simulator,id=FA2064AC-5A8C-47A6-A131-77E80195D1D4'
-destination 'platform=iOS Simulator,id=4CD5897F-F46B-4279-B6F2-A8BF89A4CF9C'
-derivedDataPath '/Users/nico/Library/Developer/Xcode/DerivedData/Uangku-fwiimgqssqptgdbrmdznmhdkisea'
-enableCodeCoverage NO
-jobs 1
-maximum-concurrent-test-simulator-destinations 1
GCC_PREPROCESSOR_DEFINITIONS="POD_CONFIGURATION_DEBUG=1 COCOAPODS=1 DEBUG=1 GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1"
-only-testing:UangkuTests/TopupMethodViewControllerTests
-skip-testing:UangkuDemoUITests
clean test |
I'm having similar issues, not only with xcpretty but with xcodebuild itself. If I use |
I haven't seen much activity on this, so I've been prototyping some changes to formatting/reporting: https://github.com/Hammertime38/xcpretty/tree/feature/parallel-testing-support @alessandroorru I've been seeing the same issues; it seems that |
Any updates on this issue? |
Hey, Any update on this? Big blocker for me. |
@astasiak93 You can always use @johnhammerlund's PR : #299. It's been ok for our use (parallel testing on iOS11 simulators and real devices).
(ebay's version 2b68127 removes somes deprecation warnings from the logs) |
@marcc-orange Hey, thank you for the snippet - it works for successful test sessions but for failures it is only displayed : Build error 1 on the scan slack notification message. |
How is this still an issue lol. I feel so many people use |
Does |
Yes, you can pass a list of devices and it works except the output doesn’t show correctly due to xcpretty not supporting it. |
You can get some output directly from junit report file using:
Then you will get output like:
I am using for now: |
Any update on that issue? |
Having this issue too - would be nice not to have to use that specific gem workaround |
I would strongly advise everyone facing this issue to start using https://github.com/TitouanVanBelle/XCTestHTMLReport since there has been no update on this issue. The report that i am talking about has support for screenshots and also supports reporting when running tests in parallel on multiple devices or simulators. |
Are there any plans to update this for Xcode 10? |
Any updates on this ? |
Any updates on this? 🤔 |
+1 |
2 similar comments
+1 |
+1 |
We've migrated to https://github.com/KrauseFx/trainer for ingestion. keeping xcpretty for formatting for now. |
https://github.com/TitouanVanBelle/XCTestHTMLReport can also produce JUnit output and a nice HTML report output. It handles parallel simulators with no problems |
For anyone comes across this, please give xcbeautify a try. |
Alternatively, for a clean build output, |
Due to xcpretty/xcpretty#295 we are fixing the xcpretty version (official gem omits tests results when running multiple destinations)
Any update? |
Without using XCPretty in fastlane am able to run the tests but since fastlane is heavily dependent on XCPretty, it is frustrating Parallel Testing doesn't work with XCPretty. Would this issue be even fixed? |
@ghost I've been using the unofficial gem suggested a few posts above by @tomaszmnich for the same reason and it works fine, have you tried it? |
I think xcpretty shouldn't be producing any reports (junit or html) and just focus on making the output pretty. Creating reports is best left to other tools that read the results bundle that Xcode produces at the end. If using Fastlane and you want to use parallel testing set these options in
Tbh at this point almost might as well just use xcodebuild directly. It would be nice if Fastlane scan was more modular/allowed more control.. |
I just came across this issue when I was looking to file this myself. Any update on this 😬? |
@thii xcbeautify works like a charm when running UI Tests in parallel with |
How has this not been fixed yet? |
Btw it is still relevant 🙃 |
any code snippet somewhere that can help to get started soon? |
This can probably be closed since the latest release of fastlane removes xcpretty: https://github.com/fastlane/fastlane/releases/tag/2.201.0 |
Any updates on this issue? |
hi @li-arezkisidali , I think this can be closed like @gmoraleda said (cc @joshdholtz) -- fastlane users just have to use xcbeautify as the just make sure xcbeautify is installed on your build machine, then fastlane should use it by default 👍 |
Xcode 9 allows testing in multiple simulators at the same time when using multiple
--destination
flags.However, this breaks
xcpretty
as it no longer recognizes the tests in the log.The text was updated successfully, but these errors were encountered: