Collate_xcresults command question

[alexnot95]

Hi, I'm using the test-center multi_scan command and for my localization tests, I'm using Fastlane snapshot. I have 6 tests and they run through a snapshot on 16 locales, after the run, I have 16 different xcresult reports and when I try to use collect_xcresults it merges all results into one, but not correctly, if for ex. 5th locale has 2 failed tests and the last locale have all passed tests then in merged xcresult I will see all passed tests... as I see collate_xcresults using merge tool under the hood:

if xcresult_bundlepaths.size > 1
          command = [
            'xcrun',
            'xcresulttool',
            'merge',
            xcresult_bundlepaths,
            '--output-path',
            tmp_collated_xcresult_bundlepath
          ].flatten
          commands_run = sh(*command)

Do you know how I can resolve this issue? Or it's a problem of xcrun xcresulttool merge command, it cannot understand that it's a different tests? Because the name of the tests are the same on 16 locales...
Thank you

[lyndsey-ferguson]

Yes, the xcresulttool tool is doing all of the collation. From your findings, it appears that the xcresulttool cannot differentiate between different locales. What if you tried a different device for each locale? Maybe the tool could reason through that?

Otherwise, you could turn off collation, and then specify which xcresults to merge in the input parameters.

Otherwise, the next thing to try, which feels ugly to me, is make a separate test target for each.

[lyndsey-ferguson]

I'm going to convert this into a GitHub Discussion question in case someone else has a suggestion.