Scan doesn't give you correct test results if you run concurrent simulators

[zacoid55]

New Issue Checklist

• Updated fastlane to the latest version
• I read the Contribution Guidelines
• I read docs.fastlane.tools
• I searched for existing GitHub issues

Issue Description

When testing a UI test scheme and specifying multiple devices in a Scanfile (taken from the functionality from fastlane snapshot and format from a Snapfile). The "Number of Tests" are coming back as 0 (as seen in screenshot) but it should be showing 2.

This was in Xcode 9

 devices ([
      "iPhone 8",
      "iPhone 7"
    ])

Note: The tests run perfectly in a concurrent state on Xcode with scan

✅ fastlane environment ✅

Stack

Key	Value
OS	10.12.6
Ruby	2.3.1
Bundler?	true
Git	git version 2.13.5 (Apple Git-94)
Installation Source	~/.rvm/gems/ruby-2.3.1/bin/fastlane
Host	Mac OS X 10.12.6 (16G29)
Ruby Lib Dir	~/.rvm/rubies/ruby-2.3.1/lib
OpenSSL Version	OpenSSL 1.0.2l 25 May 2017
Is contained	false
Is homebrew	false
Is installed via Fabric.app	false
Xcode Path	/Applications/Xcode.app/Contents/Developer/
Xcode Version	9.0

System Locale

Variable	Value
LANG	en_GB.UTF-8	✅
LC_ALL
LANGUAGE

fastlane files:

`./Fastfile`

fastlane_version "2.50.1"

import_from_git(url: 'https://github.com/theappbusiness/MasterFastfile.git', branch: '2.2.0', path: 'Fastfile')

`./Appfile`

app_identifier = ENV['FL_UPDATE_PLIST_APP_IDENTIFIER']

fastlane gems

Gem	Version	Update-Status
fastlane	2.59.0	✅ Up-To-Date

Loaded fastlane plugins:

No plugins Loaded

Loaded gems

Gem	Version
did_you_mean	1.0.2
executable-hooks	1.3.2
bundler-unload	1.0.2
rubygems-bundler	1.4.4
bundler	1.14.6
io-console	0.4.6
CFPropertyList	2.3.5
public_suffix	2.0.5
addressable	2.5.2
babosa	1.0.2
claide	1.0.2
colored	1.2
colored2	3.1.2
highline	1.7.8
commander-fastlane	4.4.5
declarative	0.0.10
declarative-option	0.1.0
unf_ext	0.0.7.4
unf	0.1.4
domain_name	0.5.20170404
dotenv	2.2.1
excon	0.59.0
multipart-post	2.0.0
faraday	0.13.1
http-cookie	1.0.3
faraday-cookie_jar	0.0.6
faraday_middleware	0.12.2
fastimage	2.1.0
gh_inspector	1.0.3
jwt	1.5.6
little-plugger	1.1.4
multi_json	1.12.1
logging	2.2.2
memoist	0.16.0
os	0.9.6
signet	0.7.3
googleauth	0.5.3
httpclient	2.8.3
mime-types-data	3.2016.0521
mime-types	3.1
uber	0.1.0
representable	3.0.4
retriable	3.1.1
google-api-client	0.13.6
json	1.8.6
mini_magick	4.5.1
multi_xml	0.6.0
plist	3.3.0
rubyzip	1.2.1
security	0.1.3
slack-notifier	1.5.1
terminal-notifier	1.8.0
unicode-display_width	1.3.0
terminal-table	1.8.0
tty-screen	0.5.0
word_wrap	1.0.0
nanaimo	0.2.3
xcodeproj	1.5.2
rouge	2.0.7
xcpretty	0.2.8
xcpretty-travis-formatter	0.0.4

generated on: 2017-09-27

[fastlane-bot]

It seems like you have not included the output of fastlane env

To make it easier for us help you resolve this issue, please update the issue to include the output of fastlane env 👍

[kviksilver]

From what i understand when running on multiple sims xcodebuild doesnt output test logs to standard output so piping it trough xcpretty as scan does wont do anything. my solution was to manually pipe all the logs in Logs/Test/Diagnostics trough xcpretty after tests and let it generate reports:

private_lane :generate_junit_reports do | options |
      diagnostics = "#{options[:derived_data_path]}/Logs/Test/Diagnostics"
      folders = Dir.entries(diagnostics).select{ |folder| !folder.start_with?('.') }
      prefix = options.has_key?(:test_report_prefix) ? options[:test_report_prefix] : ""
      if folders.count == 1
          FileUtils.mv "#{options[:reports_path]}/report.junit", "#{options[:reports_path]}/#{prefix}_report.junit", :force => true
          FileUtils.mv "#{options[:reports_path]}/report.html", "#{options[:reports_path]}/#{prefix}_report.html", :force => true
          UI.message "Found single device, report generated at #{options[:reports_path]} 📨 "      "
        next
      end
      folders.each do |folder|
          destination = "#{options[:reports_path]}/#{prefix}#{folder}"
          sh "cat #{diagnostics}/#{folder}/StandardOutputAndStandardError.txt | xcpretty -r junit -o '#{destination}.junit' -r html -o '#{destination}.html' > /dev/null"
          UI.message "Report generated at #{destination} 📨 "
      end
end

hope this helps as a temporary solution...

[Niko-r]

Issue reported in xcpretty
xcpretty/xcpretty#295

[fastlane-bot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

[fastlane-bot]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍

[niveus]

We're still waiting on this. Most people are going to run tests on multiple simulators and Fastlane has been broken for months. Maybe xcpretty should be dropped if it's not going to be supported anymore?

[Buju77]

yes, we are also waiting for this to be fixed. are there any plans how to move forward? are there any alternatives considered?

[lemonkey]

I would also like an option to disable xcpretty as I would like to be able to extract overall performance test information from the job run more easily.