sauce-service - Before hook timeouts are reported as success on saucelabs

[christian-bromann]

From @trinckler on August 15, 2017 17:53

The problem

SAUCE SERVICE is not updating the job when the test fails inside the before hook.
We try to open a page with an specific timeout, after that timeout the test fails but the sauce labs job is not updated.

Environment

• WebdriverIO version:4.8.0
• Node.js version:v6.10.2
• Standalone mode or wdio testrunner: WDIO
• if wdio testrunner, running synchronous or asynchronous tests: synchronous
• Additional wdio packages used (if applicable): Mocha

Details

We have a 40 concurrency suit and some pages never finish loading. Using browser.timeouts('page load',60000); and the retries we skip that problem. if the page fails to load , the sauce service update_job is not called and the saucelabs test is marked as success.

Link to Selenium/WebdriverIO logs

https://gist.github.com/trinckler/a6993afad9c175a5cb1b44d8107c1a29
https://gist.github.com/trinckler/399a46cc150762a875bbea26bfa41fd9

Code To Reproduce Issue [ Good To Have ]

describe(' describe text', () => {
before( () => {
browser.timeouts('page load', 900);
browser.url('http://webdriver.io/');
}, 3);
it('should be true', () => {
// something true
});
});

Copied from original issue: webdriverio/webdriverio#2243