[Bug]: IOS Real Device Tests started failing #815

agray · 2024-08-10T01:51:44Z

Description

Real Device tests started failing after upgrade to Appium dotnet client 5.1.0 and Selenium 4.23.0 on Saturday 10/08 on SauceLabs.

Environment

.NET client build version or git revision if you use some snapshot: 5.1.0 and Selenium 4.23.0
Mobile platform/version under test: IOS devices
Real device or emulator/simulator: Real Devices

Details

SauceLabs tests for iPhone 15 Pro Max Real Device started failing on Saturday 10th August.

Please provide more details, if necessary.
Error returned from SauceLabs: "Status Code: 500 - Error while starting appium session: An unknown server-side error occurred while processing the command. Original error: No webview has been detected after 13926ms."

Dor-bl · 2024-08-10T04:30:32Z

@agray, have you checked the https://github.com/appium/dotnet-client#compatibility-matrix?

agray · 2024-08-10T08:39:18Z

I am using Selenium 4.23.0

Dor-bl · 2024-08-10T09:21:41Z

And dotnet client version 5.0.0, works for you?

KazuCocoa · 2024-08-10T20:56:56Z

Could you attache the full Appium server log?

No webview has been detected after 13926ms thinks me that it was client independent behavior. It would be helpful to attach working Appium server log as well.

agray · 2024-08-11T00:37:02Z

Rolled back to Appium 5.0.0 and Selenium 4.22.0 and even less works. (I had originally skipped Selenium 4.22.0.)

Sauceforge/Saucery@32ef411

NUnit Virtual: Tests launch (except IOS Virtual)
XUnit Virtual: Tests don't launch
NUnit Real Devices: Tests don't launch
XUnit Real Devices: Tests don't launch

On Appium 5.1.0 and Selenium 4.23.0:

NUnit Virtual: Tests launch
XUnit Virtual: Tests launch
NUnit Real Devices: Tests launch (only Android works)
XUnit Real Devices: Tests launch (only Android works)

agray · 2024-08-11T00:54:46Z

Could you attache the full Appium server log?

No webview has been detected after 13926ms thinks me that it was client independent behavior. It would be helpful to attach working Appium server log as well.

https://app.saucelabs.com/tests/5988caa82b3d4cbb858bf745f5ca7d24#0

agray · 2024-08-11T00:57:32Z

It last worked on August 1st:
https://app.saucelabs.com/builds/rdc/ccadaa7715f33b228031f806c6f56087

IPhone 15 Pro Max (IOS 17.4.1)
https://app.saucelabs.com/tests/13c99c1b6c674651af2f952ec2113e2d

I'll even provide the video from SauceLabs:
https://github.com/user-attachments/assets/bae2ea75-58b7-4333-b885-a558926f034d

Dor-bl · 2024-08-11T01:28:20Z

So what was the last version that worked for you properly?

agray · 2024-08-11T01:32:02Z

So what was the last version that worked for you properly?

I think it was pre 5.0.0 (or one of the 5 betas) and Selenium 4.21.0

Dor-bl · 2024-08-11T01:42:16Z

Does other webViews works for you? I have no major changes in mind that can cause this.

agray · 2024-08-11T01:52:17Z

Does other webViews works for you? I have no major changes in mind that can cause this.

Webviews?

KazuCocoa · 2024-08-11T03:54:43Z

No available web pages after 10 retries: Could not connect to a valid webapp. Make sure it is debuggable and has at least one active page.

This is appium log from the failed one, 5988ca....
It looks like the appium xcuitest failed to get webviews. I haven't checked logs well yet but it is client independent.

agray · 2024-08-11T04:00:35Z

It was working on 1st August and I have only updated nuget packages. This code base has worked for many many years.

Dor-bl · 2024-08-11T05:36:35Z

Does other webViews works for you? I have no major changes in mind that can cause this.

Webviews?

https://developer.apple.com/design/human-interface-guidelines/web-views

KazuCocoa · 2024-08-11T06:18:22Z

worked one:

{"level":"VERBOSE","message":"[Appium] Requiring driver at /root/appium/appium2-202408010003/node_modules/appium-xcuitest-driver/build/index.js","timestamp":"2024-08-01 10:18:19.926"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Appium v2.11.2 creating new XCUITestDriver (v7.24.6) session","timestamp":"2024-08-01 10:18:19.927"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Checking BaseDriver versions for Appium and XCUITestDriver","timestamp":"2024-08-01 10:18:19.928"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Appium's BaseDriver version is 9.11.2","timestamp":"2024-08-01 10:18:19.928"}
...
ebugger] Notified that an application has been updated","timestamp":"2024-08-01 10:18:22.677"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Current applications available:","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]     Application: \"PID:504\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         id: \"PID:504\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isProxy: false","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         name: \"Safari\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         bundleId: \"com.apple.mobilesafari\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isActive: true","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isAutomationEnabled: true","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         pageArray:","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]           - id: 1","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             title: \"\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             url: \"about:blank\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             isKey: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]     Application: \"PID:209\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         id: \"PID:209\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isProxy: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         name: \"amsengagementd\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         bundleId: \"com.apple.amsengagementd\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isActive: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isAutomationEnabled: \"Unknown\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Finally selecting app PID:504","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Selected app after 10ms","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][XCUITestDriver@4505] Picking webview 'WEBVIEW_504.1' after 292ms","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][XCUITestDriver@4505] Attempting to set context to 'WEBVIEW_504.1' from 'NATIVE_APP'","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Selecting page '1' on app 'PID:504' and forwarding socket setup","timestamp":"2024-08-01 10:18:22.681"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Sending '_rpc_forwardIndicateWebView:' message to app 'PID:504', page '1' (id: 4): 'indic

issue's one:

{"level":"VERBOSE","message":"[AppiumDriver@45ad] Appium v2.11.3 creating new XCUITestDriver (v7.24.14) session","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] Checking BaseDriver versions for Appium and XCUITestDriver","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] Appium's BaseDriver version is 9.11.3","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] XCUITestDriver's BaseDriver version is 9.11.3","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[XCUITestDriver@52b5] ","timestamp":"2024-08-11 00:51:07.671"}
...
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Received response from send (id: 2): '[\"PID:606\",{}]'","timestamp":"2024-08-11 00:51:14.406"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Sending to Web Inspector took 5ms","timestamp":"2024-08-11 00:51:14.407"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Skipping app 'PID:606'. Original error: Empty page dictionary received","timestamp":"2024-08-11 00:51:14.414"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.415"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.533"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.535"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.537"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Current applications available:","timestamp":"2024-08-11 00:51:14.915"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]     Application: \"PID:606\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         id: \"PID:606\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isProxy: false","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         name: \"Safari\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         bundleId: \"com.apple.mobilesafari\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isActive: true","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isAutomationEnabled: true","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         pageArray: undefined","timestamp":"2024-08-11 00:51:14.917"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]     Application: \"PID:398\"","timestamp":"2024-08-11 00:51:14.917"}

Your automation used different version of xcuitest driver. I wondered if recent refactoring in remote debugger affected pageArray selection

cc @mykola-mokhnach (just to share the remote debugger's thing)

Or... https://github.com/appium/appium-xcuitest-driver/blob/master/CHANGELOG.md#7249-2024-08-01 affected the behavior? My recent local test with browserName safari worked though. Need to debug

KazuCocoa · 2024-08-11T06:31:48Z

Could you test again with XCUITest driver v7.24.6 once? @agray

agray · 2024-08-11T06:52:56Z

I have never specified the version of XCUITest

See SauceLabs platform configurator documentation
https://app.saucelabs.com/platform-configurator

SUT is page hosted by SauceLabs:
http://saucelabs.com/test/guinea-pig

mykola-mokhnach · 2024-08-11T07:27:05Z

@agray Is the issue persistent, e. g. does it happen always or rather randomly? Are you able to reproduce the issue locally?

Could you please provide an example of a failed log with appium:safariLogAllCommunication capability set to true?

mykola-mokhnach · 2024-08-11T07:28:04Z

Also please review logs attached above - they contain your user name and the access token. I don't think you want these values being shared

agray · 2024-08-11T07:36:35Z

so please review logs attached above - they contain your user name and the access token. I don't think you want these values being shared

deleted

agray · 2024-08-11T07:36:58Z

@agray Is the issue persistent, e. g. does it happen always or rather randomly? Are you able to reproduce the issue locally?

Could you please provide an example of a failed log with appium:safariLogAllCommunication capability set to true?

Yes its persistent and I get the same exception locally while trying to initialise the driver on line 93 of SauceryBase.cs trying to instantiate a new IOSDriver. This is an integration test so it has to talk to SauceLabs.

mykola-mokhnach · 2024-08-11T07:44:55Z

Does it also happen on a single device or multiple devices? Have you tried to reboot the device if it is always the same and run the test again?

Eventually, there is also a possibility on Sauce to explicitly provide the Appium snapshot version to run on (https://docs.saucelabs.com/mobile-apps/automated-testing/appium/appium-versions/). You may try to run with an older version to verify the assumption whether the issue has to do with Appium-specific changes

agray · 2024-08-11T07:46:39Z

Does it also happen on a single device or multiple devices? Have you tried to reboot the device if it is always the same and run the test again?

The "device" is the one requested via DesiredOptions on SauceLabs. SauceLabs spins up a new instance for EVERY test.

mykola-mokhnach · 2024-08-11T07:49:02Z

The "device" is the one requested via DesiredOptions on SauceLabs. SauceLabs spins up a new instance for EVERY test.

Sorry, my question was a bit different. Do you always request the same device name from your test or every time the test is executed it runs on a different device from the cloud?

agray · 2024-08-11T07:50:19Z

Sorry, my question was a bit different. Do you always request the same device name from your test or every time the test is executed it runs on a different device from the cloud?

Same type of device, yes. How the selenium grid (which is what SauceLabs is) provides that device it up to the grid and part of the grid "service".

agray · 2024-08-11T07:53:23Z

Sauceforge/Saucery@54da2c8

https://github.com/Sauceforge/Saucery/actions/runs/10338099541

appiumRequests.json
appium.log
device.log

mykola-mokhnach · 2024-08-11T08:01:19Z

Lets sum up what needs to be done step by step:

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)
Run the same test with an older appium snapshot via appiumVersion sauce option
Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session)
Run the same test on a different device

KazuCocoa · 2024-08-13T02:58:47Z

Hm, I don't have 17.4.1 but have 17.5.1 iPhone. It returned an empty safari view as below with browserName: safari, so I was not able to see pageArray: undefined case as attached in this ticket. I changed web inspector config as well though. The base xcuitest was 7.24.14.
https://gist.github.com/KazuCocoa/4e2b14918e57cc1e3df932cc1cf789b3

Possibly the device has a specific condtion

agray · 2024-08-18T23:48:05Z

Any updates on this?

Dor-bl · 2024-08-19T04:59:27Z

Any updates on this?

Did you performed the below steps?

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)
Run the same test with an older appium snapshot via appiumVersion sauce option
Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session)
Run the same test on a different device

mykola-mokhnach · 2024-08-19T06:39:45Z

I tend to think this is an issue in iOS itself. I was able to reproduce the same behaviour locally with iOS 17.4 and 17.5 Simulators via the Safari remote debugger.
The only viable solution was to upgrade iOS version to 17.6 or 18.0

Btw, downgrade to 17.2 or 17.3 also did the trick

agray · 2024-08-19T12:12:23Z

Any updates on this?

Did you performed the below steps?

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)

Run the same test with an older appium snapshot via appiumVersion sauce option

Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session)
Run the same test on a different device

Have you?

agray · 2024-08-20T23:38:34Z

Had to rollback to IPhone 14 Pro Max on IOS 16.2 on emulated (IPhone 15 Pro Max on IOS 17.0 just hangs)
Had to rollback to IPhone 14 Pro Max on IOS 16 on real devices (IPhone 15 Pro Max on IOS 17 just hangs)

agray changed the title ~~[Bug]: <title>~~ [Bug]: IOS Real Device Tests started failing Aug 10, 2024

Dor-bl added ThirdParty iOS labels Aug 19, 2024

Dor-bl added the NeedsInfo label Aug 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: IOS Real Device Tests started failing #815

[Bug]: IOS Real Device Tests started failing #815

agray commented Aug 10, 2024 •

edited

Loading

Dor-bl commented Aug 10, 2024

agray commented Aug 10, 2024

Dor-bl commented Aug 10, 2024

KazuCocoa commented Aug 10, 2024

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

Dor-bl commented Aug 11, 2024

agray commented Aug 11, 2024 •

edited

Loading

Dor-bl commented Aug 11, 2024

agray commented Aug 11, 2024

KazuCocoa commented Aug 11, 2024

agray commented Aug 11, 2024

Dor-bl commented Aug 11, 2024

KazuCocoa commented Aug 11, 2024 •

edited

Loading

KazuCocoa commented Aug 11, 2024

agray commented Aug 11, 2024 •

edited

Loading

mykola-mokhnach commented Aug 11, 2024

mykola-mokhnach commented Aug 11, 2024

agray commented Aug 11, 2024

agray commented Aug 11, 2024 •

edited

Loading

mykola-mokhnach commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024

mykola-mokhnach commented Aug 11, 2024

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

mykola-mokhnach commented Aug 11, 2024

KazuCocoa commented Aug 13, 2024

agray commented Aug 18, 2024

Dor-bl commented Aug 19, 2024 •

edited

Loading

mykola-mokhnach commented Aug 19, 2024 •

edited

Loading

agray commented Aug 19, 2024

agray commented Aug 20, 2024

[Bug]: IOS Real Device Tests started failing #815

[Bug]: IOS Real Device Tests started failing #815

Comments

agray commented Aug 10, 2024 • edited Loading

Description

Environment

Details

Dor-bl commented Aug 10, 2024

agray commented Aug 10, 2024

Dor-bl commented Aug 10, 2024

KazuCocoa commented Aug 10, 2024

agray commented Aug 11, 2024 • edited Loading

agray commented Aug 11, 2024 • edited Loading

agray commented Aug 11, 2024 • edited Loading

Dor-bl commented Aug 11, 2024

agray commented Aug 11, 2024 • edited Loading

Dor-bl commented Aug 11, 2024

agray commented Aug 11, 2024

KazuCocoa commented Aug 11, 2024

agray commented Aug 11, 2024

Dor-bl commented Aug 11, 2024

KazuCocoa commented Aug 11, 2024 • edited Loading

KazuCocoa commented Aug 11, 2024

agray commented Aug 11, 2024 • edited Loading

mykola-mokhnach commented Aug 11, 2024

mykola-mokhnach commented Aug 11, 2024

agray commented Aug 11, 2024

agray commented Aug 11, 2024 • edited Loading

mykola-mokhnach commented Aug 11, 2024 • edited Loading

agray commented Aug 11, 2024

mykola-mokhnach commented Aug 11, 2024

agray commented Aug 11, 2024 • edited Loading

agray commented Aug 11, 2024 • edited Loading

mykola-mokhnach commented Aug 11, 2024

KazuCocoa commented Aug 13, 2024

agray commented Aug 18, 2024

Dor-bl commented Aug 19, 2024 • edited Loading

mykola-mokhnach commented Aug 19, 2024 • edited Loading

agray commented Aug 19, 2024

agray commented Aug 20, 2024

agray commented Aug 10, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

KazuCocoa commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

mykola-mokhnach commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

agray commented Aug 11, 2024 •

edited

Loading

Dor-bl commented Aug 19, 2024 •

edited

Loading

mykola-mokhnach commented Aug 19, 2024 •

edited

Loading