Issue with Cloudflare Detection in Headless Mode using SeleniumBase

[benfrankie]

Hi, Until recently, everything was working fine. However, a few hours ago, I started facing challenges with Cloudflare triggering a security screen in headless mode on Windows 11, and SB fails to bypass it. When running the script in the standard (non-headless) mode, this issue does not occur.

Here's a brief overview of my setup:

I'm using the uc_open_with_reconnect method provided by SeleniumBase for navigating to URLs.
The issue arises when attempting to access the login page of a website (in my case, fetlife.com ).
This seems to be a recent development, as there were no such issues until a few hours ago.

Any recommendations or workarounds you could suggest to help mitigate this issue in headless uc mode?
Thank you

driver = Driver(
    uc=True,
    headless=True,
    proxy=proxy_ip_port
)

driver.uc_open_with_reconnect("https://example.com/", reconnect_time=10)
 

login_url = "https://example.com/users/sign_in"    
driver.uc_open_with_reconnect(login_url, reconnect_time=20)

[mdmintz]

Hello @benfrankie, it might be due to #2523

If you can help me debug this, I can get a fix out quickly. I'll need you to run the following:

print(driver.execute_script("return navigator.userAgent;"))

There was a change made between 4.23.6 and 4.23.7, which affected headless UC Mode, but maybe that change is only needed on Mac/Linux and not on Windows.

Try both of those versions of SeleniumBase in headless UC Mode, get the user agent, and let me know what it is for the different situations. Also let me know the user agent of a regular Chrome browser for you. Also, in case the user agent is affected by the proxy server you used, let me know that too. It'll help me debug this and put in the correct fix.

[benfrankie]

Hi Michael @mdmintz

The issue persisted with all versions.

4.23.6:
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.0.0 Safari/537.36
4.23.7
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.0.0 Safari/537.36

The user agent of a regular Chrome browser:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

The user agent was not altered regardless of whether proxies are used or not.

The issue seems to have started after an automatic restart of my computer. This restart was probably due to Windows and Chrome updates a few hours ago.

Please advise. Thank you!

[mdmintz]

@benfrankie, I need to know why this line wasn't reached:

SeleniumBase/seleniumbase/core/browser_launcher.py

Line 3475 in b2ec928

user_agent = user_agent.replace(

It replaces "Headless" with "" in the user-agent so that "HeadlessChrome" becomes "Chrome". If you can clone SeleniumBase and drop in a breakpoint() somewhere before that, maybe we can figure out why your user-agent wasn't changed.

The quick fix would be setting: agent="User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" inside the Driver() initialization. (Removing the "Headless" part directly.) So that's the quick fix. For the long-term fix, I need to know why that line wasn't reached. Are you familiar with the breakpoint() command? Once I know why that line wasn't reached for you, I can fix it for the long term.

[benfrankie]

@mdmintz I've been debugging the issue with SeleniumBase in headless UC mode as per your guidance. Here are my findings:

Execution Point: The execution successfully reaches the breakpoint in the browser_launcher.py file.

Before Breakpoint: The user agent was Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.0.0 Safari/537.36.
After Breakpoint: The user agent remained the same: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.0.0 Safari/537.36.
It appears that the code meant to modify the user agent (specifically to remove "Headless") is not having the intended effect. This might be contributing to the issues I'm experiencing with headless mode and Cloudflare.

Non-Headless Mode: Interestingly, when running in non-headless mode, the script doesn't hit the breakpoint but continues running without issue.

Any insights or further steps I should take would be greatly appreciated.

[mdmintz]

@benfrankie OK, so this code here... (which comes right after the breakpoint)

user_agent = user_agent.replace("Headless", "")

...should have changed your user-agent from Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.0.0 Safari/537.36 to just Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 (Changing HeadlessChrome to Chrome.)

How did that change not occur? (Your user-agent had HeadlessChrome in it.) What are we missing?