Getting Requests & Responses Through CDP and Persisting Network History

[webdjoe]

I am loving this library, thank you for the hard work. I have another cdp network events question in undetected mode (I know this has been asked millions of times). There is a quirky issue where I cannot get all network traffic in UC mode. It is not due to an iframe as was the cause in another issue. You mentioned in several other issues this could be due to the disconnects and reconnects that the driver does to avoid detection but is there any way to capture all events despite this?

I've tried both adding an event handler and the --uc-cdp-events option with get_log("performance"). The event handler is successful at capturing some events but not the request I am looking for. I used the basic:

def add_cdp_listener(self):
        self.driver.add_cdp_listener(
            "Network.responseReceived",
            lambda data: pprint(data)
        )

When I use the performance logs, there are no network events logged until I call sb.refresh(), but it seems all previous network events were cleared before the refresh. I tried calling get_log at random intervals and oddly enough it returns different requests each time until nothing is returned a few seconds later. Is there a way to persist network events, like the "preserve" option in devtools? I see the network request logged in DevTools, so I know that it has definitely fired.

with SB(uc=True, uc_cdp_events=True) as sb:
    sb.open(SITE_URL)
    sb.execute_cdp_command("Network.enable", {})
   
    .....
    sb.wait(1)
    sb.get_log("performance") # Returns []
    sb.refresh()
    sb.driver.get_log("performance")
    sb.wait(0.5)
    sb.driver.get_log("performance")

Is there another way to go about this with undetected mode or would my only other option be to use an external MITM with proxy approach?

[mdmintz]

An MITM with the proxy approach is probably the best option. Data is lost in the browser during disconnects because the browser clears data that would make it detectable. If UC Mode gets people past the CAPTCHAs, I'm wondering what the "Network.enable" would be needed for if not for CAPTCHA bypass?

[webdjoe]

Thank you for getting back to me. My understanding is that it enables network events, which is required to get requests and responses through CDP. I didn't receive any network events from get_log("performance") at first, so I thought it might help but no change. Based on your question, I likely could have completely missed the mark with my understanding.

There is a way set set the preserve network logs option through chrome preferences, any idea whether that might be a viable option?

[mdmintz]

Set log_cdp_events=True (Or command-line: --log-cdp-events)

SeleniumBase/seleniumbase/plugins/sb_manager.py

Line 59 in e6e3c97

log_cdp_events=None, # Capture {"performance": "ALL", "browser": "ALL"}

For any other options you need, add them to chromium_arg. (See https://stackoverflow.com/q/47579699/7058266)

SeleniumBase/seleniumbase/plugins/sb_manager.py

Line 191 in e6e3c97

chromium_arg (str): "ARG=N,ARG2" (Set Chromium args, ","-separated.)