Nodriver proxy with authentication

[strukiii]

Hi all, I'm putting this here since there is no nodriver discussions section. I'm trying to get the browser setup with an authenticated proxy but chrome does not support credentials via browser args. I found this, where the answer from Arkadiy Bolotov is interesting. There are the same cdp methods as in his post available in Network though they are deprecated for those in Fetch. Here is my current attempt I put together this afternoon, which does intercept the requests however no authRequired events are fired as the documentation suggests.

class Scraper:
    main_tab: tab

    def __init__(self):
        loop().run_until_complete(self.run())

    async def run(self):
        browser = await start(
            browser_args=[f"--proxy-server={PROXY}"],
        )
        self.main_tab = browser.main_tab
        await self.main_tab.send(cdp.fetch.enable(handle_auth_requests=True))
        self.main_tab.add_handler(cdp.fetch.RequestPaused, self.req_paused)
        self.main_tab.add_handler(cdp.fetch.AuthRequired, self.auth_challenge_handler)
        page = await browser.get('https://www.whatismyip.com/')
        await asyncio.sleep(6000)


    async def auth_challenge_handler(self, event: cdp.fetch.AuthRequired):
        # Split the credentials
        # Respond to the authentication challenge
        await self.main_tab.send(cdp.fetch.continue_with_auth(
            request_id=event.request_id,
            auth_challenge_response=cdp.fetch.AuthChallengeResponse(
                response="ProvideCredentials",
                username=USERNAME,
                password=PASSWORD
            )
        ))


    async def req_paused(self, event: cdp.fetch.RequestPaused):
        await self.main_tab.send(cdp.fetch.continue_request(request_id=event.request_id))


if __name__ == '__main__':
    Scraper()

[WindBora]

Hi

Here 2 problems arises:

1. You add handlers after enabling domain (internal implementation overrides enabling domain) - that's the reason why handlers do not receive an event
2. When you will finally receive an event, your response will block all processes

Solutions:

1. Add handlers before enabling domain. Only after adding them do enable domain
2. For such blocking requests there is a must to just use fire-and-forget method e.g. do not await for sending response on req_paused or auth_required. Just wrap the call into asyncio.create_task

Example of working code based of of yours:

class Scraper:
    main_tab: uc.Tab

    def __init__(self):
        uc.loop().run_until_complete(self.run())

    async def run(self):
        browser = await uc.start(
            browser_args=[f"--proxy-server={PROXY}"],
        )
        self.main_tab = await browser.get("draft:,")
        self.main_tab.add_handler(uc.cdp.fetch.RequestPaused, self.req_paused)
        self.main_tab.add_handler(
            uc.cdp.fetch.AuthRequired, self.auth_challenge_handler
        )
        await self.main_tab.send(uc.cdp.fetch.enable(handle_auth_requests=True))
        page = await browser.get("https://www.whatismyip.com/")
        await asyncio.sleep(6000)

    async def auth_challenge_handler(self, event: uc.cdp.fetch.AuthRequired):
        # Split the credentials
        # Respond to the authentication challenge
        asyncio.create_task(
            self.main_tab.send(
                uc.cdp.fetch.continue_with_auth(
                    request_id=event.request_id,
                    auth_challenge_response=uc.cdp.fetch.AuthChallengeResponse(
                        response="ProvideCredentials",
                        username=USERNAME,
                        password=PASSWORD,
                    ),
                )
            )
        )

    async def req_paused(self, event: uc.cdp.fetch.RequestPaused):
        asyncio.create_task(
            self.main_tab.send(
                uc.cdp.fetch.continue_request(request_id=event.request_id)
            )
        )


if __name__ == "__main__":
    Scraper()

[strukiii]

Great work! Just tried it out, works perfectly so far.

[raunaqss]

Thanks so much, this is helpful for me as well