[Python] Is it possible to dynamically change proxy while scraping?

[mcharles96]

Hello guys!

I have a few codes that scrapes some websites running and working, but neither of that codes have a proxy setup. I tried some approaches over the internet to use public proxy to access websites, but my IP still being detected.

Is that possible to use a free proxy list without username and password authentication to make the requests? I'm currently using the undetected-chromedriver to do the tasks.

Thank you!

[ManiMozaffar]

You've to do that on your network, that's the best way of doing it.

Also you can do using extension.
By default, if you bound chrome to a proxy, then it keep using that.

[mcharles96]

But I have two problems/doubts:

1. And if the website blocks me or doesn't allow web scraping?
2. Assuming that 1 is true, how can I use proxy inside cloud? I'm using Databricks today to do the scrapes.

[Charlotte-br560]

Yes, you can switch proxies dynamically. Try using libraries like Requests or ProxyBroker for easy proxy management. Also, consider checking out Crawlbase for reliable proxies.

[st681]

Free proxies will be banned everywhere. You can use static/rotating residential proxies from many different providers for about 1USD/GB.

[denixyd]

can you refer to a website that provides quality residential proxies