Avoid being tracked and located while web scrapping

[francis-fm]

Select Topic Area

Question

Body

Hello community, I have a project in mind. I want to do web scrapping of several pages of retail companies. As I am extracting all the information of all the products and be able to store, I need to not leave any trace or evidence such as my IP, address of my machine, etc.. I want to do it in Python using selenium and Google Driver libraries. The goal is that every two days the code gets the information of all the products from the web pages of the main suppliers of a retail store. I want to extract all the information and avoid being tracked or maybe generate a change of internet network or dynamic IP. Among the options mentioned is for example to do it in a virtual machine but it seems to me that it leaves a disaster. I would like to know some recommendations and suggestions to be able to do this project and avoid being tracked. I would like to know how I could apply, what environment and how I could do it. I appreciate your comments and input.

[LiteBrite82]

Thanks for posting in the GitHub Community, @francis-fm !

We're happy you're here. You are more likely to get a useful response if you are posting your question in the applicable category, the Discussions category is solely related to conversations around the GitHub product Discussions. This question should be in the Programming Help category. I've gone ahead and moved it for you. Good luck!

[francis-fm]

Thanks. Right now I change

[CopperEagle]

So I mean the topic of hiding identity on the web is a huge one. It also depends on the website you talk about.

To this end, you could go full bonkers using TOR or a trustworthy VPN, Qubes OS or Tails OS, etc. But then, you also need to be aware of techniques such as browser fingerprinting that turn your request headers, settings and the browser environment (available fonts, rendering, etc.) to a pseudonym (e.g. hashing these together) to reidentify you.

And by the way, Virtual Machines alone would not change anything for most. The IP the VM has is a local one. It is translated at the NAT (Network address translation, located typically in your home router). The IP that the webserver can see typically is the one from your NAT.

So... I am not aware of a single project that suits your need. A reasonable approach may be the fallowing:

• Use some VPN. This should be good enough because few IP addresses may correspond to many people. Then, dynamic IP changes should not be that important anymore.
• Use a scrapper without Javascript (e.g. you can use plain requests from Python). Not having Javascript can disable some forms of browser fingerprinting but not all of them. If you allow me some shameless advertising: I wrote a in Python using Selenium that allows you to enable Javascript on a per (sub) domain basis and profiles to modify your digital signature, like the access pattern. Note that since many websites nowadays fetch scripts from many domains (like googleanalytics.com), it may take some experimentation to allow only those you actually need. It uses the Firefox driver though. It is the equivalent of noscript for scrapers.

Pro tip: Your frequency of access can reveal whether you are a human. If you keep accessing websites in a near perfect one second interval, you're likely a bot. Now there are usually way fewer bots on the website than human visitors. Thus, you are easier to reidentify.

You should make sure that you scrapping the targeted website is legal, by the way. This is not legal advice by any means.

[MarouaneBenbetka]

I personally use Selenium for interactions [with an undecteced driver], and then BeautifulSoup for parsing elements.
it's like accesing the web pages using your real profile, so mosot of the websites won't detect you.

hope this answers your questions, if so mark it as answered.

import undetected_chromedriver as uc

options = uc.ChromeOptions()
options.headless = False
options.add_argument(
    r"--user-data-dir=C:/PATH/TO/YOUR/CHROME/PROFILE")
driver = uc.Chrome(options=options)

GOOD LUCK.