New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crawling duplicated url #22

Open

hbakhtiyor opened this issue Nov 18, 2023 · 1 comment

hbakhtiyor commented Nov 18, 2023

need to ignore the url if already crawled the page, in my case same url crawled several times

18758976569 commented Feb 6, 2024

请问怎么控制相同的网址是否可以抓取多次，我需要每次抓取都去实时的抓取

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment