Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

建议添加目录及增量下载 #4

Open
kabumos opened this issue Oct 12, 2017 · 2 comments
Open

建议添加目录及增量下载 #4

kabumos opened this issue Oct 12, 2017 · 2 comments

Comments

@kabumos
Copy link

kabumos commented Oct 12, 2017

建议:

  1. 下载文件放到目录中;
  2. 添加重复处理,已下载的文件不再被下载。

建立一个缓存文件,下载后的文件URL记录下来。第二次运行时,查看缓存文件,若已存在则跳过该文件。

@kulovecc
Copy link
Owner

后面有时间可能会完善一下。
当初写这个脚本的初衷是:在一个网站上看到了爬取煎蛋图片的文章,但是只能爬取第一页,后来索性就自己写了一个,没想到被这么多人收藏了..

@stevenling
Copy link

现在还能用吗?煎蛋网是有反爬虫策略了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants