New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrawlerThread的process方法里判断当前链接是否是白名单链接逻辑有问题 #12

Open

lomoye opened this issue Aug 5, 2018 · 1 comment

lomoye commented Aug 5, 2018

// ------- pagevo ----------
if (!crawler.getRunConf().validWhiteUrl(link)) { // limit unvalid-page parse, only allow spread child
return false;
}

这一段代码返回false，如果用户设置了重试次数，会导致无意义的重试。这里应该返回true

Owner

xuxueli commented Oct 15, 2018

你好，感谢反馈！
已经合并至master，将会跟随想下个版本一同发布。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment