Scrapy + FlareSolverr #876
dancook993
started this conversation in
General
Replies: 1 comment
-
Yes, here is a simple method which you can use to get your spiders using flaresolver directly (assuming the flaresolverr container is running on your local network). Have your spider yield requests to Flaresolverr: def get_fs_body(url, method="get"):
post_body = {
"cmd": f"request.{method}",
"url": url,
"maxTimeout": 60000
}
return json.dumps(post_body) yield scrapy.Request(url=FLARE_SOLVER_URL, callback=self.callback, body=get_fs_body(url), headers={"Content-Type":"application/json"}, method="POST") And then in your callback you just need to parse the flaresolverr response. def parse_fs_response(resp):
meta = resp.meta
resp = json.loads(resp.text)
response = HtmlResponse(url=resp['solution']['url'], body=resp['solution']['response'], encoding='utf-8')
return response, meta def callback(self, response):
response, meta = parse_fs_response(response)
... rest of your callback... Another option (probably better) - would be to write a download handler. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
Has anyone successfully integrated Scrapy and FlareSolverr?
Many thanks in advance
Beta Was this translation helpful? Give feedback.
All reactions