Update the scrapy-poet config #28
Conversation
| SPIDER_MIDDLEWARES = { | ||
| "scrapy_poet.RetryMiddleware": 275, | ||
| } | ||
| REQUEST_FINGERPRINTER_CLASS = "scrapy_poet.ScrapyPoetRequestFingerprinter" |
There was a problem hiding this comment.
This interacts both with the duplicate url discarder and with scrapy-zyte-api (see https://github.com/zytedata/duplicate-url-discarder/blob/d6920865039db10d1eb0a6763d1fa3be20e3469b/duplicate_url_discarder/_addon.py#L26 and https://github.com/scrapy-plugins/scrapy-zyte-api/blob/cc3cfddcd5e38502d7cda211178aafcd0a8863d5/scrapy_zyte_api/addon.py#L76).
I don't expect any issues, but we should check somehow that it works.
There was a problem hiding this comment.
Working on a test for it, I found out it was working as expected before this change. It’s because the scrapy-zyte-api fingerprinter already uses the scrapy-poet one as fallback if scrapy-poet is installed. So this line is unnecessary.
There was a problem hiding this comment.
On a related note, the scrapy-zyte-api add-on also configures the injection middleware,
I did not detect any issue or even tested the effect of these settings (i.e. if unsure, maybe we should test them before merging), but these settings are defined in the configuration docs of scrapy-poet and they do not seem to be configured by any add-on (which makes sense, they should probably be set by a scrapy-poet add-on).