Add ability to set referer header

[HDoujinDownloader]

Environment

• FlareSolverr version: v2.1.0
• Last working FlareSolverr version: v1.2.9
• Operating system: Win10
• Are you using Docker: no
• FlareSolverr User-Agent (see log traces or / endpoint): Mozilla/5.0 (Windows NT
  6.3; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0
• Are you using a proxy or VPN? no
• Are you using Captcha Solver: no
• If using captcha solver, which one:
• URL to test this issue: https://cdn3.webtoon.xyz/manga_5e585e57af3fc/chapter-187/001.jpg

Description

FlareSolverr v2.0.0+ removed the ability to specify headers with requests. This causes issues with domains that have custom firewall rules to block requests without a whitelisted referrer, resulting in a Error 1020 page from Cloudflare.

Logged Error Messages

2021-12-22T00:00:00-00:00 INFO REQ-7 Cloudflare detected
2021-12-22T00:00:00-00:00 INFO REQ-7 Cloudflare Error: Cloudflare has blocked this request. Probably your IP is banned for this site, check in your web browser.

Screenshots

N/A

[ngosang]

FlareSolverr 2.0.0 uses Firefox instead of Chrome and this feature is not supported in Puppeteer for Firefox.

You can write your own script to:

• Make a regular GET request with FlareSolverr and save the Cookies and User-Agent returned.
• Make a POST request without using FlareSolverr. You can use cURL, Python or any language but make sure you are setting the User-Agent and Cookie headers properly.

[HDoujinDownloader]

For the record, I'm using FlareSolverr with C#, and even when the User-Agent and Cookie headers are set appropriately, requests will still often get 403'd. I think in the case where the site is using Cloudflare's HTTPS service it analyzes the TLS handshake and uses the available encryption algorithms to create a fingerprint and block the request.

The same issue happens with cURL, where the order of the headers (which cURL does not allow to be configured) is another dead giveaway that the request isn't originating from a browser.

[ngosang]

I think in the case where the site is using Cloudflare's HTTPS service it analyzes the TLS handshake and uses the available encryption algorithms to create a fingerprint and block the request.

I'm not sure about that but it could be.
Notice that the Cloudflare cookie expires in less than an hour. Maybe it helps you.

[Sunda001]

FlareSolverr 2.0.0 uses Firefox instead of Chrome and this feature is not supported in Puppeteer for Firefox.

You can write your own script to:

• Make a regular GET request with FlareSolverr and save the Cookies and User-Agent returned.
• Make a POST request without using FlareSolverr. You can use cURL, Python or any language but make sure you are setting the User-Agent and Cookie headers properly.

so, there's no plan to add headers on the next version?

[ale5000-git]

@ngosang

FlareSolverr 2.0.0 uses Firefox instead of Chrome and this feature is not supported in Puppeteer for Firefox.

Since the last version of FlareSolverr use Chrome, is there any update on this issue?