Allow to rewrite http-request value after server selection

[Et7f3]

Your Feature Request

allow http-request replace-path, http-request replace-uri or http-request replace-header to be processed after server selection. If I read the docs correctly, reqrep could fulfill this before his retirement.

What are you trying to do?

I want to have active-active configuration for my https app, behind the same fqdn (example.com)
I need server persistence (I use cookie)

The backend app check that Referrer contain his real fqdn (but it won't because client doesn't know real fqdn)

From here I can:

1. patch the app for each backend server (I don't control the code so it might imply maintenance burden)
2. ask haproxy to rewrite this header

I choose version 2

So I tried using reqrep but got as expected:
The 'reqrep' directive is not supported anymore since HAProxy 2.1. Use 'http-request replace-path', 'http-request replace-uri' or 'http-request replace-header' instead.

Not cool but trying to use:

1. http-request replace-header Referer ^https://example.com(.*)$ https://%[req.hdr(Host)]\1
Host is header send by browser not the rewritten header.
2. http-request replace-header Referer ^https://example.com(.*)$ https://%s\1
%s is <NOSRV> since it is before backend server selection.
3. http-request replace-header Referer ^https://example.com(.*)$ https://%b\1
%b is backend_app_name.
4. Tried 1 and 2 after server lines:
   It still aply before server selection and no warning like this one:
   `[WARNING] (24809) : config : parsing [/tmp/haproxy.conf:311] : a 'http-request' rule placed after a 'redirect' rule will still be processed before.
5. I can't find the variable that allow to access the persistence cookie:
   cookie SERVERID insert indirect nocache httponly secure: %[req.cook(SERVERID)] is `` when re-reading the documentation: inserted by HAProxy in server responses so yes it can't be in `req` but can't access the `res` object because we are in req phase.

Workaround:
first request the fqdn in my client: ignore the error, then send the real POST because now the SERVERID cookie is available.
I still find this unoptimal because we add a round trip: put unnecessary logic on client.
If reqrep was retired because performance issue or people having trouble with acl that are analyzed before can we have a http-request-second-expansion (like Makefile .SECONDEXPANSION), http-request-after-acl/selection or http-request-before-sending

Can we do this in lua as another workaround ?

Output of haproxy -vv

I don't know how to access those details but I have
haproxy version is 2.6.0-aloha-17d34501-flx2.12_albva
Aloha appliance v14.5.12 (1463)

[vkssv]

Hi Et7f3 !

Thanks for posting this !

Yes, reqrep is no longer supported, due to some internal architecture reworks and the most important, problems that this feature may cause in conjunction with retries, retry-on and option redispatch.

As you are using ALOHA appliance, we would prefer, that you post your requests at dedicated customer's portal here or, if you don't have yet an account at our customers portal, just send a email with a description of your problem and the output of haproxy -vv to this address [email protected]. Our support team will reply you and will create your account.

As for this feature request, it is quite risky to rewrite Referer header of the first client request (within the session) just after the server selection, because of the retry mechanism mentioned above.

Imagine, that the selected server hasn't replied or has replied badly and we've already modified Referer to point to it. Then, retry will be triggered, the request with already modified Referer will be resent to another server, but the Referer header will be not updated automatically, and so on and so forth.

So, if I've got right your idea, we are quite thoughtful about this feature at the moment.

Kind regards,

[wtarreau]

A few extra points on this topic:

• if your servers are configured with different FQDN, you're not doing load-balancing at all, but "something else" (yet to be defined). Load balancing implies that you're distributing incoming traffic to any server from a group configured similarly, so you shouldn't have to hack like this with servers expecting to see their own names in Referer.
• even if for whatever application but the server would expect such a thing, keep in mind that Referer is very rare nowadays since browsers almost always drop it, and that it would be much simpler to just unconditionally delete it on input;
• modifying headers post-LB causes a lot of trouble in various situations like redispatch after connect failures. This is supported only for one specific feature, "http-send-name-header", which probably is the feature that has caused the highest trouble-to-lines-of-code ratio over time. We've thought a few times about adding an "http-after-request" ruleset to perform such manipulations, but never implemented it due to the amount of trouble this causes when redispatching.

[Et7f3]

As you are using ALOHA appliance, we would prefer, that you post your requests at dedicated customer's portal here or, if you don't have yet an account at our customers portal, just send a email with a description of your problem and the output of haproxy -vv to this address [email protected]. Our support team will reply you and will create your account.

I opened in this open-source canal because I wanted to help (if not too time consuming) but ok I will use the dedicated channel.

As for this feature request, it is quite risky to rewrite Referer header of the first client request (within the session) just after the server selection, because of the retry mechanism mentioned above.

modifying headers post-LB causes a lot of trouble in various situations like redispatch after connect failures

We have the check directive so we shouldn't have to retry on failure ?

Then, retry will be triggered, the request with already modified Referer will be resent to another server, but the Referer header will be not updated automatically, and so on and so forth.

The rewrite rule can always be applied to the original message (this mean more memory usage because we keep original message + rewritten message to be sent). Or if we use regular expression we can then always successfully rewrite.

if your servers are configured with different FQDN, you're not doing load-balancing at all, but "something else" (yet to be defined)

I have node1.app, node2.app and node.app (this one proxy to either node). Since it is a https application it need to remember the server used to decrypt session cookies but both server display the same information. I think it is called Application layer persistence

so you shouldn't have to hack like this with servers expecting to see their own names in Referer.

and that it would be much simpler to just unconditionally delete it on input;

The application added this check to protect against CRSF.

[vkssv]

Hi Et7f3 !

Thanks a lot for your explanations !

As for protecting against CRSF, referer-based checks are generally less effective than CSRF token validation, because, returning to the previous post above, the most browsers have a trend to drop Referer, in order to protect user's privacy.

I would suggest instead to have a look to CSRF Token mechanics. In general it is implemented via modifying server-side applications: server app generates such CSRF token, puts it within a hidden HTML form field, which client obtains via GET and then POSTs back to really validate itself. So, CSRF token is sent back to server X along with sessionToken from server's X cookie. But CSRF token is more protected, than sessionToken from cookie, as it is stored on the client's side in the internal browser memory, but not together with cookies and it could be sent back to server only within a returned HTML form or via a special header.

As I've got it right, that you have haproxy in front of node1.app, node2.app and node.app ? Or haproxy co-exists with node.app on the same host/VM and then requests are re-dispatched to node1.app, node2.app ?

Kind regards,

[Et7f3]

As for protecting against CRSF, referer-based checks are generally less effective than CSRF token validation, because, returning to the previous post above, the most browsers have a trend to drop Referer, in order to protect user's privacy.

I agree but I don't control the application. If they change the mechanism then this feature request can be ignored but I haven't seen any upstream PR about it. Also I don't think it can really protect because it is a react app that store parameter in url/route so you can click on button without sending a form.

As I've got it right, that you have haproxy in front of node1.app, node2.app and node.app ? Or haproxy co-exists with node.app on the same host/VM and then requests are re-dispatched to node1.app, node2.app ?

Let's draw it:

flowchart TD;
   node.app -->|rewrite Referrer from node.app to node1.app|node1.app
   node.app -->|rewrite Referrer from node.app to node2.app|node2.app

So dns entry of node.app points directly on the haproxy instance

[wtarreau]

I'm sorry for insisting but this remains totally wrong from a basic architecture perspective. What this is saying is that both nodes are installed differently, that one thinks it runs for domain "node1.app" and the other one for domain "node2.app" while the application's address is supposed to be "node.app". There is no way an application can work this way!

The "referer" header, when present, contains the name of the site the client found the link that was clicked. In your case, assuming the client was on side "node.app", it must match "node.app" and the application must expect the entire site to be "node.app", never "node1.app" nor "node2.app". By the way, the links placed in the pages by the application contain "node.app", not "node1.app", otherwise no link would work.

Thus, if the application wants to validate the domain in the "referer" header, it must validate that it matches "node.app".

Another point, since you said that the application cannot be fixed to implement CSRF correctly, I'm not sure why you're trying to hack around it. In this case, couldn't you just implement the referer check directly in haproxy by verifying that every POST request coming with a Referer header has a Referer that matches "node.app" ? As Valentine said above, the referer method is unreliable but if you cannot do anything else it can still be better than nothing. In this case it would do approximately something like this untested config excerpt:

...
backend node.app
   # check that referer, if present, starts with http://node.app/ or https://node.app/ on all POST requests
   acl has_referer req.hdr(referer) -m found
   acl referer_ok req.hdr(referer) -m beg http://node.app/ https://node.app/
   http-request deny if METH_POST has_referer !referer_ok

This means "deny any POST request that comes with a "referer" header which doesn't start with http://node.app/ nor https://node.app/". That seems to be what you're trying to make the application do (which we apparently all agree is very light), but done the correct way by checking the correct domain and not something that would be specific to a possibly misconfigured server instance.

[Et7f3]

I'm sorry for insisting but this remains totally wrong from a basic architecture perspective.

Yes but

I agree but I don't control the application

Also the referrer in this app is checked in the code not in nginx or other webserver. Thanks for the snippet still.

However this application is open source so I can modify this application to allow multiples referrers (that way we can still access the web ui from haproxy name and from direct name).

I have found where this settings is stored and where it is used. I will open a PR to the upstream project. Since the initial motivation of this issue doesn't hold, I will close the issue and thanks for the assistance.