feature request: ARC for forwarded emails

[Mygod]

Context

Authenticated Received Chain (ARC) reduces bouncing of forwarded emails. See #3642.

Description

Have an easy switch or a guide to enable ARC in DMS.

Alternatives

Currently it seems possible to enable it through rspamd with a lot of manual tinkering (still testing?): #3642 (comment)

OpenARC is another alternative but doesn't seem quite good: trusteddomainproject/OpenARC#157

Applicable Users

Users that forward emails with DMS.

Are you going to implement it?

Yes, because I know the probability of someone else doing it is low and I can learn from it.

What are you going to contribute?

I could PR a guide to the wiki.

[georglauterbach]

I think we can easily integrate an ENV like RSPAMD_ENABLE_ARC (or something similar), if the config really is just 3 lines - or, make it a default even?

[polarathene]

Other mail projects like DMS have this feature already setup with rspamd, so you could likely refer to their source for what they're doing 😎

[georglauterbach]

Overview

1. @Mygod provided his current additional configuration:

   sign_local = false;
   allow_hdrfrom_mismatch = true;
   use_domain = "recipient";

2. Here is mailcows configuration:

   # If false, messages with empty envelope from are not signed
   allow_envfrom_empty = true;
   # If true, envelope/header domain mismatch is ignored
   allow_hdrfrom_mismatch = true;
   # If true, multiple from headers are allowed (but only first is used)
   allow_hdrfrom_multiple = false;
   # If true, username does not need to contain matching domain
   allow_username_mismatch = false;
   # If false, messages from authenticated users are not selected for signing
   sign_authenticated = false;
   # Default path to key, can include '$domain' and '$selector' variables
   path = "/data/dkim/keys/$domain.dkim";
   # Default selector to use
   selector = "dkim";
   # If false, messages from local networks are not selected for signing
   sign_local = false;
   # Symbol to add when message is signed
   symbol = "ARC_SIGNED";
   # Whether to fallback to global config
   try_fallback = true;
   # Domain to use for DKIM signing: can be "header" or "envelope"
   use_domain = "recipient";
   # Whether to normalise domains to eSLD
   use_esld = false;
   # Whether to get keys from Redis
   use_redis = true;
   # Hash for DKIM keys in Redis
   key_prefix = "DKIM_PRIV_KEYS";
   # Selector map
   selector_prefix = "DKIM_SELECTORS";
   sign_inbound = true;
   use_domain_sign_inbound = "recipient";

3. Comparing to my currently running DMS instance (no explicit ARC configuration happening here):

   root@mail:/# rspamadm configdump arc
   *** Section arc ***
   sign_networks [
       "127.2.4.7",
   ]
   sign_authenticated = false;
   use_esld = true;
   selector = "arc";
   allow_envfrom_empty = true;
   allow_hdrfrom_multiple = false;
   sign_inbound = true;
   sign_local = false;
   sign_symbol = "ARC_SIGNED";
   use_domain = "recipient";
   allow_hdrfrom_mismatch = true;
   use_redis = false;
   allow_username_mismatch = false;
   key_prefix = "ARC_KEYS";
   try_fallback = true;
   
   *** End of section arc ***

4. The official Rspamd configuration of the ARC module:

   # If false, messages with empty envelope from are not signed
   allow_envfrom_empty = true;
   # If true, envelope/header domain mismatch is ignored
   allow_hdrfrom_mismatch = true;
   # If true, multiple from headers are allowed (but only first is used)
   allow_hdrfrom_multiple = false;
   # If true, username does not need to contain matching domain
   allow_username_mismatch = false;
   # If false, messages from authenticated users are not selected for signing
   sign_authenticated = false;
   # Default path to key, can include '$domain' and '$selector' variables
   path = "/data/dkim/keys/$domain.dkim";
   # Default selector to use
   selector = "dkim";
   # If false, messages from local networks are not selected for signing
   sign_local = false;
   # Symbol to add when message is signed
   symbol = "ARC_SIGNED";
   # Whether to fallback to global config
   try_fallback = true;
   # Domain to use for DKIM signing: can be "header" or "envelope"
   use_domain = "recipient";
   # Whether to normalise domains to eSLD
   use_esld = false;
   # Whether to get keys from Redis
   use_redis = true;
   # Hash for DKIM keys in Redis
   key_prefix = "DKIM_PRIV_KEYS";
   # Selector map
   selector_prefix = "DKIM_SELECTORS";
   sign_inbound = true;
   use_domain_sign_inbound = "recipient";

Current Status Quo

It seems all the changes @Mygod applied are already defaults. The way I understood this issue is that @Mygod stated they had to add the configuration posted explicitly. @Mygod did you match path = "/data/dkim/keys/$domain.dkim"; (from the ARC configuration) for the path to your DKIM keys by chance? I guess the ARC module still needs access to the keys, and we'd need to provide the path explicitly? Otherwise, I do not see how the configuration you applied explicitly @Mygod enabled ARC, because the way I understand our and your setup, ARC is never disabled in the first place. @Mygod can you please post the output of rspamadm configdump dkim_signing arc? :)

[Mygod]

Ah I see I haven't actually enabled the module. I believe I've mistaken gmail's arc headers as rspamd. Is there a way to test arc?

[georglauterbach]

I think you will need to provide path explicitly. The configuration is similar to DKIM. This should probably do the trick (I hope)... At the moment, Rspamd evaluates ARC properly already. Testing whether Rspamd adds an ARC header would be done by having DMS relay an email and check on the final receiver's side whether DMS added ARC headers.

[Mygod]

I copied the domain section from dkim_signing to arc.conf and restarted DMS. I didn't see anything changed in the ARC headers in my gmail client. 🤔

[georglauterbach]

I copied the domain section from dkim_signing to arc.conf and restarted DMS. I didn't see anything changed in the ARC headers in my gmail client. 🤔

Did DMS forward an e-mail from another mail server to your GMail account?

[Mygod]

Yep. Gmail -> my server -> Gmail.

[georglauterbach]

I don't know at the moment whether copying the whole domain section works.. I guess ARC only needs the public private key? I never worked with ARC before.

EDIT: Seems like copying from DKIM signing is just fine.

EDIT2: Just out of curiosity, how do you restart DMS? The container has to be destroyed properly, otherwise your changes may not have any effect.

[Mygod]

I did docker-compose down && docker-compose up -d.

I enabled debug log. Looks like some misconfiguration: arc; lua_dkim_tools.lua:195: mail is ineligible for signing

[Mygod]

I managed to make it work now (I think). Somehow sign_local = true; is also needed or the arc module would refuse to sign (maybe because of docker networking? It looks like dkim_signing has this enabled too.). I updated #3642 (comment) with the config.

Before: (this is presumably gmail's auth result)

After:

[georglauterbach]

The default in DMS for sign_local in dkim_signing is actually false

docker-mailserver/target/bin/rspamd-dkim

Lines 218 to 241 in aba218e

	cat >"${DEFAULT_CONFIG_FILE}" << EOF
	# documentation: https://rspamd.com/doc/modules/dkim_signing.html
	enabled = true;
	sign_authenticated = true;
	sign_local = false;
	try_fallback = false;
	use_domain = "header";
	use_redis = false; # don't change unless Redis also provides the DKIM keys
	use_esld = true;
	allow_username_mismatch = true;
	check_pubkey = true; # you want to use this in the beginning
	domain {
	${DOMAIN} {
	path = "${PRIVATE_KEY_FILE}";
	selector = "${SELECTOR}";
	}
	}
	EOF

Do you have a special Docker setup (64NAT or something alike)? There are multiple reasons the message is regard as "local", and it is not unlikely that this stems from the way DMS handles forwarding. sign_local does not hurt here, so enabling it is not a problem, but I still wonder why it is required.

EDIT: In my production setup, sign_local is set to true for DKIM signing as well... so I guess setting is required.

[Mygod]

Not according to this:

docker-mailserver/docs/content/config/best-practices/dkim_dmarc_spf.md

Line 152 in aba218e

sign_local = true;

[georglauterbach]

Not according to this:

docker-mailserver/docs/content/config/best-practices/dkim_dmarc_spf.md

Line 152 in aba218e

sign_local = true;

Can you be more specific, please?:)

[Mygod]

I was trying to say that I followed this tutorial and that's why I had this sign_local enabled. Probably should add sign_authenticated = true as well. :)

I don't remember my Docker setup. It's probably default other than enabling ip6tables I think?

[georglauterbach]

I was trying to say that I followed this tutorial and that's why I had this sign_local enabled. Probably should add sign_authenticated = true as well. :)

👍🏼

I don't remember my Docker setup. It's probably default other than enabling ip6tables I think?

And there we go - that's very likely the reason :D If the default 64NAT is applied, then the message appears as if it is coming from a local network (Docker's), hence the requirement. Please note that this may also be the reason your SPF checks fail - the origin IP is not correct when mail servers connect to your instance via IPv6. (There are a million other reasons SPF could not check out; the usual advice I give to people: don't use IPv6 😆).

[georglauterbach]

Alright - I guess ARC works pretty much out of the box except for

1. sign_local when 64NAT is active (but this also breaks SPF, i.e., it should be resolved by the user)
2. sign_authenticated is probably a good idea
3. the domain section of DKIM signing should be in the ARC configuration as well

Did I miss anything @Mygod?

[Mygod]

LGTM. 👍

[georglauterbach]

I'll look into it when my other PRs are merged :)

[georglauterbach]

We can look into this when #3778 is completed, which itself is a subtask of #3630. Any help is appreciated.

[dustinmoris]

Yep. Gmail -> my server -> Gmail.

@Mygod How did you set up the forwarding from DMS to Gmail? I set up Rspamd with ARC in my DMS and from my limited testing the ARC headers are being added but my emails which are being forwarded to Gmail are still bouncing. I use Mailgun as a relay though, which unfortunately I have to do because port 25 is blocked for outgoing traffic in Google Cloud.

Could you share a bit more details on your setup which seems to work? For example how did you set up the forwarding? I am using postfix-virtual.cf alias rules for that. Is there a different way which I could try that might work?

---

Nevermind, I got ARC working. Emails arrive at my Gmail now. My issue was that I had to configure relay settings through a config file rather than the ENV vars. I needed to set these vars in order for it to work with Mailgun:

Screenshot taken from this guide:
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-mail-relay-with-postfix-and-mailgun-on-ubuntu-16-04

This is my arc.conf which I mount into /etc/rspamd/local.d/arc.conf in my container:

sign_authenticated = false;
sign_local = false;
sign_inbound = true;

sign_symbol = "ARC_SIGNED";

allow_hdrfrom_mismatch = true;

try_fallback = true;

use_domain = "recipient";
use_esld = true;
use_redis = false;

selector = "mail";
path = "/tmp/docker-mailserver/rspamd/dkim/$domain/$selector.private";

sign_headers = "(o)from:(o)sender:(o)reply-to:(o)subject:(o)date:(o)message-id:(o)to:(o)cc:(o)mime-version:(o)content-type:(o)content-transfer-encoding:resent-to:resent-cc:resent-from:resent-sender:resent-message-id:(o)in-reply-to:(o)references:list-id:list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature"

The only reason why I have use_redis = false; is because I configured a cluster shared Redis instance via redis.conf, so you can ignore this.

[polarathene]

My issue was that I had to configure relay settings through a config file rather than the ENV vars. I needed to set these vars in order for it to work with Mailgun

You likely won't need to worry about that as much when v14 is out. Quite a few relay host feature support issues were resolved:

docker-mailserver/CHANGELOG.md

Line 52 in 849293f

- The relay host feature was refactored ([#3845](https://github.com/docker-mailserver/docker-mailserver/pull/3845))

docker-mailserver/CHANGELOG.md

Lines 98 to 101 in 849293f

	- Relay host feature refactored ([#3845](https://github.com/docker-mailserver/docker-mailserver/pull/3845))
	- `DEFAULT_RELAY_HOST` ENV can now also use the `RELAY_USER` + `RELAY_PASSWORD` ENV for supplying credentials.
	- `RELAY_HOST` ENV no longer enforces configuring outbound SMTP to require credentials. Like `DEFAULT_RELAY_HOST` it can now configure a relay where credentials are optional.
	- Restarting DMS should not be required when configuring relay hosts without these ENV, but solely via `setup relay ...`, as change detection events now apply relevant Postfix setting changes for supporting credentials too.

---

You'd want to go with the DEFAULT_RELAY_HOST ENV, however this will only properly configure from v14 IIRC. Our current edge version docs for the relay host feature page + the relay section at our ENV docs page were also rewritten accordingly.

Here's the location of the config settings in your screenshot to show that we handle that for you 👍

docker-mailserver/target/scripts/helpers/relay.sh

Lines 94 to 101 in 849293f

	# Enable credential lookup + SASL authentication to relayhost:
	# - `noanonymous` enforces authentication requirement
	# - `encrypt` enforces requirement for a secure connection (prevents sending credentials over cleartext, aka mandatory TLS)
	postconf \
	'smtp_sasl_password_maps = texthash:/etc/postfix/sasl_passwd' \
	'smtp_sasl_auth_enable = yes' \
	'smtp_sasl_security_options = noanonymous' \
	'smtp_tls_security_level = encrypt'

docker-mailserver/target/postfix/main.cf

Line 56 in 849293f

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination

smtp_sasl_mechanism_filter should not need to be set.

[dustinmoris]

@polarathene That is fantastic, thank you for the information, I'm looking forward to v14!

[Mygod]

I think my set up is still not quite right. I am still getting some bounced emails...

Mar 22 01:16:14 sparta postfix/smtp[3272020]: 3B9BA58B0: to=<[email protected]>, orig_to=<[email protected]>, relay=gmail-smtp-in.l.google.com[74.125.199.26]:25, delay=2.6, delays=1.7/0.01/0.18/0.7, dsn=5.7.26, status=bounced (host gmail-smtp-in.l.google.com[74.125.199.26] said: 550-5.7.26 This mail has been blocked because the sender is unauthenticated. 550-5.7.26 Gmail requires all senders to authenticate with either SPF or DKIM. 550-5.7.26 550-5.7.26 Authentication results: 550-5.7.26 DKIM = did not pass 550-5.7.26 SPF [sender server host] with ip: [my MX server IP] = did not 550-5.7.26 pass 550-5.7.26 550-5.7.26 For instructions on setting up authentication, go to 550 5.7.26 https://support.google.com/mail/answer/81126#authentication r3-20020a62e403000000b006e73e2f50b3si1418318pfh.193 - gsmtp (in reply to end of DATA command))

EDIT: I added sign_authenticated = true. Let's see how it goes.

[dustinmoris]

@Mygod Did you set up dkim signing via Rspamd as well?

Your error message suggests you might have not done it or misconfigured it?

Gmail requires all senders to authenticate with either SPF or DKIM

My dkim_signing.conf looks as following:

  dkim_signing.conf: |
    enabled = true;

    symbol = "DKIM_SIGNED";

    try_fallback = true;

    sign_authenticated = true;
    sign_local = true;

    allow_envfrom_empty = true;
    allow_hdrfrom_mismatch = false;
    allow_hdrfrom_multiple = false;
    allow_username_mismatch = false;

    use_domain = "header";
    use_redis = false;
    use_esld = true;
    check_pubkey = false;

    selector = "mail";
    path = "/tmp/docker-mailserver/rspamd/dkim/$domain/$selector.private";

Then I have mounted it like so:

- name: files
  subPath: dkim_signing.conf
  mountPath: /etc/rspamd/override.d/dkim_signing.conf
  readOnly: true

[Mygod]

I think docker-mailserver supports configuring dkim_signing already. rspamadm configdump dkim_signing also shows that mine is enabled.

[Mygod]

After some debugging, I see the email in question only supports SPF but not DKIM (this was sent by an external server). I wonder if ARC could be helpful at all for getting this email through gmail...

[Mygod]

Hmm this might be relevant. My mail server is attaching ARC-Authentication-Results: i=1; <mail server>; none. This seems wrong.

[Mygod]

EDIT: I fixed it. It turns out I forgot to properly configure IPv6 support for my container.

[Mygod]

I am still seeing ARC not being applied in some cases, more specifically, if the sender passes SPF but does not have DKIM, ARC fails for some reason. Gmail simply bounces the forwarded email, and in Microsoft Outlook I see this:

ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=softfail; dmarc=none action=none; dkim=none (message not signed); arc=fail (47)

Other emails with DKIM have arc=pass as expected. How should I debug this? 🤔

[georglauterbach]

Is this possibly expected behavior? I mean, if there is no DKIM, can ARC actually work at all?

[Mygod]

Hmm I'm not sure but I thought it was supposed to help. Microsoft's ARC-Seal also reports cv=fail so maybe there's something wrong with the signature?

ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=fail; b=...

[georglauterbach]

The example you posted in #3759 (comment) failed SPF, DKIM and DMARC. I'd guess ARC is not supposed to work then...

[georglauterbach]

This seems to indicate DKIM (or at least the DNS record) is a requirement for ARC to work.

[Mygod]

@georglauterbach Yes but that's the downstream validation result. Upstream result has SPF pass but missing DKIM and DMARC.

ARC-Authentication-Results: i=1;
	my.mailserver.com;
	dkim=none;
	spf=pass (omitted);
	dmarc=none

[Mygod]

This seems to indicate DKIM (or at least the DNS record) is a requirement for ARC to work.

Note: For the ARC signatures to be valid, the DNS records required for DKIM to work must be configured properly for the domain hosting the forwarding mail account.

I think it is saying that the domain hosting the forwarding mail needs to have DKIM, which I have. I am unsure if it is required for the sender to have working DKIM. (To be honest, if ARC requires DKIM on the sender then ARC seems a bit pointless?)

[georglauterbach]

You're right, my bad.

One question: Can you please explain to me why having DKIM on the sender's side as a requirement would render ARC mood?

[Mygod]

I'm not saying this because I saw this documented anywhere. I just think that if the original email is already signed with DKIM then we can already know the authenticity of the email, right?

[polarathene]

I just think that if the original email is already signed with DKIM then we can already know the authenticity of the email, right?

It's about the receiver trusting the intermediary that forwards the original senders mail. DKIM should help with that trust for the original sender, but ARC is for trust with the intermediary inbetween.

https://en.wikipedia.org/wiki/Authenticated_Received_Chain

Authenticated Received Chain (ARC) is an email authentication system designed to allow an intermediate mail server like a mailing list or forwarding service to sign an email's original authentication results.
This allows a receiving service to validate an email when the email's SPF and DKIM records are rendered invalid by an intermediate server's processing.

and

However, a strict DMARC policy may block legitimate emails sent through a mailing list or forwarder, as the DKIM signature will be invalidated if the message is modified, such as by adding a subject tag or footer, and the SPF check will either fail (if the forwarder didn't change the bounce address) or be aligned with the mailing list domain and not with the message author's domain (unless the mailing list rewrites the From: header field.)

ARC was devised to solve this problem by giving intermediate servers a way to sign the original message's validation results. Even if the SPF and DKIM validation fail, the receiving service can choose to validate the ARC chain. If it indicates that the original message passed the SPF and DKIM checks, and the only modifications were made by intermediaries trusted by the receiving service, the receiving service may choose to accept the email.

---

It seems the above was roughly already understood in this discussion, but quoting with bolded emphasis for when ARC is relevant. The wiki entry also notes that it's up to the receiver to place trust in the ARC validation, but for the most part you're providing validation on any modifications to the forwarded mail such that it still allows for SPF and DKIM checks to pass validation despite the intermediary server that delivered it.

A good use-case where ARC is probably helpful is the scenario described in this discussion, where an intermediary receives the mail to forward to a private instance for storage purposes. However that complicates validation on SPF/DKIM AFAIK which would fail, so their approach was to blindly trust anything sent from the intermediary, skipping those security checks.

That discussion was summarized into a guide for our v14 docs, you can presently view that on our edge docs here for reference if it's of interest.

If the intermediary was instead a DMS instance and used ARC, then the private DMS instance should be able to perform the proper security checks on the forwarded mail if it was configured correctly.

[Mygod]

Ah right. So the arc=fail that I was getting would probably indicate that there was something wrong with the intermediary signatures? I wonder how to debug that...