[TODO]: Tests - test_helper.bats refactor mailqueue test cases

[polarathene]

Description

This was identified during recent review feedback, that the test cases could probably be refactored:

• "wait_for_empty_mail_queue_in_container fails when timeout reached"
• "wait_for_empty_mail_queue_in_container succeeds within timeout"

Presently it's skipped for being unreliable like some other tests have been (Dovecot Quota). The test file also needs to be adapted to our newer test format.

We can configure Postfix to defer mail to a transport, keeping it stuck in the queue until we decide to flush it. This is the main focus of the test-case which relied on unreliable means to try fill the queue up momentarily.

---

docker-mailserver/test/tests/serial/test_helper.bats

Lines 183 to 192 in aba218e

	SECONDS=0
	# no mails -> should return immediately
	TEST_TIMEOUT_IN_SECONDS=5 wait_for_empty_mail_queue_in_container "${CONTAINER_NAME}"
	[[ ${SECONDS} -lt 5 ]]
	# fill the queue with a message
	docker exec "${CONTAINER_NAME}" /bin/sh -c "nc 0.0.0.0 25 < /tmp/docker-mailserver-test/emails/amavis-virus.txt"
	# that should still be stuck in the queue
	! TEST_TIMEOUT_IN_SECONDS=0 wait_for_empty_mail_queue_in_container "${CONTAINER_NAME}"

docker-mailserver/test/tests/serial/test_helper.bats

Lines 215 to 221 in aba218e

	# fill the queue with a message
	docker exec "${CONTAINER_NAME}" /bin/sh -c "nc 0.0.0.0 25 < /tmp/docker-mailserver-test/emails/amavis-virus.txt"
	# give it some time to clear the queue
	SECONDS=0
	wait_for_empty_mail_queue_in_container "${CONTAINER_NAME}"
	[[ ${SECONDS} -gt 0 ]]

[polarathene]

Using the Postfix queue in a controlled manner (block delivery so mail is held in queue until we decide to decide to release it via postqueue -f / postqueue -s domain-part).

This comment isn't specific to this issue, but collects enough information on the subject to go forward with an implementation, which may be useful elsewhere in testing (if the Queue ID is important, it is much easier to retrieve assuming mail is accepted for delivery).

---

Presently we use mailq to check the status of the mail queue:

docker-mailserver/test/helper/common.bash

Lines 319 to 328 in aba218e

	function _wait_for_empty_mail_queue_in_container() {
	local CONTAINER_NAME=$(__handle_container_name "${1:-}")
	local TIMEOUT=${TEST_TIMEOUT_IN_SECONDS}
	# shellcheck disable=SC2016
	_repeat_in_container_until_success_or_timeout \
	"${TIMEOUT}" \
	"${CONTAINER_NAME}" \
	/bin/bash -c '[[ $(mailq) == "Mail queue is empty" ]]'
	}

postqueue is a nicer interface over that which can optionally provide JSON format output:

# Equivalent output of mailq command:
$ postqueue -p
-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
8551B132C2*     802 Wed Jan 10 00:32:03  [email protected]
                                         [email protected]

C293F132B1*     802 Wed Jan 10 00:29:19  [email protected]
                                         [email protected]

AC00D132B3*     802 Wed Jan 10 00:29:48  [email protected]
                                         [email protected]

-- 2 Kbytes in 3 Requests.

# JSON output supported:
$ postqueue -j
{"queue_name": "active", "queue_id": "8551B132C2", "arrival_time": 1704846723, "message_size": 802, "forced_expire": false, "sender": "[email protected]", "recipients": [{"address": "[email protected]"}]}
{"queue_name": "active", "queue_id": "C293F132B1", "arrival_time": 1704846559, "message_size": 802, "forced_expire": false, "sender": "[email protected]", "recipients": [{"address": "[email protected]"}]}
{"queue_name": "active", "queue_id": "AC00D132B3", "arrival_time": 1704846588, "message_size": 802, "forced_expire": false, "sender": "[email protected]", "recipients": [{"address": "[email protected]"}]}

# Use jaq query on the JSON to match a specific sender and return their queue_id:
$ postqueue -j | jaq -r 'select(.sender == "[email protected]") | .queue_id'
C293F132B1

# `postqueue -i <QUEUE ID>` triggers delivery of deferred queue item,
# This uses jaq to find a JSONL entry with a recipient we expect, then returns the queue id:
$ postqueue -i $(postqueue -j | jaq -r 'select(.recipients[].address == "[email protected]") | .queue_id')

# Mail was released from queue:
$ postqueue -j
{"queue_name": "active", "queue_id": "8551B132C2", "arrival_time": 1704846723, "message_size": 802, "forced_expire": false, "sender": "[email protected]", "recipients": [{"address": "[email protected]"}]}
{"queue_name": "active", "queue_id": "C293F132B1", "arrival_time": 1704846559, "message_size": 802, "forced_expire": false, "sender": "[email protected]", "recipients": [{"address": "[email protected]"}]}

So if we want to hold mail in the queue for local and lmtp transports, this works:

postconf defer_transports=local,lmtp

Mail has to reach that point of making it to the point it's queued for delivery to the transport I think. If that's not viable because mail is rejected earlier, you may only be able to track mail in the logs where getting the Queue ID is less convenient.

Definition of a transport

transport_maps uses the transport table:

The transport field specifies the name of a mail delivery transport (the first name of a mail delivery service entry in the Postfix master.cf file).

Likewise local, virtual, relay, smtp are referenced here:

transport

• The delivery agent to use.
• This is the first field of an entry in the master.cf file.

smtp is similar to lmtp (that wasn't mentioned in that reference). While DMS has the virtual delivery agent configured to use the lmtp transport with virtual_transport = lmtp:unix:/var/run/dovecot/lmtp. the master.cf config doesn't convey that:

$ grep -E 'lmtp|virtual' /etc/postfix/master.cf
virtual        unix  -       n       n       -       -       virtual
lmtp           unix  -       -       n       -       -       lmtp

# Won't hold mail in queue:
$ postconf defer_transports=virtual

# Will hold mail for Dovecot in queue:
$ postconf defer_transports=lmtp

# Will hold mail for local agent (@$myhostname) in queue:
$ postconf defer_transports=local

$ grep -E 'local|relay' /etc/postfix/master.cf
local          unix  -       n       n       -       -       local
relay          unix  -       -       n       -       -       smtp

$ postconf | grep -E '_transport = (smtp|local|relay|lmtp|virtual)'
default_transport = smtp
local_transport = local:$myhostname
relay_transport = relay
virtual_transport = lmtp:unix:/var/run/dovecot/lmtp

---

Extra insights

We can use transport maps config to direct mail for a recipient to a particular transport (here is equivalent for matching on sender), or redirect all incoming mail to a specific fallback transport instead. If the transport is omitted it'll be selected by the appropriate address class. We can add our own custom transport to master.cf and use transport_maps to direct to that if we needed that kind of control.

content_filter if not already in use technically works to at direct to defer "transport", but I have noticed this fails to flush the queue (with postqueue -f) even when content_filter has been unset 🤔

postsuper -d ALL will clear the queue when necessary (when postqueue -f doesn't flush the queue successfully).

The postsuper docs also make mention of Queue ID gotchas with default enable_long_queue_ids=no setting (more info on that setting).

postcat can be a useful utility to read the mail and other metadata related to it while it's in this queue. You would reference it with the location from postconf queue_directory + <queue name>/<queue_id>:

# Take the two fields needed and prefix the queue directory
postqueue -j | jaq -r \
  --arg queue_dir "$(postconf -h queue_directory)" \
  '[$queue_dir, .queue_name, .queue_id] | join("/")'

# Output would be 1 or more queue items like this
/var/spool/postfix/deferred/0BBAB132FA

# Pass path to postcat to view the queued items info / content:
postcat /var/spool/postfix/deferred/0BBAB132FA