Unacked messages are not redelivered

[programmer445]

Describe the bug

When we re-start our .net 7 application using the rabbitmq.client package, at least one queue gets stuck with unacked messages.

We have a debug log implemented as first step after the consumer received a message (e.g. Consumer received message 1928618 with deliveryTag 1191 for queue X.), and see that for this stuck queue we never received any message after the restart, although according to the rabbitmq web managment 20 messages are unacked (the prefetch count is set to 20 per queue).

What looks strange that in the overview of the queue there are 20 unacked messages, but the consumer channel has 0 messages unacknowledged.

All other queue consumers on this consumer connection are working fine with related log messages.
We have 2 .net 7 applications, one is only producing messages over one producer connection and one is producing messages over one producer connection and consuming messages over one consumer connection, by 1 consumer per queue for total 149 queues.
With publisher confirms and manual acks every 10 messages.

Reproduction steps

1. Create 149 quorum queues with about 5000 messages ready evenly distributed (prefetch=20)
2. Start the one consumer per queue, one after one

Expected behavior

We expect that the messages will be returned to the server if the consumer is disconnected.

Additional context

RabbitMQ -v 3.13.1
Erlang -v 26.2.2
RabbitMQ.Client -v 6.5.0
Mirror: quorum (3 nodes)

[programmer445]

More info:

When we close the connection in the rabbitmq web managment, the messages of this queue are received and processed successfully

After the windows service of the .net app was restarted and the consumer disappeared in the web view, the unacked messages went back into state ready.

But after the consumer started, there are again 20 unacked messages and no log that one message for this queue was received.

For all other 148 queues everything is working perfectly.

Two log entries on the rabbitmq server node where the .net app channel is connected to:

[michaelklishin]

We expect that the messages will be returned to the server if the consumer is disconnected

We need an executable way to reproduce. You haven't provided any evidence that automatic requeueing does not happen, and we haven't see anyone report this earlier.

In fact, I cannot reproduce this behavior when I terminate a few connections with consumed and unacknowledged messages.

Connections going away is [detected after a period of time](https://www.rabbitmq.com/docs/heartbeats, so that can be a factor. But a more likely scenario is this:

1. A delivery is consumed but not acknowledged due to a consumer environment issue (e.g. it cannot access a data store of sorts and this does not acknowledge)
2. A connection is closed
3. The message is requeued and instantly redelivered to another consumer which is in the same situation as consumer at step 1

In fact, the consumer on step 3 can be the same consumer if it reconnects before or concurrently with the auto-requeueing step performed by the server.

[michaelklishin]

Two log entries on the rabbitmq server node where the .net app channel is connected to

You haven't provided any evidence of that connection being closed. According to the log snippet shared, it was successfully accepted and that's it.

In 99.9% of cases such behavior is an indication of a stuck consumer. Double checking error handling and exception reporting in your consumer code is where I'd start producing an executable way for us to reproduce.

And please share logs as files with sensitive data obfuscated, not as screenshots.

[michaelklishin]

the prefetch count is set to 20 per queue

Prefetch cannot be set for a queue, it can be set for a channel or for a consumer (the latter is a fairly rare thing to see).

What looks strange that in the overview of the queue there are 20 unacked messages, but the consumer > channel has 0 messages unacknowledged

An explanation can be as simple as: there are multiple consumers on a channel and these "stuck" messages were instantly redelivered to another consumer on another channel.

Again, we do not guess in this community so without an executable way to reproduce, above is a number of hypothesis for you to investigate.

[michaelklishin]

Finally, one very easy way to verify if this queue is "special" in any way is to

1. See its status in rabbitmqctl list_queues
2. Grep logs for its name
3. Use automatic acknowledgements and see if anything is "stuck" that way. If not, then it is very likely a problem on the acknowledging end, which is your application

To further confirm 3 and every hypothesis above, a traffic capture will tell you exactly what RabbitMQ and connected clients send and receive.

[programmer445]

Thank you for the fast response!

The connection from the first screenshot was definitely closed:
2024-05-14 12:14:23.478959+02:00 [info] <0.19276495.0> closing AMQP connection <0.19276495.0> (X:39237 -> X:5672, vhost: '/', user: 'admin').

Since there is only one application consuming these queues, I cannot think that it is a requeuing problem.

In code we attached to all exception & error events, and nothing occurred.

I will dig into it with all your provided steps.
Again, thank you very much!

[programmer445]

@michaelklishin I guess I have found the problem.

Some of our quorum queues do not have all members online, and when a consumer opens a connection to a node that does not have that queue, the problems described occur.
By analyzing the network traffic with Wireshark, I am now also sure that the client never received these "stuck" messages.

In the logs two weeks ago I found now some information about corrupt segment files related to these sometimes misleadingly acting queues, and also some crashes with the reason 'reached_max_restart_intensity'.

[mkuratczyk]

As Michael said - please provide executable way of triggering this issue.

[michaelklishin]

@programmer445 a quorum queue that does not have a majority of replicas online will not be available, by design.