rabbitmq pod crash report - 1 pod in a 3 pod cluster

[vjvel]

rabbitmq-1 pod has crashed with below message and its not becoming the member automatically. we again did forgetnode to join the cluster.

rabbitmq image - 3.8.5-debian-10-r38

** {{timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,215}]},{aten_detector,handle_info,2,[{file,"src/aten_detector.erl"},{line,97}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}

[vjvel]

2022-07-12 08:35:56.288 [error] <0.231.0> Supervisor aten_sup had child aten_detector started with aten_detector:start_link() at <0.26978.5510> exit with reason {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in context child_terminated

2022-07-12 08:35:55.338 [error] <0.31578.5460> CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 

We are not able to understand this crash report. can you please help us to know about this issue and workaround.

[johanrhodin]

You can see some more digging about this here: https://groups.google.com/g/rabbitmq-users/c/IMWJGPLzDwE/m/RuEvgaaZAAAJ, it is most likely a symptom and not a cause. Upgrade RabbitMQ since we see it much more rarely in later releases.

[michaelklishin]

RabbitMQ 3.8 is out of general support and will go completely out of support by July 31st.

The function in question does very little and uses a timeout of 5s. Your node must be starved of CPU resources.

[vjvel]

Thanks @michaelklishin for the response

Is there any config parameter needs to be added? its running in k8s. We have resource limits configured. What is the stable version we can upgrade now from 3.8.5 ?

[michaelklishin]

That timeout is not configurable. Given that this is the first time I recall seeing Aten time out, I don't think it should be. Provisioning more CPU resources or paying attention to their use should help.

[michaelklishin]

I'm copying my response in another discussion on the same topic.

This is a symptom that, most likely, the node in question is is low on/out of CPU resources often enough for RabbitMQ's failure detector used by Raft-based features to time out.

Besides the fact that it is usually observed on older versions (3.8.x through 3.10.x), see monitoring data, increase the number of available cores (listed in rabbitmq-diagnostics cluster_status in modern versions) and follow the docs if you have a lot of queues, channels, connections that are mostly idle.