After upgrading to 3.13, management interface shows 500 for /api/deprecated-features/used #10640

leonjoosse · 2024-02-27T12:29:17Z

leonjoosse
Feb 27, 2024

Describe the bug

Today, we upgraded our rabbitmq server from 3.12.x to 3.13.0.
When loading the management interface after upgrading, the page display a fat banner at the bottom, saying "Management API returned status code 500 -".
Inspecting the Network tab in the developer tools reveals that /api/deprecated-features/used returns a 500 Internal Server Error.
When looking in the rabbitmq logs, I can't find anything related to this call failing.

All feature flags are enabled (apart from the experimental khepri_db).
We are on RabbitMQ 3.13.0 and Erlang 26.2.2, running on Ubuntu 20.04.6.

I can imagine one needs more information to find the cause, please let me know what information is needed. I'm happy to provide more insight on configuration or logs.

Reproduction steps

Open RabbitMQ management page, login as normal
Go to Admin tab

...

Expected behavior

I'd expect to have a functioning Deprecated Features endpoint.

Additional context

Here is the failing request, copied from the browser (ourcompany.com is obviously not our real domain)

curl 'https://rabbitmqdev.ourcompany.com/api/deprecated-features/used' \
  -H 'Accept: */*' \
  -H 'Accept-Language: en,en-GB;q=0.9,en-US;q=0.8,nl;q=0.7' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: m=59b9:true' \
  -H 'Referer: https://rabbitmqdev.ourcompany.com/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36' \
  -H 'authorization: Basic ...' \
  -H 'content-type: application/json' \
  -H 'sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  --compressed

And that responds

Content-Length: 0

The response body is empty.

This is what the Admin tab looks like

I believe this rabbitmqctl cmd belongs to the new deprecated features page, its output may be helpful:

..@rabbitmqdev:~$ sudo rabbitmqctl list_deprecated_features
Listing deprecated features ...
name                      deprecation_phase
classic_queue_mirroring   permitted_by_default
global_qos                permitted_by_default
management_metrics_collection   permitted_by_default
ram_node_type             permitted_by_default
transient_nonexcl_queues  permitted_by_default

Answered by lukebakken

May 14, 2024

@cvijdea-bd @johnnycederholm this was fixed recently - #11193

#11192

View full answer

michaelklishin · 2024-02-27T12:41:01Z

michaelklishin
Feb 27, 2024
Maintainer

@leonjoosse what does the node log when this happens?

0 replies

leonjoosse · 2024-02-27T12:50:12Z

leonjoosse
Feb 27, 2024
Author

Thanks for the swift reply @michaelklishin

I had a better look at the logs. I thought this was unrelated, but it is printed everytime I execute the request via Insomnia.

Using the cURL request from the start post, this is logged:

2024-02-27 12:47:20.871970+00:00 [info] <0.89778.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-02-27 12:47:20.872151+00:00 [info] <0.89778.0> Successfully synced tables from a peer
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>   crasher:
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     initial call: cowboy_stream_h:request_process/3
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     pid: <0.89778.0>
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     registered_name: []
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     exception error: bad argument
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in function  lists:keymember/3
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>          called as lists:keymember(<<"ha-mode">>,1,
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>                                    #{<<"max-length-bytes">> => 5242880})
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>          *** argument 3: not a list
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from rabbit_mirror_queue_misc:does_policy_configure_cmq/1 (rabbit_mirror_queue_misc.erl, line 1014)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from lists:any/2 (lists.erl, line 1535)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from rabbit_mirror_queue_misc:are_cmqs_used1/0 (rabbit_mirror_queue_misc.erl, line 997)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from rabbit_deprecated_features:'-list/1-fun-1-'/2 (rabbit_deprecated_features.erl, line 370)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from maps:filter_1/3 (maps.erl, line 334)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from maps:filter/2 (maps.erl, line 320)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>       in call from rabbit_depr_ff_extra:cli_info/1 (rabbit_depr_ff_extra.erl, line 41)
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     ancestors: [<0.89777.0>,<0.2226.0>,<0.2225.0>,<0.2224.0>,<0.2222.0>,
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>                   rabbit_web_dispatch_sup,<0.2172.0>]
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     message_queue_len: 0
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     messages: []
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     links: [<0.89777.0>]
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     dictionary: []
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     trap_exit: false
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     status: running
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     heap_size: 6772
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     stack_size: 28
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>     reductions: 4665
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>   neighbours:
2024-02-27 12:47:20.872444+00:00 [error] <0.89778.0>
2024-02-27 12:47:20.873234+00:00 [error] <0.89777.0> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15671}, connection process <0.89777.0>, stream 1 had its request process <0.89778.0> exit with reason badarg and stacktrace [{lists,keymember,[<<"ha-mode">>,1,#{<<"max-length-bytes">> => 5242880}],[{error_info,#{module => erl_stdlib_errors}}]},{rabbit_mirror_queue_misc,does_policy_configure_cmq,1,[{file,"rabbit_mirror_queue_misc.erl"},{line,1014}]},{lists,any,2,[{file,"lists.erl"},{line,1535}]},{rabbit_mirror_queue_misc,are_cmqs_used1,0,[{file,"rabbit_mirror_queue_misc.erl"},{line,997}]},{rabbit_deprecated_features,'-list/1-fun-1-',2,[{file,"rabbit_deprecated_features.erl"},{line,370}]},{maps,filter_1,3,[{file,"maps.erl"},{line,334}]},{maps,filter,2,[{file,"maps.erl"},{line,320}]},{rabbit_depr_ff_extra,cli_info,1,[{file,"rabbit_depr_ff_extra.erl"},{line,41}]}]

0 replies

SimonUnge · 2024-02-27T21:56:16Z

SimonUnge
Feb 27, 2024
Collaborator

Interesting. Would it be possible to see the policies you have running too @leonjoosse ?

0 replies

dumbbell · 2024-02-28T09:17:42Z

dumbbell
Feb 28, 2024
Maintainer

Thank you for the report!

I see the internal format of the policy is an Erlang map instead of the expected Erlang proplist. How do you declare the policy in the first place?

0 replies

leonjoosse · 2024-02-28T12:52:58Z

leonjoosse
Feb 28, 2024
Author

@SimonUnge @dumbbell Thanks for getting back on this. Here's the policies we defined. Only added them via the UI, they are there for quite a while already. Unfortunately, I don't know on which RabbitMQ version we added them.

The colleague usually managing the system got back today, on his advice I removed both policies. The error as described in the start post was immediately gone. I added both policies again, using the UI, with the exact same parameters.

This works for me, so closed the issue for now. If you have more questions, I'm happy to answer them!

0 replies

dumbbell · 2024-02-28T14:00:50Z

dumbbell
Feb 28, 2024
Maintainer

Thank you for the additional feedback!

So somehow, the policy has been stored internally in an invalid format, despite the validity checks. The web UI itself can’t bypass these checks, so there must be something else going on.

Another question: did you used the definitions import/export feature?

0 replies

michaelklishin · 2024-02-28T14:46:36Z

michaelklishin
Feb 28, 2024
Maintainer

Moved to a discussion since we are still narrowing down what may be going on and where.

0 replies

SimonUnge · 2024-02-28T18:10:36Z

SimonUnge
Feb 28, 2024
Collaborator

@leonjoosse If you happen to have the information on what version the policy was added, it would help narrow it down, or perhaps you have some idea at least of which version of RabbitMQ you started out with at some point?

1 reply

leonjoosse Feb 29, 2024
Author

Unfortunately, I can't answer that straight away. I'll ask around to my colleagues. Hope to provide an answer in 2 or 3 days.

cvijdea-bd · 2024-05-14T09:48:39Z

cvijdea-bd
May 14, 2024

I got hit by this too, also fixed by re-creating all policies

I had 1 user policy and 1 operator policy, both most likely created long ago on rabbitmq 3.8 via the management UI

0 replies

johnnycederholm · 2024-05-14T13:51:42Z

johnnycederholm
May 14, 2024

Even if it's not related to upgrading from an earlier version of RabbitMQ I will take the chance to ask if this is related somehow to the issue that started this thread?

By just creating a fresh instance of RabbitMQ 3.13.2, using Docker on my local machine, I'm seem to be able to reproduce the issue by importing definitions created on same version of RabbitMQ.

The steps I have performed to reproduce it is the following

Create a new container using docker run -it -p 5672:5672 -p 15672:15672 rabbitmq:3.13.2-management
Setup a new operator policy using the management interface. In my case I just took the example policy shown in the documentation
Export the definitions using the definitions end-point curl -u guest:guest -X GET http://localhost:15672/api/definitions > definitions.json
Throw away my running instance of RabbitMQ and create a new one
Login to the management interface
Import the definitions into the new instance curl -u guest:guest -H "Content-Type: application/json" -X POST -T definitions.json http://localhost:15672/api/definitions
Go back to the management interface just to see the banner in the bottom and the following output in the container log of RabbitMQ

2024-05-14 13:48:06.485061+00:00 [info] <0.1160.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-05-14 13:48:06.485175+00:00 [info] <0.1160.0> Successfully synced tables from a peer
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>   crasher:
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     initial call: cowboy_stream_h:request_process/3
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     pid: <0.1160.0>
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     registered_name: []
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     exception error: bad argument
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in function  lists:keymember/3
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>          called as lists:keymember(<<"ha-mode">>,1,
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>                                    #{<<"expires">> => 1800000})
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>          *** argument 3: not a list
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from rabbit_mirror_queue_misc:does_policy_configure_cmq/1 (rabbit_mirror_queue_misc.erl, line 1014)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from lists:any/2 (lists.erl, line 1535)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from rabbit_mirror_queue_misc:are_cmqs_used1/0 (rabbit_mirror_queue_misc.erl, line 997)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from rabbit_deprecated_features:'-list/1-fun-1-'/2 (rabbit_deprecated_features.erl, line 370)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from maps:filter_1/3 (maps.erl, line 334)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from maps:filter/2 (maps.erl, line 320)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>       in call from rabbit_depr_ff_extra:cli_info/1 (rabbit_depr_ff_extra.erl, line 41)
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     ancestors: [<0.1150.0>,<0.770.0>,<0.764.0>,<0.763.0>,<0.761.0>,
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>                   rabbit_web_dispatch_sup,<0.711.0>]
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     message_queue_len: 0
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     messages: []
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     links: [<0.1150.0>]
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     dictionary: []
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     trap_exit: false
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     status: running
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     heap_size: 4185
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     stack_size: 28
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>     reductions: 4519
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>   neighbours:
2024-05-14 13:48:06.485242+00:00 [error] <0.1160.0>
2024-05-14 13:48:06.485671+00:00 [error] <0.1150.0> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15672}, connection process <0.1150.0>, stream 6 had its request process <0.1160.0> exit with reason badarg and stacktrace [{lists,keymember,[<<"ha-mode">>,1,#{<<"expires">> => 1800000}],[{error_info,#{module => erl_stdlib_errors}}]},{rabbit_mirror_queue_misc,does_policy_configure_cmq,1,[{file,"rabbit_mirror_queue_misc.erl"},{line,1014}]},{lists,any,2,[{file,"lists.erl"},{line,1535}]},{rabbit_mirror_queue_misc,are_cmqs_used1,0,[{file,"rabbit_mirror_queue_misc.erl"},{line,997}]},{rabbit_deprecated_features,'-list/1-fun-1-',2,[{file,"rabbit_deprecated_features.erl"},{line,370}]},{maps,filter_1,3,[{file,"maps.erl"},{line,334}]},{maps,filter,2,[{file,"maps.erl"},{line,320}]},{rabbit_depr_ff_extra,cli_info,1,[{file,"rabbit_depr_ff_extra.erl"},{line,41}]}]
2024-05-14 13:48:06.485671+00:00 [error] <0.1150.0>

0 replies

cvijdea-bd · 2024-05-14T13:53:43Z

cvijdea-bd
May 14, 2024

Yes, I can confirm that for me the issue is also related to definitions import (the affected policies are actually defined in a definitions.json file that I import on every startup of rabbitmq)

0 replies

lukebakken · 2024-05-14T14:57:53Z

lukebakken
May 14, 2024
Maintainer

@cvijdea-bd @johnnycederholm this was fixed recently - #11193

#11192

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After upgrading to 3.13, management interface shows 500 for /api/deprecated-features/used #10640

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 12 comments 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

After upgrading to 3.13, management interface shows 500 for /api/deprecated-features/used #10640

leonjoosse Feb 27, 2024

Describe the bug

Reproduction steps

Expected behavior

Additional context

Replies: 12 comments · 1 reply

michaelklishin Feb 27, 2024 Maintainer

leonjoosse Feb 27, 2024 Author

SimonUnge Feb 27, 2024 Collaborator

dumbbell Feb 28, 2024 Maintainer

leonjoosse Feb 28, 2024 Author

dumbbell Feb 28, 2024 Maintainer

michaelklishin Feb 28, 2024 Maintainer

SimonUnge Feb 28, 2024 Collaborator

leonjoosse Feb 29, 2024 Author

cvijdea-bd May 14, 2024

johnnycederholm May 14, 2024

cvijdea-bd May 14, 2024

lukebakken May 14, 2024 Maintainer

leonjoosse
Feb 27, 2024

Replies: 12 comments 1 reply

michaelklishin
Feb 27, 2024
Maintainer

leonjoosse
Feb 27, 2024
Author

SimonUnge
Feb 27, 2024
Collaborator

dumbbell
Feb 28, 2024
Maintainer

leonjoosse
Feb 28, 2024
Author

dumbbell
Feb 28, 2024
Maintainer

michaelklishin
Feb 28, 2024
Maintainer

SimonUnge
Feb 28, 2024
Collaborator

leonjoosse Feb 29, 2024
Author

cvijdea-bd
May 14, 2024

johnnycederholm
May 14, 2024

cvijdea-bd
May 14, 2024

lukebakken
May 14, 2024
Maintainer