Rabbitmq broker API times out #1112
Assignees
Labels
Comments
|
@mher I am looking into this one as it is I think the last remaining issue to fix and get new release so please assign me. |
|
Results of investigation:
Tested if similar happens for Redis and it does although the broker tab does not result in spinning wheel of death and no Timeout is logged - still the dashboard, tasks tabs show no content and broker tab shows no real content. Potential solutions:
@mher Can you point me at a place in the code where this issue takes place if you remember :)? Also when I boot Flower and there is no broker running, would you expect an exception being raised or an error logged in flower? Is there code that already checks that and stopped working? |
|
This is fixed in #1113 by making a connection check when booting. |
|
d4ea1dc fixes the issue |
|
Thanks @Tomasz-Kluczkowski , you can keep tests as well. |
|
I'm commenting just to say, as a fellow open source developer, that I'm impressed by you guys going so smoothly between two implementation variations - as well as contributing it on a Sunday. So yeah, people are indeed watching (and loving) your work :) |
|
@mher I have tested your change (the added timeouts) and while yes, we no longer get the spinning wheel of death in the browser Flower still is no longer capable of connecting to the broker if it was booted without the broker running. I have tested with rabbitmq and redis as brokers. Test:
If broker is started first and then Flower all is good. I think we may need to merge my changes from #1113 with waiting for available broker as an improvement of sorts. Please let me know what you think. |
|
I will check one more thing and report back. |
|
I tried and did not achieve anything @mher , I think making a check for broker connection on Flower's boot is the best option for now. Please let me know if you want to merge it as it looks like I need to fix merge conflicts there. |
|
I'm not able to reproduce. Running without RabbitMQ, then starting, stoping RabbitMQ works as expected. Broker tab shows celery queue when RabbitMQ is available. |
|
I have to check if it makes the difference but my testing was done solely with rabbitmq/redis in docker containers. This is to simulate closer to our production where it runs in a kubernetes pod. Are you running rabbitmq for this test inside of a container too? If not maybe that's the difference that makes or breaks Flower. If that is the case I think a lot use cases are in containerised environments nowadays and Flower should cater to that too. I will investigate more tomorrow evening. |
Yes |
|
Hello, I tried initially using pycharm configuration to start the rabbitmq , then deleted the container/image and begun starting rabbitmq via command line with: and I start flower not in a container which maybe is the reason for all of this trouble? Are you running Flower in a docker container for this test too @mher? Is the latest Flower image based on latest master so I can test too, or have to build it myself? |
No, only RabbitMQ server. Flower is running from the master version with |
|
Hopefully it has nothing to do with this (but I have seen issues with host OS making our own app work differently), but are you testing on Linux? |
|
I'm testing on OSX. Have you tried to use |
|
just did now, and I can see that then it used redis as the broker. |
|
I think because that is the celeryconfig.py inside flower/examples |
|
I am changing the celeryconfig.py to use rabbitmq as the broker_url and test again. |
|
You can change broker with env vars |
|
Ok , run with those env vars on both my linux machine and macbook pro with rabbitmq in docker and starting flower first and broker after creates the connection issue. I am really at a loss now what is the difference that makes it run for you @mher. maybe it's the difference in how you start rabbitmq in docker? |
|
I attach the logs |
|
You can see before I start rabbitmq in docker, it says: Then I start rabbitmq and we get: but then I go to the broker tab and still we get: :( |
|
The difference I see is that in your logs above after it connected it did all those inspect calls again: I do not see that in my logs after it connected, but not sure if this makes any difference |
|
are you running latest rabbitmq docker image @mher? my command for local testing (I use is this how you run yours? |
|
Try to reproduce using this docker-compose
|
|
I think this is the reason why it may be working for you. Are you testing as I mention below? What we need to test is strictly manually start flower, wait for it to finish boot, then manually start rabbitmq in docker - for me on both linux and mac it then never connects to broker and shows timeout error. point number 2 regarding this docker compose - it uses old flower image / celery 4.4.7 (I exec'd into it): Could you try the manual starting and then see if it fails for you same way as for me please? If you already did then I have no more ideas for now grrrr :(/ |
|
I stop both rabbitmq and flower then start flower |
|
Ok but it runs incorrect flower version with old celery. I guess we need to be strict here and use actual newest versions.
I will create a new publicly available image of flower based on its newest master and then we can do all these tests using docker-compose with it.
Still if you could be so kind and try just manual starting of newest master of flower from command line and then manual start of rabbitmq inside docker, maybe the issue would appear as well…
|
|
@mher Ok, I have made a public image of Flower based on its newest master here: https://hub.docker.com/layers/155902451/kilthar/flower/docker-comms-test/images/sha256-3e5ea07f05afb3cad5f86e2399c5ce2c8edad6748b73a3644c8b07af375e3590?context=explore It uses this And the Fortunately for me during first boot the issue appeared, I attach the complete logs (sorry they are long, lots of spam from rabbitmq :). For the sake of making sure, I went through 3 cycles of docker-compose up, down, up. I was lucky that Flower started first each time and the timeout issue when connecting to rabbitmq appeared straight away but one can also start flower_flower_1 container manually first and then rabbitmq to make sure - result is the same - connection problem. Notice that flower boots first, way ahead of rabbitmq, which is the condition that triggers the bug. but still the broker tab times out: So it looks like there is a change of behaviour of either celery or kombu in the newest celery package that causes this. My solution to fix which waits for broker is working (we have it running in prod). Please let me know if you were able to reproduce using this Complete logs of |
|
you can use the image mher/flower with the tag latest, I see it's being built regularly, https://hub.docker.com/r/mher/flower/tags?page=1&ordering=last_updated LE: Oh! I'm wrong actually, I see it's doing pip install flower :-s This explains some of the things I'm seeing :) |
|
Yeah that's why I built a new one. I wonder if anyone else has same connection issue depending on order of booting of Flower/broker. We definitely had it in our kubernetes cluster until I used the my fork of Flower with waiting for broker on boot... |
|
Yep, same issue here during local development, via docker-compose. |
|
You can use this docker-compose file: version: '3.4' *** remember not to use single quotation for CELERY_BROKER_URL value!! the below config will raise error: |
|
Though of posting here, seems to be a similar problem I'm having. I have a When looking at the logs flower is connecting to the default broker I might be doing something wrong, but that has been my experience so far after updating to Celery 5. EDIT: Got it working when setting the CELERY_BROKER_URL env on the Flower manifest. Maybe the lack of the CELERY_... env was making it override the |
|
This is happening when I run Flower/RabbitMQ too. Compose File: queue:
# image: rabbitmq:3
# Using the below image instead is required to enable the "Broker" tab in the flower UI:
image: rabbitmq:3-management
#
# You also have to change the flower command
flower:
image: mher/flower
env_file:
# This just sets login details
- .env
environment:
- CELERY_BROKER_URL=amqp://guest@queue:5672//
command:
- "celery"
- "--broker=amqp://guest@queue:5672//"
- "flower"
# For the "Broker" tab to work in the flower UI, uncomment the following command argument,
# and change the queue service's image as well
- "--broker_api=http://guest:guest@queue:15672/api//"Versions: $ docker compose exec flower python -c "import flower, celery; print(f'Flower {flower.__version__}'); print(f'Celery {celery.__version__}')"
Flower 1.0.0
Celery 5.1.2Logs from RabbitMQ (
|
|
I have troubleshot it a bit I got exception http_client.fetch(url) but at the same time, if i open the same URL in the browser and get it with requests - it works - see attached pic
|
|
the last line where i can catch an exception is tornado.httpclient
|
|
Could you check this breakpoint when everything is ok (broker was started before Flower and connection is made correctly)? I wonder what is host then, is it also just 'localhost' ? |
|
I had the same issue. I have to change the broker to Redis instead of RabbitMQ to make flower dashboard work. |
|
Well this is still present @mher , would you be willing to merge this: #1113 which makes flower wait for the broker if I fix merge conflicts? I did try to investigate why on earth tornado http client is unable to make this request even though simple |
|
I see the same connection error in my project |
|
Any update on this? Another problem concern the broker tab, I get information with N/A but no errors in flower logs ... Any idea ? App2: That doesnt work Broker tab: My docker-compose: logs of flower for app2: Read a lot of issues about flower, I read maybe there is some bugs with rabbitmq but not with redis, could be the case here ? thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When using
rabbitmqas the broker and specifyingbroker_apifor Flower to use the request forapi/queuestimes out with:[E 210605 19:25:16 broker:78] RabbitMQ management API call failed: Timeout while connectingUsing identical url from the browser to reach the
api/queuesendpoint of rabbitmq management api is successful.To Reproduce
Steps to reproduce the behavior:
celery -A tasks -broker=amqp://guest:guest@localhost:5672// flower --debug --broker_api=http://guest:guest@localhost:15672/api/brokertab, notice the spinning browser's connection wheelSimilar situation happens if flower is started first and redis after.
Expected behavior
The rabbitmq's api is responding correctly and Flower is able to use it.
Screenshots
If applicable, add screenshots to help explain your problem.
System information
Output of
python -c 'from flower.utils import bugreport; print(bugreport())'commandThe text was updated successfully, but these errors were encountered: