-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Messages dropped because of a full thread queue #816
Comments
Hi @krisdigital Thanks for raising this. We're looking into this and will update the thread as soon as we can. |
Hi @krisdigital Thank you for your patience as we investigated this issue further. The issue seems to stem from the worker thread terminating. Are you aware of any reasons why this may be happening? if so, could you please elaborate? For background, the thread should stay alive until we stop it in an bugsnag-ruby/lib/bugsnag/delivery/thread_queue.rb Lines 59 to 63 in 990f835
|
Hi @clr182, thank you for looking into it! I don't know why the thread terminated in our case, sadly. It may have been a bad message in an exception? But it is hard to tell. The problem is that in this case the error reporting silently stops working. Would it maybe make sense to check if the thread is still running when a new message is pushed on the queue? |
Hi Kris, We do believe the worker thread dying is the root cause of your issue in this case. As previously stated, this thread should always be alive until wewe stop it in an |
Hi @clr182, all right thank you, we will continue to look for the reason of the thread exiting! |
Hi @krisdigital As there hasn't been any activity on the thread for a while, we are now going to close this issue. If you continue to experience issues with this, or have any other questions, please feel free to reopen this or open a ticket with us directly by contacting [email protected] with further details or relevant information. |
Describe the bug
When the thread in the thread queue dies, messages are pushed on the queue and never delivered until the process restarts.
Steps to reproduce
See the example code - we found out by accident that messages where stuck and not delivered to Bugsnag. We got the message "Dropping notification, 101 outstanding requests" in the logs.
Environment
Example code snippet
This example code mirrors the behaviour in lib/bugsnag/delivery/thread_queue.rb:
Output:
Question: Could you maybe check if the
worker_thread
is still alive before pushing new messages and if not start a new thread? In the code example we can see, that after the exception the thread reports to be dead. So maybe this could be used as indicator to create a new worker thread?The text was updated successfully, but these errors were encountered: