Disconnect / Errors from stream #25
Comments
|
Hey @Tsopic thanks for checking out IBind and letting me know of this issue 👍 I was not aware of some of these status responses returned by IBKR. I'll be sure to add them when I find a minute. As for handling this gracefully, you say:
Can you expand on this? What do you mean it crashes? From what I see in the logs, the authentication status seems to be lost with IBKR - which is not uncommon - and the IbkrWsClient indeed handles it gracefully by logging the incorrect authentication status. Should the authentication be recreated (assuming you're using some automated way such as IBeam) the client would recreate the subscriptions automatically. Does that not happen? If so, I'd appreciate if you'd share a full output log and the code you use to create the subscription - thanks! |
|
So far it seems: IBEAM at some point does restore the authorization, but the streamed data if any, does not show up. For logs, I need to wait a little bit, to encounter this again |
|
hey, thanks for sharing that - it seems all in order. I've pushed v0.1.6 which should handle the competing:true, though that has nothing to do with the resubscriptions. Please try to share the full log once you encounter that same issue next time 👍 |
|
This is using v0.1.6 @Voyz Adding some more logs, Just for full context, I removed the SSL Warnings for cleaner logs. |
|
Hey @Tsopic thanks for providing all of this log - it helped me understand what's happening. I've noticed that the connection is never re-established and subscriptions are not recreated. There is a 'check_health' method on both the I run it along the lines of: authenticated = self._ibkr_client.check_health()
if not authenticated:
_LOGGER.warning(f'IBeam is not healthy: {self.ibkr_url}')
ws_healthy = self._ibkr_ws_client.ready() and self._ibkr_ws_client.check_health()
if not ws_healthy:
_LOGGER.warning(f'IBeam websocket is not healthy: {self.ibkr_websocket_url}')As for the log itself, it seems that the corruption is causing some strange responses to be received from IBKR. It seems that |
|
Okay, adding this as the next iteration of healthcheck. Currently running the health check with 60 sec interval with Uptime Kuma ;) |
|
Great, let me know how it goes! 🔥 |
|
sidecomment - gateway_result = ibkr_client.tickle() |
|
It is - I noticed it too. I thought you wanted to return the various authentication-related messages from that endpoint hence done it twice. Both ways are valid 👍 |
|
@Voyz Sending you the logs over email, still encountering some strange behavior. On the first restart it hangs after like 10min, and on the second restart seems to work fine for hours. |
|
hey @Tsopic I'd appreciate if you could share the logs here for consistency and for other users who may run into this issue in the future. If you feel it's too long you can make it collapsible in the following fashion: Which will look like this: Note the dot Thanks! |
|
So current state is that it still hangs up on the stream. There is a strange behavior, where for about 10 minutes at the beginning of the day it works after restarting. And then hangs and stops sending the price updates. On second Restart it runs a lot longer. I reduced the logs for the "commit" and "consume" thread quite a bit. 01:31:18,825 timestamp is where it hangs. . |
|
Thanks for sharing these 👍 If I could ask you a favour: in the future if you would have a chance to disable or remove the As for the log itself - I cannot see any indication of the IbkrWsClient trying to reset the connection, which leads me to believe that something unexpected is happening here. Usually when these errors appeared, the ping and heartbeat would fail and after one minute cause the re-subscription of all channels. This isn't happening. Strangely, your In the log you attached I cannot see any websocket messages being consumed - I'm assuming you're doing it elsewhere and not outputting anything into this log. In the situation we're facing it would be useful to see when the data is coming in correctly, and when it stops. Can you either try adding message consumption logs to your log output, or confirm in which areas of the above log the data is coming through as expected and when not? Lastly, if you detect such an anomaly using your Let me know how it goes 👍 |
|
|
@Voyz Logs, in the last comment, it is with the ard_reset(restart=True) method. Now it's complaining about Invalid WS returned |
|
Could you point me in right direction how to catch this error in my client?
Would be useful to handle it then on client side |
|
I'm sorry @Tsopic I'm travelling at the moment. I should be able to investigate within the next couple of weeks. Thanks for bearing with me in the meantime and for keeping this thread active 👍 |
|
Hey @Tsopic thanks for your patience 🙏 I've spent a while investigating the reason behind the In short:
The rest of your log tells me that where you're calling Try the |
|
Trying the ibind==0.1.8rc1 Will let know, in few days |
|
Still encountering the same behavior with new ibind version. Attaching the latest state of health check endpoint code. . |
|
Many thanks fo reporting back, appreciate it 👍
When you say 'same', can you elaborate? From what I can see the Other than that I've just pushed As for the rest of your log - it seems to indicate that your session loses authentication and that needs to be fixed externally, possibly by IBeam. Then I can see a call to health server indicating that the authentication is restored. Does the IbkrWsClient proceed to output anything past that point? Have you attempted to use the websocket connection after that point, and if so to what result? A separate interpretation could be that despite receiving If it's A) then I'd suggest that you send a few of these health-check requests over a course of a minute or two, and see what happens. If it's B) then I have no idea what to do 😅 We'd have to talk to IBKR and ask them why does that happen. Thanks for sharing your code too, it looks reasonable. Though here: ws_health = ws_client.check_health()
# Handle ws_health as a boolean
if isinstance(ws_health, bool):I'm a little confused. Does |
|
By same I mean it still disconnects from the MD stream and does not successfully reestablish the streams. By my definition, it would be solved when It's able to restart / resubscribe itself whenever the datastreams get interrupted. I'm now switching to
I think I just added this to satisfy the linting tools for type check |
|
Okay, I think I've got some good news. I found the core issue here. The IbkrWsClient has a lot of code in place to handle drops in connectivity, but it wasn't handling authentication drop as gracefully. While IBKR WS API sends a message when authentication is dropped, it doesn't send a message when authentication is restored, leaving us thinking that authentication continues to be false. I've spent a while to add a bunch of code for handling authentication drops. In short, it will close the WS connection completely, as this will get it into a reauthentication loop, attempting to reopen and resubscribe every few seconds. Once the authentication is restored, this should eventually succeed. I've tested it with Market Data channel and can report that it is restored successfully. Please try the
On the master branch. |
|
Updated to |
|
Heh :) It seems solved for now. I will monitor until Friday, and close this issue, if success keeps repeating 👍 |
|
Music to my ears! Keep me posted, hopefully we got it this time 👍 |
|
Gonna keep monitoring it for one more week. Getting IBeam re-authentication flow happening more often now now, wanna pinpoint the scenario :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey,
I tend to have this problem, where the ´md´ subscription works for a while but then crashes in the background and is not handled properly.
Are there any best practices for reconnecting to WS gracefully?
The text was updated successfully, but these errors were encountered: