Connections not closing, runs out of file descriptors

[mattventura]

I occasionally run into an issue where after running for a while, connections that haven't been cleanly closed from one end or the other accumulate, and eventually stops accepting more connections. I see a bunch of these in the log:

Aug 22 19:36:25 tobaru[538]: [2023-08-23T02:36:22Z ERROR tobaru::tcp] Accept failed: Os { code: 24, kind: Uncategorized, message: "No file descriptors available" }
Aug 22 19:36:25 tobaru[538]: [2023-08-23T02:36:22Z ERROR tobaru::tcp] Accept failed: Os { code: 24, kind: Uncategorized, message: "No file descriptors available" }
Aug 22 19:36:25 tobaru[538]: [2023-08-23T02:36:22Z ERROR tobaru::tcp] Accept failed: Os { code: 24, kind: Uncategorized, message: "No file descriptors available" }

I think something is causing connections to not properly be cleaned up. For example, right now, the server running Tobaru is showing 250+ TCP connections to/from Tobaru, while the host that Tobaru is redirecting them to is showing only 20 connections on that port. Stopping the service for a few seconds and starting it again fixes the problem.

If the underlying problem can't be fixed, then there are several potential workarounds, including:

• Timeout for old connections
• Limit for connections, where new connections will close old connections once full
• Option to have the service exit if this condition is hit so that it can be restarted automatically via systemd or others

[cfal]

how are you checking that there are only 20 connections on that port? for the 250+ TCP connections to/from tobaru, what state do you see them in when checking with a tool like ss? (eg TIME-WAIT, CLOSE-WAIT)

also - do you have a server setup that forwards to both TLS and non-TLS servers? there was previously no timeout set for the peek TLS stage, which was just added.

i'd recommend testing again at v0.9.0 which has quite a few fixes, but unfortunately requires a few config changes: https://github.com/cfal/tobaru/blob/master/UPGRADING.md

[mattventura]

Sorry, my notifications broke so I didn't see this until now.

I'm not doing any TLS, just simple TCP+UDP. The connections I saw were in ESTABLISHED state.

One thing that I don't see in the UPGRADING doc - do I need to change anything if I'm just using the URL configuration on the command line?

[mattventura]

Okay, I saw that the URL config no longer supports UDP, so I switched to YAML configuration and version 0.9.0. Still seeing the issue.

[cfal]

I'm not doing any TLS, just simple TCP+UDP. The connections I saw were in ESTABLISHED state.

if you're seeing it in ESTABLISHED state, that means that both sides of the connection (the proxy connection from you to tobaru, and the server connection from tobaru to your server) is still alive, and tobaru is behaving as expected. if either side of the pipe is not readable, the connection would be closed.