Too Many Pending Connections

[MohammadrezaNasrabadi]

Title: Too Many Pending Connections

On a Couchdb cluster with three nodes, the number of connections in
CLOSE-WAIT state to one of the nodes gradually increases. This is
what I get right now (Couchdb has been running for about four days):

$ ss -nt | grep CLOSE-WAIT | wc -l
4514
$ ss -nt | grep CLOSE-WAIT | head
CLOSE-WAIT 1      0       192.168.40.2:35938    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:44694    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:60596    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:34240    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:40818    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:43376    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:56616    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:47966    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:48366    192.168.40.2:5984
CLOSE-WAIT 1      0       192.168.40.2:59302    192.168.40.2:5984

Maybe Couchdb does not close connections sometimes? We use Couchdb 3.3.3.

[nickva]

From what I remember from a while back, close-wait means the remote client closed the connection, then, our (server) kernel found out about it, and is now waiting on the application code (CouchDB) to notice as well and close it.

So, perhaps maybe that one node is overwhelmed, sometimes is blocked? (maybe a firewall?), or not getting enough CPU time to run?. Check your resource usage (cpu, memory, open file handles) and the logs to see if there are exceptions or errors on that one node. At least you have a few other nodes to compare to where this doesn't happen.

[MohammadrezaNasrabadi]

We don't apply any configuration about the firewall to have limitation or control over the connections.

On the other hand, We didn't see huge CPU or RAM usage on the nodes hosting CouchDB.

Do you have any benchmark results based on the number of shards, replicas, amount of database requests and the available resources? It will be useful to compare our CouchDB infrastructure status with the ideal resource requirements.

I will share the detailed information of our infrastructure if needed.

[nickva]

It's hard to compare architectures, it depends on the requirements. For instance some of our larger clusters have 30+ nodes and our load balancer handles 100k concurrent open connections. Even there, on the load balancer usually there almost no CLOSE_WAIT connections. At most I could see a short temporary increase in CLOSE_WAIT up to 3k out of 100k established connections, then they get cleared back to a few hundred only.

It's good that your resource usage is low it seems. Since you have other nodes to compare to, and others don't seem to show the same behavior try to see what's different between them (configuration, hardware resources). Maybe check your logs for any errors on some vs others?

[MohammadrezaNasrabadi]

We performed experiments to find the main reason for this problem.
The problem (connections in CLOSE-WAIT state) happens even without
a CouchDB cluster and with a local loopback connection. It seems
that CouchDB does not close some of its connections after the
client closes it; therefore, the connections remain in CLOSE-WAIT
state.

We connect to CouchDB from an Elixir application using the hackney
library. After a few weeks of running, we have thousands of CouchDB
connections in CLOSE-WAIT state that disappear only by restarting
CouchDB. We have not yet found which of the connections made by
hackney remain in CLOSE-WAIT state when hackney closes it.