Socketioxide feedback with large number of connections

[Henrriky]

Recently I had problems with Socketio to scale a company system, I decided to look for similar alternatives, as the application was written based on Socketio. Fortunately, there was this version in Rust that has much of the functionality of the official Javascript library.

I come here to give feedback on Socketioxide with a large number of clients, managing rooms and some sockets through Redis. In addition to a considerable number of rooms and event handlers created.

The process was managed at all times by PM2. Apparently memory consumption has a linear growth along with user growth

10K users triggering events every 1000ms

• Each thread 10% CPU
• 7.5% RAM
• Connection time: 0.0005ms
• Average number of rooms per user: 6

30K users triggering events every 1000ms

• Each thread 19% CPU
• 25% RAM usage
• Connection time: 0.0105ms
• Average number of rooms per user: 6

[tausifcreates]

Hey, this data is so awesome and insightful! Thank you for sharing with us! Just yesterday I was talking with @Totodore about heavy ram usage and looks like it isn't too taxing on ram 😄

[Henrriky]

No problem, here are the hardware specs:

Taking advantage of this, I would like to know if you have any plans for the launch of an adapter for use with multiple servers. I ask this question because our customer base is growing very quickly and there will probably come a time when we will run into the limitation of just one server.

[Henrriky]

Just one detail about Socketioxide. With a large number of simultaneous connections, at exactly the same time and without the redis that I implemented to fetch user information from the same room, I faced a problem of high cpu usage, approximately 80% each thread and the events did not arrive after a time.

Another thing, there seems to be something that remains in RAM memory, I just can't say what. When there is a peak in connected users, let's assume that memory remains at 20%. After they suffered the disconnection, the right thing to do was for it to decrease, right? However, usage remains at 20% even after users disconnect and the count drops to 0

[Totodore]

I currently can't work much on soxletioxide. I'll try to implement an async adapter for July and eventually the redis remote adapter for September.
Any help will be welcomed for other adapters.

[Totodore]

There is probably a memory leak somewhere.
When you enable debug logs. There is a log when a socket is removed and how many remains. Could you check that at the end of your run (wait the end of the ack timeout), remaining sockets are at 0?
I'm currently adding more tooling to detect memory leaks and ram consumption.

[Henrriky]

There is probably a memory leak somewhere.
When you enable debug logs. There is a log when a socket is removed and how many remains. Could you check that at the end of your run (wait the end of the ack timeout), remaining sockets are at 0?
I'm currently adding more tooling to detect memory leaks and ram consumption.

I will try to activate these logs

[tausifcreates]

@Henrriky Would you mind sharing the Nodejs socketio server performance on the same audience? It would be nice to see the real life performance comparisons :)

[Henrriky]

Absolutely, in fact, to be quite frank, I tried everything to implement NodeJS Socketio, as it is the programming language I know the most. So much so that it was the first time I used Rust, I had never used it before, so it took me 1 week and a half to learn it.
Anyway, talking about SocketIO, I was having performance problems on the same machine, I tried literally everything, the same application. The specifications of the computer used for this were:

Here is a link to the issue I opened on SocketIO: socketio/socket.io-cluster-adapter#11

The results I got with SocketIO were terrible. The development team tried everything: Redis Adapter, MongoDB Adapter, Cluster Adapter, PostgreeSQL Adapter, storing users in Redis. Unfortunately for me, none of them worked, I tried everything, looking at Github issues, posting an issue, and to my surprise several people were facing the same problem. I even tried to explore the low level of the library to see if I could find anything and nothing.

Apparently the problem is mainly in the rooms and in calling functions like fetchSockets.

But I'll tell you what results I achieved with the test example used by SocketIO itself:

=======SINGLE SERVER (POOLING TRASPORT)=======

==== 3K users triggering events every 1000ms

• CPU 100%
• Connection time: Only the first users connected, after that it started to give connection errors: Timeout, XHR Post error, among many others. The application stopped in production and users were no longer able to connect. Then I had to restart the server.
• Average number of rooms per user: 6

==== 10K users triggering events every 1000ms with 1 Instance Without Sticky Sessions and Multiple Servers:

• CPU 130% (Questionable)
• Connection time: Only the first users connected, after that it started to give connection errors: Timeout, XHR Post error, among many others. To be honest, the server even crashed. I didn't even try any more connections, as it was shooting in the dark.
• Average number of rooms per user: 6

======MULTIPLE SERVERS (ALL ADAPTERS AND POOLING TRANSPORT)======

==== 5K users triggering events every 1000ms

• Each thread 12% CPU
• Connection time: The connection in this case was fast, but when there were many concurrent connections, I started to face timeout errors, disconnections and problems mainly with the fetchSockets function.
• Average number of rooms per user: 6
• ERRO DESCRIPTION: Error: timeout reached: only 14 responses received out of 15
  at Timeout._onTimeout (/home/producao/teste/performance-tests/javascript-websocket/node_modules/@socket.io/cluster-adapter/dist/index.js:358:28)
  at listOnTimeout (node:internal/timers:573:17)
  at process.processTimers (node:internal/timers:514:7)

==== 10K users triggering events every 1000ms

• Each thread 38% CPU
• Connection time: The connection in this case was fast, but when there were many concurrent connections, I started to face timeout errors, disconnections and problems mainly with the fetchSockets function.
• Average number of rooms per user: 6
• ERRO DESCRIPTION: Error: timeout reached: only 14 responses received out of 15
  at Timeout._onTimeout (/home/producao/teste/performance-tests/javascript-websocket/node_modules/@socket.io/cluster-adapter/dist/index.js:358:28)
  at listOnTimeout (node:internal/timers:573:17)
  at process.processTimers (node:internal/timers:514:7)

NOTE: I can't describe the exact values ​​of the tests, because I don't remember well, but I can guarantee one thing, the problems I faced with SocketIO were several, and I tried everything, from the simplest application (like a example from their own website) even more complex ones. These tests were also carried out with the artillery. If you want, you can contact me via email so we can have a call and I can explain in more detail, although my English isn't very good and I'm new to programming: [email protected].

[tausifcreates]

Thanks so much for providing all the info in detail :) I will try to summarize this discussion and open a pr to include in the README. what do you say @Totodore ?

[Totodore]

The results and the discussion is very interesting but I would like to only include non-comparative benchmarks (for the moment) with the code provided and graph results.

However @Henrriky it would be really nice if you could share your code for the benchmark (more precisely the artillery code).

[Henrriky]

I attached a link to a Github issue in SocketIO here in this thread