Sync + Server thread hitch starting with artifact 8096

[manzarek123]

What happened?

Starting with artifact 8096, every time a new player is connecting to the server there is a sync + server thread hitch.
This happens at high player count and every time a new player is connecting to the server.
People are complaining of more lags but is is probably because there are lots of new players connecting all the time.

This started with build 8096, I have tested the following builds:
7900 ok
8075 ok
8085 ok
8096 not ok
8139 not ok
8181 not ok

Let me know if I can help to provide more info

[ script:bay_queue] [Queue] Allowed jeuke from VIP Queue (size: 2)
[ citizen-server-impl] sync thread hitch warning: timer interval of 185 milliseconds
[ citizen-server-impl] server thread hitch warning: timer interval of 195 milliseconds

script:bay_queue] [Queue] Allowed Satoshi (Fr) from VIP Queue (size: 1)
[ citizen-server-impl] server thread hitch warning: timer interval of 171 milliseconds
[ citizen-server-impl] sync thread hitch warning: timer interval of 109 milliseconds

[ script:bay_queue] [Queue] Allowed ogu from Queue (size: 43)
[ citizen-server-impl] server thread hitch warning: timer interval of 195 milliseconds
[ citizen-server-impl] sync thread hitch warning: timer interval of 215 milliseconds

Expected result

No sync and server thread hitch

Reproduction steps

1. Have a server wiith arctifact 8096
2. Connect to the server

Importancy

Slight inconvenience

Area(s)

FXServer

Specific version(s)

Server 8096

Additional information

No response

[gottfriedleibniz]

Would it be possible to get a ETW Trace? May be possible that reverting Lua back to use the systems allocator is causing a bit of lock contention.

[manzarek123]

I will update to 8096 this afternoon and make the trace this evening !

[manzarek123]

Here is a trace, let me know if it is good
https://1drv.ms/u/c/16cac549e8207bcc/Ea7_wWYPgS1JqcQVSYq_5w0B5InYZvxSJ6cJ-jb0Q48nKQ?e=dQD7KC

[gottfriedleibniz]

Thanks. Seems to be the case: garbage collection step deallocating ~800 objects and this is contending with some other threads. The Windows allocator is pretty slow in general so Lua's custom allocator was probably carrying a lot of weight.

This trace showed a few areas where heap interactions can be reduced/optimized. Hopefully I'll be able to get to this sooner rather than later.

[Korioz]

Up, this affects a lot of servers, we cannot update to newer builds because of those massive hitches, this cause our players to see this misleading "Server artifact version is outdated" message.

[gottfriedleibniz]

Would it be possible to test artifact 8450. If you are still running into issues, would it be possible to send another ETW trace?

There's still some heap contention between the other threads that needs to be optimized.

[manzarek123]

Sure, I'll update now and report this evening when player count is high

[manzarek123]

there are no issues with build 8450