Replace Timings with Spark

[kashike]

Based on an internal discussion in August of 2022, we are now entering the planning stages to remove the Timings functionality from the Paper server software, and to include Spark in the distribution instead. More information will be shared once plans have solidified, this issue is being opened now to begin public tracking of our plans.

Some of the reasons for this are as follows:

• Timings has been unmaintained for multiple years now, with only very few additional timings categories added since. Because of this, it is not ideal for actually identifying causes of lag in specific plugins or parts of the server outside of simple categories like synchronous chunk loads.
• Timings are difficult to read for people that are new to it, and there are no short and easy to digest tutorials on it either. Spark is a lot less confusing in terms of UI and has more beginner-friendly documentation.
• Enabling timings can sometimes have non-negligible hits on server performance, whereas Spark is lighter and also has an asynchronous profiling mode.

While all of this is still open for discussion, here are potential todos:

• Deprecate public Timings classes for removal #8949 ?
• Add runtime nags for uses of Timings API: 32875b4
• Figure out how exactly to bundle or include spark
• Make sure spark properly replaces the useful functionality timings had

[hendoulmou9]

Figure out how exactly to bundle or include spark: for example, we can create a new timing using Spark's Spark.createTask() method, which takes a string as a parameter representing the name of the timing task. We then start the timing using the startTiming() method, and run some code. Finally, we stop the timing using the stopTiming() method.

[Phoenix616]

I feel like one requirement for a replacement should be to have equally user-friendly names for entries like Timings has right now which is a major benefit of it over Spark: That someone who doesn't know anything about internals is able to quickly spot what is wrong and not have to bother data that doesn't matter for them.

Having just the (sometimes confusing or even wrong) Mojang names of classes/methods doesn't really help people that are not developers and right now Timing names also provide additional context like the type of entity or the default tick counts that are currently not accessible via Spark or (easy) other means.

• Timings are difficult to read for people that are new to it, and there are no short and easy to digest tutorials on it either. Spark is a lot less confusing in terms of UI and has more beginner-friendly documentation.

Basically I'm counteracting this point. Spark profile entries are a lot harder to read than timings entries are. Timings (in contrast to spark) even provides the parts that cause issues directly at the top as well as point out lag directly, something that Spark doesn't do (at least as far as I know)

If anything the Mojang profiler information (which is in places more granular than timings) should be used. (Or could even be pulled into timings directly to solve the "Timings has been unmaintained for mulitple years now" part)

Other requirements (besides better user-friendlieness/usability) for a new system that I see that Spark doesn't provide (right now):

• A way for plugins to specify human-readable names for stuff like Timings did would be really appreciated, a user is not going to understand what some loop/stream structure means but they can easily know that with plugin-author provided information
• Automatic pointing out stuff that causes lag (similar to Timings default lag view and the ordering of the entries by the time spent. Spark (by default) uses a tree view which you need to manually expand)
• Easily accessible Entity count information per region (Timings region tab, Spark has that information but the interface there right now is really convoluted and hard to use)

[regulad]

You should consider contributing to Spark to bring it closer to feature parity with Timings, as the Timings library has been unmaintaned for quite some time now (see OC issue) and only becomes more archaic and harder to work with as time goes on. Spark, on the other hand, is still being actively maintained and having features added. Other Paper forks like Pufferfish/Purpur have already replaced their timings with Spark, and do not seem to have lost any functionality or ease-of-use.

[Phoenix616]

@regulad I fail to see how your comment helps at all with the discussion how Spark can/should be included in Paper directly and how it compares to Timings. Of course if Spark should be used directly and changes need to be made to bring it up to the usability of Timings then that could be contributed directly to their project (although it would probably still be necessary to have some changes be done directly in the server and not just Spark alone, e.g. Spark would not be able to automatically know the context that their data is gathered from without markers directly in the server code indicating that in a human-readable way similar to how Timings work right now)

[Fillup99999]

This is the spark jar file. May want to look into another solution...

[electronicboy]

Antivirus software is generally useless about jar files and often just flags stuff somewhat blindly at best. I don’t think that the malicious result from a singular piece of software stands for annything against an open source project in which you can vet every single line of code in the output.

[Codixer]

This is the spark jar file. May want to look into another solution...

This is the worst example of a "Hey this file is a virus" warning I have ever seen.
I would really check who created Spark, considering they also created one of the most famous permission plugins in the world. And with 63 anti-virus's who say it's not a virus? I doubt it would be a virus as all.

How do I know for sure it's owned by the same guy? Well:

• https://spark.lucko.me/
• https://github.com/lucko

Case and point. If you make an accusation of something being a virus, do some proper testing and sandboxing first.

[JIBSIL]

After the migration would there be a way to re-enable timings? I personally find Spark completely ineffective for actually finding sources of lag. Right now, though it's touted as being beginner-friendly, Timings is basically readable by people who can't code at all- and Spark lacks valuable integrations with plugins such as Skript. Lag was such a big problem with my server that I had to switch Papermc forks because Purpur disabled timings.

If you do decide to switch Timings with Spark, would there be a commandline flag or plugin that would re-enable it?

[Phoenix616]

I personally find Spark completely ineffective for actually finding sources of lag. Right now, though it's touted as being beginner-friendly, Timings is basically readable by people who can't code at all- and Spark lacks valuable integrations with plugins such as Skript. Lag was such a big problem with my server that I had to switch Papermc forks because Purpur disabled timings.

As often pointed out when this point is made it would be good to bring your feedback to the Spark developers directly regarding what you think is missing from it to make it useable for you personally.

While I personally also agree with parts of your statement (and have forwarded my issues that I see to Spark devs already, especially the plugin-integration thing) it will at some point probably not be possible to re-enable it seeing as completely removing it adds more value (as the devs would no longer need to maintain and potentially work around it) than leaving it available but in a disabled state.

[dcorbe]

I have spark installed on my papermc server (via plugins folder) but I still get the spark warning on startup as of 1.20.1. Is there something more I have to do to configure it?

[Warriorrrr]

The warning is because timings is still enabled, if you want to get rid of it you'll need to disable it in the config.

[Chevels]

The big problem with Spark is that it is not compatible with Java OpenJ9...
I had to remove it a long time ago because we are running Java OpenJ9 for performance reasons (to fix docker issue with GC and RAM) and unfortunately Spark killed the Java machine on startup. It still the case en October 2023.
The Paper devs will abandon support for Timings to make way for Spark but if it is not OpenJ9 compatible some servers will no longer have effective profiling solutions :/

In addition, when I read a report from Spark I do not understand the terms in the proposed tree structure. However, I have been administering my server for 13 years, but despite my little experience I find no useful information for someone who does not know the meaning of the classes or terms given by Spark (in fact I do not even know what this is when I see for example "libc.so.6.__futex_abstimed_wait_common()" ). How to read such a report without being a developer?

Spark therefore brings together 2 big problems, and not the least...

[l3nnartt]

Paper is not OpenJ9 compatible too

[Chevels]

Paper is not OpenJ9 compatible too

I've been using Paper on OpenJ9 for years and it works perfectly.
Moreover, out of 86 plugins, all work in OpenJ9, except Spark...

[Sytm]

This sounds like an issue that should be reported over at the GitHub of Spark with some logs and reproduction steps to get this incompatibility actually fixed

[Chevels]

This sounds like an issue that should be reported over at the GitHub of Spark with some logs and reproduction steps to get this incompatibility actually fixed

Indeed you are right. I informed Luck on Spigot but he doesn't respond there...
I'll go to Github, thanks

[l3nnartt]

Paper is not OpenJ9 compatible too

I've been using Paper on OpenJ9 for years and it works perfectly. Moreover, out of 86 plugins, all work in OpenJ9, except Spark...

I had the same problems a few months ago and was told on the Paper Discord that there was no support for OpenJ9.
But if you find out more, please let me know, ive been running on hotspot since then and the memory consumption is significantly higher. :/

https://discord.com/channels/289587909051416579/289587909051416579/1117841776594657341 (discord.gg/papermc)

[MiniDigger]

Paper is not OpenJ9 compatible too

I've been using Paper on OpenJ9 for years and it works perfectly. Moreover, out of 86 plugins, all work in OpenJ9, except Spark...

I had the same problems a few months ago and was told on the Paper Discord that there was no support for OpenJ9. But if you find out more, please let me know, ive been running on hotspot since then and the memory consumption is significantly higher. :/

discord.com/channels/289587909051416579/289587909051416579/1117841776594657341 (discord.gg/papermc)

there are two different levels of support. we will not support you if you use openj9, as in, we will not help you with problems because we dont know if its caused by openj9 (which it often is). we also only test on officially validated distributions of openjdk/hotspot.
that doesnt mean that paper doesnt run on openj9. it just means that if it doesnt, but does on hotspot, you need to go to IBM, not us for support.

[Chevels]

Paper is not OpenJ9 compatible too

I've been using Paper on OpenJ9 for years and it works perfectly. Moreover, out of 86 plugins, all work in OpenJ9, except Spark...

I had the same problems a few months ago and was told on the Paper Discord that there was no support for OpenJ9. But if you find out more, please let me know, ive been running on hotspot since then and the memory consumption is significantly higher. :/
discord.com/channels/289587909051416579/289587909051416579/1117841776594657341 (discord.gg/papermc)

there are two different levels of support. we will not support you if you use openj9, as in, we will not help you with problems because we dont know if its caused by openj9 (which it often is). we also only test on officially validated distributions of openjdk/hotspot. that doesnt mean that paper doesnt run on openj9. it just means that if it doesnt, but does on hotspot, you need to go to IBM, not us for support.

Which is really a shame because OpenJ9 manages memory much much better and doesn't have any docker related issues.
As for the problems you are talking about related to OpenJ9 with Paper (when you say "which it often is"), I don't see which ones there could have been because in several years I have never encountered one.
In reality, the only thing that doesn't work with OpenJ9 is Spark, which is a shame for such a plugin that's supposed to be so good...

[SnowzNZ]

OpenJ9 is slower aint it?
https://github.com/brucethemoose/Minecraft-Performance-Flags-Benchmarks/blob/main/Benchmarks/2022-10-20_21-44-05_OpenJ9_vs_OpenJDK.json

[emilyy-dev]

This issue is for planning and discussing Spark's future integration into Paper, not HotSpot's ability to run on docker or OpenJ9's performance metrics.

[Chevels]

This issue is for planning and discussing Spark's future integration into Paper, not HotSpot's ability to run on docker or OpenJ9's performance metrics.

Except that many servers run on OpenJ9 because OpenJDK has a significant memory leak due to docker, especially among hosts using Ptero. So if Spark does not run on J9, by making the decision to integrate Spark instead of timings, you will put all these servers in difficulty. This problem is however known (an example):
pterodactyl/panel#4635
I think many here don't realize the real problem.

For example, on my server, like many others, OpenJDK always reserves more and more memory, without ever lowering its allocation until the server crashes. If I currently install OpenJDK, my RAM consumption will increase to 13GB in a few hours and reach 16GB in approximately 12 hours. The same server on OpenJ9 consumes a maximum of 5-6GB and the memory allocation is perfectly managed. The server can run for 2 weeks without reboot (for example, even if in reality it restarts 1 time per day, but it would be possible) which is inconceivable on JDK and this is independent of the server itself.

Ignoring this and brushing it aside is irrelevant, especially since J9 compatibility does not seem infeasible since apart from Spark, all the other plugins that I have used in 13 years on my server are compatible (actually 85, except Spark).

So yes, it's the same subject, whether you like it or not :/

[electronicboy]

I've not heard of a memory leak inside of OpenJDK, just a general misunderstanding of how javas memory model works (and the odd once in a blue moon native leak nobody has ever managed to figure out), and its application of such when running in a resource-constrained environment.

But, even if there was a leak, that generally does not change the fact that we do not support non-hotspot JVMs. You are more than welcome to use them, many dozen people run paper inside of Graal or OpenJ9, however, we have 0 inclination to impede the rest of the community for a small % of servers running in an unsupported environment. We're not going to go out of our way to hamper running those environments, but, timings is a tool that has generally outgrown its utilization, especially in an era where a few ms a tick is a sizable hit vs years ago when the performance cost of timings far was far outweighed by its utility, that is far from the case in this era.

You'd do better pleading to the async-profiler folk to consider supporting non-hotspot environments, but I have no idea what APIs IBM have butchered/removed in their conquest.

[Chevels]

I understand but I would like to point out that the servers experiencing this problem have nothing to do with it, they depend on the choices of their server hosts and for ordinary people, it is impossible to know in advance if this or that host will have an environment incompatible :/

In the end just need to make Spark compatible (I posted the issue on the Spark Github, wait and see now...), after all the other plugin developers are getting there so it shouldn't be insurmountable.

[electronicboy]

It's not our role to go out of our way to cater towards hosts enforcing unsupported environments onto people, especially ones which are commonly known within the java community to come with their own host of headaches and potential caveats/issues.

timings is no longer a small inconsequential % of the servers mspt and is slowly becoming less and less useful over the years due to mojangs complexity meaning that we'd need to drastically increase timings coverage for it to remain to be useful, which in turn would come with get another hit to performance.

theres some interesting stuff in the lineup for async-profiler in which should hopefully get us much closer to timings in terms of its easy of understanding what certain parts of the tick loop do, async-profiler does seem to support OpenJ9 to some degree, theres a few j9 related (closed) issues on their tracker, generally looks like a bunch of it is "we need j9 to fix this"

[emilyy-dev]

Spark killed the Java machine on startup. It still the case en October 2023

I should mention that I am able to start and run a server running OpenJ9 with Spark installed just fine, use the profiler and everything, so there seems to be some other external environmental issues at play than just "it crashes on startup".

[Chevels]

Probably, however, out of 90 plugins, the one and only one that just needs to be added so that the server crashes when starting with openJ9 is Spark. So it's obvious that there is a problem with this particular plugin.
Whether it is related to an environment or not, my server is far from being the only one to be affected by this problem (and it is not hosted by a small host that no one knows...). Just type the keywords into Google to find out, in addition to the link I provided above.

I understand that you do not want to lift a finger about this problem, I will not insist any further since it is of no use and you do not seem to understand. But whether you like it or not, there is a real problem with Spark and J9, it is obvious, clear and precise. You don't have to leave a big school to realize this.

Best wishes