Macro interpreter memory leak, up to 24GB RAM usage observed

[yuxiaomao]

When compiling a macro heavy project with vscode, we observed that haxe.exe 's memory usage grows quickly at each compile (and diagnostic) when the source is modified. It grow easily up to 24 GB before display memory details and 10 GB after display (there is a Ocaml GC call inside get_memory_json) which freeze the system. Restarting language server reset the cache size but does not stay for long (devs are restarting very regularly now 😭).

We suspect memory leak in macro interpreter. With some additional tracing on each variable of "macro interpreter", it shows the following result (knowing that after ocaml GC, the total cache size after the first build is less than 400MB, and average usage of another projects is around 1.5GB):

[Simn]

I'll be happy to assist you in this quest, but it will be quite difficult for me to reliably infer much from here. Most data on evalContext.ml context itself is not open ended, so I think the most likely candidate for a leak is mutable curapi : value MacroApi.compiler_api. I've removed a lot of global state in #11483 so we might have a chance to diagnose the memory properly.

First we should make sure that we're actually looking at the right thing though. Something you could try is doing macro_interp_cache := None in serverCompilationContext.ml reset. This will be slow, but if it indeed resolves the memory leak then we can at least confirm that the data reachable from context is involved.

[yuxiaomao]

I added MacroContext.macro_interp_cache := None; just before

haxe/src/compiler/serverCompilationContext.ml

Line 47 in 68c95df

Parser.reset_state();

I could not observe memory usage, because when I build for the second time:

If i modified the source (I added a new line), I got an error from domkit
Missing super component registration flow

if I did not modified the source, I got this warning (which is a true warning) and each time it add 1 warning to the same position (which is strange)
Warning : (WDeprecated) Use LookAt instead

However I can see that GC heap words is bigger than the sum of other memory usage displayed

[Simn]

Sounds like those are separate issues to address regardless of what we end up finding here.

GC heap words is interesting because get_memory_json does have a Gc.compact() call, so this is actually meaningful (otherwise it would only show the amount of reserved memory which is pretty useless information). That then begs the question what is actually leaking.

Just to be sure, the sub-items under total cache don't show any leaks entries, right?

[yuxiaomao]

Yes I think those are separate issues (but very likely related to language server).
I thought that GC heap is the sum of others because in the first image it seems really the case. But in the second one it's just big.
The total cache item remains around 200MB and sub-items does not exceed the size of total cache (the biggest is context which is nearly the total size)

[Simn]

I think what you mean is live_words. It might be a good idea to add that to the JSON output as well.

[yuxiaomao]

I used the code below for tracing in memory.ml

"additionalSizes",jarray (
	(match !MacroContext.macro_interp_cache with
	| Some interp ->
		[
		jobject ["name",jstring "macro interpreter";"size",jint (mem_size (MacroContext.macro_interp_cache))];
		jobject ["name",jstring "macro builtins";"size",jint (mem_size (interp.builtins))];
		jobject ["name",jstring "macro debug";"size",jint (mem_size (interp.debug))];
		jobject ["name",jstring "macro curapi";"size",jint (mem_size (interp.curapi))];
		jobject ["name",jstring "macro type_cache";"size",jint (mem_size (interp.type_cache))];
		jobject ["name",jstring "macro overrides";"size",jint (mem_size (interp.overrides))];
		jobject ["name",jstring "macro array_prototype";"size",jint (mem_size (interp.array_prototype))];
		jobject ["name",jstring "macro string_prototype";"size",jint (mem_size (interp.string_prototype))];
		jobject ["name",jstring "macro vector_prototype";"size",jint (mem_size (interp.vector_prototype))];
		jobject ["name",jstring "macro instance_prototypes";"size",jint (mem_size (interp.instance_prototypes))];
		jobject ["name",jstring "macro static_prototypes";"size",jint (mem_size (interp.static_prototypes))];
		jobject ["name",jstring "macro constructors";"size",jint (mem_size (interp.constructors))];
		jobject ["name",jstring "macro file_keys";"size",jint (mem_size (interp.file_keys))];
		jobject ["name",jstring "macro toplevel";"size",jint (mem_size (interp.toplevel))];
		jobject ["name",jstring "macro eval";"size",jint (mem_size (interp.eval))];
		jobject ["name",jstring "macro evals";"size",jint (mem_size (interp.evals))];
		jobject ["name",jstring "macro exception_stack";"size",jint (mem_size (interp.exception_stack))];
		jobject ["name",jstring "gc live_words";"size",jint (stat.live_words)];
		]
	| None ->
		[jobject ["name",jstring "macro interpreter";"size",jint (mem_size (MacroContext.macro_interp_cache))];]
	)
	@
	[
		(* jobject ["name",jstring "macro stdlib";"size",jint (mem_size (EvalContext.GlobalState.stdlib))];
		jobject ["name",jstring "macro macro_lib";"size",jint (mem_size (EvalContext.GlobalState.macro_lib))]; *)
		jobject ["name",jstring "last completion result";"size",jint (mem_size (DisplayException.last_completion_result))];
		jobject ["name",jstring "Lexer file cache";"size",jint (mem_size (Lexer.all_files))];
		jobject ["name",jstring "GC heap words";"size",jint (int_of_float size)];
	]
);

[yuxiaomao]

And after 2 compilation (with some code modified manually)

[yuxiaomao]

I just saw that Gc.quick_stat() does not compute live_words, will check if it can gives more informations

[Simn]

Maybe I have to RTFM again, don't know what's going on with the live words... (Edit: Ah, just saw you next comment, makes sense!)

But this is useful information, so the problem appears to be in the eval data structure. One crude but effective approach here would be to keep going, i.e. adding more nested information for env, thread and such. Eventually this should allow us to find the culprit, and that would then help with this kind of problem in the future too.

[Simn]

Let's keep this open, I want to investigate why the stack frames aren't popped in the first place.