Open some org files and the memory usage will rise to 100% until it is stuck

[178me]

Describe the bug

I try to use orgmode, but opening some org files will fill up my computer's memory until it dies

Steps to reproduce

1. open file
2. Observe computer memory

Expected behavior

Open it without jamming

Emacs functionality

No response

Minimal init.lua

Used the provided minimal_init.lua (no my configuration). link

Screenshots and recordings

No response

OS / Distro

ArchLinux

Neovim version/commit

0.8.2

Additional context

No response

[hardenedapple]

Just wanted to +1 this.
I have quite a few orgmode files for work (that unfortunately can't share since it's work), and while orgmode doesn't crash on opening them it make my neovim memory footprint much larger (such that when I'm compiling Linux often decides to OOM-kill neovim).
Also (in case it helps) I've noticed that during my work day the memory footprint increases significantly -- don't know what actions trigger this, but it does seem to be while working on the orgmode buffers.

[kristijanhusak]

@hardenedapple can you provide some metadata regarding your files? How many lines? Do you have any source blocks (BEGIN_SRC/END_SRC)? Does your memory increases only while the Neovim is opened, or it happens even after you closed Neovim?

[hardenedapple]

Sure -- the files are about 40k lines long.
They have quite a lot of #+begin_example blocks but not #+begin_src.
I tend to have quite large blocks of quoted text in the #+begin_example blocks, without any filetype annotations (essentially transcribed shell sessions including the output I saw).
(Just checked one file, 32k lines of the 43k lines are within these blocks)
With a random test right now it seems that just opening a file takes up about 200mb, then if I edit it a bit (adding a few titles and typing some text) neovim usage grows to 800mb RAM. However after a while further edits seem to add less and things seem to be stable at ~900MB Resident RAM (Rss) for this one file.

[kristijanhusak]

I'll investigate, but those are fairly big files.
There is some memoization involved in everything, so that's why it takes some of the memory, but GC should release the old cache after any changes to the file.

[hardenedapple]

Thanks! I appreciate you looking into it (especially since this can't be a super simple thing to track down and I know how difficult it is to find time to work on unpaid things).

Just since your comment made me realise the size of the file is useful information I forgot to mention earlier:
The size of the file I tested is 3.2mb (c.f. to ~900mb of RAM that it eventually consumes).

[kristijanhusak]

@hardenedapple can you pull latest master and see how it works now? I did some refactoring around memoization to clean up unused cache, so hopefully it should not take so much memory. Let me know.

[hardenedapple]

It's certainly helped a lot thanks!
That 900mb number after a few edits has dropped to about 480mb.
I'm still having to be a little careful about memory and the OOM killer, but much less careful 👍

[kristijanhusak]

Awesome! Keep an eye on it and let me know if you get to a 100% memory usage.

[kristijanhusak]

Also in your description you wrote that you use Neovim 0.8.2. We support 0.9+ versions, but it is suggested to switch to 0.10 since it performs much better in terms of tree-sitter parsing.

[hardenedapple]

Also in your description you wrote that you use Neovim 0.8.2. We support 0.9+ versions, but it is suggested to switch to 0.10 since it performs much better in terms of tree-sitter parsing.

Ah -- I think this was the OP of the issue (that I seem to have hijacked). I am using v0.10.