Handle .bin file of big size

[svex99]

I'm running a profiling to a script that runs about 600 seconds, commiting a bulk of entries to a DB and other tasks. I run the profiling and it works fine but the file .bin generated is about 5.0 GB of size. My laptop has around 8.0 GB of RAM and when I run any of the reporters (tree, summary, flamegraph, etc) the process is killed, so I cannot display the results.

Is there some way to display the results or reduce the size of the profiling output file?

Thanks!

[pablogsal]

Hi @svex99 and thanks for opening an issue!

In the next memray release that we are preparing there are three things that will greatly help with this:

• File sizes will be reduced by more than 95%! 🚀
• Files will now be compressed on disk using lz4 📦
• Reporters (tree, summary, flamegraph, etc) won't need to read the entire file in memory at the same time, so although it may take time, you will be able to analyze even gigantic files 😮

We will make a release next week, so hopefully, this will help :)

Thanks a lot for using memray :)

[svex99]

Thank you for your fast reply. Great news, I'll wait the release!

[godlygeek]

The new version was released yesterday - give Memray 1.1.0 a shot (pip install --upgrade memray if you're using pip) and see if things work better now.

[svex99]

I just tried it and is amazing, only 94 MB of size for the generated .bin. Great job and thanks for this tool!

[tvalentyn]

I have used memray with fairly large .bin files with memray table command. I noticed that the generated html output also grows in size significantly with the source file growth. If I profile a program for 10x longer, the bin file is 10x larger and so is the html table file, however the number of rows in the table stays roughly the same. In an extreme example, I had 20gb of profiled data in (--native mode), and the table html was ~700 MB, which was unusable at that point.