ONNX runtime on CPU - huge difference between heap vs resident size

[ogencoglu]

I am running an inference with a neural network model in ONNX format on CPU. I am on Mac with 32GB memory. The report shows peak heap size 53.7 GB. I am not getting any OOM errors. Do I understand correctly that the actual utilized memory is the resident size? What may be the reason with such huge difference between heap and resident with ONNX runtime?

[ogencoglu]

I guess it is related to this: microsoft/onnxruntime#11627 (comment)

[sarahmonod]

Do I understand correctly that the actual utilized memory is the resident size?

The "actual utilized memory" can mean different things. Instead of guessing what you meant by this, I think we should instead focus on what both of the metrics Memray reports mean:

1. The Resident size (also called Resident Set Size, or RSS) is the memory that is currently loaded in RAM. It does not include memory that is swapped out (saved to disk). It does include memory from shared libraries as long as the pages from those libraries are loaded in RAM. It includes memory allocated on the stack and the heap of the program.
2. The Heap size is the size of the heap portion of the Virtual Memory. This includes all of the (heap) memory that the process can access, including memory that is swapped out and memory that is allocated, but not used. It does not include the stack memory, the shared libraries, nor the memory of the binary being executed itself. In other words, this is all of the memory that was allocated through malloc or mmap.

In your case, we know for sure that a significant chunk of the memory reported in Heap size is swapped out or unused because the value reported is higher than the available RAM.

See more about the difference between Resident Set Size and Virtual Memory Size here: https://softwareperformancenotes.github.io/rssvsz

[ogencoglu]

Thanks for the detailed reply! Appreciated!

[godlygeek]

To wrap this up, we now have some documentation on how to understand both heap memory and RSS memory. It helps to explain why it's possible for heap memory usage to be either higher or lower than the resident set size, and hopefully helps people understand how to interpret these two different ways of measuring the process's memory usage.