Heap size in memory plot

[Kacper-Pietkun]

Hi there,

Recently, I've been profiling memory usage of dgl (https://github.com/dmlc/dgl), with memray, and I've got some strange results – heap size chart generated by the flamegraph reporter was exceeding memory available on my machine.

Profiling was done on a machine (Ubuntu 20.04.3 LTS (Focal Fossa)) with 126GB of RAM (and 4GB swap), however amount of heap size exceeds almost 5TB (Peak memory usage: 4.8 TB).

Am I misinterpreting the meaning of heap size chart, or does it classify as a bug?

[pablogsal]

This is very possible. You can request more memory than what you have and the kernel will have different ways to act here. For instance, it can simply do nothing until you write to this memory, in which case it will do the actual allocation. Another thing it can do is use SWAP space to satisfy the memory that you are requesting. Something else than can happen if that the program is calling mmap with file-backed storage.

You can read more about this here.

Without more data is difficult to know exactly what's happening but my guess here is that your program either has file-backed mmap-ed chunks or is requesting a ton of memory that is not used.

If you give us a very small reproducer that we can run we can investigate further.

[Kacper-Pietkun]

Thanks for the explanation! I managed to achieve these results only when installing dgl from source. When I install it with pip, I get different results (heap size does not exceed available RAM). If I manage to reproduce this behavior with dgl installed via pip, I will definitely share a reproducer here.