Profiling stops when bin file hits 129mb?

[cakir-enes]

I'm using python api to start the profiler with the following options (its a flask app, and some celery modules)
After bin files hit 129mb it seems they are not updated anymore? Size doesnt increase and last touch time doesnt change.

-rw-r--r--  1 root root 129M May  5 20:00 0.8451901132856101.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.5528604572353127.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.0005046702400007819.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.20149761944142108.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.7285573534574442.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.9211960565905064.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.9200296351836896.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.33875160830970574.bin
-rw-r--r--  1 root root 129M May  5 20:00 0.17866675118930586.bin
-rw-r--r--  1 root root  14M May  5 20:00 main.bin

This is probably my bad, Im wrapping whole flask app like this:

with memray.Tracker("main.bin", follow_fork=True, native_traces=True):
 ...
  if __name__ == "__main__":
     app.run...

Doing the same thing for the file that celery -A <module_name> commands run, wrapping the whole file within with memray. block.

How should I be actually doing this? (Im using the api because we are using python modules called by uwsgi and celery not directly)

[pablogsal]

Could you give us a link to the code you are running to see if we can reproduce it?

[cakir-enes]

Sorry, I can't :G I'll try to come up with a minimal thing that reproduces it. BTW it was also running inside docker container (host is linux as well)

[godlygeek]

I have a guess here - can you run ulimit -a in the Docker container and paste the output? I've got a hunch you won't see

-f: file size (blocks)              unlimited

but instead something like

-f: file size (blocks)              32768

or something like that - which would mean you have some cap on the maximum file size you can create.

[godlygeek]

Perhaps an easier test, in fact: what happens if you run:

dd if=/dev/zero of=test.bin bs=1024 count=204800

If that creates a 200 MB file, you're able to create files larger than 128 MB and my theory is wrong.

If that creates a 128 MB file, something in your environment is preventing you from creating large files.

[cakir-enes]

Here are the outputs of the commands, It did create 200 mb file

root@fc6dbe20fa62:/app# dd if=/dev/zero of=test.bin bs=1024 count=204800
204800+0 records in
204800+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.393632 s, 533 MB/s
root@fc6dbe20fa62:/app# ls -alrth *.bin
-rw-r--r-- 1 root root 200M May  7 07:58 test.bin
root@fc6dbe20fa62:/app# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 506577
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 200000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 506577
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[godlygeek]

Oh, the other possibility - are the workers short lived? If your workers are legitimately getting restarted every N requests, this might be what you would see, as gunicorn keeps spawning new workers, which keep creating new Trackers, which keep being closed after writing 128 MB.

[cakir-enes]

But still last modified date would be changing, no? And harcoded name exists only for uwsgi workers. For celery workers their names generated randomly on with block.