-
Notifications
You must be signed in to change notification settings - Fork 575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
memory leak present even in trivial tests #3975
Comments
I think this is probably the conjecture data cache in There is a memory/performance trade-off here, and the benefits of this cache are mainly in the shrinking phase. |
FWIW: Changing to |
I wanted to see when memory consumption flattens out so I've modified stateful_minimal.py.gz with
Occasionally this fails with:
Sometimes it does finish, after about 5 minutes:
At this point it ate 440 MB of memory. Here's memory consumption over time, with 10 second sampling interval:
(Generated with trivial It is possible that it caches all 69508 examples? Or why it should be still consuming more memory? Please note I'm not saying this is some huge problem - at least until we get to hypofuzzing state machines (which is the original motivation for my experiment). |
So, I ran the test in memray to get better measurements, and in summary I don't see strong evidence of leaks. What I think we're seeing is that memory consumption has two main drivers:
As evidence (running the stateful test above, about 70k examples): Possible actions:
|
I think either of you has a better intuition for memory size than I. I would say it is very possible memory usage has increased recently due to ir-related churn and we should do a detailed pass one things settle. In particular 1kb/example for |
That's larger than I was expecting, but makes sense once I actually think about the data structures and objects involved. I agree that it's plausible the IR migration is causing temporarily higher memory usage, both because we're carrying around two representations now, and because the IR hasn't really been memory-optimized yet. Not worth trying to fix that before we finish migrating though. Adding |
Good idea to compare with pre-ir! You'll notice immediately that we use a lot less memory in the post-ir world: the (probable) Less obviously, we produce more reference cycles, so the GC works harder. It's worth keeping an eye on this during development/migration, since data structures are harder than code to "fix" afterwards . |
Yep, agree on this (with both of you). We can revisit afterwards - for the record, if nothing else.
Or alternatively record and account for the pauses using |
Oh, that's a better idea, we can attribute gc time correctly in statistics and observability output too. |
Versions tested
Summary
For some reason even trivial stateless test and stateful test cases leak memory at rate of several megabytes per second.
Reproducers
Here are two test-nothing test cases:
test_given.py.gz - leaks on average ~ 2 MB/s, in the second half just 1 MB/s. On my machine the memory consumption goes form 25 MB to 145 MB at the end.
stateful_minimal.py.gz - leaks on average ~ 7 MB/s, in the second half of runtime still 7 MB/s . On my machine the memory consumption goes form 25 MB to 170 MB at the end.
Both files can be executed with
python -m unittest
so the pytest is out of equation.I've attempted memory profiling using Scalene profiler version 1.5.41 but I did not get anything conclusive. At first it pointed to
random.getstate()
, so I've butcheredinternal/entropy.py:deterministic_PRNG()
andcontrol.py:deprecate_random_in_strategy()
by commenting out all calls torandom.getstate()
but the overall memory consumption did not move. Scalene just started to claim that memory went to different places.I don't trust Scalene's memory attribution very much, but there certainly is some sort of leak because running tests in memory-constrained environment invokes OOM killer:
Simplest way to verify that more memory is being consumed, and not just "virtually", is adjusting
settings(max_examples=100)
vs.settings(max_examples=5000)
. In BASH limiting memory withulimit -d 50000
allows small test cases to finish but larger examples explode.I don't understand subtleties of
ulimit
but similar effect can be achieved usingsystemd-run --user --shell -p MemoryMax=50M -p MemoryHigh=45M -p MemorySwapMax=0B
command if you are on a system with Systemd. In that case OOM killer does not allow larger test cases to finish.The text was updated successfully, but these errors were encountered: