-
Notifications
You must be signed in to change notification settings - Fork 790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
labels method lead to memory leak #937
Comments
If you run the garbage collector in your example, most of that allocated memory is reclaimed: import gc
import tracemalloc
from prometheus_client import Counter
counter_a = Counter('request_count_a', 'request count to a', ['name'])
tracemalloc.start()
for _ in range(10):
counter_a.labels('a').inc()
curr_mem, peak_mem = tracemalloc.get_traced_memory()
print(f"curr mem: {curr_mem / 10 ** 3} Kb, peak mem: {peak_mem / 10 ** 3} Kb")
print("Run the garbage collector")
gc.collect()
curr_mem, peak_mem = tracemalloc.get_traced_memory()
print(f"curr mem: {curr_mem / 10 ** 3} Kb, peak mem: {peak_mem / 10 ** 3} Kb") gives the following:
However perhaps your real code is different, could it be that you're creating labels that have an unbounded set of possible values? Prometheus client has to keep ahold of every label value it has ever seen, so if you keep passing in different label values (eg: user IDs), memory usage will continue to grow for each new label value. See the Prometheus label documentation: |
I've also found memory leaks, which on a few computers appear to be adding 3 gigabytes of memory immediately after being pulled by prometheus |
I find metrics in my celery task will lead to memory leak. And I reproduced this problem issue locally.
It seems to be caused by the
labels
method. PS:prometheus-client==0.17.1
The text was updated successfully, but these errors were encountered: