Replies: 4 comments 19 replies
-
|
I suppose I could use this as my help discussion. The guilds is fine and expected. The issue is, the resident memory size of the python process steadily increases from between 1 and 2GB all the way up until it runs out of memory. So I suppose this is the stack memory if the heap is happily doing nothing out of the ordinary, do you know how I can debug and deal with this? |
Beta Was this translation helpful? Give feedback.
-
How much memory does it reach before it runs out of memory?
That seems unlikely to me. The Python interpreter really doesn't store anything large on the stack, and the call stack's depth is guaranteed to be fairly limited thanks to |
Beta Was this translation helpful? Give feedback.
-
It is ran in two separate (almost identical, sharding for a Discord bot) processes, this example was only about 3GB (worried it would spike up overnight so I wanted to get the file early). It has (from metrics) gone all the way to 6GB (when the other process was OOM killed).
This is what the grafana+prometheus metrics looks like for resident memory, it just steadily rises over several days, and this is what it looks like right now:
I'll take your advice of running with the native flag, hopefully it doesn't hurt performance so bad this time with pre-aggregated stats. Hopefully I'll see you in a few days with a bit of a happier story in #226 or at least some better debuggability ❤️ |
Beta Was this translation helpful? Give feedback.
-
|
Here is my summary, |
Beta Was this translation helpful? Give feedback.
-
|
That's telling us that, at the point where that allocation happened, there were no Python frames on the stack. It could be a C extension causing that: if an extension spawns its own thread using the native C/C++/Rust threading APIs, without going through the Python |
Beta Was this translation helpful? Give feedback.
-
|
I had some issues trying to run
FROM python:3.11-slim-bookworm
WORKDIR /bot
RUN apt-get update && apt-get install git netcat-openbsd iproute2 debuginfod python3.11-dbg -yqq && rm -rf /var/lib/apt/lists/*
ENV DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
ENV POETRY_VIRTUALENVS_CREATE="false"
ENV PYTHONPATH="/usr/local/lib/python311.zip:/usr/local/lib/python3.11:/usr/local/lib/python3.11/lib-dynload:/usr/local/lib/python3.11/site-packages"
RUN pip install poetry
COPY vibr/pyproject.toml vibr/poetry.lock ./
RUN poetry install --no-root --only main
COPY . .
RUN mkdir wow
CMD ["sh", "vibr/entrypoint.sh"]command: error: |
Beta Was this translation helpful? Give feedback.
-
|
That error doesn't look like it has anything to do with Memray... what happens if you run: instead? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sticking with me. I do have yet another issue though, upon exiting I have Managed to get the undefined symbol error to work with the following (janky) dockerfile: FROM python:3.11-slim-bookworm
WORKDIR /bot
RUN apt-get update && apt-get install git netcat-openbsd iproute2 debuginfod python3.11-dbg python3-pip -yqq && rm -rf /var/lib/apt/lists/*
ENV DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
ENV PYTHONPATH="/usr/lib/python3.11:/usr/lib/python3.11/lib-dynload:/usr/lib/python3.11/site-packages"
RUN python3.11-dbg -m pip install poetry --break-system-packages
COPY vibr/pyproject.toml vibr/poetry.lock ./
RUN poetry install --no-root --only main
COPY . .
RUN mkdir wow
CMD ["sh", "vibr/entrypoint.sh"]
|
Beta Was this translation helpful? Give feedback.
-
|
It doesn't seem like you should be touching |
Beta Was this translation helpful? Give feedback.
-
|
Ignore, reply solved this speculation. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think this will give you any more information. The flame graph and summary report are both showing you information about the point in time when the process was at its heap memory high water mark, so even large allocations which were later freed will show up. |
Beta Was this translation helpful? Give feedback.
-
|
@godlygeek I return with a couple of 220M native+aggregated files made via alpine and python3.11 debug.
Traceback (most recent call last):
File "/usr/bin/memray", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/lib/python3.11/site-packages/memray/commands/__init__.py", line 137, in main
arg_values.entrypoint(arg_values, parser)
File "/usr/lib/python3.11/site-packages/memray/commands/common.py", line 264, in run
self.write_report(
File "/usr/lib/python3.11/site-packages/memray/commands/common.py", line 171, in write_report
reporter = self.reporter_factory(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/memray/reporters/table.py", line 37, in from_snapshot
list(record.hybrid_stack_trace(max_stacks=1))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/memray/_memray.pyx", line 338, in memray._memray.AllocationRecord.hybrid_stack_trace
File "src/memray/_memray.pyx", line 245, in memray._memray.hybrid_stack_trace
AssertionErrorDockerfileFROM alpine
WORKDIR /bot
RUN apk add --no-cache --update git netcat-openbsd iproute2 python3 python3-dbg py3-pip python3-dev gcc
# ENV DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
RUN python3 -m pip install poetry --break-system-packages
# COPY vibr/pyproject.toml vibr/poetry.lock ./
COPY vibr/requirements.txt ./
RUN pip install -r requirements.txt
COPY . .
RUN mkdir wow
CMD ["sh", "vibr/entrypoint.sh"]entrypoint.sh# unrelated migrations + network waiting
python3 -m memray run --aggregate --native -o wow/memray.$((1 + $RANDOM % 999)).bin -m vibrrequirements.txtaiodns==3.0.0 ; python_version >= "3.10" and python_version < "4.0"
aiohttp==3.8.4 ; python_version >= "3.10" and python_version < "4.0"
aiohttp[speedups]==3.8.4 ; python_version >= "3.10" and python_version < "4.0"
aiosignal==1.3.1 ; python_version >= "3.10" and python_version < "4.0"
async-spotify @ git+https://github.com/ooliver1/AsyncSpotify@dc09920#subdirectory=src ; python_version >= "3.10" and python_version < "4.0"
async-timeout==4.0.2 ; python_version >= "3.10" and python_version < "4.0"
asyncache==0.3.1 ; python_version >= "3.10" and python_version < "4.0"
asyncpg==0.27.0 ; python_version >= "3.10" and python_version < "4.0"
attrs==23.1.0 ; python_version >= "3.10" and python_version < "4.0"
black==23.3.0 ; python_version >= "3.10" and python_version < "4.0"
brotli==1.0.9 ; python_version >= "3.10" and python_version < "4.0"
cachetools==5.3.0 ; python_version >= "3.10" and python_version < "4.0"
certifi==2023.5.7 ; python_version >= "3.10" and python_version < "4.0"
cffi==1.15.1 ; python_version >= "3.10" and python_version < "4.0"
charset-normalizer==3.1.0 ; python_version >= "3.10" and python_version < "4.0"
click==8.1.3 ; python_version >= "3.10" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0"
delarva @ git+https://github.com/ooliver1/delarva@0391e39 ; python_version >= "3.10" and python_version < "4.0"
dnspython==2.3.0 ; python_version >= "3.10" and python_version < "4.0"
docker==6.1.2 ; python_version >= "3.10" and python_version < "4.0"
docstring-parser==0.15 ; python_version >= "3.10" and python_version < "4.0"
email-validator==2.0.0.post2 ; python_version >= "3.10" and python_version < "4.0"
frozenlist==1.3.3 ; python_version >= "3.10" and python_version < "4.0"
hiredis==2.2.3 ; python_version >= "3.10" and python_version < "4.0"
humanfriendly==10.0 ; python_version >= "3.10" and python_version < "4.0"
idna==3.4 ; python_version >= "3.10" and python_version < "4.0"
inflection==0.5.1 ; python_version >= "3.10" and python_version < "4.0"
jinja2==3.1.2 ; python_version >= "3.10" and python_version < "4.0"
mafic @ git+https://github.com/ooliver1/mafic@master ; python_version >= "3.10" and python_version < "4.0"
markdown-it-py==3.0.0 ; python_version >= "3.10" and python_version < "4.0"
markupsafe==2.1.2 ; python_version >= "3.10" and python_version < "4.0"
mdurl==0.1.2 ; python_version >= "3.10" and python_version < "4.0"
memray==1.9.1 ; python_version >= "3.10" and python_version < "4.0"
mslex==0.3.0 ; python_version >= "3.10" and python_version < "4.0" and sys_platform == "win32"
multidict==6.0.4 ; python_version >= "3.10" and python_version < "4.0"
mypy-extensions==1.0.0 ; python_version >= "3.10" and python_version < "4.0"
nextcord @ git+https://github.com/boting-devs/nextcord@vibr ; python_version >= "3.10" and python_version < "4.0"
nextcord-ext-menus==1.5.6 ; python_version >= "3.10" and python_version < "4.0"
nextcord[speed] @ git+https://github.com/boting-devs/nextcord@vibr ; python_version >= "3.10" and python_version < "4.0"
ooliver-botbase[db] @ git+https://github.com/ooliver1/botbase@no-psutil ; python_version >= "3.10" and python_version < "4.0"
orjson==3.8.14 ; python_version >= "3.10" and python_version < "4.0"
packaging==23.1 ; python_version >= "3.10" and python_version < "4.0"
pathspec==0.11.1 ; python_version >= "3.10" and python_version < "4.0"
piccolo[orjson,postgres,uvloop]==0.111.1 ; python_version >= "3.10" and python_version < "4.0"
platformdirs==3.5.1 ; python_version >= "3.10" and python_version < "4.0"
prometheus-async==22.2.0 ; python_version >= "3.10" and python_version < "4.0"
prometheus-client==0.17.0 ; python_version >= "3.10" and python_version < "4.0"
pycares==4.3.0 ; python_version >= "3.10" and python_version < "4.0"
pycparser==2.21 ; python_version >= "3.10" and python_version < "4.0"
pydantic[email]==1.10.8 ; python_version >= "3.10" and python_version < "4.0"
pygments==2.16.1 ; python_version >= "3.10" and python_version < "4.0"
pyreadline3==3.4.1 ; sys_platform == "win32" and python_version >= "3.10" and python_version < "4.0"
python-dotenv==1.0.0 ; python_version >= "3.10" and python_version < "4.0"
pywin32==306 ; python_version >= "3.10" and python_version < "4.0" and sys_platform == "win32"
pyyaml==6.0.1 ; python_version >= "3.10" and python_version < "4.0"
redis[hiredis]==4.5.5 ; python_version >= "3.10" and python_version < "4.0"
requests==2.31.0 ; python_version >= "3.10" and python_version < "4.0"
rich==13.5.2 ; python_version >= "3.10" and python_version < "4.0"
targ==0.3.8 ; python_version >= "3.10" and python_version < "4.0"
tomli==2.0.1 ; python_version >= "3.10" and python_version < "4.0"
typing-extensions==4.6.1 ; python_version >= "3.10" and python_version < "4.0"
urllib3==2.0.2 ; python_version >= "3.10" and python_version < "4.0"
uvloop==0.17.0 ; python_version >= "3.10" and python_version < "4.0"
websocket-client==1.5.2 ; python_version >= "3.10" and python_version < "4.0"
wrapt==1.15.0 ; python_version >= "3.10" and python_version < "4.0"
yarl==1.9.2 ; python_version >= "3.10" and python_version < "4.0"memray.zip |
Beta Was this translation helpful? Give feedback.
-
|
@pablogsal I appareciate the in-depth replies. I've been running it in the same docker container as before, with a few extra libraries installed to get memray to compile. I have lots of ignoring record logs, some missing shared objects (using alpine as the development build clashes with debian), and the flamegraph only shows the expected memory usage (using both
How I've been running this container locally is building it and assigning the local path to a volume within docker, and simply running |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if I understand correctly the conclusion to your last comment but here are some observations:
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Instead of sending the project, try to trim your program as much as possible while showing the same problem and then try to produce a minimal self contained reproducer. Even if you send us the entire project we are unlikely to be able to help if is too big or requires a lot of work to reproduce. You don't need to produce the leak, just something that crashes memray when checking the results with that assertion. I know it sounds complicated but unfortunately we won't be able to do a lot without something like that we can investigate in a reasonable amount of time. Maybe you can start running memray with a request or two and checking if that works. |
Beta Was this translation helpful? Give feedback.
-
|
I can't even get it to happen on my machine when running locally, it is only something that happens with a ton of users using it constantly, so I suppose I'm stuck for now. Thank you for helping though! It may happen if I run it for a few hours, but I can't really do that right now. Update: I have a couple of ideas for it and have managed to reproduce it locally with enough usage, thankfully I can finally start to strip away at the project, and I will hopefully be back soon. |
Beta Was this translation helpful? Give feedback.
-
This initial post is outdated, this is the current comment thread.
I have a 91M memray bin file i converted into html using flamegraph, which itself is 605M in size. How do you think I should process this?
Table is basically the same. I would like to know if there is a way to see if the browser is even attempting to process this data or not, since I see it is all in a script.
Beta Was this translation helpful? Give feedback.
All reactions