Memray in armv7 environments

[pazzarpj]

Hi all,

I'm trying to profile my python code on an ARMv7 system. I seem to hit different problems getting memray to run depending on how I build/run it.

The closest I have gotten is to docker buildx bake build for the python docker image that I use and then run from there.

Docker bake

target "memray37" {
  dockerfile = "docker/dockerfiles/memray-build.Dockerfile"
  contexts = {
    base = "docker-image://python:3.7.16-buster"
    pyawareb = "target:pyawareb37"
  }
  output = ["type=docker,dest=memray37.tar"]
  tags = ["pyaware-memray-37:latest"]
  platforms = ["linux/arm/v7"]
}

memray-build

FROM base
COPY --from=pyawareb dist ./dist
RUN apt-get update && apt-get -y install build-essential python3-dev libunwind-dev liblz4-dev
RUN pip3 wheel memray -w ./dist
RUN pip3 install pyaware memray --no-index --find-links ./dist
WORKDIR /opt/pyaware
CMD ["memray", "run", "-m", "pyaware", "/etc/aware"]

This sucessfully builds the memray.whl file for the architecture. And it seems to run on a Hello world python file.

However, when I run against the actual codebase, I get a memory error.

ampcontrol@Moxa:/usr/share/aware$ ./run_profile.sh
Writing profile results into memray-pyaware.1.bin
Memray WARNING: Correcting symbol for mmap from 0xb6b8bad1 to 0xb6b8ba81
[memray] Successfully generated profile results.

You can now generate reports from the stored allocation records.
Some example commands to generate reports:

/usr/local/bin/python -m memray flamegraph memray-pyaware.1.bin
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/site-packages/memray/__main__.py", line 6, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/memray/commands/__init__.py", line 138, in main
    arg_values.entrypoint(arg_values, parser)
  File "/usr/local/lib/python3.7/site-packages/memray/commands/run.py", line 339, in run
    _run_with_file_output(args)
  File "/usr/local/lib/python3.7/site-packages/memray/commands/run.py", line 191, in _run_with_file_output
    post_run_message=example_report_generation_message,
  File "/usr/local/lib/python3.7/site-packages/memray/commands/run.py", line 65, in _run_tracker
    runpy.run_module(args.script, run_name="__main__", alter_sys=True)
  File "/usr/local/lib/python3.7/runpy.py", line 201, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/local/lib/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/local/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.7/site-packages/pyaware/__init__.py", line 14, in <module>
    from pyaware.logger import runtime_logger
  File "/usr/local/lib/python3.7/site-packages/pyaware/logger.py", line 8, in <module>
    import pyaware.config
  File "/usr/local/lib/python3.7/site-packages/pyaware/config.py", line 18, in <module>
    import ifaddr
  File "/usr/local/lib/python3.7/site-packages/ifaddr/__init__.py", line 24, in <module>
    from ifaddr._shared import Adapter, IP
  File "/usr/local/lib/python3.7/site-packages/ifaddr/_shared.py", line 22, in <module>
    import ctypes
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 551, in <module>
    _reset_cache()
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
    CFUNCTYPE(c_int)(lambda: None)
MemoryError

Running the exact same image but with "python -m pyaware /etc/aware" instead of "python -m memray run -m pyaware /etc/aware"
I get the expected running logs.

ampcontrol@Moxa:/usr/share/aware$ ./run_profile_image_no_profile.sh
2024-05-12 00:48:09,780 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/events/events.py:455      Subscribed to mqtt_command_response:3062291752
2024-05-12 00:48:09,791 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/events/events.py:455      Subscribed to trigger_send:3062995104
2024-05-12 00:48:10,434 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/mqtt/client.py:490      Starting Gateway Heartbeat
2024-05-12 00:48:10,499 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/mqtt/client.py:554      Gateway state change detected. Sending new state.
2024-05-12 00:48:10,504 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/maintenance.py:324      Checking service setup
2024-05-12 00:48:10,510 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/maintenance.py:329      Detected service systemd
2024-05-12 00:48:10,557 MainThread      INFO     /usr/local/lib/python3.7/site-packages/transitions/core.py:131      Finished processing state new exit callbacks.
2024-05-12 00:48:10,562 MainThread      INFO     /usr/local/lib/python3.7/site-packages/transitions/core.py:125      Finished processing state connected enter callbacks.
2024-05-12 00:48:10,570 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:407      Waiting 2 seconds for configuration over mqtt
2024-05-12 00:48:12,576 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:414      Failed to receive remote configuration
2024-05-12 00:48:12,581 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:416      Proceeding to load local configuration
2024-05-12 00:48:12,990 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/maintenance.py:238      Version set to manual, skipping bootstrap
2024-05-12 00:48:13,019 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/maintenance.py:137      Skipping auto update as version is set to manual
2024-05-12 00:48:13,059 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/store/disk.py:56       Initialising database @ /etc/aware/cache.db
2024-05-12 00:48:13,137 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/store/disk.py:60       Initialised database @ /etc/aware/cache.db
2024-05-12 00:48:13,142 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:600      Initialising serial
2024-05-12 00:48:13,234 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:616      Initialised serial
2024-05-12 00:48:13,239 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:600      Initialising ethernet
2024-05-12 00:48:13,273 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:616      Initialised ethernet
2024-05-12 00:48:13,278 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/config.py:600      Initialising imac2_auto
2024-05-12 00:48:15,613 MainThread      INFO     /usr/local/lib/python3.7/site-packages/pyaware/mqtt/client.py:1257     Scheduling backlog to start in 1656s
2024-05-12 00:48:15,675 MainThread      INFO     /usr/local/lib/python3.7/site-packages/aiomodbus/tcp.py:60       Modbus Client connected at 10.1.1.10

Does anyone have any insight on how to get this working?

[godlygeek]

I don't think Memray has ever been tested on aarch32 or ARMv7. I'd suggest trying to run Memray's test suite, and see whether it passes. That will tell us whether this is a problem of Memray not working properly on that architecture, or of Memray not playing nicely with the package you're testing it with. From a clone of the repo, the test dependencies can be installed with pip3 install -r requirements-test.txt, and from there a python3 -m pytest tests/ will run the test suite.

[pazzarpj]

I guess I have my answer. Its untested/unsupported.

Using a new docker buildx build I've managed to run the tests and have some failures.
Although only 5 failures for an unsupported operating architecture isn't too bad.

1 is the sigint
4 seem to be related to subprocess/multiprocessing.
With this insight, I can probably mock out the things that call out to that as my core code doesn't need multi-processing, and it is the ifaddr library to determine the network interfaces, probably does a subprocess call to determine those interfaces.

Thank you very much. Will report back if I get it working on my codebase, also for anyone else wanting to run it unsupported.

Details

Dockerfile

FROM base
RUN apt-get update && apt-get -y install build-essential python3-dev libunwind-dev liblz4-dev
COPY requirements-test.txt requirements-extra.txt requirements-docs.txt /tmp/
RUN pip3 install -r /tmp/requirements-extra.txt -r /tmp/requirements-test.txt
COPY . .
RUN pip3 install -e .
RUN pip3 wheel memray -w ./dist
RUN python3 -m pytest tests/

[+] Building 2910.3s (13/13) FINISHED                                                                                                                                                                                                                                                 docker-container:frosty_blackwell
 => [internal] load local bake definitions                                                                                                                                                                                                                                                                         0.0s 
 => => reading docker-bake.hcl 219B / 219B                                                                                                                                                                                                                                                                         0.0s
 => [internal] load build definition from memray-build.Dockerfile                                                                                                                                                                                                                                                  0.0s
 => => transferring dockerfile: 403B                                                                                                                                                                                                                                                                               0.0s 
 => [context base] load metadata for python:3.7.16-buster                                                                                                                                                                                                                                                          1.5s 
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                    0.0s 
 => [context base] python:3.7.16-buster                                                                                                                                                                                                                                                                            0.3s 
 => => resolve docker.io/library/python:3.7.16-buster                                                                                                                                                                                                                                                              0.3s 
 => [internal] load build context                                                                                                                                                                                                                                                                                  0.0s 
 => => transferring context: 31.66kB                                                                                                                                                                                                                                                                               0.0s
 => CACHED [1/8] RUN apt-get update && apt-get -y install build-essential python3-dev libunwind-dev liblz4-dev                                                                                                                                                                                                     0.0s 
 => [2/8] COPY requirements-test.txt requirements-extra.txt requirements-docs.txt /tmp/                                                                                                                                                                                                                            0.0s 
 => [3/8] RUN pip3 install -r /tmp/requirements-extra.txt -r /tmp/requirements-test.txt                                                                                                                                                                                                                          466.4s 
 => [4/8] COPY . .                                                                                                                                                                                                                                                                                                 0.2s
 => [5/8] RUN pip3 install -e .                                                                                                                                                                                                                                                                                  709.1s
 => [6/8] RUN pip3 wheel memray -w ./dist                                                                                                                                                                                                                                                                        711.0s
 => ERROR [7/8] RUN python3 -m pytest tests/                                                                                                                                                                                                                                                                    1021.4s
------
 > [7/8] RUN python3 -m pytest tests/:
15.56 ============================= test session starts ==============================
15.56 platform linux -- Python 3.7.16, pytest-7.4.4, pluggy-1.2.0
15.56 rootdir: /
15.56 configfile: pyproject.toml
15.56 plugins: syrupy-3.0.6, cov-4.1.0, textual-snapshot-0.4.0
15.56 collected 606 items
15.56
15.56 tests/test_utils.py .......                                              [  1%]
15.63 tests/integration/test_api.py .......                                    [  2%]
15.92 tests/integration/test_attach.py ssssssss                                [  3%]
16.04 tests/integration/test_extensions.py .........                           [  5%]
41.78 tests/integration/test_greenlet.py ..                                    [  5%]
44.57 tests/integration/test_ipython.py ..........                             [  7%]
47.98 tests/integration/test_main.py ......................................... [ 13%]
194.7 ................................F....                                    [ 19%]
319.3 tests/integration/test_native_tracking.py ............                   [ 21%]
341.5 tests/integration/test_processes.py FFFF                                 [ 22%]
343.5 tests/integration/test_socket.py ...........                             [ 24%]
357.5 tests/integration/test_threads.py ..                                     [ 24%]
357.7 tests/integration/test_tracing.py ...........................            [ 29%]
360.6 tests/integration/test_tracking.py ..................................... [ 35%]
368.3 ........................................................................ [ 47%]
377.6 ...........                                                              [ 49%]
383.6 tests/unit/test_allocation_lifetime_aggregator.py ...................    [ 52%]
384.1 tests/unit/test_attach.py .                                              [ 52%]
384.3 tests/unit/test_cli.py ................................................. [ 60%]
390.5 ..........................                                               [ 64%]
391.7 tests/unit/test_flamegraph_reporter.py ................................. [ 70%]
395.7 ...                                                                      [ 70%]
395.8 tests/unit/test_frame_tools.py .........................                 [ 74%]
396.7 tests/unit/test_high_water_mark_aggregator.py .......................... [ 79%]
397.4 ......                                                                   [ 80%]
397.6 tests/unit/test_highwatermark_command.py ..............                  [ 82%]
400.7 tests/unit/test_reader.py ..s..                                          [ 83%]
401.3 tests/unit/test_stats_reporter.py ..........                             [ 84%]
401.7 tests/unit/test_summary_reporter.py .....                                [ 85%]
403.7 tests/unit/test_table_reporter.py .....                                  [ 86%]
403.8 tests/unit/test_templates.py ......                                      [ 87%]
404.1 tests/unit/test_tracker.py ..                                            [ 87%]
404.4 tests/unit/test_transform_reporter.py ...........                        [ 89%]
404.7 tests/unit/test_tree_reporter.py .............press 'u' (char='u')
432.6 .press 'i' (char='i')
445.1 press 'i' (char='i')
445.1 ..press 'i' (char='i')
460.6 .press 'i' (char='i')
473.1 press 'i' (char='i')
473.1 .press 'e' (char='e')
482.5 ....press 'down' (char=None)
529.6 press 'down' (char=None)
529.6 .press 'down' (char=None)
546.9 press 'down' (char=None)
546.9 press 'down' (char=None)
546.9 ..press 'down' (char=None)
579.2 press 'down' (char=None)
579.2 press 'down' (char=None)
579.2 press 'down' (char=None)
579.2 press 'e' (char='e')
579.2 .press 'i' (char='i')
592.0 .press 'u' (char='u')
604.9 .press 'u' (char='u')
619.8 press 'i' (char='i')
619.8 .press 'down' (char=None)
637.2 press 'down' (char=None)
637.2 press 'down' (char=None)
637.2 ..press 'end' (char=None)
671.5 .        [ 94%]
671.5 tests/unit/test_tui_reporter.py ..................press '' (char=None)
814.0 press 'o' (char='o')
814.0 press 'a' (char='a')
814.0 press 't' (char='t')
814.0 .press '' (char=None)
842.3 press 'greater_than_sign' (char='>')
842.3 press 'greater_than_sign' (char='>')
842.3 press 'greater_than_sign' (char='>')
842.3 press 'less_than_sign' (char='<')
842.3 press 'less_than_sign' (char='<')
842.3 press 'less_than_sign' (char='<')
842.3 ..press 'tab' (char='\t')
881.7 ...press 'space' (char=' ')
941.1 press 'tab' (char='\t')
941.1 .press 'space' (char=' ')
966.2 press 'tab' (char='\t')
966.2 .press 'space' (char=' ')
990.2 press 'tab' (char='\t')
990.2 .....          [100%]
1014.1
1014.1 =================================== FAILURES ===================================
1014.1 _ TestLiveRemoteSubcommand.test_live_client_exits_properly_on_sigint_before_connecting _
1014.1
1014.1 self = <tests.integration.test_main.TestLiveRemoteSubcommand object at 0x435e0950>
1014.1 free_port = 51533
1014.1
1014.1     def test_live_client_exits_properly_on_sigint_before_connecting(self, free_port):
1014.1         # GIVEN
1014.1         client = subprocess.Popen(
1014.1             [
1014.1                 sys.executable,
1014.1                 "-m",
1014.1                 "memray",
1014.1                 "live",
1014.1                 str(free_port),
1014.1             ],
1014.1             stderr=subprocess.PIPE,
1014.1             stdout=subprocess.PIPE,
1014.1             # Explicitly reset the signal handler for SIGINT to work around any signal
1014.1             # masking that might happen on Jenkins.
1014.1             preexec_fn=lambda: signal.signal(signal.SIGINT, signal.default_int_handler),
1014.1         )
1014.1
1014.1         # Ensure that it's waiting on the socket
1014.1         _wait_until_process_blocks(client.pid)
1014.1
1014.1         # WHEN
1014.1         client.send_signal(signal.SIGINT)
1014.1         try:
1014.1             client.wait(timeout=TIMEOUT)
1014.1         except subprocess.TimeoutExpired:
1014.1             client.terminate()
1014.1
1014.1         # THEN
1014.1 >       assert client.returncode == 0
1014.1 E       assert 1 == 0
1014.1 E        +  where 1 = <subprocess.Popen object at 0x46872a70>.returncode
1014.1
1014.1 tests/integration/test_main.py:1625: AssertionError
1014.1 ____________________ test_allocations_with_multiprocessing _____________________
1014.1
1014.1 tmpdir = local('/tmp/pytest-of-root/pytest-0/test_allocations_with_multipro0')
1014.1
1014.1     @pytest.mark.no_cover
1014.1     def test_allocations_with_multiprocessing(tmpdir):
1014.1         # GIVEN
1014.1         output = Path(tmpdir) / "test.bin"
1014.1         allocator = MemoryAllocator()
1014.1
1014.1         # WHEN
1014.1         with Tracker(output):
1014.1 >           with Pool(3) as p:
1014.1
1014.1 tests/integration/test_processes.py:47:
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:119: in Pool
1014.1     context=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:158: in __init__
1014.1     self._setup_queues()
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:251: in _setup_queues
1014.1     self._inqueue = self._ctx.SimpleQueue()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:112: in SimpleQueue
1014.1     return SimpleQueue(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/queues.py:332: in __init__
1014.1     self._rlock = ctx.Lock()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:67: in Lock
1014.1     return Lock(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:162: in __init__
1014.1     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1
1014.1 self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1
1014.1
1014.1     def __init__(self, kind, value, maxvalue, *, ctx):
1014.1         if ctx is None:
1014.1             ctx = context._default_context.get_context()
1014.1         name = ctx.get_start_method()
1014.1         unlink_now = sys.platform == 'win32' or name == 'fork'
1014.1         for i in range(100):
1014.1             try:
1014.1                 sl = self._semlock = _multiprocessing.SemLock(
1014.1                     kind, value, maxvalue, self._make_name(),
1014.1 >                   unlink_now)
1014.1 E                   OSError: [Errno 22] Invalid argument
1014.1
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:59: OSError
1014.1 _____________ test_allocations_with_multiprocessing_following_fork _____________
1014.1
1014.1 tmpdir = local('/tmp/pytest-of-root/pytest-0/test_allocations_with_multipro1')
1014.1
1014.1     @pytest.mark.no_cover
1014.1     def test_allocations_with_multiprocessing_following_fork(tmpdir):
1014.1         # GIVEN
1014.1         output = Path(tmpdir) / "test.bin"
1014.1         allocator = MemoryAllocator()
1014.1
1014.1         # WHEN
1014.1         with Tracker(output, follow_fork=True):
1014.1 >           with Pool(3) as p:
1014.1
1014.1 tests/integration/test_processes.py:85:
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:119: in Pool
1014.1     context=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:158: in __init__
1014.1     self._setup_queues()
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:251: in _setup_queues
1014.1     self._inqueue = self._ctx.SimpleQueue()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:112: in SimpleQueue
1014.1     return SimpleQueue(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/queues.py:332: in __init__
1014.1     self._rlock = ctx.Lock()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:67: in Lock
1014.1     return Lock(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:162: in __init__
1014.1     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1
1014.1 self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1
1014.1
1014.1     def __init__(self, kind, value, maxvalue, *, ctx):
1014.1         if ctx is None:
1014.1             ctx = context._default_context.get_context()
1014.1         name = ctx.get_start_method()
1014.1         unlink_now = sys.platform == 'win32' or name == 'fork'
1014.1         for i in range(100):
1014.1             try:
1014.1                 sl = self._semlock = _multiprocessing.SemLock(
1014.1                     kind, value, maxvalue, self._make_name(),
1014.1 >                   unlink_now)
1014.1 E                   OSError: [Errno 22] Invalid argument
1014.1
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:59: OSError
1014.1 _____________________ test_pymalloc_allocations_after_fork _____________________
1014.1
1014.1 tmpdir = local('/tmp/pytest-of-root/pytest-0/test_pymalloc_allocations_afte0')
1014.1
1014.1     @pytest.mark.no_cover
1014.1     def test_pymalloc_allocations_after_fork(tmpdir):
1014.1         # GIVEN
1014.1         output = Path(tmpdir) / "test.bin"
1014.1
1014.1         # WHEN
1014.1         with Tracker(output, follow_fork=True, trace_python_allocators=True):
1014.1 >           with Pool(3) as p:
1014.1
1014.1 tests/integration/test_processes.py:140:
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:119: in Pool
1014.1     context=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:158: in __init__
1014.1     self._setup_queues()
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:251: in _setup_queues
1014.1     self._inqueue = self._ctx.SimpleQueue()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:112: in SimpleQueue
1014.1     return SimpleQueue(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/queues.py:332: in __init__
1014.1     self._rlock = ctx.Lock()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:67: in Lock
1014.1     return Lock(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:162: in __init__
1014.1     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1
1014.1 self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1
1014.1
1014.1     def __init__(self, kind, value, maxvalue, *, ctx):
1014.1         if ctx is None:
1014.1             ctx = context._default_context.get_context()
1014.1         name = ctx.get_start_method()
1014.1         unlink_now = sys.platform == 'win32' or name == 'fork'
1014.1         for i in range(100):
1014.1             try:
1014.1                 sl = self._semlock = _multiprocessing.SemLock(
1014.1                     kind, value, maxvalue, self._make_name(),
1014.1 >                   unlink_now)
1014.1 E                   OSError: [Errno 22] Invalid argument
1014.1
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:59: OSError
1014.1 ________________________ test_stack_cleanup_after_fork _________________________
1014.1
1014.1 tmpdir = local('/tmp/pytest-of-root/pytest-0/test_stack_cleanup_after_fork0')
1014.1
1014.1     @pytest.mark.no_cover
1014.1     def test_stack_cleanup_after_fork(tmpdir):
1014.1         """Test that we don't crash miserably when we try to write pending Python
1014.1         frames when the profile function is deactivated if the tracker has been
1014.1         destroyed after a fork without `follow_fork=True`"""
1014.1
1014.1         # GIVEN
1014.1         output = Path(tmpdir) / "test.bin"
1014.1
1014.1         def foo():
1014.1             with Pool() as pool:
1014.1                 result = pool.map_async(sys.setprofile, [None])
1014.1                 return result.get(timeout=1)
1014.1
1014.1         # WHEN
1014.1
1014.1         with Tracker(output, follow_fork=False):
1014.1 >           result = foo()
1014.1
1014.1 tests/integration/test_processes.py:181:
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1 tests/integration/test_processes.py:174: in foo
1014.1     with Pool() as pool:
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:119: in Pool
1014.1     context=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:158: in __init__
1014.1     self._setup_queues()
1014.1 usr/local/lib/python3.7/multiprocessing/pool.py:251: in _setup_queues
1014.1     self._inqueue = self._ctx.SimpleQueue()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:112: in SimpleQueue
1014.1     return SimpleQueue(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/queues.py:332: in __init__
1014.1     self._rlock = ctx.Lock()
1014.1 usr/local/lib/python3.7/multiprocessing/context.py:67: in Lock
1014.1     return Lock(ctx=self.get_context())
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:162: in __init__
1014.1     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
1014.1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
1014.1
1014.1 self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1
1014.1
1014.1     def __init__(self, kind, value, maxvalue, *, ctx):
1014.1         if ctx is None:
1014.1             ctx = context._default_context.get_context()
1014.1         name = ctx.get_start_method()
1014.1         unlink_now = sys.platform == 'win32' or name == 'fork'
1014.1         for i in range(100):
1014.1             try:
1014.1                 sl = self._semlock = _multiprocessing.SemLock(
1014.1                     kind, value, maxvalue, self._make_name(),
1014.1 >                   unlink_now)
1014.1 E                   OSError: [Errno 22] Invalid argument
1014.1
1014.1 usr/local/lib/python3.7/multiprocessing/synchronize.py:59: OSError
1014.1 Textual Snapshot Report
1014.1
1014.1 18 mismatched snapshots
1014.1
1014.1 View the failure report.
1014.1
1014.2 /snapshot_report.html
1014.2
1017.6 --------------------------- snapshot report summary ----------------------------
1017.6 18 snapshots failed. 1 snapshot passed.
1017.6 =========================== short test summary info ============================
1017.6 FAILED tests/integration/test_main.py::TestLiveRemoteSubcommand::test_live_client_exits_properly_on_sigint_before_connecting
1017.6 FAILED tests/integration/test_processes.py::test_allocations_with_multiprocessing
1017.6 FAILED tests/integration/test_processes.py::test_allocations_with_multiprocessing_following_fork
1017.6 FAILED tests/integration/test_processes.py::test_pymalloc_allocations_after_fork
1017.6 FAILED tests/integration/test_processes.py::test_stack_cleanup_after_fork - O...
1017.6 ============ 5 failed, 592 passed, 9 skipped in 1014.42s (0:16:54) =============
------
memray-build.Dockerfile:8
--------------------
   6 |     RUN pip3 install -e .
   7 |     RUN pip3 wheel memray -w ./dist
   8 | >>> RUN python3 -m pytest tests/
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m pytest tests/" did not complete successfully: exit code: 1

[godlygeek]

Hm... It seems more likely to me that multiprocessing.Lock is broken in this environment than that Memray is somehow breaking multiprocessing.Lock and nothing else, but if we are, that would be very interesting to know, and worth tracking down...

Does python3 -c "import multiprocessing as mp; mp.Lock()" succeed?
Does python3 -m memray run -c "import multiprocessing as mp; mp.Lock()" fail?

If the first succeeds and the second fails, that would definitely suggest a bug of some sort in Memray...

[pazzarpj]

Yes it worked on mp.Lock() for no memray but failed on memray run

Details

 => [internal] load local bake definitions                                                                                                                                                                                                                                                                         0.0s 
 => => reading docker-bake.hcl 219B / 219B                                                                                                                                                                                                                                                                         0.0s
 => [internal] load build definition from memray-build.Dockerfile                                                                                                                                                                                                                                                  0.0s
 => => transferring dockerfile: 537B                                                                                                                                                                                                                                                                               0.0s 
 => [context base] load metadata for python:3.7.16-buster                                                                                                                                                                                                                                                          1.8s 
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                    0.0s 
 => [context base] python:3.7.16-buster                                                                                                                                                                                                                                                                            0.3s 
 => => resolve docker.io/library/python:3.7.16-buster                                                                                                                                                                                                                                                              0.3s 
 => [internal] load build context                                                                                                                                                                                                                                                                                  0.0s 
 => => transferring context: 32.42kB                                                                                                                                                                                                                                                                               0.0s
 => CACHED [1/8] RUN apt-get update && apt-get -y install build-essential python3-dev libunwind-dev liblz4-dev                                                                                                                                                                                                     0.0s
 => CACHED [2/8] COPY requirements-test.txt requirements-extra.txt requirements-docs.txt /tmp/                                                                                                                                                                                                                     0.0s 
 => CACHED [3/8] RUN pip3 install -r /tmp/requirements-extra.txt -r /tmp/requirements-test.txt                                                                                                                                                                                                                     0.0s 
 => [4/8] COPY . .                                                                                                                                                                                                                                                                                                 0.2s 
 => [5/8] RUN pip3 install -e .                                                                                                                                                                                                                                                                                  709.7s 
 => [6/8] RUN python3 -c "import multiprocessing as mp; mp.Lock()"                                                                                                                                                                                                                                                 0.7s 
 => ERROR [7/8] RUN python3 -m memray run -c "import multiprocessing as mp; mp.Lock()"                                                                                                                                                                                                                             2.5s 
------
 > [7/8] RUN python3 -m memray run -c "import multiprocessing as mp; mp.Lock()":
2.334 Memray WARNING: Correcting symbol for mmap from 0x40c0aad1 to 0x40c0aa81
2.452 [Errno 22] Invalid argument
2.453 Writing profile results into memray-string.8.bin
2.453 [memray] Successfully generated profile results.
2.453
2.453 You can now generate reports from the stored allocation records.
2.453 Some example commands to generate reports:
2.453
2.453 /usr/local/bin/python3 -m memray flamegraph memray-string.8.bin
------
memray-build.Dockerfile:8
--------------------
   6 |     RUN pip3 install -e .
   7 |     RUN python3 -c "import multiprocessing as mp; mp.Lock()"
   8 | >>> RUN python3 -m memray run -c "import multiprocessing as mp; mp.Lock()"
   9 |     #
  10 |     # RUN pip3 wheel memray -w ./dist
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m memray run -c \"import multiprocessing as mp; mp.Lock()\"" did not complete successfully: exit code: 1

[godlygeek]

Hm, ok. So... something about running under Memray is definitely breaking multiprocessing.Lock().

multiprocessing.Lock() winds up making a sem_open call under the hood, which is apparently failing with EINVAL...

OK, a few things to try. See if setting these two environment variables before building Memray changes anything:

ENV NO_MEMRAY_FAST_TLS=1 MEMRAY_MINIMIZE_INLINING=1
RUN pip3 install -e .

I'm guessing it won't, but it'd be nice if it did 😅. On the off chance that it does change the behavior, try each of those environment variables one at a time, and report which one fixes the issue. Otherwise, if things are still failing, I'd love it if you could share strace logs of both of those processes:

RUN apt-get -y install strace
RUN strace -o no_memray.out -s 1024 -f python3 -c "import multiprocessing as mp; mp.Lock()"
RUN strace -o with_memray.out -s 1024 -f python3 -m memray run -c "import multiprocessing as mp; mp.Lock()" || true

and then https://gist.github.com/ the two .out files for me. That should give us a peek under the hood to see what syscall is giving the EINVAL, and what arguments it was passed... (and whether they were in fact invalid, which may give us the hint we need about what's causing them to be invalid...)