mamba 2 solver performance regression

[jgukelberger]

Troubleshooting docs

• My problem is not solved in the Troubleshooting docs

Anaconda default channels

• I do NOT use the Anaconda default channels (pkgs/* etc.)

How did you install Mamba?

Mambaforge or latest Miniforge

Search tried in issue tracker

slow

Latest version of Mamba

• My problem is not solved with the latest version

Tried in Conda?

Not applicable

Describe your issue

Creating an environment with the following spec list is 20x slower with mamba 2.0.5 than with 1.5.11. Memory usage is 2x higher.

$ mamba --version
mamba 1.5.11
conda 24.11.0
$ /usr/bin/time mamba create -n tmptest --dry-run --override-channels --channel conda-forge 'python 3.11.*' adaptive adaptive-scheduler arviz astropy atomicwrites black 'blas * mkl' botorch cachetools cloudpickle conda-build 'dask >=2024' datashader deprecation fenics flake8 flaky 'freecad <0.21.2' gitpython gmsh gpytorch h5netcdf holoviews imagemagick ipympl ipyvtklink 'ipywidgets >=8.1' jupyter jupyterlab jupytext 'kwant * mpi*' mamba matplotlib memory_profiler meshio mkl mkl-service nb_conda_kernels nbconvert networkx nodejs notebook numba numpy numpy-indexed numpy-stl opencensus opencensus-ext-azure opt_einsum 'pandas <2.2' panel pfapack 'pip *' plotly portalocker pre-commit protobuf pycairo pydocstyle pydoe2 pymc pytables pytablewriter pytest pytest-cov pytest-datadir pytest-regressions pytest-xdist python-gmsh python-graphviz python-xxhash 'pytorch * *cpu*' pyvista rich rope ruamel.yaml scikit-learn scipy seaborn shapely
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
Transaction

[...]
  Summary:

  Install: 694 packages

  Total download: 242MB

────────────────────────────────────────────────────────────────────────────────────────────────────────────


Dry run. Exiting.

[...]

91.01user 7.55system 2:58.31elapsed 55%CPU (0avgtext+0avgdata 3577092maxresident)k
362544inputs+67032outputs (0major+783942minor)pagefaults 0swaps

$ mamba --version
2.0.5
$ /usr/bin/time mamba create -n tmptest --dry-run --override-channels --channel conda-forge 'python 3.11.*' adaptive adaptive-scheduler arviz astropy atomicwrites black 'blas * mkl' botorch cachetools cloudpickle conda-build 'dask >=2024' datashader deprecation fenics flake8 flaky 'freecad <0.21.2' gitpython gmsh gpytorch h5netcdf holoviews imagemagick ipympl ipyvtklink 'ipywidgets >=8.1' jupyter jupyterlab jupytext 'kwant * mpi*' mamba matplotlib memory_profiler meshio mkl mkl-service nb_conda_kernels nbconvert networkx nodejs notebook numba numpy numpy-indexed numpy-stl opencensus opencensus-ext-azure opt_einsum 'pandas <2.2' panel pfapack 'pip *' plotly portalocker pre-commit protobuf pycairo pydocstyle pydoe2 pymc pytables pytablewriter pytest pytest-cov pytest-datadir pytest-regressions pytest-xdist python-gmsh python-graphviz python-xxhash 'pytorch * *cpu*' pyvista rich rope ruamel.yaml scikit-learn scipy seaborn shapely

Transaction

[...]

  Summary:

  Install: 694 packages

  Total download: 254MB

────────────────────────────────────────────────────────────────────────────────────────────────────────────


Dry run. Not executing the transaction.
2982.72user 599.17system 1:01:51elapsed 96%CPU (0avgtext+0avgdata 7134392maxresident)k
826680inputs+1226616outputs (0major+10458732minor)pagefaults 0swaps

mamba info / micromamba info

/home/jan/opt/miniconda3/lib/python3.11/site-packages/conda/base/context.py:200: FutureWarning: Adding 'defaults' to channel list implicitly is deprecated and will be removed in 25.3.

To remove this warning, please choose a default channel explicitly with conda's regular configuration system, e.g. by adding 'defaults' to the list of channels:

  conda config --add channels defaults

For more information see https://docs.conda.io/projects/conda/en/stable/user-guide/configuration/use-condarc.html

  deprecated.topic(

          mamba version : 1.5.11
     active environment : base
    active env location : /home/jan/opt/miniconda3
            shell level : 5
       user config file : /home/jan/.condarc
 populated config files : /home/jan/.condarc
          conda version : 24.11.0
    conda-build version : 24.11.1
         python version : 3.11.10.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=x86_64_v4
                          __conda=24.11.0=0
                          __glibc=2.28=0
                          __linux=4.18.0=0
                          __unix=0=0
       base environment : /home/jan/opt/miniconda3  (writable)
      conda av data dir : /home/jan/opt/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/jan/opt/miniconda3/pkgs
                          /home/jan/.conda/pkgs
       envs directories : /home/jan/opt/miniconda3/envs
                          /home/jan/.conda/envs
               platform : linux-64
             user-agent : conda/24.11.0 requests/2.32.3 CPython/3.11.10 Linux/4.18.0-425.19.2.el8_7.x86_64 almalinux/8.7 glibc/2.28 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.11
                UID:GID : 528442005:528442005
             netrc file : None
           offline mode : False



       libmamba version : 2.0.5
          mamba version : 2.0.5
           curl version : libcurl/8.11.1 OpenSSL/3.4.0 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.1 nghttp2/1.64.0
     libarchive version : libarchive 3.7.7 zlib/1.3.1 liblzma/5.6.3 bz2lib/1.0.8 liblz4/1.10.0 libzstd/1.5.6
       envs directories : /home/jan/micromamba/envs
          package cache : /home/jan/micromamba/pkgs
                          /home/jan/.mamba/pkgs
            environment : /home/jan/opt/miniconda3/envs/mamba2 (active)
           env location : /home/jan/opt/miniconda3/envs/mamba2
      user config files : /home/jan/.mambarc
 populated config files : /home/jan/.condarc
       virtual packages : __unix=0=0
                          __linux=4.18.0=0
                          __glibc=2.28=0
                          __archspec=1=x86_64_v4
               channels :
       base environment : /home/jan/micromamba
               platform : linux-64

Logs

No response

environment.yml

No response

~/.condarc

channels: []

[jjerphan]

I am trying to reproduce the performance regression.

Currently I cannot reproduce the issue, profiling with samply:

• 1.5.11 (with conda 24.11.1) takes 3min22s to create the environment
• 2.0.4 and 2.0.5 takes 2min30s to create the environment

In particular, running the provided command for 2.0.5 gives:

Dry run. Not executing the transaction.
52.35user 2.28system 0:55.66elapsed 98%CPU (0avgtext+0avgdata 3471264maxresident)k
0inputs+384outputs (0major+937219minor)pagefaults 0swaps

Both takes around 2GiB or RAM to run.

The solutions are slightly different:

1c1
< name: 1.5
---
> name: 2.0
53a54
>   - boltons=24.0.0=pyhd8ed1ab_1
56c57
<   - botorch=0.8.5=pyhd8ed1ab_0
---
>   - botorch=0.11.3=pyhd8ed1ab_0
82,83c83,84
<   - conda=22.9.0=py311h38be061_2
<   - conda-build=3.25.0=py311h38be061_0
---
>   - conda=23.7.4=py311h38be061_0
>   - conda-build=24.11.2=py311h38be061_1
142a144
>   - frozendict=2.4.6=py311h9ecbd09_0
163d164
<   - glob2=0.7=py_0
172c173
<   - gpytorch=1.13=pyhd8ed1ab_0
---
>   - gpytorch=1.12=pyhd8ed1ab_0
219c220
<   - jaxtyping=0.2.19=pyhd8ed1ab_0
---
>   - jaxtyping=0.2.36=pyhd8ed1ab_0
228a230
>   - jsonpatch=1.33=pyhd8ed1ab_1
247c249
<   - kernel-headers_linux-64=2.6.32=he073ed8_17
---
>   - kernel-headers_linux-64=3.10.0=he073ed8_18
305c307
<   - liblief=0.12.3=h27087fc_0
---
>   - liblief=0.14.1=h5888daf_2
358c360
<   - linear_operator=0.5.3=pyhd8ed1ab_0
---
>   - linear_operator=0.5.2=pyhd8ed1ab_0
380a383
>   - menuinst=2.2.0=py311h38be061_0
495c498
<   - py-lief=0.12.3=py311ha362b79_0
---
>   - py-lief=0.14.1=py311hfdbb021_2
515,516c518,519
<   - pymc=5.16.1=hd8ed1ab_0
<   - pymc-base=5.16.1=pyhd8ed1ab_0
---
>   - pymc=5.18.0=hd8ed1ab_0
>   - pymc-base=5.18.0=pyhd8ed1ab_0
527,528c530,531
<   - pytensor=2.23.0=py311h4332511_0
<   - pytensor-base=2.23.0=py311hd037940_0
---
>   - pytensor=2.25.2=py311h7babd2d_0
>   - pytensor-base=2.25.2=py311h4a3439e_0
574c577
<   - ruamel.yaml=0.18.6=py311h9ecbd09_1
---
>   - ruamel.yaml=0.17.40=py311h459d7ec_0
576d578
<   - ruamel_yaml=0.15.80=py311h459d7ec_1009
581c583
<   - scipy=1.14.1=py311he9a78e4_2
---
>   - scipy=1.13.1=py311h517d4fd_0
612c614
<   - sysroot_linux-64=2.12=he073ed8_17
---
>   - sysroot_linux-64=2.17=h4a8ded7_18
631c633
<   - typeguard=4.4.1=pyhd8ed1ab_1
---
>   - typeguard=2.13.3=pyhd8ed1ab_0

[jgukelberger]

Thanks for looking into this! Interesting that you can't reproduce at all.

I just tried again on the same system (a VM running Alma Linux 8.7) and got similar results (1h 11min, 7.1 GB with mamba 2.0.5).

Then I tried on my laptop (WSL2 running Ubuntu 20.04.6 LTS). Here the difference is smaller but still significant:

mamba 1.5.11:

66.07user 2.88system 1:11.05elapsed 97%CPU (0avgtext+0avgdata 3722632maxresident)k
267808inputs+229712outputs (0major+150055minor)pagefaults 0swaps

mamba 2.0.5:

379.87user 4.93system 6:25.17elapsed 99%CPU (0avgtext+0avgdata 3500024maxresident)k
42512inputs+1228608outputs (270major+342214minor)pagefaults 0swaps

So, similar memory usage but still more than 5x slower.

Any ideas how to get to the bottom of this?

[jjerphan]

Could you provide a YAML environnement specification, all the pieces of configuration, the exact commands again?

Alternatively, could you provide runs of samply record on those commands?