Installing packages in different order gives totally different versions

[avouacr]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Hi, I'm a bit puzzled by this problem, any help would be appreciated. I installed mamba through mambaforge and use it to build Python virtual environments.

If I do the following :

mamba create -n test s3fs boto3 python=3.11

Mamba ends up installing 48 packages with the following versions :

  + _libgcc_mutex           0.1  conda_forge         conda-forge     Cached
  + python_abi             3.11  4_cp311             conda-forge     Cached
  + ld_impl_linux-64       2.40  h41732ed_0          conda-forge     Cached
  + ca-certificates    2024.2.2  hbcca054_0          conda-forge     Cached
  + libstdcxx-ng         13.2.0  h7e041cc_5          conda-forge     Cached
  + libgomp              13.2.0  h807b86a_5          conda-forge     Cached
  + _openmp_mutex           4.5  2_gnu               conda-forge     Cached
  + libgcc-ng            13.2.0  h807b86a_5          conda-forge     Cached
  + libxcrypt            4.4.36  hd590300_1          conda-forge     Cached
  + libexpat              2.6.1  h59595ed_0          conda-forge     Cached
  + libzlib              1.2.13  hd590300_5          conda-forge     Cached
  + libffi                3.4.2  h7f98852_5          conda-forge     Cached
  + bzip2                 1.0.8  hd590300_5          conda-forge     Cached
  + ncurses                 6.4  h59595ed_2          conda-forge     Cached
  + openssl               3.2.1  hd590300_0          conda-forge     Cached
  + libuuid              2.38.1  h0b41bf4_0          conda-forge     Cached
  + libnsl                2.0.1  hd590300_0          conda-forge     Cached
  + xz                    5.2.6  h166bdaf_0          conda-forge     Cached
  + tk                   8.6.13  noxft_h4845f30_101  conda-forge     Cached
  + libsqlite            3.45.1  h2797004_0          conda-forge     Cached
  + readline                8.2  h8228510_1          conda-forge     Cached
  + tzdata                2024a  h0c530f3_0          conda-forge     Cached
  + python               3.11.8  hab00c5b_0_cpython  conda-forge     Cached
  + wheel                0.42.0  pyhd8ed1ab_0        conda-forge     Cached
  + setuptools           69.1.1  pyhd8ed1ab_0        conda-forge     Cached
  + pip                    24.0  pyhd8ed1ab_0        conda-forge     Cached
  + six                  1.16.0  pyh6c4a22f_0        conda-forge     Cached
  + idna                    3.6  pyhd8ed1ab_0        conda-forge     Cached
  + pysocks               1.7.1  pyha2e5f31_6        conda-forge     Cached
  + jmespath              1.0.1  pyhd8ed1ab_0        conda-forge     Cached
  + attrs                23.2.0  pyh71513ae_0        conda-forge     Cached
  + typing_extensions    4.10.0  pyha770c72_0        conda-forge     Cached
  + fsspec             2024.2.0  pyhca7485f_0        conda-forge     Cached
  + python-dateutil       2.9.0  pyhd8ed1ab_0        conda-forge     Cached
  + aioitertools         0.11.0  pyhd8ed1ab_0        conda-forge     Cached
  + brotli-python         1.1.0  py311hb755f60_1     conda-forge     Cached
  + multidict             6.0.5  py311h459d7ec_0     conda-forge     Cached
  + frozenlist            1.4.1  py311h459d7ec_0     conda-forge     Cached
  + wrapt                1.16.0  py311h459d7ec_0     conda-forge     Cached
  + yarl                  1.9.4  py311h459d7ec_0     conda-forge     Cached
  + urllib3               2.0.7  pyhd8ed1ab_0        conda-forge     Cached
  + aiosignal             1.3.1  pyhd8ed1ab_0        conda-forge     Cached
  + botocore            1.34.51  pyge310_1234567_0   conda-forge     Cached
  + s3transfer           0.10.0  pyhd8ed1ab_0        conda-forge     Cached
  + boto3               1.34.51  pyhd8ed1ab_0        conda-forge     Cached
  + aiohttp               3.9.3  py311h459d7ec_0     conda-forge     Cached
  + aiobotocore          2.12.1  pyhd8ed1ab_0        conda-forge     Cached
  + s3fs               2024.2.0  pyhd8ed1ab_0        conda-forge     Cached

Now if I just change the order of the packages when installing from :

mamba create -n test s3fs boto3 python=3.11

to :

mamba create -n test boto3 python=3.11 s3fs 

mamba installs only 37 packages and I get the following versions specs :

  + _libgcc_mutex          0.1  conda_forge         conda-forge     Cached
  + ld_impl_linux-64      2.40  h41732ed_0          conda-forge     Cached
  + ca-certificates   2024.2.2  hbcca054_0          conda-forge     Cached
  + libstdcxx-ng        13.2.0  h7e041cc_5          conda-forge     Cached
  + python_abi            3.11  4_cp311             conda-forge     Cached
  + libgomp             13.2.0  h807b86a_5          conda-forge     Cached
  + _openmp_mutex          4.5  2_gnu               conda-forge     Cached
  + libgcc-ng           13.2.0  h807b86a_5          conda-forge     Cached
  + libxcrypt           4.4.36  hd590300_1          conda-forge     Cached
  + libexpat             2.6.1  h59595ed_0          conda-forge     Cached
  + libzlib             1.2.13  hd590300_5          conda-forge     Cached
  + libffi               3.4.2  h7f98852_5          conda-forge     Cached
  + bzip2                1.0.8  hd590300_5          conda-forge     Cached
  + ncurses                6.4  h59595ed_2          conda-forge     Cached
  + openssl              3.2.1  hd590300_0          conda-forge     Cached
  + libuuid             2.38.1  h0b41bf4_0          conda-forge     Cached
  + libnsl               2.0.1  hd590300_0          conda-forge     Cached
  + xz                   5.2.6  h166bdaf_0          conda-forge     Cached
  + tk                  8.6.13  noxft_h4845f30_101  conda-forge     Cached
  + libsqlite           3.45.1  h2797004_0          conda-forge     Cached
  + readline               8.2  h8228510_1          conda-forge     Cached
  + tzdata               2024a  h0c530f3_0          conda-forge     Cached
  + python              3.11.8  hab00c5b_0_cpython  conda-forge     Cached
  + wheel               0.42.0  pyhd8ed1ab_0        conda-forge     Cached
  + setuptools          69.1.1  pyhd8ed1ab_0        conda-forge     Cached
  + pip                   24.0  pyhd8ed1ab_0        conda-forge     Cached
  + six                 1.16.0  pyh6c4a22f_0        conda-forge     Cached
  + pysocks              1.7.1  pyha2e5f31_6        conda-forge     Cached
  + fsspec            2024.2.0  pyhca7485f_0        conda-forge     Cached
  + jmespath             1.0.1  pyhd8ed1ab_0        conda-forge     Cached
  + python-dateutil      2.9.0  pyhd8ed1ab_0        conda-forge     Cached
  + brotli-python        1.1.0  py311hb755f60_1     conda-forge     Cached
  + urllib3              2.0.7  pyhd8ed1ab_0        conda-forge     Cached
  + botocore           1.34.60  pyge310_1234567_0   conda-forge     Cached
  + s3transfer          0.10.0  pyhd8ed1ab_0        conda-forge     Cached
  + boto3              1.34.60  pyhd8ed1ab_0        conda-forge     Cached
  + s3fs                 0.4.2  py_0                conda-forge     Cached

Now what I don't understand is how such a small change of order can have such a dramatic impact on what gets installed in the end. In particular, the version of s3fs installed in the second configuration is almost 4 years old, whereas it's the latest one in the first configuration.

There seems to be somewhat a conflict of dependencies between s3fs et boto3 because if I create the environment in the second configuration and then runs mamba install s3fs=2024.2.0 in it, I get the following changes :

  Upgrade:
────────────────────────────────────────────────────────────────────────────

  - s3fs                  0.4.2  py_0               conda-forge     Cached
  + s3fs               2024.2.0  pyhd8ed1ab_0       conda-forge     Cached

  Downgrade:
────────────────────────────────────────────────────────────────────────────

  - botocore            1.34.60  pyge310_1234567_0  conda-forge     Cached
  + botocore            1.34.51  pyge310_1234567_0  conda-forge     Cached
  - boto3               1.34.60  pyhd8ed1ab_0       conda-forge     Cached
  + boto3               1.34.51  pyhd8ed1ab_0       conda-forge     Cached

But in the optimization process that mamba does, I don't understand how a huge major downgrading of s3fs can be prioritized over downgrading boto3 from a few patches, and especially why the result differs so much even when they are simultaneously installed.

Thank's !

Conda Info

active environment : test8
    active env location : /opt/mamba/envs/test8
            shell level : 9
       user config file : /home/onyxia/.condarc
 populated config files : /opt/mamba/.condarc
          conda version : 23.11.0
    conda-build version : not installed
         python version : 3.11.6.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake_avx512
                          __conda=23.11.0=0
                          __glibc=2.35=0
                          __linux=6.1.0=0
                          __unix=0=0
       base environment : /opt/mamba  (writable)
      conda av data dir : /opt/mamba/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/mamba/pkgs
                          /home/onyxia/.conda/pkgs
       envs directories : /opt/mamba/envs
                          /home/onyxia/.conda/envs
               platform : linux-64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.6 Linux/6.1.0-16-amd64 ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.5
                UID:GID : 1000:100
             netrc file : None
           offline mode : False

Conda Config

==> /opt/mamba/.condarc <==
channels:
  - conda-forge

Conda list

No response

Additional Context

No response

[jaimergp]

Hi! This is the issue tracker for conda-libmamba-solver, a conda plugin that happens to use libmamba behind the scenes. mamba also relies on the same libmamba library, but it's not developed here. You can voice your concerns at https://github.com/mamba-org/mamba/issues

There's a related issue there at mamba-org/mamba#3058.

If you use conda these days, you get the same solver as when using mamba, so performance should be comparable. conda (via conda-libmamba-solver) does sort the input specs before delegating to libmamba, so you'll get consistent results.

Can you check the output of conda create -n test s3fs boto3 python=3.11 and conda create -n test boto3 python=3.11 s3fs?

[avouacr]

Hi @jaimergp and thank you for your answer. Duly noted for the relevant repository, I'll open the issue there.

Regarding your question, the outcome is indeed different when I use conda to create my environment : it is consistent across orderings in the sense that the same number of packages and versions are installed. However, in both configurations, it installs the version 0.4.2 of s3fs, prioritizing getting the latest of boto3 while accepting a very outdated version for s3fs.

I wonder if it could be related to the fact that s3fs changed their versioning system at some point on conda-forge (see here), going from usual semantic versioning to date-based one. But still the major of the subsequent versions are (way) higher than the previous ones, so I don't see why this should produce this outcome.

[jaimergp]

They probably left a lower bound open at some point, and a too restrictive one in recent versions, so the solver "finds its way" easier in the old builds.

conda-libmamba-solver will prioritize the most constrained specs first, then alphabetically. So if you have python numpy=1.20, the solver will prioritize numpy=1.20 first, then python. My recommendation for now is to use a constraint on s3fs (e.g. s3fs >=2020) so you actually get something recent.

The long term fix is to submit a repodata patch adding the required constraints so "that door" to old builds is no longer open, but findind out which restraints need to be added is tricky.

[avouacr]

Understood, thank you for the explanation. For now, I'll use a loose constraint as you suggest.