PackagesNotFoundError listing more than missing package

[tl-hbk]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

I'm unsure if this is a bug in conda-libmamba-solver or libmamba but in recent versions when trying to solve for a package missing in one channel it reports all requested packages as missing even if they're not.

I'm able to reproduce it with the following:

channels:
  - conda-forge
  - defaults
dependencies:
  - defaults::_libgcc_mutex=0.1=main
  - defaults::_openmp_mutex=5.1=1_gnu
  - defaults::bzip2=1.0.8=h5eee18b_5
  - defaults::ca-certificates=2023.12.12=h06a4308_0
  - defaults::ld_impl_linux-64=2.38=h1181459_1
  - defaults::libffi=3.4.4=h6a678d5_0
  - defaults::libgcc-ng=11.2.0=h1234567_1
  - defaults::libgomp=11.2.0=h1234567_1
  - defaults::libstdcxx-ng=11.2.0=h1234567_1
  - defaults::libuuid=1.41.5=h5eee18b_0
  - defaults::ncurses=6.4=h6a678d5_0
  - defaults::openssl=3.0.13=h7f8727e_0
  - defaults::pip=23.3.1=py311h06a4308_0
  - defaults::python=3.11.8=h955ad1f_0
  - defaults::readline=8.2=h5eee18b_0
  - defaults::setuptools=68.2.2=py311h06a4308_0
  - defaults::sqlite=3.41.2=h5eee18b_0
  - defaults::tk=8.6.12=h1ccaba5_0
  - defaults::tzdata=2024a=h04d1e81_0
  - defaults::wheel=0.41.2=py311h06a4308_0
  - defaults::xz=5.4.6=h5eee18b_0
  - defaults::zlib=1.2.13=h5eee18b_0

Given this valid environment in ok.yml.

conda env create --name ok --file ok.yml --dry-run succeeds as expected

After copying the ok.yml file to broken.yml and changing one of these pins to conda-forge

channels:
  - conda-forge
  - defaults
dependencies:
  - defaults::_libgcc_mutex=0.1=main
  - defaults::_openmp_mutex=5.1=1_gnu
  - defaults::bzip2=1.0.8=h5eee18b_5
  - defaults::ca-certificates=2023.12.12=h06a4308_0
  - defaults::ld_impl_linux-64=2.38=h1181459_1
  - defaults::libffi=3.4.4=h6a678d5_0
  - defaults::libgcc-ng=11.2.0=h1234567_1
  - defaults::libgomp=11.2.0=h1234567_1
  - defaults::libstdcxx-ng=11.2.0=h1234567_1
  - defaults::libuuid=1.41.5=h5eee18b_0
  - defaults::ncurses=6.4=h6a678d5_0
  - conda-forge::openssl=3.0.13=h7f8727e_0  # Should not exist
  - defaults::pip=23.3.1=py311h06a4308_0
  - defaults::python=3.11.8=h955ad1f_0
  - defaults::readline=8.2=h5eee18b_0
  - defaults::setuptools=68.2.2=py311h06a4308_0
  - defaults::sqlite=3.41.2=h5eee18b_0
  - defaults::tk=8.6.12=h1ccaba5_0
  - defaults::tzdata=2024a=h04d1e81_0
  - defaults::wheel=0.41.2=py311h06a4308_0
  - defaults::xz=5.4.6=h5eee18b_0
  - defaults::zlib=1.2.13=h5eee18b_0

conda env create --name broken --file broken.yml --dry-run fails with the following error message

Solving environment: ...working... failed

PackagesNotFoundError: The following packages are not available from current channels:

  - pkgs/main::_libgcc_mutex==0.1=main
  - pkgs/main::_openmp_mutex==5.1=1_gnu
  - pkgs/main::bzip2==1.0.8=h5eee18b_5
  - pkgs/main::ca-certificates==2023.12.12=h06a4308_0
  - pkgs/main::ld_impl_linux-64==2.38=h1181459_1
  - pkgs/main::libffi==3.4.4=h6a678d5_0
  - pkgs/main::libgcc-ng==11.2.0=h1234567_1
  - pkgs/main::libgomp==11.2.0=h1234567_1
  - pkgs/main::libstdcxx-ng==11.2.0=h1234567_1
  - pkgs/main::libuuid==1.41.5=h5eee18b_0
  - pkgs/main::ncurses==6.4=h6a678d5_0
  - conda-forge::openssl==3.0.13=h7f8727e_0
  - pkgs/main::pip==23.3.1=py311h06a4308_0
  - pkgs/main::python==3.11.8=h955ad1f_0
  - pkgs/main::readline==8.2=h5eee18b_0
  - pkgs/main::setuptools==68.2.2=py311h06a4308_0
  - pkgs/main::sqlite==3.41.2=h5eee18b_0
  - pkgs/main::tk==8.6.12=h1ccaba5_0
  - pkgs/main::tzdata==2024a=h04d1e81_0
  - pkgs/main::wheel==0.41.2=py311h06a4308_0
  - pkgs/main::xz==5.4.6=h5eee18b_0
  - pkgs/main::zlib==1.2.13=h5eee18b_0

In the past I would've seen something like this instead

Solving environment: ...working... failed

PackagesNotFoundError: The following packages are not available from current channels:

  - conda-forge::openssl==3.0.13=h7f8727e_0

Here's the issue reproduced in a github action build

https://github.com/tl-hbk/conda-solver-error-message/actions/runs/8190938627/job/22399027702

Conda Info

active environment : None
       user config file : /home/runner/.condarc
 populated config files : /home/runner/.condarc
          conda version : 24.1.2
    conda-build version : not installed
         python version : 3.12.1.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen2
                          __conda=24.1.2=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /usr/share/miniconda3  (writable)
      conda av data dir : /usr/share/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/runner/conda_pkgs_dir
       envs directories : /usr/share/miniconda3/envs
                          /home/runner/.conda/envs
               platform : linux-64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.12.1 Linux/6.5.0-1015-azure ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.3
                UID:GID : 1001:127
             netrc file : None
           offline mode : False

Conda Config

==> /home/runner/.condarc <==
auto_update_conda: False
auto_activate_base: True
notify_outdated_conda: False
changeps1: False
pkgs_dirs:
  - /home/runner/conda_pkgs_dir
always_yes: True

Conda list

# packages in environment at /usr/share/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
archspec                  0.2.1              pyhd3eb1b0_0    defaults
boltons                   23.0.0          py312h06a4308_0    defaults
brotli-python             1.0.9           py312h6a678d5_7    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
c-ares                    1.19.1               h5eee18b_0    defaults
ca-certificates           2023.12.12           h06a4308_0    defaults
certifi                   2024.2.2        py312h06a4308_0    defaults
cffi                      1.16.0          py312h5eee18b_0    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
conda                     24.1.2          py312h06a4308_0    defaults
conda-content-trust       0.2.0           py312h06a4308_0    defaults
conda-libmamba-solver     23.12.0            pyhd3eb1b0_1    defaults
conda-package-handling    2.2.0           py312h06a4308_0    defaults
conda-package-streaming   0.9.0           py312h06a4308_0    defaults
cryptography              41.0.7          py312hdda0065_0    defaults
distro                    1.8.0           py312h06a4308_0    defaults
expat                     2.5.0                h6a678d5_0    defaults
fmt                       9.1.0                hdb19cb5_0    defaults
icu                       73.1                 h6a678d5_0    defaults
idna                      3.4             py312h06a4308_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
krb5                      1.20.1               h143b758_1    defaults
ld_impl_linux-64          2.38                 h1181459_1    defaults
libarchive                3.6.2                h6ac8c49_2    defaults
libcurl                   8.5.0                h251f7ec_0    defaults
libedit                   3.1.20230828         h5eee18b_0    defaults
libev                     4.33                 h7f8727e_1    defaults
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libmamba                  1.5.3                haf1ee3a_0    defaults
libmambapy                1.5.3           py312h2dafd23_0    defaults
libnghttp2                1.57.0               h2d74bed_0    defaults
libsolv                   0.7.24               he621ea3_0    defaults
libssh2                   1.10.0               hdbd6064_2    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
libxml2                   2.10.4               hf1b16e4_1    defaults
lz4-c                     1.9.4                h6a678d5_0    defaults
menuinst                  2.0.2           py312h06a4308_0    defaults
ncurses                   6.4                  h6a678d5_0    defaults
openssl                   3.0.13               h7f8727e_0    defaults
packaging                 23.1            py312h06a4308_0    defaults
pcre2                     10.42                hebb0a14_0    defaults
pip                       23.3.1          py312h06a4308_0    defaults
platformdirs              3.10.0          py312h06a4308_0    defaults
pluggy                    1.0.0           py312h06a4308_1    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.6           py312h5eee18b_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pysocks                   1.7.1           py312h06a4308_0    defaults
python                    3.12.1               h996f2a0_0    defaults
readline                  8.2                  h5eee18b_0    defaults
reproc                    14.2.4               h295c915_1    defaults
reproc-cpp                14.2.4               h295c915_1    defaults
requests                  2.31.0          py312h06a4308_1    defaults
ruamel.yaml               0.17.21         py312h5eee18b_0    defaults
setuptools                68.2.2          py312h06a4308_0    defaults
sqlite                    3.41.2               h5eee18b_0    defaults
tk                        8.6.12               h1ccaba5_0    defaults
tqdm                      4.65.0          py312he106c6f_0    defaults
truststore                0.8.0           py312h06a4308_0    defaults
tzdata                    2023d                h04d1e81_0    defaults
urllib3                   2.1.0           py312h06a4308_1    defaults
wheel                     0.41.2          py312h06a4308_0    defaults
xz                        5.4.5                h5eee18b_0    defaults
yaml-cpp                  0.8.0                h6a678d5_0    defaults
zlib                      1.2.13               h5eee18b_0    defaults
zstandard                 0.19.0          py312h5eee18b_0    defaults
zstd                      1.5.5                hc292b87_0    defaults

Additional Context

No response

[tl-hbk]

This seems to be caused by _pool.create_whatprovides() being called a second time on the same pool object

https://github.com/conda/conda-libmamba-solver/blob/24.1.0/conda_libmamba_solver/solver.py#L376

https://github.com/mamba-org/mamba/blob/mamba-1.5.7/libmamba/src/core/solver.cpp#L38

With verbose logging on the 1st solve attempt I see

info     libsolv  4281 of 697306 installable solvables considered for solving

and on the 2nd attempt

info     libsolv  6 of 697306 installable solvables considered for solving

If I add index._pool.create_whatprovides() before solve is called in _solve_attempt this causes the 1st solve attempt to do the exact same thing as the second.

https://github.com/conda/conda-libmamba-solver/blob/24.1.0/conda_libmamba_solver/solver.py#L406

[tl-hbk]

Looking at the comments added in this PR it seems like the order of when create_whatprovides is called matters.

https://github.com/mamba-org/mamba/pull/2428/files

[tl-hbk]

add_channel_specific_matchspec uses pool.find_string(repr) to find already added matchspec at the start and returns the original DependencyId.

The problem is that the whatprovides relationships set in add_channel_specific_matchspec by pool.add_to_whatprovides(repr_id, pool.add_to_whatprovides_data(selected_pkgs)) are cleared by pool_createwhatprovides according to the comments in libmamba

            // FRAGILE This get deleted when calling ``pool_createwhatprovides`` so care
            // must be taken to do it before

pool_createwhatprovides is called on each solve attempt due to solver initialization but the later attempts don't readd the whatprovides relationships due to the cache being hit.

I'll need to step through libmamba with gdb and verify this behavior

[jaimergp]

Thanks for the detailed investigation! If there's a workaround we can add in Python, we can handle that here. If it has to be done in C++ land, then we'll need to contribute a patch to libmamba. However, the mamba devs are currently busy with shipping v2, so maybe this will be fixed only in the new version (if it's indeed happening there; maybe it doesn't).

[tl-hbk]

I think the change would have to be in the libmamba side unless we recreate a fresh pool instance for every solve attempt.

I was able to confirm my previous comment by having add_channel_specific_matchspec always create the whatprovides relationships every call even if the string is already in the cache which ended up resolving the issue.

[tl-hbk]

Semi-related to this issue was that I noticed in the error messages that the channel pins would add duplicate entries

Current channels:

  - https://conda.anaconda.org/conda-forge/linux-64
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://conda.anaconda.org/conda-forge
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults
  - defaults

I created a PR for this #449

[tl-hbk]

The issue is resolved in the mamba PR that's been merged so I'm closing this issue.

[jaimergp]

I tried current 1.x locally and it works 🚀 Thanks!