Conda build 3.28.1 fails with libmamba solver 23.12.0 with KeyError

[desilinguist]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

We use a local Artifactory instance to host our internal packages and a conda-forge cache. Just like with #363, a KeyError is generated when building a conda package as shown below:

Traceback

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda/exception_handler.py", line 17, in __call__
        return func(*args, **kwargs)
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda/cli/main.py", line 83, in main_subshell
        exit_code = do_call(args, parser)
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 170, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/plugin.py", line 10, in build
        execute(*args, **kwargs)
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/cli/main_build.py", line 572, in execute
        outputs = api.build(
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/api.py", line 254, in build
        return build_tree(
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/build.py", line 3793, in build_tree
        packages_from_this = build(
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/build.py", line 2577, in build
        create_build_envs(top_level_pkg, notest)
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/build.py", line 2417, in create_build_envs
        environ.create_env(
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda_build/environ.py", line 1050, in create_env
        display_actions(actions, index)
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda/exports.py", line 214, in display_actions
        actions["LINK"] = [index[d] for d in actions["LINK"]]
      File "/home/ec2-user/miniconda/lib/python3.10/site-packages/conda/exports.py", line 214, in <listcomp>
        actions["LINK"] = [index[d] for d in actions["LINK"]]
    KeyError: Dist(channel='defaults', dist_name='_libgcc_mutex-0.1-conda_forge', name='_libgcc_mutex', fmt='.tar.bz2', version='0.1', build_string='conda_forge', build_number=0, base_url=None, platform=None)

I can build the package successfully if I set the CONDA_SOLVER environment variable to classic when building the package.

Conda Info

active environment : base
    active env location : /home/ec2-user/miniconda
            shell level : 1
       user config file : /home/ec2-user/.condarc
 populated config files : /home/ec2-user/miniconda/.condarc
                          /home/ec2-user/.condarc
          conda version : 23.11.0
    conda-build version : 3.28.1
         python version : 3.10.13.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake_avx512
                          __conda=23.11.0=0
                          __glibc=2.34=0
                          __linux=6.1.61=0
                          __unix=0=0
       base environment : /home/ec2-user/miniconda  (writable)
      conda av data dir : /home/ec2-user/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://artifactory.ets.org/artifactory/api/conda/conda-virtual/linux-64
                          https://artifactory.ets.org/artifactory/api/conda/conda-virtual/noarch
                          https://artifactory.ets.org/artifactory/api/conda/conda-virtual/conda-forge/linux-64
                          https://artifactory.ets.org/artifactory/api/conda/conda-virtual/conda-forge/noarch
          package cache : /home/ec2-user/data/conda/pkgs
       envs directories : /home/ec2-user/data/conda/envs
                          /home/ec2-user/miniconda/envs
                          /home/ec2-user/.conda/envs
               platform : linux-64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.10.13 Linux/6.1.61-85.141.amzn2023.x86_64 amzn/2023 glibc/2.34 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.5
                UID:GID : 1000:1000
             netrc file : /home/ec2-user/.netrc
           offline mode : False

Conda Config

==> /home/ec2-user/.condarc <==
auto_update_conda: False
changeps1: True
envs_dirs:
  - /home/ec2-user/data/conda/envs
pkgs_dirs:
  - /home/ec2-user/data/conda/pkgs
channel_alias: https://artifactory.ets.org/artifactory/api/conda/conda-virtual
channel_priority: strict
channels:
  - https://artifactory.ets.org/artifactory/api/conda/conda-virtual
default_channels:
  - https://artifactory.ets.org/artifactory/api/conda/conda-virtual
conda-build:
  root-dir: /home/ec2-user/data/conda/build

Conda list

# packages in environment at /home/ec2-user/miniconda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    https://conda.anaconda.org/conda-forge
_openmp_mutex             4.5                       2_gnu    https://conda.anaconda.org/conda-forge
anaconda-client           1.12.1             pyhd8ed1ab_1    /
anaconda-project          0.11.1             pyhd8ed1ab_0    /
archspec                  0.2.2              pyhd8ed1ab_0    /
attrs                     23.1.0             pyh71513ae_1    /
beautifulsoup4            4.12.2             pyha770c72_0    /
black                     23.11.0         py310hff52083_0    /
boltons                   23.0.0             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    /
bzip2                     1.0.8                hd590300_5    /
c-ares                    1.23.0               hd590300_0    /
ca-certificates           2023.11.17           hbcca054_0    /
certifi                   2023.11.17         pyhd8ed1ab_0    /
cffi                      1.16.0          py310h2fee648_0    /
chardet                   5.2.0           py310hff52083_1    /
charset-normalizer        3.3.2              pyhd8ed1ab_0    /
click                     8.1.7           unix_pyh707e725_0    /
clyent                    1.2.2              pyhd8ed1ab_2    /
colorama                  0.4.6              pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
conda                     23.11.0         py310hff52083_1    /
conda-build               3.28.1          py310hff52083_0    /
conda-index               0.3.0              pyhd8ed1ab_1    /
conda-libmamba-solver     23.12.0            pyhd8ed1ab_0    /
conda-pack                0.7.1              pyhd8ed1ab_0    /
conda-package-handling    2.2.0              pyh38be061_0    https://conda.anaconda.org/conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
cryptography              41.0.7          py310hb8475ec_1    /
curl                      8.5.0                hca28451_0    /
defusedxml                0.7.1              pyhd8ed1ab_0    /
distro                    1.8.0              pyhd8ed1ab_0    /
filelock                  3.13.1             pyhd8ed1ab_0    /
flake8                    6.1.0              pyhd8ed1ab_0    /
fmt                       10.1.1               h00ab1b0_1    /
freetype                  2.12.1               h267a509_2    /
gettext                   0.21.1               h27087fc_0    /
git                       2.43.0          pl5321h7bc287a_0    /
icu                       73.2                 h59595ed_0    https://conda.anaconda.org/conda-forge
idna                      3.6                pyhd8ed1ab_0    /
importlib_resources       6.1.1              pyhd8ed1ab_0    /
isort                     5.13.1             pyhd8ed1ab_0    /
jinja2                    3.1.2              pyhd8ed1ab_1    /
jq                        1.7                  hd590300_0    /
jsonpatch                 1.33               pyhd8ed1ab_0    /
jsonpointer               2.4             py310hff52083_3    /
jsonschema                4.20.0             pyhd8ed1ab_0    /
jsonschema-specifications 2023.11.2          pyhd8ed1ab_0    /
jupyter_core              5.5.0           py310hff52083_0    /
keyutils                  1.6.1                h166bdaf_0    https://conda.anaconda.org/conda-forge
krb5                      1.21.2               h659d440_0    https://conda.anaconda.org/conda-forge
lcms2                     2.16                 hb7c19ff_0    /
ld_impl_linux-64          2.40                 h41732ed_0    https://conda.anaconda.org/conda-forge
lerc                      4.0.0                h27087fc_0    /
libarchive                3.7.2                h2aa1ff5_1    /
libcurl                   8.5.0                hca28451_0    /
libdeflate                1.19                 hd590300_0    /
libedit                   3.1.20191231         he28a2e2_2    https://conda.anaconda.org/conda-forge
libev                     4.33                 hd590300_2    /
libevent                  2.1.10               h28343ad_4    /
libexpat                  2.5.0                hcb278e6_1    /
libffi                    3.4.2                h7f98852_5    https://conda.anaconda.org/conda-forge
libgcc-ng                 13.2.0               h807b86a_3    /
libgomp                   13.2.0               h807b86a_3    /
libiconv                  1.17                 hd590300_1    /
libjpeg-turbo             3.0.0                hd590300_1    /
liblief                   0.12.3               h27087fc_0    /
libmamba                  1.5.5                had39da4_0    /
libmambapy                1.5.5           py310h39ff949_0    /
libnghttp2                1.58.0               h47da74e_1    /
libnsl                    2.0.1                hd590300_0    /
libpng                    1.6.39               h753d276_0    /
libsolv                   0.7.27               hfc55251_0    /
libsqlite                 3.44.2               h2797004_0    /
libssh2                   1.11.0               h0841786_0    https://conda.anaconda.org/conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    /
libtiff                   4.6.0                ha9c0a0a_2    /
libuuid                   2.38.1               h0b41bf4_0    https://conda.anaconda.org/conda-forge
libuv                     1.46.0               hd590300_0    /
libwebp-base              1.3.2                hd590300_0    /
libxcb                    1.15                 h0b41bf4_0    /
libxml2                   2.12.3               h232c23b_0    /
libzlib                   1.2.13               hd590300_5    https://conda.anaconda.org/conda-forge
lz4-c                     1.9.4                hcb278e6_0    https://conda.anaconda.org/conda-forge
lzo                       2.10              h516909a_1000    https://conda.anaconda.org/conda-forge
mamba                     1.5.5           py310h51d5547_0    /
markupsafe                2.1.3           py310h2372a71_1    /
mccabe                    0.7.0              pyhd8ed1ab_0    /
menuinst                  2.0.1           py310hff52083_0    /
more-itertools            10.1.0             pyhd8ed1ab_0    /
mypy_extensions           1.0.0              pyha770c72_0    /
nbformat                  5.9.2              pyhd8ed1ab_0    /
ncurses                   6.4                  h59595ed_2    /
nodejs                    20.9.0               hb753e55_0    /
oniguruma                 6.9.9                hd590300_0    /
openjpeg                  2.5.0                h488ebb8_3    /
openssl                   3.2.0                hd590300_1    /
packaging                 23.2               pyhd8ed1ab_0    /
patch                     2.7.6             h7f98852_1002    /
patchelf                  0.17.2               h58526e2_0    /
pathspec                  0.12.1             pyhd8ed1ab_0    /
pcre2                     10.42                hcad00b1_0    /
perl                      5.32.1          4_hd590300_perl5    /
pillow                    10.1.0          py310h01dd4db_0    /
pip                       23.3.1             pyhd8ed1ab_0    /
pkginfo                   1.9.6              pyhd8ed1ab_0    /
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    /
platformdirs              4.1.0              pyhd8ed1ab_0    /
pluggy                    1.3.0              pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
psutil                    5.9.5           py310h2372a71_1    /
pthread-stubs             0.4               h36c2ea0_1001    /
py-lief                   0.12.3          py310hd8f1fbe_0    /
pybind11-abi              4                    hd8ed1ab_3    https://conda.anaconda.org/conda-forge
pycodestyle               2.11.1             pyhd8ed1ab_0    /
pycosat                   0.6.6           py310h2372a71_0    /
pycparser                 2.21               pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
pydocstyle                6.3.0              pyhd8ed1ab_0    /
pyflakes                  3.1.0              pyhd8ed1ab_0    /
pyopenssl                 23.3.0             pyhd8ed1ab_0    /
pysocks                   1.7.1              pyha2e5f31_6    https://conda.anaconda.org/conda-forge
python                    3.10.13         hd12c33a_0_cpython    /
python-dateutil           2.8.2              pyhd8ed1ab_0    /
python-fastjsonschema     2.19.0             pyhd8ed1ab_0    /
python-libarchive-c       5.0             py310hff52083_2    /
python_abi                3.10                    4_cp310    /
pytz                      2023.3.post1       pyhd8ed1ab_0    /
pyyaml                    6.0.1           py310h2372a71_1    /
readline                  8.2                  h8228510_1    https://conda.anaconda.org/conda-forge
referencing               0.32.0             pyhd8ed1ab_0    /
reproc                    14.2.4.post0         hd590300_1    /
reproc-cpp                14.2.4.post0         h59595ed_1    /
requests                  2.31.0             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_0    /
ripgrep                   14.0.3               he8a937b_1    /
rpds-py                   0.13.2          py310hcb5633a_0    /
ruamel.yaml               0.18.5          py310h2372a71_0    /
ruamel.yaml.clib          0.2.7           py310h2372a71_2    /
ruamel_yaml               0.15.80         py310h2372a71_1009    /
setuptools                68.2.2             pyhd8ed1ab_0    /
six                       1.16.0             pyh6c4a22f_0    /
snowballstemmer           2.2.0              pyhd8ed1ab_0    /
soupsieve                 2.5                pyhd8ed1ab_1    /
tk                        8.6.13          noxft_h4845f30_101    /
tmux                      3.3                  h385fc29_0    /
tomli                     2.0.1              pyhd8ed1ab_0    /
toolz                     0.12.0             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
tornado                   6.3.3           py310h2372a71_1    /
tqdm                      4.66.1             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
traitlets                 5.14.0             pyhd8ed1ab_0    /
truststore                0.8.0              pyhd8ed1ab_0    /
typing_extensions         4.9.0              pyha770c72_0    /
tzdata                    2023c                h71feb2d_0    https://conda.anaconda.org/conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    /
wheel                     0.42.0             pyhd8ed1ab_0    /
xorg-libxau               1.0.11               hd590300_0    /
xorg-libxdmcp             1.1.3                h7f98852_0    /
xz                        5.2.6                h166bdaf_0    https://conda.anaconda.org/conda-forge
yaml                      0.2.5                h7f98852_2    /
yaml-cpp                  0.8.0                h59595ed_0    /
zipp                      3.17.0             pyhd8ed1ab_0    /
zlib                      1.2.13               hd590300_5    /
zstandard                 0.22.0          py310h1275a96_0    /
zstd                      1.5.5                hfc55251_0    https://conda.anaconda.org/conda-forge

Additional Context

No response

[jaimergp]

I have questions about the config.

• Why do you need to redefine default_channels?
• If you want to redefine the default_channels, then you don't need to define channels again. Choose one oe the other.
• channel_alias should be set to the parent URL, not the channel itself

Sorry for the brevity, typing from my phone 😅

[desilinguist]

Thanks for your comments! The config is coming straight from Artifactory documentation itself. Additionally, we have been using this config for more than 2 years now for both installing and building packages and it has worked just fine until now with the new solver. If you still think it's an issue with the config, I am happy to test whatever changes you might suggest.

[jaimergp]

Yes, we should revisit that configuration. Maybe it worked for classic, but there are certain inconsistencies that we have been trying to address with libmamba. Let me elaborate on why I think some aspects of your configuration are redundant:

• default_channels is used to redefine what -c defaults means. By default, in conda, it's a list of two (three on Windows) channels on https://repo.anaconda.com: pkgs/main and pkgs/r. On Windows, add pkgs/msys2. The full URL is thus https://repo.anaconda.com/pkgs/main.
• channels is used to define which channels to use by default. The factory config for this is [defaults]. Confusing, I know. Users can redefine this to use other community channels by default, like conda-forge or bioconda. Name channels like those are assumed to be available under conda.anaconda.org. e.g. conda-forge is served from https://conda.anaconda.org/conda-forge.
• channel_alias can be used to change that base URL assumption. Instead of expecting https://conda.anaconda.org, you can point it to your mirror.

Let's recap with examples:

• If you installed conda via Miniconda, you'll be using the default configuration. This means you'll get your packages from defaults, aka https://repo.anaconda.com/pkgs/main and friends.
• If you installed conda via Miniforge, you'll be using a custom configuration that sets conda-forge as the default channel. You'll get your packages from https://conda.anaconda.org/conda-forge. You can still get stuff from https://repo.anaconda.com/pkgs/main by adding -c defaults to your CLI.

Note, you can ALSO fetch packages from conda-forge by setting default_channels to ["https://conda.anaconda.org/conda-forge"]. In this case, you won't be able to use -c defaults to refer to repo.anaconda.com/pkgs/main.

---

Now, back to your configuration. I'm not sure how Artifactory works, but I assume conda-virtual is an aggregated mirror of channels that have been approved to use.

To use conda-virtual from Artifactory by default (i.e. what it's used with a simpler conda install package) but leaving everything else as in factory settings (you can still fetch from anaconda.org via -c conda-forge):

# channel_alias; remove
channels:
- https://artifactory.ets.org/artifactory/api/conda/conda-virtual
# default_channels; remove

Same, but -c defaults means Artifactory instead of pkgs/main:

# channel_alias; remove
# channels; remove
default_channels:
- https://artifactory.ets.org/artifactory/api/conda/conda-virtual

To only use Artifactory as a source and allow -c defaults as pkgs/main:

channel_alias: https://artifactory.ets.org/artifactory/api/conda
channels:
- conda-virtual
# default_channels; remove to restore `-c defaults` meaning Anaconda's `pkgs/main`

Same but -c defaults means Artifactory too:

channel_alias: https://artifactory.ets.org/artifactory/api/conda
channels:
- conda-virtual
default_channels:
- https://artifactory.ets.org/artifactory/api/conda/conda-virtual

There are more ways to configure things, but the main issue I see is having channel_alias and channels with the same value. You need channel_alias as the "base url" and channels as a list of directory names in "base url".

Hope that's clear to an extent 😬

[jaimergp]

Btw, https://conda.io/projects/conda/en/latest/configuration.html lists and describes all the config options available. Search for channels there to make more sense of what I wrote, if needed :)

[desilinguist]

Thank you for the detailed explanation 🙏! I'll try out some of the things you suggested and report back.

[desilinguist]

Okay, I finally had a chance to try out your suggestion. I changed my $HOME/.condarc to look like the following:

channel_alias: https://artifactory.ets.org/artifactory/api/conda
channels:
  - conda-virtual
default_channels:
  - https://artifactory.ets.org/artifactory/api/conda/conda-virtual

However, if I try to build the package now, I get a 500 error like so:

CondaHTTPError: HTTP 500 INTERNAL SERVER ERROR for url <https://artifactory.ets.org/artifactory/api/conda/conda-forge/linux-64/repodata.json>

We are using conda from Miniforge3 so I guess it still tries to access the named conda-forge channel and fails because the channel alias is now different. Just to double check, I tried creating a test environment using mamba create with this updated .condarc and got the same 500 error. What's interesting is that the same mamba create command works just fine with my original .condarc.

[jaimergp]

I see, yes, your assessment is correct. You can either delete the condarc file in your miniforge folder or add this custom_channel:

custom_channels:
  conda-forge: https://conda.anaconda.org

[desilinguist]

That works perfectly! Thank you so much all your help!