Adding a new channel results in too many package updates #242

jaimergp · 2023-07-20T10:30:59Z

Description

Will close #240

Checklist - did you ...

Add a file to the news directory (using the template) for the next release's release notes?
Add / update necessary tests?
Add / update outdated documentation?

jaimergp · 2023-07-20T11:09:06Z

This PR only adds a test that reproduces the behavior reported in the issue. For debugging purposes, one can also do this:

$ conda create -n test --override-channels -c defaults conda python  # set 'base' env
$ conda install -n test --override-channels -c conda-forge -c defaults libzlib --solver=libmamba -vvv --dry-run

These are the logs for the 2nd step with libmamba 1.4.1 and 1.4.9:

libmamba141.txt.zip (zipped because 1.4.1 produces waaay more output)
libmamba149.txt

I'll take a look and see if I can find the cause in the logs, and hopefully provide a fix.

cc @costrouc, in case you have some time, could you also look into this?

costrouc · 2023-07-20T21:14:52Z

Debugging this issue with the example you provided @jaimergp.

conda install -n test --override-channels -c conda-forge -c defaults libzlib --solver=libmamba --dry-run

When I look at the tasks passed to libmamba

('USERINSTALLED', 2560) ['pkgs/main/linux-64::python==3.11.4=h955ad1f_0', 'pkgs/main/linux-64::conda==23.5.2=py311h06a4308_0', 'pkgs/main/linux-64::ca-certificates==2023.05.30=h06a4308_0', 'pkgs/main/linux-64::certifi==2023.5.7=py311h06a4308_0', 'pkgs/main/linux-64::openssl==3.0.9=h7f8727e_0'] 

('UPDATE | ESSENTIAL', 131840) ['pkgs/main/linux-64::python==3.11.4=h955ad1f_0', 'pkgs/main/linux-64::conda==23.5.2=py311h06a4308_0', 'ca-certificates', 'certifi', 'openssl'] 

('UPDATE', 768) ['pkgs/main/linux-64::zstandard==0.19.0=py311h5eee18b_0', 'pkgs/main/linux-64::zlib==1.2.13=h5eee18b_0', 'pkgs/main/linux-64::_openmp_mutex==5.1=1_gnu', 'pkgs/main/linux-64::conda-package-streaming==0.8.0=py311h06a4308_0', 'pkgs/main/linux-64::cryptography==39.0.1=py311h9ce1e76_2', 'pkgs/main/linux-64::requests==2.29.0=py311h06a4308_0', 'pkgs/main/linux-64::ruamel.yaml==0.17.21=py311h5eee18b_0', 'pkgs/main/linux-64::wheel==0.38.4=py311h06a4308_0', 'pkgs/main/linux-64::libstdcxx-ng==11.2.0=h1234567_1', 'pkgs/main/linux-64::_libgcc_mutex==0.1=main', 'pkgs/main/noarch::pycparser==2.21=pyhd3eb1b0_0', 'pkgs/main/linux-64::boltons==23.0.0=py311h06a4308_0', 'pkgs/main/linux-64::cffi==1.15.1=py311h5eee18b_3', 'pkgs/main/linux-64::pysocks==1.7.1=py311h06a4308_0', 'pkgs/main/linux-64::idna==3.4=py311h06a4308_0', 'pkgs/main/linux-64::toolz==0.12.0=py311h06a4308_0', 'pkgs/main/linux-64::readline==8.2=h5eee18b_0', 'pkgs/main/linux-64::setuptools==67.8.0=py311h06a4308_0', 'pkgs/main/linux-64::libffi==3.4.4=h6a678d5_0', 'pkgs/main/linux-64::ncurses==6.4=h6a678d5_0', 'pkgs/main/linux-64::xz==5.4.2=h5eee18b_0', 'pkgs/main/linux-64::sqlite==3.41.2=h5eee18b_0', 'pkgs/main/noarch::jsonpointer==2.1=pyhd3eb1b0_0', 'pkgs/main/linux-64::libgcc-ng==11.2.0=h1234567_1', 'pkgs/main/noarch::tzdata==2023c=h04d1e81_0', 'pkgs/main/noarch::jsonpatch==1.32=pyhd3eb1b0_0', 'pkgs/main/linux-64::libuuid==1.41.5=h5eee18b_0', 'pkgs/main/linux-64::tqdm==4.65.0=py311h92b7b1e_0', 'pkgs/main/linux-64::libgomp==11.2.0=h1234567_1', 'pkgs/main/linux-64::tk==8.6.12=h1ccaba5_0', 'pkgs/main/linux-64::packaging==23.0=py311h06a4308_0', 'pkgs/main/noarch::charset-normalizer==2.0.4=pyhd3eb1b0_0', 'pkgs/main/linux-64::urllib3==1.26.16=py311h06a4308_0', 'pkgs/main/linux-64::pluggy==1.0.0=py311h06a4308_1', 'pkgs/main/linux-64::brotlipy==0.7.0=py311h5eee18b_1002', 'pkgs/main/linux-64::ld_impl_linux-64==2.38=h1181459_1', 'pkgs/main/linux-64::pycosat==0.6.4=py311h5eee18b_0', 'pkgs/main/linux-64::pyopenssl==23.0.0=py311h06a4308_0', 'pkgs/main/linux-64::bzip2==1.0.8=h7b6447c_0', 'pkgs/main/linux-64::conda-package-handling==2.1.0=py311h06a4308_0', 'pkgs/main/linux-64::pip==23.1.2=py311h06a4308_0'] 

('INSTALL', 256) ['libzlib']

This is too much. I see your commit above @jaimergp which talks about this issue as well with the UPDATE task.

I inserted a breakpoint here. And then modified the solve for libmamba deleting

  del tasks[('UPDATE', 768)]
  del tasks[('UPDATE | ESSENTIAL', 131840)]

I got similar but not exactly the same solve

classic

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
    _openmp_mutex-4.5          |       2_kmp_llvm           6 KB  conda-forge
    libgcc-ng-13.1.0           |       he5830b7_0         758 KB  conda-forge
    libzlib-1.2.13             |       hd590300_5          60 KB  conda-forge
    llvm-openmp-14.0.6         |       h9e868ea_0         4.4 MB
    zlib-1.2.13                |       hd590300_5          91 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         5.3 MB

The following NEW packages will be INSTALLED:

  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  llvm-openmp        pkgs/main/linux-64::llvm-openmp-14.0.6-h9e868ea_0

The following packages will be REMOVED:

  libgomp-11.2.0-h1234567_1

The following packages will be UPDATED:

  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-13.1.0-he5830b7_0
  openssl               pkgs/main::openssl-3.0.9-h7f8727e_0 --> conda-forge::openssl-3.1.1-hd590300_1
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_0 --> conda-forge::zlib-1.2.13-hd590300_5

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_kmp_llvm
  ca-certificates    pkgs/main::ca-certificates-2023.05.30~ --> conda-forge::ca-certificates-2023.5.7-hbcca054_0
  certifi            pkgs/main/linux-64::certifi-2023.5.7-~ --> conda-forge/noarch::certifi-2023.5.7-pyhd8ed1ab_0

libsolve

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
    _openmp_mutex-4.5          |            2_gnu          23 KB  conda-forge
    libgcc-ng-13.1.0           |       he5830b7_0         758 KB  conda-forge
    libgomp-13.1.0             |       he5830b7_0         409 KB  conda-forge
    libzlib-1.2.13             |       hd590300_5          60 KB  conda-forge
    zlib-1.2.13                |       hd590300_5          91 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.3 MB

The following NEW packages will be INSTALLED:

  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5

The following packages will be UPDATED:

  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-13.1.0-he5830b7_0
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-13.1.0-he5830b7_0
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_0 --> conda-forge::zlib-1.2.13-hd590300_5

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu

The classic solver still opted for updating the openssl, ca-certificates, certifi all to conda-forge and interestingly downgraded ca-certificates. Next I'm going to see how messing with tasks[('UPDATE | ESSENTIAL', 131840)] could get a simliar solve.

costrouc · 2023-07-20T21:23:42Z

When I delete 'pkgs/main/linux-64::python==3.11.4=h955ad1f_0', 'pkgs/main/linux-64::conda==23.5.2=py311h06a4308_0' from ('UPDATE | ESSENTIAL', 131840) I get the following

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
    _openmp_mutex-4.5          |            2_gnu          23 KB  conda-forge
    libgcc-ng-13.1.0           |       he5830b7_0         758 KB  conda-forge
    libgomp-13.1.0             |       he5830b7_0         409 KB  conda-forge
    libzlib-1.2.13             |       hd590300_5          60 KB  conda-forge
    zlib-1.2.13                |       hd590300_5          91 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.3 MB

The following NEW packages will be INSTALLED:

  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5 

The following packages will be UPDATED:

  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-13.1.0-he5830b7_0 
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-13.1.0-he5830b7_0 
  openssl               pkgs/main::openssl-3.0.9-h7f8727e_0 --> conda-forge::openssl-3.1.1-hd590300_1 
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_0 --> conda-forge::zlib-1.2.13-hd590300_5 

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu 
  certifi            pkgs/main/linux-64::certifi-2023.5.7-~ --> conda-forge/noarch::certifi-2023.5.7-pyhd8ed1ab_0

It feels like conda-libmamba-solver with the tasks being set is causing all the libraries that are dependencies of ('UPDATE | ESSENTIAL', 131840) to update.

costrouc · 2023-07-21T01:49:46Z

Inspecting the mamba install command and the libsolve inputs that are used

['libzlib'] 256                INSTALL
['ca-certificates'] 768        UPDATE
['certifi'] 768                UPDATE
['openssl'] 768                UPDATE
ADD_PIN python 3.11.*
ADD_JOBS pin-1  0

I've checked conda-libmamba-solver if I override the tasks with the following below and I get the same result.

tasks[('INSTALL', 256)] = ['libzlib']
tasks[('UPDATE', 768)] = ['ca-certificates', 'certifi', 'openssl']
tasks[('USERINSTALLED', 2560)] = ['python==3.11.*']

jaimergp · 2023-07-21T08:34:50Z

Yea, we are finding mostly the same stuff. Making this new test pass is a matter of "doing less" as you have seen, but then some other integration tests fail.

Interestingly, using libmamba 1.4.9 (conda-forge tests), we manage to pass almost everything, but still not perfect. We might need to revisit then "broken by libmamba 1.4.2" tests then.

The problem is that we were using targeted UPDATE tasks as a way of pinning but then those get downgraded to non-pinning updates after a conflict and hence the "update --all"-like behaviour, I think.

costrouc · 2023-07-21T13:50:21Z

conda_libmamba_solver/solver.py

                    # This is "just" an essential job, so it gets higher priority in the solver
                    # conflict resolution. We do this because these are "protected" packages
                    # (history, aggressive updates) that we should try not messing with if
                    # conflicts appear
-                    key = ("UPDATE | ESSENTIAL", api.SOLVER_UPDATE | api.SOLVER_ESSENTIAL)
+                    key = f"{key[0]} | ESSENTIAL", key[1] | api.SOLVER_ESSENTIAL


I think this line changes

('UPDATE | ESSENTIAL', ...) ['ca-certificates', 'certifi', 'openssl']

to

('INSTALL | ESSENTIAL', 131328) ['ca-certificates', 'certifi', 'openssl']

mamba does

('UPDATE ', ...) ['ca-certificates', 'certifi', 'openssl']

It seems like either way we get the same result.

costrouc · 2023-07-28T04:09:50Z

I'm seeing failures across the board on os's windows/linux/osx. Additionally I'm seeing some new failures in CI from recent new tests in conda e.g. tests/test_priority.py::test_reorder_channel_priority. I've been running the CI and trying things out locally but it seems like more things started failing when I merged main into this PR.

jaimergp · 2023-07-28T08:22:27Z

We only need to worry about the conda-forge failures, since defaults only provides libmamba 1.4.1 and that one is known to segfault with some error reporting.

I'll focus on finding the error that causes a segfault and workaround it, if possible
You handle the errors on unit tests:

FAILED ../conda-libmamba-solver-src/tests/test_modified_upstream.py::test_conda_downgrade
FAILED ../conda-libmamba-solver-src/tests/test_modified_upstream.py::test_python2_update

costrouc · 2023-08-10T03:57:59Z

I've been investigating tests/test_modified_upstream.py::test_conda_downgrade to see what has changed to make the test fail. I can narrow the failure down to the task being given for line https://github.com/conda/conda-libmamba-solver/blob/aggressive-channel-update/tests/test_modified_upstream.py#L778.

Using main I get the following task

1. (first attempt) {('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py37_0', 'channel-4/linux-64::conda==4.5.10=py37_0', 'channel-4/linux-64::python==3.7.0=hc3d631a_0'], ('UPDATE | ESSENTIAL', 131840): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", "python[version='>3.7.0']"], ('INSTALL', 256): ['itsdangerous']}
2. (2nd attempt passes) {('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py37_0', 'channel-4/linux-64::conda==4.5.10=py37_0', 'channel-4/linux-64::python==3.7.0=hc3d631a_0'], ('UPDATE | ESSENTIAL', 131840): ['conda-build', 'ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", 'python'], ('INSTALL', 256): ['itsdangerous'], ('DISFAVOR', 3328): ['conda-env', 'libffi', 'libgcc-ng', 'libstdcxx-ng', 'zlib', 'ncurses', 'patchelf', 'tk', 'xz', 'yaml', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'beautifulsoup4', 'chardet', 'cryptography-vectors', 'filelock', 'glob2', 'idna', 'markupsafe', 'pkginfo', 'psutil', 'pycosat', 'pycparser', 'pysocks', 'pyyaml', 'ruamel_yaml', 'six', 'cffi', 'setuptools', 'cryptography', 'jinja2', 'pyopenssl', 'urllib3', 'requests'], ('ALLOWUNINSTALL', 2816): ['conda-env', 'libffi', 'libgcc-ng', 'libstdcxx-ng', 'zlib', 'ncurses', 'patchelf', 'tk', 'xz', 'yaml', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'beautifulsoup4', 'chardet', 'cryptography-vectors', 'filelock', 'glob2', 'idna', 'markupsafe', 'pkginfo', 'psutil', 'pycosat', 'pycparser', 'pysocks', 'pyyaml', 'ruamel_yaml', 'six', 'cffi', 'setuptools', 'cryptography', 'jinja2', 'pyopenssl', 'urllib3', 'requests'], ('UPDATE', 768): ['conda-env', 'libffi', 'libgcc-ng', 'libstdcxx-ng', 'zlib', 'ncurses', 'patchelf', 'tk', 'xz', 'yaml', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'beautifulsoup4', 'chardet', 'cryptography-vectors', 'filelock', 'glob2', 'idna', 'markupsafe', 'pkginfo', 'psutil', 'pycosat', 'pycparser', 'pysocks', 'pyyaml', 'ruamel_yaml', 'six', 'cffi', 'setuptools', 'cryptography', 'jinja2', 'pyopenssl', 'urllib3', 'requests']}

vs. new PR aggressive-channel-update

1. {('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py37_0', 'channel-4/linux-64::conda==4.5.10=py37_0', 'channel-4/linux-64::python==3.7.0=hc3d631a_0'], ('LOCK | ESSENTIAL', 132608): ['conda-build==3.12.1=py37_0'], ('INSTALL | ESSENTIAL', 131328): ['ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", "python[version='>3.7.0']"], ('INSTALL', 256): ['itsdangerous']}

with fails with error ValueError: Unknown message: - nothing provides requested python >3.7.0 - package conda-...

I've been spending some time investigating which tasks have an effect on the solve.

costrouc · 2023-08-10T04:13:44Z

I've made several attempts at modifying the tasks to see what works

# attempt 1 (tried to modify `install | essential` -> `update | essential` this did nothing

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0',         'channel-4/linux-64::openssl==1.0.2p=h14c3975_0',         'channel-4/linux-64::certifi==2018.8.13=py37_0',         'channel-4/linux-64::conda==4.5.10=py37_0',         'channel-4/linux-64::python==3.7.0=hc3d631a_0'     ],    ('LOCK | ESSENTIAL', 132608): ['conda-build==3.12.1=py37_0'],    ('UPDATE | ESSENTIAL', 131840): ['ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", "python[version='>3.7.0']"],    ('INSTALL', 256): ['itsdangerous']}

# attempt 2 (tried to modify python constraint to empty, failed to solve but looped)

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0',         'channel-4/linux-64::openssl==1.0.2p=h14c3975_0',         'channel-4/linux-64::certifi==2018.8.13=py37_0',         'channel-4/linux-64::conda==4.5.10=py37_0',         'channel-4/linux-64::python==3.7.0=hc3d631a_0'     ],    ('LOCK | ESSENTIAL', 132608): ['conda-build==3.12.1=py37_0'],    ('UPDATE | ESSENTIAL', 131840): ['ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", "python"],    ('INSTALL', 256): ['itsdangerous']}

# attempt 3 (tied to modify the conda-build constraint "lock" and convert to update essential. failed to solve but looped)

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0',         'channel-4/linux-64::openssl==1.0.2p=h14c3975_0',         'channel-4/linux-64::certifi==2018.8.13=py37_0',         'channel-4/linux-64::conda==4.5.10=py37_0',         'channel-4/linux-64::python==3.7.0=hc3d631a_0'     ],  ('UPDATE | ESSENTIAL', 131840): ['conda-build==3.12.1=py37_0', 'ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", "python"],    ('INSTALL', 256): ['itsdangerous']}

# attempt 4 (already relaxed for us, see if works, didn't work but looped)

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py37_0', 'channel-4/linux-64::conda==4.5.10=py37_0', 'channel-4/linux-64::python==3.7.0=hc3d631a_0'], ('INSTALL | ESSENTIAL', 131328): ['conda-build', 'ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", "python[version='>3.7.0']"], ('INSTALL', 256): ['itsdangerous']}

# attempt 5 (already relaxed more for us)

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda-build==3.12.1=py37_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py37_0', 'channel-4/linux-64::conda==4.5.10=py37_0', 'channel-4/linux-64::python==3.7.0=hc3d631a_0'], ('INSTALL | ESSENTIAL', 131328): ['conda-build', 'ca-certificates', 'openssl', 'certifi', "conda[version='<4.4.10']", 'python'], ('INSTALL', 256): ['itsdangerous', 'conda-env', 'libffi', 'libgcc-ng', 'libstdcxx-ng', 'zlib', 'ncurses', 'patchelf', 'tk', 'xz', 'yaml', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'beautifulsoup4', 'chardet', 'cryptography-vectors', 'filelock', 'glob2', 'idna', 'markupsafe', 'pkginfo', 'psutil', 'pycosat', 'pycparser', 'pysocks', 'pyyaml', 'ruamel_yaml', 'six', 'cffi', 'setuptools', 'cryptography', 'jinja2', 'pyopenssl', 'urllib3', 'requests'], ('DISFAVOR', 3328): ['conda-env', 'libffi', 'libgcc-ng', 'libstdcxx-ng', 'zlib', 'ncurses', 'patchelf', 'tk', 'xz', 'yaml', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'beautifulsoup4', 'chardet', 'cryptography-vectors', 'filelock', 'glob2', 'idna', 'markupsafe', 'pkginfo', 'psutil', 'pycosat', 'pycparser', 'pysocks', 'pyyaml', 'ruamel_yaml', 'six', 'cffi', 'setuptools', 'cryptography', 'jinja2', 'pyopenssl', 'urllib3', 'requests'], ('ALLOWUNINSTALL', 2816): ['conda-env', 'libffi', 'libgcc-ng', 'libstdcxx-ng', 'zlib', 'ncurses', 'patchelf', 'tk', 'xz', 'yaml', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'beautifulsoup4', 'chardet', 'cryptography-vectors', 'filelock', 'glob2', 'idna', 'markupsafe', 'pkginfo', 'psutil', 'pycosat', 'pycparser', 'pysocks', 'pyyaml', 'ruamel_yaml', 'six', 'cffi', 'setuptools', 'cryptography', 'jinja2', 'pyopenssl', 'urllib3', 'requests']}

So from what I am seeing the main behavior failed to solve too on the first try but after relaxing the constraints it did solve. I see the same behavior for this branch that if it is able to loop several times and relax the constraints it solves too.

So I think this has to do with not catching the new error message possibly? I'm going to attempt to catch the error message

costrouc · 2023-08-10T04:27:54Z

Stopping at the point when the solve fails https://github.com/conda/conda-libmamba-solver/blob/aggressive-channel-update/tests/test_modified_upstream.py?rgh-link-date=2023-08-10T03%3A57%3A59Z#L778.

The state is

$ self.solver.problems_to_str()
'Encountered problems while solving:\n  - nothing provides requested python >3.7.0  - package conda-build-3.12.1-py37_0 requires python >=3.7,<3.8.0a0, but none of the providers can be installed'
$ out_state.conflicts
{}
$ self._parse_problems(self.solver.problems_to_str())
*** ValueError: Unknown message: - nothing provides requested python >3.7.0  - package conda-build-3.12.1-py37_0 requires python >=3.7,<3.8.0a0, but none of the providers can be installed

When checking the error that main gets

$ self.solver.problems.to_str()
'Encountered problems while solving:\n  - nothing provides requested channel-4/linux-64::conda-build ==3.12.1 py37_0  - nothing provides requested python >3.7.0'

I've tested that if this branch got the message 'Encountered problems while solving:\n - nothing provides requested channel-4/linux-64::conda-build ==3.12.1 py37_0 - nothing provides requested python >3.7.0' instead of the unknown message it would solve for the environment after loosening the requirements.

With results in ``self._parse_problems(self.solver.problems_to_str()) -> {'conflicts': {}, 'not_found': {'channel-4/linux-64::conda-build': MatchSpec("channel-4/linux-64::conda-build==3.12.1")}}

costrouc · 2023-08-10T05:22:37Z

Okay on to something. In the aggresive-channel-update branch the message 'Encountered problems while solving:\n - nothing provides requested python >3.7.0 - package conda-build-3.12.1-py37_0 requires python >=3.7,<3.8.0a0, but none of the providers can be installed' is missing a newline which making it not parse the message properly.

If I add a newline and change it to 'Encountered problems while solving:\n - nothing provides requested python >3.7.0\n - package conda-build-3.12.1-py37_0 requires python >=3.7,<3.8.0a0, but none of the providers can be installed'. It solves.

For all these tests I am using:

latest conda c691f729a @ upstream/main [pre-commit.ci] pre-commit autoupdate (#12975)
latest mamba be105ee7 @ upstream/main Safe Curl opt in url.cpp (#2734)

This commit bit of code was touched 3 months ago https://github.com/mamba-org/mamba/blame/be105ee79de0073a13357b9bca7d3e6f965d3c52/libmamba/src/core/solver.cpp#L453 in mamba-org/mamba#2544. And it looks to me not to include newlines.

Yeup inspecting the diff there is a missing newline https://github.com/mamba-org/mamba/pull/2544/files#diff-6a1aa709238cca66bca724329d07cd1a9da34068b774c70416245c929aed963dL609-R469.

So I think this is the breakage for this specific test.

jaimergp · 2023-08-10T08:12:23Z

Wow, nice find @costrouc! I guess we can strip the header manually and move on from there at parse time.

jaimergp · 2023-08-10T08:26:08Z

Ah wait, that's not the header, but the error message instead. Now we might need to split by - 😬 This was introduced in 1.4.5. Let's see if they accept an easy fix for 1.4.10? We would need to mark 1.4.5 <-> 1.4.9 as incompatible.

Edit: Opened mamba-org/mamba#2743

costrouc · 2023-08-18T19:33:05Z

Can confirm that mamba-org/mamba#2743 solves tests/test_modified_upstream.py::test_conda_downgrade

costrouc · 2023-08-18T19:38:31Z

Tests Identified in #242 (comment). I will check them off as I verify they work with the latest on mamba-org/mamba#main and conda/conda#main.

conda-libmamba-solver-src/tests/test_modified_upstream.py::test_conda_downgrade
conda-libmamba-solver-src/tests/test_modified_upstream.py::test_python2_update
tests/test_priority.py::test_reorder_channel_priority[with pinned_package] (might be fixed if we generalize this block to any libmamba version)
- this test is failing on main right now too

From checking the main branch has the same issue from what I can tell with tests/test_priority.py::test_reorder_channel_priority[with pinned_package]

I am focusing on tests/test_modified_upstream.py::test_python2_update

costrouc · 2023-08-24T03:48:58Z

Spent time today debugging the faliing test tests/test_modified_upstream.py::test_python2_update. I discovered that this test is not failing on the latest on main. The fail from a simple difference between INSTALL vs. UPDATE.

this branch

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda==4.5.10=py27_0', 'channel-4/linux-64::python==2.7.15=h1571d57_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py27_0'], 
 ('INSTALL | ESSENTIAL', 131328): ['conda', 'python=3', 'ca-certificates', 'openssl', 'certifi'], 
 ('DISFAVOR', 3328): ['conda-env', 'libgcc-ng', 'libstdcxx-ng', 'libffi', 'ncurses', 'tk', 'yaml', 'zlib', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'chardet', 'cryptography-vectors', 'enum34', 'futures', 'idna', 'ipaddress', 'pycosat', 'pycparser', 'pysocks', 'ruamel_yaml', 'six', 'cffi', 'cryptography', 'pyopenssl', 'urllib3', 'requests'],
 ('ALLOWUNINSTALL', 2816): ['conda-env', 'libgcc-ng', 'libstdcxx-ng', 'libffi', 'ncurses', 'tk', 'yaml', 'zlib', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'chardet', 'cryptography-vectors', 'enum34', 'futures', 'idna', 'ipaddress', 'pycosat', 'pycparser', 'pysocks', 'ruamel_yaml', 'six', 'cffi', 'cryptography', 'pyopenssl', 'urllib3', 'requests'],
 ('INSTALL', 256): ['conda-env', 'libgcc-ng', 'libstdcxx-ng', 'libffi', 'ncurses', 'tk', 'yaml', 'zlib', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'chardet', 'cryptography-vectors', 'enum34', 'futures', 'idna', 'ipaddress', 'pycosat', 'pycparser', 'pysocks', 'ruamel_yaml', 'six', 'cffi', 'cryptography', 'pyopenssl', 'urllib3', 'requests']}

main branch

{('USERINSTALLED', 2560): ['channel-4/linux-64::conda==4.5.10=py27_0', 'channel-4/linux-64::python==2.7.15=h1571d57_0', 'channel-4/linux-64::ca-certificates==2018.03.07=0', 'channel-4/linux-64::openssl==1.0.2p=h14c3975_0', 'channel-4/linux-64::certifi==2018.8.13=py27_0'],
 ('UPDATE | ESSENTIAL', 131840): ['conda', 'python=3', 'ca-certificates', 'openssl', 'certifi'],
 ('DISFAVOR', 3328): ['conda-env', 'libgcc-ng', 'libstdcxx-ng', 'libffi', 'ncurses', 'tk', 'yaml', 'zlib', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'chardet', 'cryptography-vectors', 'enum34', 'futures', 'idna', 'ipaddress', 'pycosat', 'pycparser', 'pysocks', 'ruamel_yaml', 'six', 'cffi', 'cryptography', 'pyopenssl', 'urllib3', 'requests'],
 ('ALLOWUNINSTALL', 2816): ['conda-env', 'libgcc-ng', 'libstdcxx-ng', 'libffi', 'ncurses', 'tk', 'yaml', 'zlib', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'chardet', 'cryptography-vectors', 'enum34', 'futures', 'idna', 'ipaddress', 'pycosat', 'pycparser', 'pysocks', 'ruamel_yaml', 'six', 'cffi', 'cryptography', 'pyopenssl', 'urllib3', 'requests'],
 ('UPDATE', 768): ['conda-env', 'libgcc-ng', 'libstdcxx-ng', 'libffi', 'ncurses', 'tk', 'yaml', 'zlib', 'libedit', 'readline', 'sqlite', 'asn1crypto', 'chardet', 'cryptography-vectors', 'enum34', 'futures', 'idna', 'ipaddress', 'pycosat', 'pycparser', 'pysocks', 'ruamel_yaml', 'six', 'cffi', 'cryptography', 'pyopenssl', 'urllib3', 'requests']}

The removal of this line caused the regression https://github.com/conda/conda-libmamba-solver/pull/242/files#diff-e6794455e7cacff948cf95915e290bc1f742360246a7a96f809b7627bbc379c4L460.

Coincidentally this is one of those tests where self._command is NULL. I'll commit the change to this branch.

Added in bda6219

costrouc · 2023-08-24T04:10:03Z

@jaimergp after making that change it regressed and broke several other tests. Is there a better place this UPDATE key should be updated?

jaimergp · 2023-08-24T07:41:43Z

Thanks @costrouc, I'll take a look. If it gets too brittle, I am inclined to start a new logic that doesn't follow conda's internal logic as much, but does cover the expectations of the users. We'll see.

jaimergp · 2023-08-24T09:58:49Z

mamba 1.5.0 released an hour ago 👀 Let's see if we get some fixes.

jaimergp · 2023-08-29T21:26:48Z

This will be probably superseded by #270

jaimergp · 2023-09-15T16:37:22Z

Superseded by #270. Thanks @costrouc!

jaimergp added 3 commits July 20, 2023 12:27

add reproducing test

3d80726

only linux

7883b6d

drop env var just in case

4126f0c

conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Jul 20, 2023

jaimergp added 9 commits July 20, 2023 13:47

use pins instead of UPDATE jobs

08945d6

pre-commit

64c8378

ignore None tasks

721d5b9

INSTALL vs UPDATE only depends on command used

5b68c95

do not disfavor requested

bee0a74

LOCK or PIN strict specs that match installed

93c22d3

mark protected as ESSENTIAL

817e081

parse this new type of conflict

d3d1a76

shorten line

4e4efcf

costrouc self-assigned this Jul 20, 2023

costrouc reviewed Jul 21, 2023

View reviewed changes

jaimergp added this to the July 2023 release milestone Jul 24, 2023

Merge branch 'main' into aggressive-channel-update

077306f

jaimergp added 6 commits July 28, 2023 10:54

cache this

c591108

anticipate segfault?

7e9daf8

debug segfaulting error message

3dcb9f4

check in all_problems?

a029312

try conflicting requests

b82fe73

revert debugging stuff

cbb2e62

jaimergp mentioned this pull request Aug 9, 2023

Make conda-libmamba-solver the default solver conda/conda#12984

Merged

3 tasks

This was referenced Aug 10, 2023

Add missing newline in legacy errors mamba-org/mamba#2743

Merged

Do not assume channels are always supplied #256

Merged

Merge branch 'main' into aggressive-channel-update

1b16849

jezdez modified the milestones: July 2023 release, September 2023 release Aug 23, 2023

Ensure that if special keys are not set use UPDATE

bda6219

jaimergp closed this Aug 24, 2023

jaimergp reopened this Aug 24, 2023

This was referenced Aug 25, 2023

Investigate compatibility with libmamba 1.5.0 #266

Merged

Specifying channel::pkg_name errors under certain conditions #269

Closed

Alternative logic for solver creation tasks #270

Merged

jaimergp closed this Sep 15, 2023

jaimergp deleted the aggressive-channel-update branch November 7, 2023 10:26

github-actions bot added the locked [bot] locked due to inactivity label Nov 7, 2024

github-actions bot locked as resolved and limited conversation to collaborators Nov 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a new channel results in too many package updates #242

Adding a new channel results in too many package updates #242

jaimergp commented Jul 20, 2023

jaimergp commented Jul 20, 2023

costrouc commented Jul 20, 2023 •

edited

Loading

costrouc commented Jul 20, 2023 •

edited

Loading

costrouc commented Jul 21, 2023 •

edited

Loading

jaimergp commented Jul 21, 2023 •

edited

Loading

costrouc Jul 21, 2023

costrouc commented Jul 28, 2023

jaimergp commented Jul 28, 2023

costrouc commented Aug 10, 2023

costrouc commented Aug 10, 2023 •

edited

Loading

costrouc commented Aug 10, 2023 •

edited

Loading

costrouc commented Aug 10, 2023 •

edited

Loading

jaimergp commented Aug 10, 2023

jaimergp commented Aug 10, 2023 •

edited

Loading

costrouc commented Aug 18, 2023

costrouc commented Aug 18, 2023 •

edited

Loading

costrouc commented Aug 24, 2023 •

edited

Loading

costrouc commented Aug 24, 2023

jaimergp commented Aug 24, 2023

jaimergp commented Aug 24, 2023

jaimergp commented Aug 29, 2023

jaimergp commented Sep 15, 2023

Adding a new channel results in too many package updates #242

Adding a new channel results in too many package updates #242

Conversation

jaimergp commented Jul 20, 2023

Description

Checklist - did you ...

jaimergp commented Jul 20, 2023

costrouc commented Jul 20, 2023 • edited Loading

classic

libsolve

costrouc commented Jul 20, 2023 • edited Loading

costrouc commented Jul 21, 2023 • edited Loading

jaimergp commented Jul 21, 2023 • edited Loading

costrouc Jul 21, 2023

Choose a reason for hiding this comment

costrouc commented Jul 28, 2023

jaimergp commented Jul 28, 2023

costrouc commented Aug 10, 2023

costrouc commented Aug 10, 2023 • edited Loading

costrouc commented Aug 10, 2023 • edited Loading

costrouc commented Aug 10, 2023 • edited Loading

jaimergp commented Aug 10, 2023

jaimergp commented Aug 10, 2023 • edited Loading

costrouc commented Aug 18, 2023

costrouc commented Aug 18, 2023 • edited Loading

costrouc commented Aug 24, 2023 • edited Loading

costrouc commented Aug 24, 2023

jaimergp commented Aug 24, 2023

jaimergp commented Aug 24, 2023

jaimergp commented Aug 29, 2023

jaimergp commented Sep 15, 2023

costrouc commented Jul 20, 2023 •

edited

Loading

costrouc commented Jul 20, 2023 •

edited

Loading

costrouc commented Jul 21, 2023 •

edited

Loading

jaimergp commented Jul 21, 2023 •

edited

Loading

costrouc commented Aug 10, 2023 •

edited

Loading

costrouc commented Aug 10, 2023 •

edited

Loading

costrouc commented Aug 10, 2023 •

edited

Loading

jaimergp commented Aug 10, 2023 •

edited

Loading

costrouc commented Aug 18, 2023 •

edited

Loading

costrouc commented Aug 24, 2023 •

edited

Loading