Ecosystem-wide test suite for solver conformance

[jaimergp]

Checklist

• I added a descriptive title
• I searched open issues and couldn't find a duplicate

What?

We propose creating a backend agnostic collection of test cases so different members of the ecosystem (e.g. conda’s own classic and libmamba solvers, but also external tools like micromamba and pixi) can assess their compliance to the underspecified solver behavior “expected” by the conda community.

Why?

• The solver tests for conda are spread across different parts of the test suite as unit and integration tests. In many cases, they incur IO overheads that are not necessary and cause a general slowdown of the test suite completion and development feedback loop.
• The PyPI ecosystem has already set a precedent in this aspect (see https://github.com/zanieb/packse and https://github.com/pradyunsg/pip-resolver-benchmarks), and the conda ecosystem should also step up.

User impact

More consistent behavior across solvers

Goals

• Set a dedicated repository with solver test cases
• Integrate it in conda/conda
• Integrate it in conda/conda-libmamba-solver

Tasks

• Create repository
• Design test case schema. Should be declarative and language agnostic (e.g. YAML or JSON)
• Create / consolidate test data (repodata, channels, etc) so there's no need to access live channels.
• Audit conda/conda for potential test cases
• Audit conda/conda-libmamba-solver for potential test cases
• Audit mamba-org/mamba for potential test cases
• Audit mamba-org/rattler for potential test cases
• Consolidate audits and write table of test cases with assumed inputs and expected outcomes
• Implement test cases

This epic is blocked by:

N/A

This epic blocks:

N/A

[jjerphan]

Thank you for starting the discussions, @jaimergp.

Set a dedicated repository with solver test cases

Which GitHub organization should welcome this repository to you?

[jaimergp]

Which GitHub organization should welcome this repository to you?

I was assuming we would start in conda-incubator.

[jjerphan]

List of known edge cases (feel free to edit this comment):

• Really old ipopt installed by pixi when installing "assimp boost ipopt vtk" prefix-dev/pixi#1708
• OpenMPI dependency resolving to external build conda-forge/openmpi-feedstock#153
• Co-installing openmpi and pocl prevents pocl build 5 from being used conda-forge/pocl-feedstock#101
• external variants pulled in first conda-forge/mpich-feedstock#95
• Solving mlflow=2.12.2=py39hb3b8efb_0 takes a lot of time rattler#684
• Known hard environments to solve: https://github.com/conda-forge/rubinenv-feedstock/blob/main/recipe/meta.yaml
• Submitted by Jaime: https://github.com/conda/conda-libmamba-solver/blob/155fe28acbd5cd2f57fd0259c5180fbf4daefa6f/tests/test_performance.py#L131
• Submitted by Jaime: https://github.com/conda/conda-libmamba-solver/blob/main/tests/test_solver_differences.py
• https://github.com/mamba-org/resolvo/pull/61/files#diff-75b7e1b2ff6f03636607e715f060aabb96639be642130ad9f4128b1cb3614377R1399-R1433
• mamba-org/mamba/blob/main/libmamba/tests/src/solver/libsolv/test_solver.cpp
• Downgrades on repo2docker:
  • tests/conda/downgrade/verify::verify
  • tests/conda/py-r-postbuild-file/verify::verify
  • tests/conda/py2/verify::verify
  • tests/conda/py310-requirements-file/verify::verify
  • tests/conda/py35-binder-dir/verify::verify

[jjerphan]

Design test case schema. Should be declarative and language agnostic (e.g. YAML or JSON)

Would conda environment YAML specifications be sufficient?

[jaimergp]

I'm afraid that's only one possible input. I was picturing a schema where we can express things like:

• Input packages:
  • List of MatchSpecs
  • Installed packages
  • History
  • Aggressive updates
  • Pins
  • Non-manageable dependencies
• Action(s) (create/install/update/remove)
• Solver flags (no-deps, only-deps, update-all, freeze-installed)
• Channels and priorities
• Subdirs
• Expected list of records (fully resolved environment; aka lockfile contents)

A single test case might take more than one action.

[jjerphan]

Thank you for those pieces of information.

For full disclosure and as discussed with @zklaus, I am interested in this PR being resolved, but I am leaving Klaus resolve it since he is assigned to it.

[jaimergp]

@zklaus has started a repo with some prototypes and pytest plugins at https://github.com/zklaus/pytest-conda-solvers.