zstandard related install failures (need zstandard 0.21?) #241
Comments
analog-cbarber
added
the
type::bug
|
Not sure whether a conda-forge version of zstandard 0.21 will fix all of this. also see: |
|
I am also curious about the note: However, I don't think this is strictly conda-libmamba-solver related, so this might belong better in conda proper or conda-package-{handling,streaming}. That warning comes from this block: , which is swallowing the actual exception. I am going to see if I can inspect the error there, locally. |
|
0.19 Works ok on my machine: So I think it might have to do with the fact that you are mixing |
|
To debug, please add the following statements to your CI config, before the problematic line: In the meantime, I am going to see if I can reproduce with the Docker image. |
|
Yea, it's an ABI incompatibility:
|
|
I can fix this by dropping |
|
Thanks for responding. I don't actually want the anaconda defaults but didn't bother to remove them or choose a different docker image, but switching to condaforge/miniforge3 does fix the issue. Although not recommended, we have getting away with mixing anoconda and conda-forge in our CI jobs for years now, so obviously this kind of problem is not that common, perhaps because most low-level libraries have usually been compatible between the two. I wonder if better error or warning behavior would be useful here.... |
|
So using miniforge3 fixes my toy reproducible example but when I try it on my real repo, I now am getting a I will see if I can get a reproducible case I can give you. Note that we are installing a package from our own custom channel here: |
|
That parse state error goes away when I remove the conda autoupdate restriction. |
|
@dholth interesting thing, have you seen this before? |
|
I have seen problems with zstd/zstandard mismatch before but not quite like this.
|
|
See also conda-forge/zstandard-feedstock#48 |
|
conda-forge's zstandard uses an external zstd, but defaults' uses the bundled one. conda-forge's zstandard is picking defaults' zstd, which is not guaranteed to be the same because the previous builds (<_2) did not pin zstd like build number 2 does. I don't know if we can reliably get a repodata patch for the previous builds (might need to inspect the rendered recipe inside the package for each one). |
|
Added this patch to prevent this incompatibility from occurring just in case: conda-forge/conda-forge-repodata-patches-feedstock#484 |
|
The patch above was merged so this particular issue is now fixed. Note that other conflicts in dependencies when defaults and conda-forge are mixed (e.g. libarchive) are still possible, though. |
|
With zstandard 0.19.0 in my base environment, conda showed a warning. conda also wouldn't update and asserted that "All requested packages already installed" (but not their updated versions). All have been fixed by first updating zstandard. P.S. now I'm looking for the feedstock where I can request updating https://anaconda.org/intel/zstandard |
|
No @dholth, both of those feedstocks show the current release's version is 0.21.0. Intel's feedstock has packages (including an older zstandard) for its version of python. |
|
I see that the AnacondaRecipes version is still 0.19. I'm confused about whether they bundle or link to zstd, since I don't see any recent recipe changes to that effect, but I remember a discussion about the license and changing to a link model. I also see a conditional in the conda-forge recipe, seems like the comment could be left out on this line |
|
@dholth anacondarecipes shows 0.21.0, but the public release is the older version. I don't install from anaconda's channel, but I think you're suggesting that Intel may be forked from anaconda's channel. P.S. you commented about zstd's license in a closed issue. |
|
I see https://github.com/AnacondaRecipes/zstandard-feedstock/blob/master/recipe/meta.yaml#L2 The forked recipe appears to be linking to a conda-forge status badge. I also see that zstandard includes a cffi backend and an ordinarily-used "c extension" or cext backend ( |
|
Note that you can get into this incompatible-versions state without needing to mix channels; running |
Checklist
What happened?
In our CI builds, we noticed that after configuring the libmamba solver and installing certain packages in the base environment that we get failures installing some packages that appear to be related to the failure to import zstandard.
For example:
Running the following CI script (this is using gitlab CI):
The last step fails to find the package. There are warnings about not being able to import zstandard,
and that might be leading to the failure to find packages.
Note that the most recent zstandard from conda-forge is 0.19, but the latest from pip is 0.21.
If I install the pip one just before the conda create, it appears to fix the problem:
The full CI log is:
The text was updated successfully, but these errors were encountered: