Critical libmamba filesystem error due to permissions on caches (Operation not permitted)

[darynwhite]

Troubleshooting docs

• My problem is not solved in the Troubleshooting docs

Anaconda default channels

• I do NOT use the Anaconda default channels (pkgs/* etc.)

How did you install Mamba?

Mambaforge or latest Miniforge

Search tried in issue tracker

critical libmamba filesystem error: in permissions:

Latest version of Mamba

• My problem is not solved with the latest version

Tried in Conda?

I do not have this problem with Conda, just with Mamba

Describe your issue

After updating mamba to 2.0.4 to solve the mamba:8: command not found: __conda_reactivate issue, mamba completely fails to do anything with packages.

mamba info / micromamba info

libmamba version : 2.0.4
          mamba version : 2.0.4
           curl version : libcurl/8.10.1 OpenSSL/3.4.0 (SecureTransport) zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.64.0
     libarchive version : libarchive 3.7.4 zlib/1.2.13 liblzma/5.2.6 bz2lib/1.0.8 liblz4/1.9.3 libzstd/1.5.6
       envs directories : /Users/white/conda/envs
          package cache : /Users/white/conda/pkgs
                          /Users/white/.mamba/pkgs
            environment : work (active)
           env location : /Users/white/conda/envs/work
      user config files : /Users/white/.mambarc
 populated config files : /Users/white/.condarc
                          /Users/white/conda/.condarc
       virtual packages : __unix=0=0
                          __osx=15.1.1=0
                          __archspec=1=arm64
               channels : https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/conda-forge/osx-arm64
       base environment : /Users/white/conda
               platform : osx-arm64

Logs

❯ mamba update -v --all
info     libmamba Searching index cache file for repo 'conda-forge/noarch'
info     libmamba No valid cache found
info     libmamba Searching index cache file for repo 'conda-forge/osx-arm64'
info     libmamba No valid cache found
critical libmamba filesystem error: in permissions: Operation not permitted ["/Users/white/conda/pkgs/cache"]

environment.yml

No response

~/.condarc

auto_update_conda: false
auto_activate_base: false
channels:
  - conda-forge

[jjerphan]

Can you report the file permissions of /Users/white/conda/pkgs/cache?

[darynwhite]

Ah, yes.. apologies:

❯ ls -la
total 544728
drwxrwxr-x    10 white  staff        320 Dec  5 22:25 .
drwxr-xr-x  1026 white  staff      32832 Dec  5 22:30 ..
-rw-r--r--     1 white  staff        464 Dec  5 22:25 09cdf8bf.info.json
-rw-r--r--     1 white  staff  112772924 Dec  5 22:25 09cdf8bf.json
-rw-r--r--     1 white  staff   33936137 Dec  5 22:25 09cdf8bf.solv
-rw-r--r--     1 white  staff        375 Dec  5 22:25 09cdf8bf.state.json
-rw-r--r--     1 white  staff        467 Dec  5 22:25 a850f475.info.json
-rw-r--r--     1 white  staff  105170726 Dec  5 22:25 a850f475.json
-rw-r--r--     1 white  staff   26994212 Dec  5 22:25 a850f475.solv
-rw-r--r--     1 white  staff        378 Dec  5 22:25 a850f475.state.json

[fperez]

I just ran into this same problem - was doing a clean install with the exact same script/worfklow I've used dozens of time, including earlier this week. Something went belly up very recently...

And same here as @darynwhite is reporting, I'm on a mac with conda installed in my home directory, with all paths owned by my user, and not having changed anything otherwise.

[darynwhite]

@fperez I'm glad to hear this, because I tried deleting and rebuilding my mamba/conda build about a half dozen times trying to figure out if it was something I switched by accident.

I do have mamba running stably at 1.5.9 for now.

[fperez]

Yup @darynwhite - I also ended up pinning mamba to 1.5.9 so I could actually do anything. Right now doing a single mamba update after a miniforge install breaks everything irreparably.

[cpelley]

Potential duplicate of #3606

[SylvainCorlay]

Thanks all. We are actively looking into it.

For now, we are not reproducing the issue (I also work on OS X and have been primarily using micromamba 2.0 since the release candidates). I tried various scenarios with miniforge, install some packages, updating mamba to 2.0, installing some (old) packages again and requesting an update of all, and things appear to work for now without any issue.

[SylvainCorlay]

Note that this says "Operation not permitted" rather than "Permission denied".

I wonder if there are multiple filesystems involved in your setup @darynwhite ? Hard links cannot be created across file systems, and it's possible that conda has a fall back for a full copy.

[darynwhite]

@SylvainCorlay thanks for looking into this.

It is only when I upgrade to 2.0.4 that the problem arises. I haven't tested any other 2.0.x version yet.

I don't know of any hard links as this is my local build within in my user directory. I don't think I even have symbolic links anywhere. I run it out of a dedicated directory that I can rebuild at will. See an example here: MambaTooling

@cpelley I did read over #3606 but as this isn't a shared setup or system, I didn't think there was direct overlap.

[SylvainCorlay]

Thanks. I was refering to hard links because conda and mamba use them internally. (Basically, if a package is installed in two environments, it only uses the disk space once). Hard links don't work across file systems, so if the environment is not on the same drive as the package cache it would not work, hence my question. From an earlier conversation with Fernando who reported the same issue, it's not the problem.

I am still looking into it. I also am an OS X user and am using micromamba or mamba 2.0 since the RCs - so I don't have a reproducer yet.

[durack1]

@SylvainCorlay looks like I might be your reproducer.. Just ran a mamba update mamba and have tripped over myself just like the above, 2.0.4 is now giving me

machine:~ user$ which mamba
/userpath/miniforge3/condabin/mamba
machine~ user$ mamba create -n aa --dry-run -c conda-forge intake-esgf
warning  libmamba You have not set the root prefix environment variable.
    To permanently modify the root prefix location, either:
      - set the 'MAMBA_ROOT_PREFIX' environment variable
      - use the '-r,--root-prefix' CLI option
      - use 'mamba shell init ...' to initialize your shell
        (then restart or source the contents of the shell init script)
    Continuing with default value: "/userpath/miniforge3"
critical libmamba filesystem error: in permissions: Operation not permitted ["/userpath/miniforge3/pkgs/cache"]
machine:~ user$ mamba create --dry-run -n aa -c conda-forge intake-esgf
warning  libmamba You have not set the root prefix environment variable.
    To permanently modify the root prefix location, either:
      - set the 'MAMBA_ROOT_PREFIX' environment variable
      - use the '-r,--root-prefix' CLI option
      - use 'mamba shell init ...' to initialize your shell
        (then restart or source the contents of the shell init script)
    Continuing with default value: "/userpath/miniforge3"
critical libmamba filesystem error: in permissions: Operation not permitted ["/userpath/miniforge3/pkgs/cache"]

[jjerphan]

I cannot reproduce the issue using mamba 2.0.4 with a new miniforge installation on macOS 14.

In particular, I cannot reproduce the issue given #3661 (comment).

[jjerphan]

Edit: this is not due to System Integrity Protection, it is enabled on my machine but I cannot reproduce the issue.

Do you still have the problem if you grant mamba or micromamba 2.0.4 Full Disk Access (which can be done via the System Preferences on macOS)?

[durack1]

Can you report if you have System Integrity Protection enabled?

@jjerphan I believe yes is the answer.

machine:~ user$ csrutil status
System Integrity Protection status: enabled.

Regarding the update path, the issue was caused in issuing the below, which begins with installing Miniforge3-24.7.1-2-MacOSX-arm64.sh (available here).

machine:~ user$ mamba update mamba 

Looking for: ['mamba']

conda-forge/osx-arm64                               13.4MB @ 566.8kB/s 23.6s
conda-forge/noarch                                  17.7MB @ 562.2kB/s 31.5s

Pinned packages:
  - python 3.12.*


Transaction

  Prefix: /userpath/miniforge3

  Updating specs:

   - mamba
   - ca-certificates
   - certifi
   - openssl


  Package                       Version  Build            Channel           Size
──────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────

  + cpp-expected                  1.1.0  hffc8910_0       conda-forge       24kB
  + simdjson                     3.10.1  h7b3277c_0       conda-forge      201kB
  + spdlog                       1.14.1  h6d8af72_1       conda-forge      164kB
  + nlohmann_json                3.11.3  h00cdb27_1       conda-forge      123kB

  Upgrade:
──────────────────────────────────────────────────────────────────────────────────

  - openssl                       3.3.2  h8359307_0       conda-forge     Cached
  + openssl                       3.4.0  h39f12f2_0       conda-forge        3MB
  - reproc                 14.2.4.post0  h93a5062_1       conda-forge     Cached
  + reproc                 14.2.5.post0  h5505292_0       conda-forge       32kB
  - reproc-cpp             14.2.4.post0  h965bd2d_1       conda-forge     Cached
  + reproc-cpp             14.2.5.post0  h286801f_0       conda-forge       25kB
  - libmamba                     1.5.10  h66a2e1b_1       conda-forge     Cached
  + libmamba                      2.0.4  h4621f14_0       conda-forge        1MB
  - libmambapy                   1.5.10  py312he1e5f57_1  conda-forge     Cached
  + libmambapy                    2.0.4  py312hd07f1d4_0  conda-forge      535kB
  - mamba                        1.5.10  py312h14bc7db_1  conda-forge     Cached
  + mamba                         2.0.4  h4ae28c3_0       conda-forge      373kB
  - conda-libmamba-solver        24.7.0  pyhd8ed1ab_0     conda-forge     Cached
  + conda-libmamba-solver       24.11.1  pyhd8ed1ab_0     conda-forge       41kB

  Summary:

  Install: 4 packages
  Upgrade: 7 packages

  Total download: 6MB

[SylvainCorlay]

Mamba update mamba works for me and Julien locally, on various OS X systems.

We are launching a reproduction challenge with everyone who has an OS X system on the team.

[durack1]

Do you still have the problem if you grant mamba or micromamba 2.0.4 Full Disk Access (which can be done via the System Preferences on macOS)?

Apologies @jjerphan I am a little lost about this - I install using the Miniforge3* installer as noted in #3661 (comment), with mamba or micromamba is this an alternate installer?

[darynwhite]

Just checked with mamba == 2.0.5:

❯ mamba update --all -y -vv
debug    libmamba Loading configuration
debug    libmamba 118 configurables computed
debug    libmamba Checking if '/Users/white/conda/pkgs' is writable
debug    libmamba '/Users/white/conda/pkgs' writable
info     libmamba Searching index cache file for repo 'conda-forge/noarch'
info     libmamba No valid cache found
info     libmamba Searching index cache file for repo 'conda-forge/osx-arm64'
info     libmamba No valid cache found
critical libmamba filesystem error: in permissions: Operation not permitted ["/Users/white/conda/pkgs/cache"]