rm: cannot remove 'target': Device or resource busy

[richb-hanover]

What happened?

This is a refined report from #4974. It appears that the problem with my M2 Mac is that something is wrong with the file system in the Dev Container, and that files/directories cannot be modified.

I have removed all containers, images, and volumes for PRQL and the caches to rebuild the entire Dev Container. I don't intentionally have any external processes running on the Mac that would occupy the Dev Container file system.

@max-sixty @eitsupi @vanillajonathan Here's the evidence I have. What else could I collect? Thanks.

• wasm-pack is not installed, running cargo install wasm-pack gives this error:
  error: failed to create directory /usr/local/cargo/registry/cache/index.crates.io-6f17d22bba15001f

• task web:run-book gives this error:

  vscode ➜ /workspaces/prql (main) $ task web:run-book
  task: [web:run-book] mdbook serve --port=3000 -n 0.0.0.0
  2024-11-12 11:46:14 [INFO] (mdbook::book): Book building has started
      Updating crates.io index
    Downloaded colorchoice v1.0.1
  error: failed to create directory `/usr/local/cargo/registry/cache/index.crates.io-6f17d22bba15001f`
  

• cd /workspaces/prql; rm -rf target gives this error:

vscode ➜ /workspaces/prql (main) $ ls -al
total 296
drwxr-xr-x 35 vscode vscode   1120 Nov 12 00:49 .
drwxr-xr-x  3 root   root     4096 Nov  5 03:29 ..
-rw-r--r--  1 vscode vscode   1924 Oct 19 22:37 bacon.toml
drwxr-xr-x  3 vscode vscode     96 Oct 19 22:37 .cargo
-rw-r--r--  1 vscode vscode 122537 Nov 12 00:49 Cargo.lock
-rw-r--r--  1 vscode vscode   1670 Nov 12 00:49 Cargo.toml
-rw-r--r--  1 vscode vscode  59433 Nov  5 03:42 CHANGELOG.md
drwxr-xr-x  7 vscode vscode    224 Nov  5 03:42 .config
drwxr-xr-x  4 vscode vscode    128 Oct 19 22:37 .devcontainer
-rw-r--r--  1 vscode vscode  10244 Nov 12 03:20 .DS_Store
-rw-r--r--  1 vscode vscode   4571 Nov  5 03:42 flake.lock
-rw-r--r--  1 vscode vscode   2882 Nov  5 03:42 flake.nix
drwxr-xr-x 16 vscode vscode    512 Nov 12 00:49 .git
-rw-r--r--  1 vscode vscode    640 Oct 19 22:37 .gitattributes
drwxr-xr-x 10 vscode vscode    320 Oct 19 22:37 .github
-rw-r--r--  1 vscode vscode    809 Oct 19 22:37 .gitignore
drwxr-xr-x  9 vscode vscode    288 Nov  5 00:45 grammars
-rw-r--r--  1 vscode vscode  10273 Oct 19 22:37 LICENSE
drwxr-xr-x  9 vscode vscode    288 Nov  5 03:35 lutra
-rw-r--r--  1 vscode vscode    535 Oct 19 22:37 .markdownlint-cli2.yaml
-rw-r--r--  1 vscode vscode   1928 Oct 19 22:37 .mega-linter.yaml
drwxr-xr-x 12 vscode vscode    384 Nov  5 05:26 node_modules
-rw-r--r--  1 vscode vscode   3086 Nov 12 00:49 .pre-commit-config.yaml
-rw-r--r--  1 vscode vscode    486 Oct 19 22:37 .prettierignore
-rw-r--r--  1 vscode vscode    183 Oct 19 22:37 .prettierrc.yaml
drwxr-xr-x 10 vscode vscode    320 Nov  5 03:42 prqlc
-rw-r--r--  1 vscode vscode   8076 Nov  5 03:42 README.md
-rw-r--r--  1 vscode vscode    285 Nov  5 03:42 rust-toolchain.toml
-rw-r--r--  1 vscode vscode     62 Oct 19 22:37 .sqlfluff
drwxr-xr-x  2 vscode vscode   4096 Nov 12 11:49 target
drwxr-xr-x  4 vscode vscode    128 Nov  5 03:30 .task
-rw-r--r--  1 vscode vscode  13869 Nov  5 03:42 Taskfile.yaml
-rw-r--r--  1 vscode vscode    546 Nov  5 03:42 .typos.toml
drwxr-xr-x  3 vscode vscode     96 Oct 19 22:37 .vscode
drwxr-xr-x  9 vscode vscode    288 Nov  5 00:45 web
vscode ➜ /workspaces/prql (main) $ ls -al target
total 4
drwxr-xr-x  2 vscode vscode 4096 Nov 12 11:49 .
drwxr-xr-x 35 vscode vscode 1120 Nov 12 00:49 ..
vscode ➜ /workspaces/prql (main) $ rm -rf target
rm: cannot remove 'target': Device or resource busy
vscode ➜ /workspaces/prql (main) $ 

PRQL input

N/A

SQL output

N/A

Expected SQL output

N/A

MVCE confirmation

• Minimal example
• New issue

Anything else?

No response

[vanillajonathan]

This is a file system permission issue. It seems you can around it by running:

sudo chown vscode /usr/local/cargo/registry/

[max-sixty]

• cd /workspaces/prql; rm -rf target gives this error:

Currently we map target into the container. Could you try rm -rf target/* or cargo clean?

[max-sixty]

This is a file system permission issue. It seems you can around it by running:

sudo chown vscode /usr/local/cargo/registry/

Yes, this seems like good advice. For context, we also map this as a volume:

prql/.devcontainer/devcontainer.json

Lines 30 to 34 in 62c7cc3

	{
	"source": "devcontainer-cargo-cache-${devcontainerId}",
	"target": "/usr/local/cargo/registry",
	"type": "volume"
	},

...though only run chown on the target path:

prql/.devcontainer/devcontainer.json

Line 42 in 62c7cc3

"set-ownership": "sudo chown vscode target",

---

For more context, we use a non-root user: https://code.visualstudio.com/remote/advancedcontainers/add-nonroot-user. (I've traditionally used a root user in all my docker containers, but ofc trust @eitsupi 's judgement here — and possibly this is standard for dev containers. I don't think it's PRQL that is setting the user to vscode. You could try "remoteUser": "root" in devcontainer.json as an alternative.)

[richb-hanover]

@max-sixty @eitsupi @vanillajonathan

Partial Answer: See #5018. Changing ownership on /usr/local/cargo/registry seems to have solved the initial problem (I can now build the Playground, Book, and Website in a Dev Container)

But cargo clean still give the "Device or resource busy" message. This doesn't seem to hurt anything, but it sure is surprising.

Thanks for your help

[richb-hanover]

Update: This "Device or resource busy" message now appears on all Dev Container platforms: My Intel macbook, my M2 Mac mini, and a Win11 box.

[vanillajonathan]

@richb-hanover Maybe you need to set ownership of more than just /usr/local/cargo/registry, maybe /usr/local/cargo or even /, I don't know.

[richb-hanover]

Me neither. Here's the state of the file system after removing all containers, etc. then rebuilding the Dev Container from scratch. (This incorporates the PR that sets the owner on /usr/local/cargo/registry)

Do any of the other permissions need to be tweaked? Thanks.

vscode ➜ /workspaces/prql (main) $ ll /usr 
total 60K
drwxr-xr-x 1 root root 4.0K Nov 22 16:12 bin
drwxr-xr-x 2 root root 4.0K Aug 14 16:10 games
drwxr-xr-x 1 root root 4.0K Nov 22 16:11 include
drwxr-xr-x 1 root root 4.0K Nov 22 16:11 lib
drwxr-xr-x 1 root root 4.0K Oct 16 18:01 libexec
drwxr-xr-x 1 root root 4.0K Nov 22 16:11 local
drwxr-xr-x 1 root root 4.0K Nov 22 16:11 sbin
drwxr-xr-x 1 root root 4.0K Nov 22 16:12 share
drwxr-xr-x 1 root root 4.0K Oct 16 18:38 src

vscode ➜ /workspaces/prql (main) $ ll /usr/local 
total 72K
drwxr-xr-x 1 root   root     4.0K Nov 22 16:12 bin
drwxrwsrwx 1 vscode rustlang 4.0K Nov 22 16:12 cargo
drwxr-xr-x 1 root   root     4.0K Oct 16 18:00 etc
drwxr-xr-x 2 root   root     4.0K Sep 26 00:00 games
drwxrwsr-x 3 vscode hugo     4.0K Nov 22 16:10 hugo
drwxr-xr-x 2 root   root     4.0K Sep 26 00:00 include
drwxr-xr-x 1 root   root     4.0K Sep 27 05:18 lib
drwxr-xr-x 3 root   root     4.0K Oct 16 18:37 libexec
lrwxrwxrwx 1 root   root        9 Sep 26 00:00 man -> share/man
drwxrwsr-x 3 vscode python   4.0K Nov 22 16:11 python
drwxrwsr-x 1 vscode pipx     4.0K Nov 22 16:11 py-utils
drwxrwsrwx 1 vscode rustlang 4.0K Sep 27 23:29 rustup
drwxr-xr-x 2 root   root     4.0K Sep 26 00:00 sbin
drwxr-xr-x 1 root   root     4.0K Nov 22 16:11 share
drwxr-xr-x 2 root   root     4.0K Sep 26 00:00 src

vscode ➜ /workspaces/prql (main) $ ll /usr/local/cargo 
total 16K
drwxrwxrwx 1 root   root     4.0K Nov 19 23:31 bin
drwxr-sr-x 2 vscode rustlang 4.0K Nov 19 23:05 binstall
-rw-rw-rw- 1 root   root      308 Sep 27 23:29 env
drwxr-sr-x 5 vscode rustlang 4.0K Nov 22 16:12 registry

vscode ➜ /workspaces/prql (main) $ ll /usr/local/cargo/registry 
total 16K
drwxr-sr-x 3 vscode rustlang 4.0K Nov 22 16:12 cache
-rw-r--r-- 1 vscode rustlang  177 Nov 19 23:05 CACHEDIR.TAG
drwxr-sr-x 3 vscode rustlang 4.0K Nov 22 16:12 index
drwxr-sr-x 3 vscode rustlang 4.0K Nov 22 16:12 src
vscode ➜ /workspaces/prql (main) $ 

[vanillajonathan]

I don't know, but I cannot delete the the target directory with sudo rm -rf target even after I use chmod to set write permissions on it, so maybe the directory is being locked by some process?

I clicked on the rust-analyzer in the VS Code status bar and it could not write to target/debug/.fingerprint/ so maybe that means that rust-analyzer ran before postCreateCommand ran?

Looking at https://containers.dev/implementors/json_reference/#lifecycle-scripts I see that there exists initializeCommand and onCreateCommand that runs before postCreateCommand, maybe we could use that?

[richb-hanover]

Closing because this no longer causes problems. The fix in #5018 seems to have solved my problems