Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISO upgrades to support user namespaces #18488

Open
mbaynton opened this issue Mar 21, 2024 · 12 comments
Open

ISO upgrades to support user namespaces #18488

mbaynton opened this issue Mar 21, 2024 · 12 comments
Labels
area/guest-vm General configuration issues with the minikube guest VM kind/feature Categorizes issue or PR as related to a new feature. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@mbaynton
Copy link

What Happened?

A new feature is nearing readiness in kubernetes/containerd/runc that enables you to run your containers in their own linux user namespace. This is beneficial for security / container isolation and also enables you to do certain operations inside containers that were previously only possible with privileged host permissions.

This feature is set to be promoted to beta in kubernetes 1.30.

It would be great to be able to use minikube to prototype setups that use user namespaces, but lots of much newer software in the ISO will be needed:

Attach the log file

n/a, trying to submit a feature request but can't find a better form

Operating System

Ubuntu

Driver

KVM2

@afbjorklund
Copy link
Collaborator

afbjorklund commented Mar 24, 2024

Sounds like an experimental feature, that doesn't really fit with the LTS versions used in the minikube ISO (or KIC)

Theoretically there could be two image versions, but that means more maintenance (and it is already struggling)

@afbjorklund afbjorklund added kind/feature Categorizes issue or PR as related to a new feature. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/discuss Items for discussion area/guest-vm General configuration issues with the minikube guest VM labels Mar 24, 2024
@afbjorklund
Copy link
Collaborator

KEP 127 looks completely untested/unimplemented for Docker, which is still the minikube default container runtime...

For Beta, the feature is tested for containerd and CRI-O in cri-tools repo using critest

@rata
Copy link
Member

rata commented Mar 25, 2024

@afbjorklund if it helps to make a decision, it doesn't seem it will be implemented in cri-dockerd anytime soon, as docker has some limitations. I've been trying them to implement it for years now, and although they like the idea, there was no progress so far: Mirantis/cri-dockerd#74

@mbaynton
Copy link
Author

doesn't really fit with the LTS versions used in the minikube ISO

Yeah at the moment it's definitely too soon to build an updated ISO for general use that enables this. I'm optimistic releases of containerd and runc will be available soon though :). But everything needs to be marked as an LTS release as well?

@afbjorklund
Copy link
Collaborator

afbjorklund commented Mar 25, 2024

But everything needs to be marked as an LTS release as well?

It is not an absolute rule, but historically minikube used Ubuntu LTS as the base for decisions for the Buildroot versions.
Other distributions such as kind, were doing latest/greatest (or Debian) and rebuilding containerd on a nightly basis...

Ubuntu Linux
16.04 4.4
18.04 4.15
20.04 5.4
22.04 5.15
24.04 6.8 (?)

But after that it was pretty arbitrary, stayed on 4.19 for a while since it was stable and then jumped to 5.10 (770d41f).
One could go for a similar jump to 6.x, perhaps timed with the OS update. The runtime versions are even more arbitrary.


The versions of containerd and runc usually came (bundled) with Docker...

https://docs.docker.com/engine/install/binaries/ (or get.docker.com)

  • dockerd 26.0.0
    • containerd 1.7.13
    • runc 1.1.12

@afbjorklund
Copy link
Collaborator

afbjorklund commented Mar 25, 2024

Note: Buildroot 2024.02 (LTS) supports these kernels:

# From https://www.kernel.org/pub/linux/kernel/v6.x/sha256sums.asc
sha256  4e43d8c5fba14f7c82597838011648056487b7550fd83276ad534559e8499b1d  linux-6.6.18.tar.xz
sha256  faa49ca22fb55ed4d5ca2a55e07dd10e4e171cfc3b92568a631453cd2068b39b  linux-6.1.79.tar.xz
# From https://www.kernel.org/pub/linux/kernel/v5.x/sha256sums.asc
sha256  bd84809a367eb400eb04e0e70294e6ba12fc03b6bfb5a7dfaca548f8947501b0  linux-5.15.149.tar.xz
sha256  4ea63c5a90fdc3c459ab35c11ee8c93d2364a7cdbfb101100f8cab70d490ef6d  linux-5.10.210.tar.xz
sha256  ff54bec6d053c7994f3bb8c45021de2858ff9f740d2ccbbcf072b87821a918cf  linux-5.4.269.tar.xz
# From https://www.kernel.org/pub/linux/kernel/v4.x/sha256sums.asc
sha256  83eeff613405d0045d0f717c6ac14c178678fe0a163c41d9dd8878ac0f73e352  linux-4.19.307.tar.xz

See https://buildroot.org/news.html (and 2024.02.x)

Going with Kernel 5.15 would be the more obvious choice.

@mbaynton
Copy link
Author

mbaynton commented Apr 2, 2024

The user namespace stuff needs features from newer kernels than 5.15. I've had success with 6.6, apparently you can go back as far as 6.3 but it looks like that's not an option for buildroot.

@rata
Copy link
Member

rata commented Apr 3, 2024

@mbaynton it should work fine with 5.12+ but you can't use tmpfs volumes (so service account tokens can't really work) and you have a higher pod start latency and storage overhead if you use 5.12 too.

@k8s-triage-robot

This comment was marked as outdated.

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 2, 2024
@k8s-triage-robot

This comment was marked as outdated.

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 1, 2024
@rata
Copy link
Member

rata commented Aug 1, 2024

/remove-lifecycle rotten

containerd 2.0 and runc 1.2 should be released this year with proper support. It seems worth keeping this open a little longer

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 1, 2024
@medyagh
Copy link
Member

medyagh commented Aug 26, 2024

I agree we should have a tutorial that lets ppl build their own ISO easily
and basicly if they want containerd 2.0 can do minkube start --iso-url=file://....

and not part of released minikube ISOs

@afbjorklund afbjorklund added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/discuss Items for discussion labels Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/guest-vm General configuration issues with the minikube guest VM kind/feature Categorizes issue or PR as related to a new feature. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants