K3s v1.31.2-k3s1 -> v1.31.3-k3s1 breaking /dev/net/tun setup (Investigation)

[robjackstewart]

Im running Gluetun in a K3d cluster and bumping the K3s image from rancher/k3s:v1.31.3-k3s1 to rancher/k3s:v1.31.2-k3s1 causes the Glueton container to fail to setup the /dev/net/tun device. For context, the bump was done via renovate.

I suspect the issue is likely to be upstream of Gluetun but since Gluetun is the only way the issue is manifesting for me, I figured it was best to open a discussion here in case anyone else is having the same issue. Ill do some debugging to see if additional capabilities are required and report back.

Details of error below.

Error on container start:

2024-12-08T12:51:29Z INFO [routing] default route found: interface eth0, gateway 10.42.0.1, assigned IP 10.42.0.15 and family v4
2024-12-08T12:51:29Z INFO [routing] adding route for 0.0.0.0/0
2024-12-08T12:51:29Z INFO [firewall] setting allowed subnets...
2024-12-08T12:51:29Z INFO [routing] default route found: interface eth0, gateway 10.42.0.1, assigned IP 10.42.0.15 and family v4
2024-12-08T12:51:29Z INFO TUN device is not available: open /dev/net/tun: no such file or directory; creating it...
2024-12-08T12:51:29Z INFO [routing] routing cleanup...
2024-12-08T12:51:29Z INFO [routing] default route found: interface eth0, gateway 10.42.0.1, assigned IP 10.42.0.15 and family v4
2024-12-08T12:51:29Z INFO [routing] deleting route for 0.0.0.0/0
2024-12-08T12:51:29Z ERROR creating tun device: unix opening TUN device file: operation not permitted (did you specify --device /dev/net/tun to your container command?)
2024-12-08T12:51:29Z INFO Shutdown successful

Making the container privileged fixes this problem but this shouldn't be the solution.

# This worked using the rancher/k3s:v1.31.2-k3s1 image but does not work in the rancher/k3s:v1.31.3-k3s1 image
# ...
containers:
  - name: gluetun
     image: ghcr.io/qdm12/gluetun:latest
     securityContext:
       capabilities:
         add:
           - NET_ADMIN
# ...

# ...
containers:
  - name: gluetun
     image: ghcr.io/qdm12/gluetun:latest
     securityContext:
       privileged: true # <--
       capabilities:
         add:
           - NET_ADMIN
# ...

Looking at the changelog for k3s v1.31.3-k3s1, I cant see anything that could affect this.

Im going to try applying the documented fix for LXC container and report back.

[robjackstewart]

I wasnt able to apply the fix. Here are some stripped down files if anyone happens to want to recreate the problem.

- k3d-config.yaml
- templates/
  - gluetun.yaml
- Chart.yaml

# k3d-config.yaml
apiVersion: k3d.io/v1alpha5
kind: Simple 
metadata:
  name: k3s-1.31.3-test
servers: 1
agents: 0
image: rancher/k3s:v1.31.2-k3s1 # changing to rancher/k3s:v1.31.3-k3s1 causes aforementioned error
network: test-network

# Chart.yaml
apiVersion: v2
name: k3s-1.31.3-test
version: 0.0.1
appVersion: v0.0.1

# templates/gluetun.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gluetun
spec:
  selector:
    matchLabels:
      app: transmission
  template:
    metadata:
      labels:
        app: transmission
    spec:
      containers:
        - name: gluetun
          image: ghcr.io/qdm12/gluetun:latest
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
          env:
            - name: VPN_SERVICE_PROVIDER
              value: mullvad
            - name: VPN_TYPE
              value: openvpn
            - name: OPENVPN_USER
              value: # redacted
            - name: OPENVPN_PASSWORD
              value: m
            - name: SERVER_COUNTRIES
              value: UK
            - name: OWNED_ONLY
              value: 'yes'

[robjackstewart]

I've created this repo for reproduction and raised this issue with K3s: k3s-io/k3s#11429

[robjackstewart]

Turns out this is a change in runc with the version bump cascading up via the a series of packages to the rancher image. containers will have to be privileged for now if you want to use k8s >1.31.2

[robjackstewart]

The wiki is being updated to reflect this.