Cannot connect to datastore with network isolation

[alpharde]

What happened?

The vcluster pod cannot connect to external databases while network isolation is enabled.

What did you expect to happen?

vcluster being able to reach the external server.

How can we reproduce it (as minimally and precisely as possible)?

Create vcluster with

(...)
vcluster:
  env:
    - name: K3S_DATASTORE_ENDPOINT
      value: mysql://user:pass@tcp(10.0.4.139:3306)/database
(...)
isolation:
  enabled: true   
  resourceQuota:
    enabled: true
    quota:
      services.nodeports: 12
      services.loadbalancers: 12
      requests.storage: "256Gi"
  networkPolicy:
    enabled: true
    outgoingConnections:
      ipBlock:
        cidr: 0.0.0.0/0
        except:
          - 10.0.4.139/32

Anything else we need to know?

vcluster log:
2023-12-26 19:39:50 INFO loghelper/klog.go:24 starting kubernetes: preparing server: creating storage endpoint: building kine: dial tcp 10.0.4.139:3306: connect: connection refused {"component": "vcluster", "component": "k3s", "time": "2023-12-26T19:39:50Z", "level": "fatal"}

Host cluster Kubernetes version

Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.5+k3s1

Host cluster Kubernetes distribution

k3s

vlcuster version

vcluster version 0.18.1

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k3s

OS and Arch

Debian bookworm (amd64)

[ishankhare07]

hi @alpharde , thanks for creating this issue. To me it seems that 10.0.4.139 is the IP of the mysql DB and you want the vcluster to use this DB as the storage. The vcluster apiserver needs to be able to access this IP

So in context of vcluster APIServer there will outgoing connections from vcluster control plane to this IP.
However if you check the network policy in the helm chart, the except block is only used in case of vcluster workloads and not for control plane.

So my guess is as of now, isolated mode and external datastore aren't supposed to work in conjunction