kubectl exec through kube-oidc-proxy fails

[smurfralf]

I have a working kube-oidc-proxy instance but when I try to run kubectl exec -it I get a failure. Is kubectl exec -it supported by the proxy?

Here is what I ran and the result:

$ kubectl exec -it mypod -n mynamespace -- /bin/sh
 error: error sending request: Post https://kube-oidc-proxy.mydomain.com/api/v1/namespaces/mynamespace/pods/mypod/exec?command=%2Fbin%2Fsh&container=maincontainer&stdin=true&stdout=true&tty=true: EOF

If make a request directly to the apiserver, it succeeds and I get a shell prompt inside the pod. But of course they are different users, since the direct call is using x509 authentication. However the kube-oidc-proxy user has a role with wildcards for every rule, I don't think permissions are the cause. There are no entries related to the request in the kube-oidc-proxy pod logs.

The image I'm using is quay.io/jetstack/kube-oidc-proxy:v0.2.0

[JoshVanL]

Hi @smurfralf,

This indeed is a network issue rather than an auth one from what I can see. The latest release v0.3.0 included a change to enable a flush interval for long running connections however it looks like we are failing on a POST request here which is odd.

Would you be able to upgrade the image tag (there should't be any breaking changes), and increase the log level of the proxy (--v=10), and report back with the output? (don't forget it remove any secrets! 😬).

[hazmei]

Hi @JoshVanL I'm experiencing the same issue however with a different error message.

hazmei.ar@Hazmeis-MBP charts % k exec dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc sh
Error from server (BadRequest): Upgrade request required
hazmei.ar@Hazmeis-MBP charts % k port-forward dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc 8080:80
error: error upgrading connection: Upgrade request required

I increased the log level of the proxy (--v=10) and this is the output for those requests

I0724 04:35:42.701980       1 handlers.go:51] authenticated request: 10.211.1.90:53230
I0724 04:35:42.702250       1 round_trippers.go:423] curl -k -v -XPOST  -H "Content-Length: 0" -H "X-Forwarded-For: 172.31.0.195, 10.211.1.90" -H "Impersonate-User: [email protected]" -H "X-Amzn-Trace-Id: Root=xxx" -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "X-Forwarded-Proto: https" -H "User-Agent: kubectl/v0.0.0 (darwin/amd64) kubernetes/$Format" -H "X-Forwarded-Port: 443" -H "Impersonate-Group: [email protected]" -H "Impersonate-Group: [email protected]" -H "Impersonate-Group: system:authenticated" -H "Authorization: Bearer authorizationbearerhere" 'https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/exec?command=sh&container=dex-k8s-authenticator&stderr=true&stdout=true'
I0724 04:35:42.710329       1 round_trippers.go:443] POST https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/exec?command=sh&container=dex-k8s-authenticator&stderr=true&stdout=true 400 Bad Request in 8 milliseconds
I0724 04:35:42.710360       1 round_trippers.go:449] Response Headers:
I0724 04:35:42.710367       1 round_trippers.go:452]     Audit-Id: a2f43851-ba92-4321-9f80-ed8913483f36
I0724 04:35:42.710373       1 round_trippers.go:452]     Cache-Control: no-cache, private
I0724 04:35:42.710378       1 round_trippers.go:452]     Content-Type: application/json
I0724 04:35:42.710383       1 round_trippers.go:452]     Date: Fri, 24 Jul 2020 04:35:42 GMT
I0724 04:35:42.710388       1 round_trippers.go:452]     Content-Length: 139

I0724 04:35:52.510581       1 handlers.go:51] authenticated request: 10.211.0.205:10340
I0724 04:35:52.510735       1 round_trippers.go:423] curl -k -v -XPOST  -H "Impersonate-User: [email protected]" -H "X-Forwarded-Proto: https" -H "X-Forwarded-Port: 443" -H "Impersonate-Group: [email protected]" -H "Impersonate-Group: [email protected]" -H "Impersonate-Group: system:authenticated" -H "X-Forwarded-For: ip-address-1, ip-address-1" -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "X-Amzn-Trace-Id: Root=xxx" -H "Authorization: Bearer authorizationbearerhere" -H "Content-Length: 0" -H "User-Agent: kubectl/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/portforward'
I0724 04:35:52.523721       1 round_trippers.go:443] POST https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/portforward 400 Bad Request in 12 milliseconds
I0724 04:35:52.523836       1 round_trippers.go:449] Response Headers:
I0724 04:35:52.523844       1 round_trippers.go:452]     Content-Length: 139
I0724 04:35:52.523852       1 round_trippers.go:452]     Audit-Id: d5bc1a29-b3b0-4cd3-9905-e8ee79ddb340
I0724 04:35:52.523856       1 round_trippers.go:452]     Cache-Control: no-cache, private
I0724 04:35:52.523861       1 round_trippers.go:452]     Content-Type: application/json
I0724 04:35:52.523913       1 round_trippers.go:452]     Date: Fri, 24 Jul 2020 04:35:52 GMT

The proxy is running on EKS behind an ALB. Besides exec and port-forward, i'm able to run the other kubectl commands like kubectl get pods/deploy/secrets/..., kubectl logs ....

[sebastienc]

Hi, we've got the same issue. Tried to expose the service on ELBs and ALBs and get the same output as the previous comment (except for hostnames). It works for everything except for kubectl exec and kubectl port-forward. Please let me know if you need any kind of logs. We are running 0.3.0.

[hazmei]

Hi, we've got the same issue. Tried to expose the service on ELBs and ALBs and get the same output as the previous comment (except for hostnames). It works for everything except for kubectl exec and kubectl port-forward. Please let me know if you need any kind of logs. We are running 0.3.0.

Hi @sebastienc . You can't run it behind ALB / ELB in AWS. kubectl exec and kubectl port-forward uses SPDY protocol which is not supported by those load balancers. I solve this by exposing the service on NLB.

[abdennour]

@hazmei My kube-apiserver is exposed thru nginx ingress.
And the nginx ingress is behind HAProxy (corporate Loabalancer on-prem)
As per your comment, i need to make sure that SPDY protocol is configured in :

• nginx ingress
• HAProxy

My question is how to configure this protocol in these 2 layers?

[febef]

Hi, I also have the same issue, but the log is different. I'm getting a 403 error. I'm using kube-oidc-proxy 0.3.0. I can't do port-forwarding. I'm using Istio -> Nginx Ingress -> kube-oidc-proxy.

[febef]

here are the logs:
I0518 02:30:38.052646 88338 loader.go:373] Config loaded from file: ./kc.yaml
I0518 02:30:38.061036 88338 round_trippers.go:463] GET https://k8s/api/v1/namespaces/argocd/services/argocd-server
I0518 02:30:38.061052 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.061059 88338 round_trippers.go:473] Accept: application/json, /
I0518 02:30:38.061063 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.749894 88338 round_trippers.go:574] Response Status: 200 OK in 688 milliseconds
I0518 02:30:38.752714 88338 round_trippers.go:463] GET https://k8s/api/v1/namespaces/argocd/pods?labelSelector=app.kubernetes.io%2Fname%3Dargocd-server
I0518 02:30:38.752751 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.752769 88338 round_trippers.go:473] Accept: application/json, /
I0518 02:30:38.752781 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.780801 88338 round_trippers.go:574] Response Status: 200 OK in 27 milliseconds
I0518 02:30:38.790906 88338 round_trippers.go:463] GET https://k8s/api/v1/namespaces/argocd/pods/argocd-server-777945846-jrsld
I0518 02:30:38.790929 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.790937 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.790944 88338 round_trippers.go:473] Accept: application/json, /
I0518 02:30:38.806739 88338 round_trippers.go:574] Response Status: 200 OK in 15 milliseconds
I0518 02:30:38.807808 88338 round_trippers.go:463] POST https://k8s/api/v1/namespaces/argocd/pods/argocd-server-777945846-jrsld/portforward
I0518 02:30:38.807816 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.807823 88338 round_trippers.go:473] X-Stream-Protocol-Version: portforward.k8s.io
I0518 02:30:38.807829 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.830910 88338 round_trippers.go:574] Response Status: 403 Forbidden in 23 milliseconds

[sickdyd]

I also get 403 when trying to use k port-forward or when trying to use k exec to connect to the proxy pod.

Any solution for this?

k port-forward -n xxx pod/redis-master-0 6379:6379 --v=6
I0921 17:31:52.061926   18020 loader.go:373] Config loaded from file:  /Users/xxx/.kube/config
I0921 17:31:52.634821   18020 round_trippers.go:553] GET https://oidc-proxy.xxx/api/v1/namespaces/xxx/pods/redis-master-0 200 OK in 542 milliseconds
I0921 17:31:52.799925   18020 round_trippers.go:553] GET https://oidc-proxy.xxx/api/v1/namespaces/xxx/pods/redis-master-0 200 OK in 127 milliseconds
I0921 17:31:53.149233   18020 round_trippers.go:553] POST https://oidc-proxy.xxx/api/v1/namespaces/xxx/pods/redis-master-0/portforward 403 Forbidden in 345 milliseconds
error: error upgrading connection:

kubectl exec -it pod/kube-oidc-proxy-kube-oidc-proxy-8555dc5985-22b8k -n kube-oidc-proxy --v=6 -- /bin/bash
I0921 17:25:35.897751   17912 loader.go:373] Config loaded from file:  /Users/xxx/.kube/config
I0921 17:25:36.394211   17912 round_trippers.go:553] GET https://oidc-proxy.xxx/api/v1/namespaces/kube-oidc-proxy/pods/kube-oidc-proxy-kube-oidc-proxy-8555dc5985-22b8k 200 OK in 486 milliseconds
I0921 17:25:36.397416   17912 podcmd.go:88] Defaulting container name to kube-oidc-proxy-kube-oidc-proxy
I0921 17:25:36.741460   17912 round_trippers.go:553] POST https://oidc-proxy.xxx/api/v1/namespaces/kube-oidc-proxy/pods/kube-oidc-proxy-kube-oidc-proxy-8555dc5985-22b8k/exec?command=%2Fbin%2Fbash&container=kube-oidc-proxy-kube-oidc-proxy&stdin=true&stdout=true&tty=true 403 Forbidden in 343 milliseconds
I0921 17:25:36.741959   17912 helpers.go:246] server response object: [{
  "metadata": {}
}]
Error from server:

[sellitforcache]

I'm also seeing this exact problem with NGINX... anybody have success yet?

[antonvigo]

In my case oidc-proxy is behind self-hosted NGINX balancer. And for kubectl-exec mode connection should be upgraded from regular HTTP/1.1 to WebSocket. So adding this to NGINX config fixed the problem:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

[akamac]

for Istio the fix is to apply the following EnvoyFilter:

  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          upgrade_configs:
          - upgrade_type: spdy/3.1