Support for Structured Authorization for Shoots >= Kubernetes v1.30

[aaronschweig]

How to categorize this issue?
/area control-plane
/kind api-change
/kind enhancement

What would you like to be added:

Hello Team 👋

We would like to benefit from the ability to use custom Kubernetes authorization webhooks in order to give a gardener provided control plane the ability to defer Authorization decisions to an external authorization engine like OpenFGA.

The ability to be able to configure the necessary kubernetes API server args via the Shoot CRD would be needed in order to achieve our desired result

Why is this needed:

This change is needed to enable users of gardener to profit form all the features that kubernetes provides in regards to authorization modes. We would like to use this feature within the context of the ORA project, so all contributers are able to use a central authorization engine, which is able to give authorization decisions in every layer of the product.

Thanks a lot and let me know in case any other input is needed!

[rfranzke]

We have to discuss to what extent we can expose this such that users cannot exclude/block gardenlet's operations on Shoots with their authz webhooks.

[rfranzke]

Let's implement this for both core.gardener.cloud/v1beta1.Shoots and operator.gardener.cloud/v1alpha1.Gardens if their Kubernetes version is at least v1.30.

I propose to extend their APIs as follows:

kubernetes:
  kubeAPIServer:
    authorization:
      structured:
        configMapName: name-of-configmap-containing-webhook-configurations

The referenced ConfigMap should contain the list of apiserver.config.k8s.io/v1beta1.AuthorizerConfigurations (we don't embed this directly in the Shoot/Garden for API-compatibility reasons, and to prevent "blowing up" the resource in case many authorization webhooks are used).
It could look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: name-of-configmap-containing-webhook-configurations
  namespace: garden-my-project
data:
  config.yaml: |
    authorizers:
    - type: Webhook
      # Name used to describe the authorizer
      # This is explicitly used in monitoring machinery for metrics
      # Note:
      #   - Validation for this field is similar to how K8s labels are validated today.
      # Required, with no default
      name: webhook
      webhook:
        # The duration to cache 'authorized' responses from the webhook
        # authorizer.
        # Same as setting `--authorization-webhook-cache-authorized-ttl` flag
        # Default: 5m0s
        authorizedTTL: 30s
        # The duration to cache 'unauthorized' responses from the webhook
        # authorizer.
        # Same as setting `--authorization-webhook-cache-unauthorized-ttl` flag
        # Default: 30s
        unauthorizedTTL: 30s
        # Timeout for the webhook request
        # Maximum allowed is 30s.
        # Required, with no default.
        timeout: 3s
        # The API version of the authorization.k8s.io SubjectAccessReview to
        # send to and expect from the webhook.
        # Same as setting `--authorization-webhook-version` flag
        # Required, with no default
        # Valid values: v1beta1, v1
        subjectAccessReviewVersion: v1
        # MatchConditionSubjectAccessReviewVersion specifies the SubjectAccessReview
        # version the CEL expressions are evaluated against
        # Valid values: v1
        # Required, no default value
        matchConditionSubjectAccessReviewVersion: v1
        # Controls the authorization decision when a webhook request fails to
        # complete or returns a malformed response or errors evaluating
        # matchConditions.
        # Valid values:
        #   - NoOpinion: continue to subsequent authorizers to see if one of
        #     them allows the request
        #   - Deny: reject the request without consulting subsequent authorizers
        # Required, with no default.
        failurePolicy: Deny
        connectionInfo: {} # NOT ALLOWED, SEE BELOW
        # matchConditions is a list of conditions that must be met for a request to be sent to this
        # webhook. An empty list of matchConditions matches all requests.
        # There are a maximum of 64 match conditions allowed.
        #
        # The exact matching logic is (in order):
        #   1. If at least one matchCondition evaluates to FALSE, then the webhook is skipped.
        #   2. If ALL matchConditions evaluate to TRUE, then the webhook is called.
        #   3. If at least one matchCondition evaluates to an error (but none are FALSE):
        #      - If failurePolicy=Deny, then the webhook rejects the request
        #      - If failurePolicy=NoOpinion, then the error is ignored and the webhook is skipped
        matchConditions:
        # expression represents the expression which will be evaluated by CEL. Must evaluate to bool.
        # CEL expressions have access to the contents of the SubjectAccessReview in v1 version.
        # If version specified by subjectAccessReviewVersion in the request variable is v1beta1,
        # the contents would be converted to the v1 version before evaluating the CEL expression.
        #
        # Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/
        #
        # only send resource requests to the webhook
        - expression: has(request.resourceAttributes)
        # only intercept requests to kube-system
        - expression: request.resourceAttributes.namespace == 'kube-system'
        # don't intercept requests from kube-system service accounts
        - expression: !('system:serviceaccounts:kube-system' in request.user.groups)

Important

We only accept authorizers of type Webhook. If we find another type, we should abort and return an error.

Similarly, the webhook.connectionInfo must similarly not be provided. Instead, we should create our own type:

import apiserver1beta1 "k8s.io/apiserver/pkg/apis/apiserver/v1beta1"

type WebhookAuthenticator struct {
  apiserverv1beta1.WebhookConfiguration `json:",inline"`
  KubeconfigSecretName *string `json:"kubeconfigSecretName,omitempty"
}

❗️ In such Secrets containing the kubeconfig for the connection, we should check that no file references are used, and return an error otherwise.

gardenlet can read this ConfigMap, generate a new ConfigMap (in the shoot namespace in the seed cluster if needed, see below), and configure kube-apiserver's --authorization-config accordingly.

Based on the KubeconfigSecretName, gardenlet can then set the connectionInfo for the webhook configurations:

authorizers:
- type: Webhook
  name: webhook
  webhook:
    connectionInfo:
      type: KubeConfig
      kubeConfigFile: <path-to-kubeconfig-secret>

❗️ Also, we should always manipulate the matchConditions[] of the webhook configurations to prevent that users exclude the gardenlet or kubeconfigs generated via the shoots/{admin,viewer}kubeconfig subresource.

Generally, gardenlet would always need to compute a ConfigMap for kube-apiserver with apiserver.config.k8s.io/v1beta1.AuthorizationConfiguration:

# for workerless shoots
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthorizationConfiguration
authorizers:
- type: RBAC
  name: rbac

# for regular shoots
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthorizationConfiguration
authorizers:
- type: Node
  name: node
- type: RBAC
  name: rbac

The additional user-provided webhook authorizers via the ConfigMap in .kubernetes.kubeAPIServer.authorization.structured.configMapName can be amened above, e.g.:

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthorizationConfiguration
authorizers:
- type: Node
  name: node
- type: RBAC
  name: rbac
- type: Webhook
  name: webhook
  webhook:
    authorizedTTL: 30s
    unauthorizedTTL: 30s
    timeout: 3s
    subjectAccessReviewVersion: v1
    matchConditionSubjectAccessReviewVersion: v1
    failurePolicy: Deny
    connectionInfo:
      type: KubeConfig
      kubeConfigFile: <path-to-kubeconfig-secret>
    matchConditions:
    # some rules that exclude the `gardenlet` and users generated via the `shoots/{admin,viewer}kubeconfig` subresource
    - expression: has(request.resourceAttributes)
    - expression: request.resourceAttributes.namespace == 'kube-system'
    - expression: !('system:serviceaccounts:kube-system' in request.user.groups)

Caution

The --authorization-mode/--authorization-webhook-* flags must no longer be provided when --authorization-config is set.

Important

gardener-operator should be adapted to make use of the new structured authorization config for the SeedAuthorizer when Garden is of >= v1.30 Kubernetes.