REP: Kubernetes Native Ray Authentication #51

andrewsykim · 2024-05-21T20:43:50Z

Add REP for supporting authentication for Ray clusters created by KubeRay using the Kubernetes TokenReview API

astefanutti · 2024-05-23T06:31:45Z

reps/2024-05-21-kuberay-authentication.md

+
+The sidecar authenticator will use the TokenReview API to authenticate tokens passed into the authorization headers of each request.
+Tokens can be created from Kubernetes ServiceAccounts or an external identity provider as long as the Kubernetes API Server is configured with an external authorizer.
+Each RayCluster will be provisioned a default ServiceAccount that KubeRay Operator will use when authenticating to the Ray head (specifically for RayJob and RayService).


It may be useful for the implementation phase, to explicitly mention here the case of RayJobs with submission mode set to K8sJobMode, for which the Ray CLI command executed in the submission Job will have to be authenticated as well.

Good point, will add that!

kevin85421 · 2024-05-28T05:21:46Z

reps/2024-05-21-kuberay-authentication.md

+3. A ServiceAccount is created per RayCluster resource
+
+The sidecar ensures only requests with the correct authorization tokens can access the Ray head. More details on how tokens are authenticated below.
+For the MVP, the sidecar will be implemented using Envoy configured with an [External Authorizer](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_authz_filter).


I also chatted with some users about this project. They have their own internal authentication solution. We should make the interface extensible and not be too opinionated about Envoy.

I agree, I don't feel too strongly about using Envoy.

I also think implementing from scratch wouldn't be that difficult, we can consider a new binary / image for the auth server.

kevin85421 · 2024-05-28T05:29:18Z

This PR looks good to me. Maybe we can organize a community design review meeting for this REP.

kevin85421 · 2024-05-28T18:54:56Z

cc @thomasdesr would you mind reviewing this PR? Thanks!

reps/2024-05-21-kuberay-authentication.md

astefanutti · 2024-06-05T09:46:21Z

reps/2024-05-21-kuberay-authentication.md

+ ...
+spec:
+ enableAuthentication: true
+ authenticationOptions:


In some organisations, RayCluster resources will be created by the "data scientist" / end-user persona, while the security aspect will be owned by the platform administrator. The current proposal makes it difficult to enforce that model with Kubernetes RBAC.

The API surface of that new feature should ideally follow usual API graduation. Including it into the RayCluster API, that's v1, will make it difficult to iterate on it and make it evolve, by complying to v1 guarantees.

The RayCluster API has already a large surface. Could the proposal explore the solution space, and research a more modular approach for the configurability aspect, that'd be more symmetrical to the sidecar implementation proposal, i.e. non intrusive, and generally try to keep the single-responsibility principle for the RayCluster API as much as possible.

In some organisations, RayCluster resources will be created by the "data scientist" / end-user persona, while the security aspect will be owned by the platform administrator. The current proposal makes it difficult to enforce that model with Kubernetes RBAC.

My assumption is that a platform admin can grant role / rolebindings that give data scientists namespace-wide access to RayClusters (both creating them and accessing them) so they don't need to manage security on a per-cluster basis. You can also make this a ClusterRole to give access to all RayClusters in the cluster. In organizations where data scientists also create RayClusters, they also have the option to by-pass the RBAC check with the fixed list of allowed users (allowedPrincipals).

The API surface of that new feature should ideally follow usual API graduation. Including it into the RayCluster API, that's v1, will make it difficult to iterate on it and make it evolve, by complying to v1 guarantees.

I think this is more of a concern around feature gates and gating changes to v1 APIs that might not be mature yet. I will try to explore adding Kubernetes-like feature gates to KubeRay as we implement this. FYI @kevin85421

The RayCluster API has already a large surface. Could the proposal explore the solution space, and research a more modular approach for the configurability aspect, that'd be more symmetrical to the sidecar implementation proposal, i.e. non intrusive, and generally try to keep the single-responsibility principle for the RayCluster API as much as possible.

Let me give this more thought and let's revisit during code review? In theory the addition of a sidecar is already modular because RayCluster lets you override the entire pod template. So my initial thinking is that enabling authentication in KubeRay should default to a solution that will work for most users out of the box (with something like kube-rbac-proxy)

We can discuss more during code review about the API.

mtaufen · 2024-06-17T23:03:49Z

reps/2024-05-21-kuberay-authentication.md

+## Access Control with TokenReview API
+
+The sidecar authenticator will use the TokenReview API to authenticate tokens passed into the authorization headers of each request.
+Tokens can be created from Kubernetes ServiceAccounts or an external identity provider as long as the Kubernetes API Server is configured with an external authorizer.


For KSA tokens, the sidecar should require them to have audiences that do not intersect with the audiences accepted by kube-apiserver. We don't want KSA tokens that grant API access included in requests destined for anywhere other than kube-apiserver. Note that audiences can be specified for mounted tokens using projected volumes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection

thanks, added a note about KSA requiring audiences

liggitt · 2024-06-18T01:03:20Z

reps/2024-05-21-kuberay-authentication.md

+apiVersion: authentication.k8s.io/v1
+kind: TokenReview
+spec:
+ token: <your_token_here>


+1 to requiring a ray-specific audience here

thanks, added a note about KSA requiring audiences

liggitt · 2024-06-18T01:04:11Z

reps/2024-05-21-kuberay-authentication.md

+apiVersion: authentication.k8s.io/v1
+kind: TokenReview
+status:
+ authenticated: true


the tokenreview response status must contain the audience the token is good for, and should be verified to intersect with the audience requested in the tokenreview spec

thanks, updated the status example to include an audience that will be verified with the audience in the spec

ahmedtd · 2024-06-18T21:57:31Z

reps/2024-05-21-kuberay-authentication.md

+
+The sidecar authenticator will use the TokenReview API to authenticate tokens passed into the authorization headers of each request.
+Tokens can be created from Kubernetes ServiceAccounts or an external identity provider as long as the Kubernetes API Server is configured with an external authorizer.
+Tokens from Kubernetes Service Accounts are required to specify an audience. For now the placeholder audience is "ray.io".


Ideally, audiences are globally unique; a token meant for interacting with ray cluster A should not be accepted by Ray cluster B

Perhaps the audience should be something like ray.io/cluster-name/foo, where foo is somehow derived from the RayCluster resource?

good point, updated to require unique auidences

ahmedtd · 2024-06-18T22:06:21Z

reps/2024-05-21-kuberay-authentication.md

+
+When submitting jobs using the Ray job CLI:
+```
+export RAY_JOB_HEADERS="Authorization: Bearer $(kubectl create token ray-cluster)"


The user will need to be given impersonation permissions for the service account default/ray-cluster. If multiple users are using the cluster, then all their access will be laundered through this service account.

The recommended usage should probably be the one from line 69, where the user authenticates to the ray cluster using the same credential they use to authenticate to the Kubernetes cluster.

The recommended usage should probably be the one from line 69, where the user authenticates to the ray cluster using the same credential they use to authenticate to the Kubernetes cluster.

Yes, we will likely recommend this for most cases, the KSA case is only supported for two cases:

For KubeRay operator itself that needs to connect to Ray clusters

For clusters that don't have external identity providers

ahmedtd · 2024-06-18T22:09:03Z

reps/2024-05-21-kuberay-authentication.md

+```
+export RAY_JOB_HEADERS="Authorization: Bearer $(kubectl create token ray-cluster)"
+OR
+export RAY_JOB_HEADERS="Authorization: Bearer $(gcloud auth print-access-token)" # example external identity provider


I was surprised that this would work, but it does seem like TokenReview will understand external tokens by calling the Webhook Token Authenticator, if one is configured.

We'll need to think about how to make this mesh with the audiences from above. GCP (and presumably AWS, Azure, etc) access tokens don't carry scopes, so they won't pass the scope check by default.

I was surprised that this would work, but it does seem like TokenReview will understand external tokens by calling the Webhook Token Authenticator, if one is configured.

I was also pleasantly surprised as well :)

We'll need to think about how to make this mesh with the audiences from above. GCP (and presumably AWS, Azure, etc) access tokens don't carry scopes, so they won't pass the scope check by default.

From my testing w/ GKE, the returned audience is something like:

audiences: - https://container.googleapis.com/v1/projects/<project>/locations/us-east4-c/clusters/<cluster>

Perhaps we need to specify a list of allowed audiences in the RayCluster if we are required to verify the audience?

OK, yes, that seems like it would work.

SubjectAccessReview API Signed-off-by: Andrew Sy Kim <[email protected]>

zhe-thoughts · 2024-07-22T21:10:29Z

We had an in-depth review meeting on this feature on June 4. The vote on ray-committers has also just passed. I'm going to merge this REP

Thanks @andrewsykim for the great contribution and @kevin85421 for shepherding

kevin85421 self-assigned this May 22, 2024

astefanutti reviewed May 23, 2024

View reviewed changes

kevin85421 reviewed May 28, 2024

View reviewed changes

kevin85421 assigned thomasdesr May 28, 2024

andrewsykim force-pushed the ray-authentication branch from e9eac24 to 552e6ce Compare June 2, 2024 02:50

rueian reviewed Jun 4, 2024

View reviewed changes

reps/2024-05-21-kuberay-authentication.md Outdated Show resolved Hide resolved

astefanutti reviewed Jun 5, 2024

View reviewed changes

andrewsykim force-pushed the ray-authentication branch from 552e6ce to e31faa3 Compare June 5, 2024 14:21

andrewsykim changed the title ~~REP: KubeRay authentication with TokenReview API~~ REP: Kubernetes Native Ray Authentication Jun 7, 2024

mtaufen reviewed Jun 17, 2024

View reviewed changes

liggitt reviewed Jun 18, 2024

View reviewed changes

andrewsykim force-pushed the ray-authentication branch from e31faa3 to 45a0b68 Compare June 18, 2024 16:41

ahmedtd reviewed Jun 18, 2024

View reviewed changes

andrewsykim force-pushed the ray-authentication branch from 45a0b68 to 90a383c Compare June 19, 2024 19:48

REP: Kubernetes Native Ray Authentication with TokenReview and

ad2b545

SubjectAccessReview API Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the ray-authentication branch from 90a383c to ad2b545 Compare June 19, 2024 20:09

kevin85421 approved these changes Jul 12, 2024

View reviewed changes

zhe-thoughts merged commit e39668b into ray-project:main Jul 22, 2024
1 check passed

kevin85421 mentioned this pull request Jul 29, 2024

[Discuss][Feature] Authenticate requests sent to Ray Serve proxy actors ray-project/kuberay#2276

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REP: Kubernetes Native Ray Authentication #51

REP: Kubernetes Native Ray Authentication #51

andrewsykim commented May 21, 2024

astefanutti May 23, 2024

andrewsykim May 23, 2024

kevin85421 May 28, 2024

andrewsykim May 28, 2024

kevin85421 commented May 28, 2024

kevin85421 commented May 28, 2024

astefanutti Jun 5, 2024

andrewsykim Jun 5, 2024

kevin85421 Jul 12, 2024

mtaufen Jun 17, 2024

andrewsykim Jun 18, 2024

liggitt Jun 18, 2024

andrewsykim Jun 18, 2024

liggitt Jun 18, 2024

andrewsykim Jun 18, 2024

ahmedtd Jun 18, 2024

andrewsykim Jun 19, 2024

ahmedtd Jun 18, 2024

andrewsykim Jun 19, 2024

ahmedtd Jun 18, 2024

andrewsykim Jun 19, 2024 •

edited

Loading

ahmedtd Jun 20, 2024

zhe-thoughts commented Jul 22, 2024

REP: Kubernetes Native Ray Authentication #51

REP: Kubernetes Native Ray Authentication #51

Conversation

andrewsykim commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin85421 commented May 28, 2024

kevin85421 commented May 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsykim Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhe-thoughts commented Jul 22, 2024

andrewsykim Jun 19, 2024 •

edited

Loading