add functions for creating ray with oauth proxy in front of the dashboard

[KPostOffice]

Issue link

closes: #174

What changes have been made

I added functions which create the necessary objects and update the appwrapper so that the SDK can create a ray cluster with an OAuth Proxy in front of the dashboard.

• OAuth sidecar container and tls secret mount
• Ingress with annotations for tls passthrough
• Service which points to OAuth port in the sidecar container and annotations for autogenerated tls certs
• Service account which contains the OAuth Callback
• ClusterRoleBinding giving OAuth sidecar necessary permissions

Verification steps

go through notebook but set openshift_oauth to true in the config

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[KPostOffice]

This isn't quite finished yet. I'm having tls related issues between the Ray API and the OAuth proxy.

[KPostOffice]

Blocked due to required changes upstream with the Ray Job Submission Client and TorchX

(edit: submission client has necessary changes. I was using an older version locally)

[blublinsky]

This URL is used not only for dashboard access but also for the APIs - currently jobs and In future also Serve. Oath proxy requires manual log in, which will break these APIs

[KPostOffice]

@blublinsky I've launched OAuth so that it supports both GUI manual login as well as authentication using an auth bearer token. Can you PTAL?

[blublinsky]

@blublinsky I've launched OAuth so that it supports both GUI manual login as well as authentication using an auth bearer token. Can you PTAL?

How do you get the auth bearer token to use?

[KPostOffice]

@blublinsky I've launched OAuth so that it supports both GUI manual login as well as authentication using an auth bearer token. Can you PTAL?

How do you get the auth bearer token to use?

https://github.com/project-codeflare/codeflare-sdk/pull/298/files#diff-5706a96c5346dd54a22370211260411363c4ed41d798fddf240eb8983b1366eaR82-R86

It is the same token used for interacting with the kubernetes api with the kubectl client. I grab it here based on the currently logged in user or in cluster config. I believe it is the same token as the one you generate from the copy login command page in the OpenShift client.

[blublinsky]

It is the same token used for interacting with the kubernetes api with the kubectl client. I grab it here based on the currently logged in user or in cluster config. I believe it is the same token as the one you generate from the copy login command page in the OpenShift client.

So basically the only thing that you are validating is that user is logged in. Not very secure, really

[KPostOffice]

It is the same token used for interacting with the kubernetes api with the kubectl client. I grab it here based on the currently logged in user or in cluster config. I believe it is the same token as the one you generate from the copy login command page in the OpenShift client.

So basically the only thing that you are validating is that user is logged in. Not very secure, really

https://github.com/project-codeflare/codeflare-sdk/pull/298/files#diff-279fd895cfcb8295f1b610826f8eb7a40fd543ff873661acd9dc452cef3d9d14R423

The OAuth Proxy supports checking for authorization based on RBAC. Here I check to make sure the authenticated user has authorization to get pods in the given namespace.

[blublinsky]

So it is basically a single cluster solution

[KPostOffice]

So it is basically a single cluster solution

I'm unsure what you are expecting. How would you propose we authenticate for a multi-cluster design here? AFAIK, the codeflare-sdk expects the user to be logged into a specific k8 cluster.

[blublinsky]

I'm unsure what you are expecting. How would you propose we authenticate for a multi-cluster design here? AFAIK, the codeflare-sdk expects the user to be logged into a specific k8 cluster.

I do not think authentication/authorization should be linked to a given cluster, It should rather be for a given user, who can be logged in any of the several clusters, but that's just my opinion

[KPostOffice]

I was running into issues with the generate CA cert from the service annotation. Putting this here so I remember to look into the ca-injector.

https://cert-manager.io/docs/concepts/ca-injector/

[KPostOffice]

I'm unsure what you are expecting. How would you propose we authenticate for a multi-cluster design here? AFAIK, the codeflare-sdk expects the user to be logged into a specific k8 cluster.

I do not think authentication/authorization should be linked to a given cluster, It should rather be for a given user, who can be logged in any of the several clusters, but that's just my opinion

Where do you propose authentication and authorization occur in this case?

[MichaelClifford]

Tested out starting a ray cluster from local machine and submitting jobs. Seems to work as expected when logging in via sdk authentication or oc login.

lgtm

@Maxusmusti could you please review as well.

Michael LGTM'd the PR already.

[anishasthana]

@KPostOffice could you create a follow-on task to switch this over to istio (once we have istio available ootb in ODH). Or at least make it configurable somehow. Alternatively, the follow-on winds up being "Remove this, and update the kuberay operator to allow auth enablement"

[Bobbins228]

Verified working on OSD cluster awesome work Kevin /approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobbins228

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Bobbins228]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment