diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 4d09c32de8..bfb60ddb80 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -17,6 +17,114 @@ namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. +## Cluster-scoped vs namespaced objects + +Most objects in this chart are namespaced and land in the release namespace. +Only two are cluster-scoped: + +| Object | Default name | +| --- | --- | +| `ClusterRole` | `-node-reader-` | +| `ClusterRoleBinding` | `-node-reader-` | + +By default the release creates both, so an install by a cluster-admin is +unchanged. On clusters where cluster-scoped RBAC is owned by a different team, +split the install in two. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount, rendered from the same values the release uses: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the release with cluster-scoped +objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) +or the equivalent `--set`: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace do not change, so the +pre-created `ClusterRoleBinding` keeps matching the release. This works with +`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows +`serviceAccount.name`, so render the admin step with the same values. + +### Which flag the installer needs + +In the default `shared` workspace mode the release also creates a namespaced +sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. +Kubernetes forbids granting permissions you do not hold, and the built-in +`admin` ClusterRole does not cover that CRD, so an installer holding only +`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not +enough and the install fails with `attempting to grant RBAC permissions not +currently held`. + +| Workspace mode | Installer holds | Use | +| --- | --- | --- | +| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | +| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | + +`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant +is needed there. + +With `rbac.create=false` the cluster-admin applies the namespaced RBAC too, +adding it to the same render: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml \ + --show-only templates/role.yaml \ + --show-only templates/rolebinding.yaml \ + --show-only templates/peer-role.yaml | kubectl apply -f - +``` + +To grant the installer the sandbox permissions instead, bind it to a Role +carrying the same rules as the chart's `openshell-sandbox` Role. + +The certgen hook and credential driver RBAC keep their own flags +(`pkiInitJob.enabled` and +`server.credentialDrivers.kubernetesSecrets.rbac.create`). + +### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of +`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change +what the `ClusterRole` contains, but they never force the namespaced release to +apply it. Re-run the cluster-admin step after changing values that affect the +`ClusterRole` rules. + ## Prerequisites > **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for @@ -260,6 +368,10 @@ discovery endpoint or its TLS CA. | probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. | | probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. | | probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. | +| rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. | +| rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `-node-reader-` default. | +| rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. | +| rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox and peer Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. | | replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. | | resources | object | `{}` | Gateway pod resource requests and limits. | | sandbox.image.digest | string | `""` | Sandbox image digest. When set, this takes precedence over tag. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 6e49260c20..1e86e0cbcf 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -17,6 +17,115 @@ namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. +## Cluster-scoped vs namespaced objects + +Most objects in this chart are namespaced and land in the release namespace. +Only two are cluster-scoped: + +| Object | Default name | +| --- | --- | +| `ClusterRole` | `-node-reader-` | +| `ClusterRoleBinding` | `-node-reader-` | + +By default the release creates both, so an install by a cluster-admin is +unchanged. On clusters where cluster-scoped RBAC is owned by a different team, +split the install in two. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount, rendered from the same values the release uses: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the release with cluster-scoped +objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) +or the equivalent `--set`: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace do not change, so the +pre-created `ClusterRoleBinding` keeps matching the release. This works with +`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows +`serviceAccount.name`, so render the admin step with the same values. + +### Which flag the installer needs + +In the default `shared` workspace mode the release also creates a namespaced +sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. +Kubernetes forbids granting permissions you do not hold, and the built-in +`admin` ClusterRole does not cover that CRD, so an installer holding only +`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not +enough and the install fails with `attempting to grant RBAC permissions not +currently held`. + +| Workspace mode | Installer holds | Use | +| --- | --- | --- | +| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | +| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | + +`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant +is needed there. + +With `rbac.create=false` the cluster-admin applies the namespaced RBAC too, +adding it to the same render: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml \ + --show-only templates/role.yaml \ + --show-only templates/rolebinding.yaml \ + --show-only templates/peer-role.yaml | kubectl apply -f - +``` + +To grant the installer the sandbox permissions instead, bind it to a Role +carrying the same rules as the chart's `openshell-sandbox` Role. + +The certgen hook and credential driver RBAC keep their own flags +(`pkiInitJob.enabled` and +`server.credentialDrivers.kubernetesSecrets.rbac.create`). + +### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of +`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change +what the `ClusterRole` contains, but they never force the namespaced release to +apply it. Re-run the cluster-admin step after changing values that affect the +`ClusterRole` rules. + + ## Prerequisites > **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for diff --git a/deploy/helm/openshell/ci/values-namespace-admin.yaml b/deploy/helm/openshell/ci/values-namespace-admin.yaml new file mode 100644 index 0000000000..7326be7328 --- /dev/null +++ b/deploy/helm/openshell/ci/values-namespace-admin.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Namespace-admin overlay — renders only namespaced objects. +# +# Use this when a cluster-admin applies the gateway ClusterRole and +# ClusterRoleBinding once, out of band, and the OpenShell release is installed +# and upgraded by an installer that holds no cluster-scoped permissions. +# +# Generate the cluster-scoped objects for the cluster-admin step from the same +# release values, then apply them as cluster-admin: +# helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \ +# --version --namespace openshell -f my-values.yaml \ +# --set rbac.create=true \ +# --set rbac.clusterScoped.create=true \ +# --set agentSandbox.preflight.enabled=false \ +# --show-only templates/clusterrole.yaml \ +# --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +# +# The gateway ServiceAccount name and release namespace are unchanged, so the +# pre-created ClusterRoleBinding still matches this release. + +rbac: + clusterScoped: + create: false diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 32e6b7cb63..ab42458759 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -86,6 +86,66 @@ default to enabled so upgrades with --reuse-values preserve the old topology. {{- if $enabled -}}true{{- end -}} {{- end }} +{{/* +Whether this chart owns gateway RBAC objects. Missing legacy values default to +enabled so upgrades with --reuse-values preserve the old topology. +*/}} +{{- define "openshell.rbacCreate" -}} +{{- $rbac := .Values.rbac | default dict -}} +{{- $create := true -}} +{{- if hasKey $rbac "create" -}} +{{- $create = get $rbac "create" -}} +{{- end -}} +{{- if $create -}}true{{- end -}} +{{- end }} + +{{/* +The rbac.clusterScoped values map, tolerating missing legacy values. +*/}} +{{- define "openshell.clusterScopedRbacValues" -}} +{{- $rbac := .Values.rbac | default dict -}} +{{- $clusterScoped := dict -}} +{{- if hasKey $rbac "clusterScoped" -}} +{{- $clusterScoped = get $rbac "clusterScoped" | default dict -}} +{{- end -}} +{{- toYaml $clusterScoped -}} +{{- end }} + +{{/* +Whether this chart owns the cluster-scoped ClusterRole and ClusterRoleBinding. +Disable for a namespace-admin install where a cluster-admin applies them +separately. Missing legacy values default to enabled. +*/}} +{{- define "openshell.clusterRbacCreate" -}} +{{- if include "openshell.rbacCreate" . -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $create := true -}} +{{- if hasKey $clusterScoped "create" -}} +{{- $create = get $clusterScoped "create" -}} +{{- end -}} +{{- if $create -}}true{{- end -}} +{{- end -}} +{{- end }} + +{{/* +Name of the gateway ClusterRole. The release namespace is part of the default +name so multiple releases on one cluster do not collide. +*/}} +{{- define "openshell.clusterRoleName" -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}} +{{- default $default (get $clusterScoped "clusterRoleName") -}} +{{- end }} + +{{/* +Name of the gateway ClusterRoleBinding. +*/}} +{{- define "openshell.clusterRoleBindingName" -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}} +{{- default $default (get $clusterScoped "clusterRoleBindingName") -}} +{{- end }} + {{/* Gateway image reference. A digest takes precedence over a tag. */}} {{- define "openshell.image" -}} {{- $image := .Values.gateway.image -}} diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index 17c483deb9..85d10c6b60 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -1,3 +1,4 @@ +{{- if include "openshell.clusterRbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -5,7 +6,7 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleName" . }} labels: {{- include "openshell.labels" . | nindent 4 }} rules: @@ -128,3 +129,4 @@ rules: - update {{- end }} {{- end }} +{{- end }} diff --git a/deploy/helm/openshell/templates/clusterrolebinding.yaml b/deploy/helm/openshell/templates/clusterrolebinding.yaml index 9b3b254586..8de246730b 100644 --- a/deploy/helm/openshell/templates/clusterrolebinding.yaml +++ b/deploy/helm/openshell/templates/clusterrolebinding.yaml @@ -1,17 +1,19 @@ +{{- if include "openshell.clusterRbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleBindingName" . }} labels: {{- include "openshell.labels" . | nindent 4 }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleName" . }} subjects: - kind: ServiceAccount name: {{ include "openshell.serviceAccountName" . }} namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/peer-role.yaml b/deploy/helm/openshell/templates/peer-role.yaml index 59ae7479ad..3aa2502f46 100644 --- a/deploy/helm/openshell/templates/peer-role.yaml +++ b/deploy/helm/openshell/templates/peer-role.yaml @@ -1,3 +1,4 @@ +{{- if include "openshell.rbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 @@ -33,3 +34,4 @@ subjects: - kind: ServiceAccount name: {{ include "openshell.serviceAccountName" . }} namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index b47efa090b..d9237e743d 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/templates/rolebinding.yaml b/deploy/helm/openshell/templates/rolebinding.yaml index 32f11644bf..49cc6f7fba 100644 --- a/deploy/helm/openshell/templates/rolebinding.yaml +++ b/deploy/helm/openshell/templates/rolebinding.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/tests/clusterrole_test.yaml b/deploy/helm/openshell/tests/clusterrole_test.yaml index 326bc2cb60..08907c614a 100644 --- a/deploy/helm/openshell/tests/clusterrole_test.yaml +++ b/deploy/helm/openshell/tests/clusterrole_test.yaml @@ -163,3 +163,42 @@ tests: apiGroups: [""] resources: ["services"] verbs: ["create", "get"] + - it: omits the ClusterRole when cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRole when all chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRole independently of the workspace mode + set: + rbac.clusterScoped.create: false + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 0 + + - it: uses the configured ClusterRole name + set: + rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader + asserts: + - equal: + path: metadata.name + value: platform-openshell-node-reader + + - it: creates the ClusterRole when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-node-reader-my-namespace diff --git a/deploy/helm/openshell/tests/clusterrolebinding_test.yaml b/deploy/helm/openshell/tests/clusterrolebinding_test.yaml index 5b9a83c04c..4cbf6d7d91 100644 --- a/deploy/helm/openshell/tests/clusterrolebinding_test.yaml +++ b/deploy/helm/openshell/tests/clusterrolebinding_test.yaml @@ -29,3 +29,48 @@ tests: kind: ServiceAccount name: openshell namespace: my-namespace + + - it: omits the ClusterRoleBinding when cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRoleBinding when all chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: uses the configured ClusterRoleBinding and ClusterRole names + set: + rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader + rbac.clusterScoped.clusterRoleBindingName: platform-openshell-node-reader-binding + asserts: + - equal: + path: metadata.name + value: platform-openshell-node-reader-binding + - equal: + path: roleRef.name + value: platform-openshell-node-reader + + - it: binds the custom gateway service account created by the namespaced release + set: + serviceAccount.create: false + serviceAccount.name: my-existing-sa + asserts: + - contains: + path: subjects + content: + kind: ServiceAccount + name: my-existing-sa + namespace: my-namespace + + - it: creates the ClusterRoleBinding when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 diff --git a/deploy/helm/openshell/tests/rbac_test.yaml b/deploy/helm/openshell/tests/rbac_test.yaml new file mode 100644 index 0000000000..3698465e27 --- /dev/null +++ b/deploy/helm/openshell/tests/rbac_test.yaml @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: Gateway RBAC creation +templates: + - templates/role.yaml + - templates/rolebinding.yaml + - templates/peer-role.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: creates the namespaced sandbox Role by default + template: templates/role.yaml + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-sandbox + + - it: creates the namespaced sandbox RoleBinding by default + template: templates/rolebinding.yaml + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-sandbox + + - it: creates the gateway peer Role and RoleBinding by default + template: templates/peer-role.yaml + asserts: + - hasDocuments: + count: 2 + - equal: + path: metadata.name + value: openshell-peer + + - it: keeps the namespaced RBAC when only cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 1 + template: templates/role.yaml + + - it: keeps the peer RBAC when only cluster-scoped RBAC is disabled + template: templates/peer-role.yaml + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 2 + + - it: omits the namespaced sandbox Role when chart-managed RBAC is disabled + template: templates/role.yaml + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the namespaced sandbox RoleBinding when chart-managed RBAC is disabled + template: templates/rolebinding.yaml + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the gateway peer Role and RoleBinding when chart-managed RBAC is disabled + template: templates/peer-role.yaml + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: creates the namespaced sandbox Role when legacy values omit the rbac block + template: templates/role.yaml + set: + rbac: null + asserts: + - hasDocuments: + count: 1 + + - it: creates the peer Role and RoleBinding when legacy values omit the rbac block + template: templates/peer-role.yaml + set: + rbac: null + asserts: + - hasDocuments: + count: 2 diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 426f3ae424..4e7e11142d 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -141,6 +141,26 @@ sandboxServiceAccount: # -- Existing service account name for sandbox pods when sandboxServiceAccount.create is false. name: "" +# RBAC objects created for the gateway ServiceAccount. Cluster-scoped objects +# can be omitted so a cluster-admin applies them once and a namespace-admin +# installs and upgrades the gateway release without cluster-scoped permissions. +rbac: + # -- Create the RBAC objects that grant the gateway ServiceAccount access. + # Disable to supply the namespaced sandbox and peer Role/RoleBinding and the + # cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook + # and credential driver RBAC keep their own flags. + create: true + clusterScoped: + # -- Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable + # for a namespace-admin install where a cluster-admin applies them + # separately; the gateway ServiceAccount name and namespace are unchanged, + # so a pre-created ClusterRoleBinding still matches. + create: true + # -- Name for the ClusterRole. Empty uses the `-node-reader-` default. + clusterRoleName: "" + # -- Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. + clusterRoleBindingName: "" + # Namespace-scoped resources needed to run sandboxes. Disable this when the # gateway and workspace prerequisites are managed as separate Helm releases # using the openshell-workspace chart. diff --git a/deploy/helm/test-split-ownership.sh b/deploy/helm/test-split-ownership.sh index 30fd7360d4..7285c61d6e 100755 --- a/deploy/helm/test-split-ownership.sh +++ b/deploy/helm/test-split-ownership.sh @@ -21,6 +21,32 @@ if yq ea -e \ exit 1 fi +helm template openshell "${repo_root}/deploy/helm/openshell" \ + --namespace openshell \ + --set agentSandbox.preflight.enabled=false \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ + --set rbac.clusterScoped.create=false \ + >"${work_dir}/namespace-admin.yaml" + +cluster_scoped_kinds="$( + yq ea -N -r \ + 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding") | + [.kind, .metadata.name] | join(" ")' \ + "${work_dir}/namespace-admin.yaml" +)" +if [[ -n "${cluster_scoped_kinds}" ]]; then + echo "gateway chart rendered cluster-scoped objects despite rbac.clusterScoped.create=false:" >&2 + echo "${cluster_scoped_kinds}" >&2 + exit 1 +fi + +if ! yq ea -e \ + 'select(.kind == "ServiceAccount" and .metadata.name == "openshell")' \ + "${work_dir}/namespace-admin.yaml" >/dev/null 2>&1; then + echo "gateway chart dropped the gateway ServiceAccount with rbac.clusterScoped.create=false" >&2 + exit 1 +fi + helm template openshell-workspace "${repo_root}/deploy/helm/openshell-workspace" \ --namespace app-a \ --set gateway.serviceAccount.name=openshell \ diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index b43ac2b073..b16065aa61 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -320,7 +320,10 @@ The chart creates the following RBAC resources in the release namespace: | ServiceAccount | Namespace | `openshell` | | ServiceAccount | Namespace | `openshell-sandbox` (for sandbox pods) | | Role + RoleBinding | Namespace | `openshell-sandbox` | -| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader` | +| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader-` | + +Every other object the chart creates is namespaced. The `ClusterRole` and +`ClusterRoleBinding` in the last row are the only cluster-scoped objects. When the Kubernetes Secrets credential driver is enabled, the chart also creates an `openshell-credential-secrets` Role and RoleBinding in the credential @@ -363,6 +366,118 @@ helm upgrade --install openshell \ The ServiceAccount must already have the Role and ClusterRole bindings described above. +### Installing without cluster-admin + +By default the release creates the `ClusterRole` and `ClusterRoleBinding`, so +the installer needs cluster-scoped permissions. When cluster-scoped RBAC is +owned by a different team, or the installer is a namespace-admin GitOps +controller, split the install into a cluster-admin step and a namespace-admin +step. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount. Render them from the same values the release uses, so the rules +match the configured workspace mode and credential driver: + +```shell +helm template openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the chart with cluster-scoped +objects omitted. The release renders only namespaced objects and needs no +permission on `clusterroles` or `clusterrolebindings`: + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace are unchanged by this flag, so +the pre-created `ClusterRoleBinding` still binds the ServiceAccount the +namespaced release creates. The same holds with `serviceAccount.create=false`: +the binding subject follows `serviceAccount.name`, so pass the same values to +both steps. + +#### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of the workspace mode. `managed` and +`operator` modes change what the `ClusterRole` grants, but they never require a +namespace-admin to apply it. Re-run the cluster-admin step after changing any +value that affects the `ClusterRole` rules. + +Set `rbac.clusterScoped.clusterRoleName` and +`rbac.clusterScoped.clusterRoleBindingName` when the cluster-admin owns the +naming. + +#### Which flag the installer needs + +In the default `shared` workspace mode the release also creates a namespaced +sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. +Kubernetes forbids granting permissions you do not hold, and the built-in +`admin` ClusterRole does not cover that CRD, so an installer holding only +`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not +enough, and the install fails with `attempting to grant RBAC permissions not +currently held`. + +| Workspace mode | Installer holds | Use | +|---|---|---| +| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | +| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | + +`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant +is needed there. + +With `rbac.create=false`, the cluster-admin applies the namespaced RBAC in the +same step by adding it to the render: + +```shell +helm template openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml \ + --show-only templates/role.yaml \ + --show-only templates/rolebinding.yaml \ + --show-only templates/peer-role.yaml | kubectl apply -f - +``` + +To grant the installer the sandbox permissions instead, bind it to a Role +carrying the same rules as the chart's `openshell-sandbox` Role. + ## Probes The gateway exposes `/healthz` for process liveness and `/readyz` for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes: