From 93278a3455e8b14803c1aa623221a3dd57cbad23 Mon Sep 17 00:00:00 2001 From: ansjindal Date: Thu, 17 Sep 2026 13:36:12 +0200 Subject: [PATCH] feat(helm): make cluster-scoped RBAC optional The gateway chart always rendered the ClusterRole and ClusterRoleBinding, so every install and upgrade required cluster-admin even when only namespaced objects were needed. Installers that are namespace-admin GitOps or platform controllers could not run the release at all, and clusters where cluster-scoped RBAC is owned by a separate team had no supported way to split the install. Add an rbac values block so a cluster-admin can apply the cluster-scoped objects once and a namespace-admin can install and upgrade the release without cluster-scoped permissions: rbac: create: true clusterScoped: create: true clusterRoleName: "" clusterRoleBindingName: "" rbac.clusterScoped.create gates the ClusterRole and ClusterRoleBinding, and is independent of the workspace mode. rbac.create additionally gates the namespaced sandbox Role and RoleBinding, which matters because Kubernetes escalation prevention stops an installer holding only the built-in admin role from creating a Role that grants agents.x-k8s.io verbs it does not itself hold. The certgen hook and credential driver RBAC keep their existing flags. Both flags default to true, so current installs are unchanged. The helpers treat a missing rbac block as enabled so upgrades with --reuse-values do not drop RBAC, matching the existing workspaceResources pattern. The ClusterRoleBinding roleRef follows clusterRoleName so a separately applied ClusterRole can carry a name the cluster-admin chooses. Document the migration for a release that already owns the cluster-scoped objects: Helm deletes objects that leave the manifest, so annotate them with helm.sh/resource-policy=keep before setting the flag, otherwise the gateway loses TokenReview until a cluster-admin re-applies them. Signed-off-by: ansjindal --- deploy/helm/openshell/README.md | 112 +++++++++++++++++ deploy/helm/openshell/README.md.gotmpl | 109 ++++++++++++++++ .../openshell/ci/values-namespace-admin.yaml | 25 ++++ deploy/helm/openshell/templates/_helpers.tpl | 60 +++++++++ .../helm/openshell/templates/clusterrole.yaml | 4 +- .../templates/clusterrolebinding.yaml | 6 +- .../helm/openshell/templates/peer-role.yaml | 2 + deploy/helm/openshell/templates/role.yaml | 2 +- .../helm/openshell/templates/rolebinding.yaml | 2 +- .../openshell/tests/clusterrole_test.yaml | 39 ++++++ .../tests/clusterrolebinding_test.yaml | 45 +++++++ deploy/helm/openshell/tests/rbac_test.yaml | 95 ++++++++++++++ deploy/helm/openshell/values.yaml | 20 +++ deploy/helm/test-split-ownership.sh | 26 ++++ docs/kubernetes/setup.mdx | 117 +++++++++++++++++- 15 files changed, 658 insertions(+), 6 deletions(-) create mode 100644 deploy/helm/openshell/ci/values-namespace-admin.yaml create mode 100644 deploy/helm/openshell/tests/rbac_test.yaml diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 4d09c32de8..bfb60ddb80 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -17,6 +17,114 @@ namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. +## Cluster-scoped vs namespaced objects + +Most objects in this chart are namespaced and land in the release namespace. +Only two are cluster-scoped: + +| Object | Default name | +| --- | --- | +| `ClusterRole` | `-node-reader-` | +| `ClusterRoleBinding` | `-node-reader-` | + +By default the release creates both, so an install by a cluster-admin is +unchanged. On clusters where cluster-scoped RBAC is owned by a different team, +split the install in two. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount, rendered from the same values the release uses: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the release with cluster-scoped +objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) +or the equivalent `--set`: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace do not change, so the +pre-created `ClusterRoleBinding` keeps matching the release. This works with +`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows +`serviceAccount.name`, so render the admin step with the same values. + +### Which flag the installer needs + +In the default `shared` workspace mode the release also creates a namespaced +sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. +Kubernetes forbids granting permissions you do not hold, and the built-in +`admin` ClusterRole does not cover that CRD, so an installer holding only +`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not +enough and the install fails with `attempting to grant RBAC permissions not +currently held`. + +| Workspace mode | Installer holds | Use | +| --- | --- | --- | +| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | +| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | + +`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant +is needed there. + +With `rbac.create=false` the cluster-admin applies the namespaced RBAC too, +adding it to the same render: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml \ + --show-only templates/role.yaml \ + --show-only templates/rolebinding.yaml \ + --show-only templates/peer-role.yaml | kubectl apply -f - +``` + +To grant the installer the sandbox permissions instead, bind it to a Role +carrying the same rules as the chart's `openshell-sandbox` Role. + +The certgen hook and credential driver RBAC keep their own flags +(`pkiInitJob.enabled` and +`server.credentialDrivers.kubernetesSecrets.rbac.create`). + +### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of +`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change +what the `ClusterRole` contains, but they never force the namespaced release to +apply it. Re-run the cluster-admin step after changing values that affect the +`ClusterRole` rules. + ## Prerequisites > **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for @@ -260,6 +368,10 @@ discovery endpoint or its TLS CA. | probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. | | probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. | | probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. | +| rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. | +| rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `-node-reader-` default. | +| rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. | +| rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox and peer Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. | | replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. | | resources | object | `{}` | Gateway pod resource requests and limits. | | sandbox.image.digest | string | `""` | Sandbox image digest. When set, this takes precedence over tag. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 6e49260c20..1e86e0cbcf 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -17,6 +17,115 @@ namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. +## Cluster-scoped vs namespaced objects + +Most objects in this chart are namespaced and land in the release namespace. +Only two are cluster-scoped: + +| Object | Default name | +| --- | --- | +| `ClusterRole` | `-node-reader-` | +| `ClusterRoleBinding` | `-node-reader-` | + +By default the release creates both, so an install by a cluster-admin is +unchanged. On clusters where cluster-scoped RBAC is owned by a different team, +split the install in two. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount, rendered from the same values the release uses: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the release with cluster-scoped +objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) +or the equivalent `--set`: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace do not change, so the +pre-created `ClusterRoleBinding` keeps matching the release. This works with +`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows +`serviceAccount.name`, so render the admin step with the same values. + +### Which flag the installer needs + +In the default `shared` workspace mode the release also creates a namespaced +sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. +Kubernetes forbids granting permissions you do not hold, and the built-in +`admin` ClusterRole does not cover that CRD, so an installer holding only +`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not +enough and the install fails with `attempting to grant RBAC permissions not +currently held`. + +| Workspace mode | Installer holds | Use | +| --- | --- | --- | +| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | +| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | + +`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant +is needed there. + +With `rbac.create=false` the cluster-admin applies the namespaced RBAC too, +adding it to the same render: + +```shell +helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --namespace openshell -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml \ + --show-only templates/role.yaml \ + --show-only templates/rolebinding.yaml \ + --show-only templates/peer-role.yaml | kubectl apply -f - +``` + +To grant the installer the sandbox permissions instead, bind it to a Role +carrying the same rules as the chart's `openshell-sandbox` Role. + +The certgen hook and credential driver RBAC keep their own flags +(`pkiInitJob.enabled` and +`server.credentialDrivers.kubernetesSecrets.rbac.create`). + +### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of +`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change +what the `ClusterRole` contains, but they never force the namespaced release to +apply it. Re-run the cluster-admin step after changing values that affect the +`ClusterRole` rules. + + ## Prerequisites > **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for diff --git a/deploy/helm/openshell/ci/values-namespace-admin.yaml b/deploy/helm/openshell/ci/values-namespace-admin.yaml new file mode 100644 index 0000000000..7326be7328 --- /dev/null +++ b/deploy/helm/openshell/ci/values-namespace-admin.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Namespace-admin overlay — renders only namespaced objects. +# +# Use this when a cluster-admin applies the gateway ClusterRole and +# ClusterRoleBinding once, out of band, and the OpenShell release is installed +# and upgraded by an installer that holds no cluster-scoped permissions. +# +# Generate the cluster-scoped objects for the cluster-admin step from the same +# release values, then apply them as cluster-admin: +# helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \ +# --version --namespace openshell -f my-values.yaml \ +# --set rbac.create=true \ +# --set rbac.clusterScoped.create=true \ +# --set agentSandbox.preflight.enabled=false \ +# --show-only templates/clusterrole.yaml \ +# --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +# +# The gateway ServiceAccount name and release namespace are unchanged, so the +# pre-created ClusterRoleBinding still matches this release. + +rbac: + clusterScoped: + create: false diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 32e6b7cb63..ab42458759 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -86,6 +86,66 @@ default to enabled so upgrades with --reuse-values preserve the old topology. {{- if $enabled -}}true{{- end -}} {{- end }} +{{/* +Whether this chart owns gateway RBAC objects. Missing legacy values default to +enabled so upgrades with --reuse-values preserve the old topology. +*/}} +{{- define "openshell.rbacCreate" -}} +{{- $rbac := .Values.rbac | default dict -}} +{{- $create := true -}} +{{- if hasKey $rbac "create" -}} +{{- $create = get $rbac "create" -}} +{{- end -}} +{{- if $create -}}true{{- end -}} +{{- end }} + +{{/* +The rbac.clusterScoped values map, tolerating missing legacy values. +*/}} +{{- define "openshell.clusterScopedRbacValues" -}} +{{- $rbac := .Values.rbac | default dict -}} +{{- $clusterScoped := dict -}} +{{- if hasKey $rbac "clusterScoped" -}} +{{- $clusterScoped = get $rbac "clusterScoped" | default dict -}} +{{- end -}} +{{- toYaml $clusterScoped -}} +{{- end }} + +{{/* +Whether this chart owns the cluster-scoped ClusterRole and ClusterRoleBinding. +Disable for a namespace-admin install where a cluster-admin applies them +separately. Missing legacy values default to enabled. +*/}} +{{- define "openshell.clusterRbacCreate" -}} +{{- if include "openshell.rbacCreate" . -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $create := true -}} +{{- if hasKey $clusterScoped "create" -}} +{{- $create = get $clusterScoped "create" -}} +{{- end -}} +{{- if $create -}}true{{- end -}} +{{- end -}} +{{- end }} + +{{/* +Name of the gateway ClusterRole. The release namespace is part of the default +name so multiple releases on one cluster do not collide. +*/}} +{{- define "openshell.clusterRoleName" -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}} +{{- default $default (get $clusterScoped "clusterRoleName") -}} +{{- end }} + +{{/* +Name of the gateway ClusterRoleBinding. +*/}} +{{- define "openshell.clusterRoleBindingName" -}} +{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}} +{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}} +{{- default $default (get $clusterScoped "clusterRoleBindingName") -}} +{{- end }} + {{/* Gateway image reference. A digest takes precedence over a tag. */}} {{- define "openshell.image" -}} {{- $image := .Values.gateway.image -}} diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index 17c483deb9..85d10c6b60 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -1,3 +1,4 @@ +{{- if include "openshell.clusterRbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -5,7 +6,7 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleName" . }} labels: {{- include "openshell.labels" . | nindent 4 }} rules: @@ -128,3 +129,4 @@ rules: - update {{- end }} {{- end }} +{{- end }} diff --git a/deploy/helm/openshell/templates/clusterrolebinding.yaml b/deploy/helm/openshell/templates/clusterrolebinding.yaml index 9b3b254586..8de246730b 100644 --- a/deploy/helm/openshell/templates/clusterrolebinding.yaml +++ b/deploy/helm/openshell/templates/clusterrolebinding.yaml @@ -1,17 +1,19 @@ +{{- if include "openshell.clusterRbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleBindingName" . }} labels: {{- include "openshell.labels" . | nindent 4 }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }} + name: {{ include "openshell.clusterRoleName" . }} subjects: - kind: ServiceAccount name: {{ include "openshell.serviceAccountName" . }} namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/peer-role.yaml b/deploy/helm/openshell/templates/peer-role.yaml index 59ae7479ad..3aa2502f46 100644 --- a/deploy/helm/openshell/templates/peer-role.yaml +++ b/deploy/helm/openshell/templates/peer-role.yaml @@ -1,3 +1,4 @@ +{{- if include "openshell.rbacCreate" . }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 @@ -33,3 +34,4 @@ subjects: - kind: ServiceAccount name: {{ include "openshell.serviceAccountName" . }} namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index b47efa090b..d9237e743d 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/templates/rolebinding.yaml b/deploy/helm/openshell/templates/rolebinding.yaml index 32f11644bf..49cc6f7fba 100644 --- a/deploy/helm/openshell/templates/rolebinding.yaml +++ b/deploy/helm/openshell/templates/rolebinding.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/tests/clusterrole_test.yaml b/deploy/helm/openshell/tests/clusterrole_test.yaml index 326bc2cb60..08907c614a 100644 --- a/deploy/helm/openshell/tests/clusterrole_test.yaml +++ b/deploy/helm/openshell/tests/clusterrole_test.yaml @@ -163,3 +163,42 @@ tests: apiGroups: [""] resources: ["services"] verbs: ["create", "get"] + - it: omits the ClusterRole when cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRole when all chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRole independently of the workspace mode + set: + rbac.clusterScoped.create: false + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 0 + + - it: uses the configured ClusterRole name + set: + rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader + asserts: + - equal: + path: metadata.name + value: platform-openshell-node-reader + + - it: creates the ClusterRole when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-node-reader-my-namespace diff --git a/deploy/helm/openshell/tests/clusterrolebinding_test.yaml b/deploy/helm/openshell/tests/clusterrolebinding_test.yaml index 5b9a83c04c..4cbf6d7d91 100644 --- a/deploy/helm/openshell/tests/clusterrolebinding_test.yaml +++ b/deploy/helm/openshell/tests/clusterrolebinding_test.yaml @@ -29,3 +29,48 @@ tests: kind: ServiceAccount name: openshell namespace: my-namespace + + - it: omits the ClusterRoleBinding when cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the ClusterRoleBinding when all chart-managed RBAC is disabled + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: uses the configured ClusterRoleBinding and ClusterRole names + set: + rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader + rbac.clusterScoped.clusterRoleBindingName: platform-openshell-node-reader-binding + asserts: + - equal: + path: metadata.name + value: platform-openshell-node-reader-binding + - equal: + path: roleRef.name + value: platform-openshell-node-reader + + - it: binds the custom gateway service account created by the namespaced release + set: + serviceAccount.create: false + serviceAccount.name: my-existing-sa + asserts: + - contains: + path: subjects + content: + kind: ServiceAccount + name: my-existing-sa + namespace: my-namespace + + - it: creates the ClusterRoleBinding when legacy values omit the rbac block + set: + rbac: null + asserts: + - hasDocuments: + count: 1 diff --git a/deploy/helm/openshell/tests/rbac_test.yaml b/deploy/helm/openshell/tests/rbac_test.yaml new file mode 100644 index 0000000000..3698465e27 --- /dev/null +++ b/deploy/helm/openshell/tests/rbac_test.yaml @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: Gateway RBAC creation +templates: + - templates/role.yaml + - templates/rolebinding.yaml + - templates/peer-role.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: creates the namespaced sandbox Role by default + template: templates/role.yaml + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-sandbox + + - it: creates the namespaced sandbox RoleBinding by default + template: templates/rolebinding.yaml + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell-sandbox + + - it: creates the gateway peer Role and RoleBinding by default + template: templates/peer-role.yaml + asserts: + - hasDocuments: + count: 2 + - equal: + path: metadata.name + value: openshell-peer + + - it: keeps the namespaced RBAC when only cluster-scoped RBAC is disabled + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 1 + template: templates/role.yaml + + - it: keeps the peer RBAC when only cluster-scoped RBAC is disabled + template: templates/peer-role.yaml + set: + rbac.clusterScoped.create: false + asserts: + - hasDocuments: + count: 2 + + - it: omits the namespaced sandbox Role when chart-managed RBAC is disabled + template: templates/role.yaml + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the namespaced sandbox RoleBinding when chart-managed RBAC is disabled + template: templates/rolebinding.yaml + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: omits the gateway peer Role and RoleBinding when chart-managed RBAC is disabled + template: templates/peer-role.yaml + set: + rbac.create: false + asserts: + - hasDocuments: + count: 0 + + - it: creates the namespaced sandbox Role when legacy values omit the rbac block + template: templates/role.yaml + set: + rbac: null + asserts: + - hasDocuments: + count: 1 + + - it: creates the peer Role and RoleBinding when legacy values omit the rbac block + template: templates/peer-role.yaml + set: + rbac: null + asserts: + - hasDocuments: + count: 2 diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 426f3ae424..4e7e11142d 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -141,6 +141,26 @@ sandboxServiceAccount: # -- Existing service account name for sandbox pods when sandboxServiceAccount.create is false. name: "" +# RBAC objects created for the gateway ServiceAccount. Cluster-scoped objects +# can be omitted so a cluster-admin applies them once and a namespace-admin +# installs and upgrades the gateway release without cluster-scoped permissions. +rbac: + # -- Create the RBAC objects that grant the gateway ServiceAccount access. + # Disable to supply the namespaced sandbox and peer Role/RoleBinding and the + # cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook + # and credential driver RBAC keep their own flags. + create: true + clusterScoped: + # -- Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable + # for a namespace-admin install where a cluster-admin applies them + # separately; the gateway ServiceAccount name and namespace are unchanged, + # so a pre-created ClusterRoleBinding still matches. + create: true + # -- Name for the ClusterRole. Empty uses the `-node-reader-` default. + clusterRoleName: "" + # -- Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. + clusterRoleBindingName: "" + # Namespace-scoped resources needed to run sandboxes. Disable this when the # gateway and workspace prerequisites are managed as separate Helm releases # using the openshell-workspace chart. diff --git a/deploy/helm/test-split-ownership.sh b/deploy/helm/test-split-ownership.sh index 30fd7360d4..7285c61d6e 100755 --- a/deploy/helm/test-split-ownership.sh +++ b/deploy/helm/test-split-ownership.sh @@ -21,6 +21,32 @@ if yq ea -e \ exit 1 fi +helm template openshell "${repo_root}/deploy/helm/openshell" \ + --namespace openshell \ + --set agentSandbox.preflight.enabled=false \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ + --set rbac.clusterScoped.create=false \ + >"${work_dir}/namespace-admin.yaml" + +cluster_scoped_kinds="$( + yq ea -N -r \ + 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding") | + [.kind, .metadata.name] | join(" ")' \ + "${work_dir}/namespace-admin.yaml" +)" +if [[ -n "${cluster_scoped_kinds}" ]]; then + echo "gateway chart rendered cluster-scoped objects despite rbac.clusterScoped.create=false:" >&2 + echo "${cluster_scoped_kinds}" >&2 + exit 1 +fi + +if ! yq ea -e \ + 'select(.kind == "ServiceAccount" and .metadata.name == "openshell")' \ + "${work_dir}/namespace-admin.yaml" >/dev/null 2>&1; then + echo "gateway chart dropped the gateway ServiceAccount with rbac.clusterScoped.create=false" >&2 + exit 1 +fi + helm template openshell-workspace "${repo_root}/deploy/helm/openshell-workspace" \ --namespace app-a \ --set gateway.serviceAccount.name=openshell \ diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index b43ac2b073..b16065aa61 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -320,7 +320,10 @@ The chart creates the following RBAC resources in the release namespace: | ServiceAccount | Namespace | `openshell` | | ServiceAccount | Namespace | `openshell-sandbox` (for sandbox pods) | | Role + RoleBinding | Namespace | `openshell-sandbox` | -| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader` | +| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader-` | + +Every other object the chart creates is namespaced. The `ClusterRole` and +`ClusterRoleBinding` in the last row are the only cluster-scoped objects. When the Kubernetes Secrets credential driver is enabled, the chart also creates an `openshell-credential-secrets` Role and RoleBinding in the credential @@ -363,6 +366,118 @@ helm upgrade --install openshell \ The ServiceAccount must already have the Role and ClusterRole bindings described above. +### Installing without cluster-admin + +By default the release creates the `ClusterRole` and `ClusterRoleBinding`, so +the installer needs cluster-scoped permissions. When cluster-scoped RBAC is +owned by a different team, or the installer is a namespace-admin GitOps +controller, split the install into a cluster-admin step and a namespace-admin +step. + +A cluster-admin applies the cluster-scoped objects once per gateway +ServiceAccount. Render them from the same values the release uses, so the rules +match the configured workspace mode and credential driver: + +```shell +helm template openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +``` + +A namespace-admin then installs and upgrades the chart with cluster-scoped +objects omitted. The release renders only namespaced objects and needs no +permission on `clusterroles` or `clusterrolebindings`: + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.clusterScoped.create=false +``` + +The gateway ServiceAccount name and namespace are unchanged by this flag, so +the pre-created `ClusterRoleBinding` still binds the ServiceAccount the +namespaced release creates. The same holds with `serviceAccount.create=false`: +the binding subject follows `serviceAccount.name`, so pass the same values to +both steps. + +#### Migrating an existing release + +Helm deletes objects that leave a release manifest, so setting +`rbac.clusterScoped.create=false` on a release that already owns the +`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses +TokenReview until a cluster-admin re-applies them. Hand ownership over first, as +cluster-admin, so nothing is deleted: + +```shell +kubectl annotate clusterrole "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate clusterrolebinding "openshell-node-reader-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flag, and the cluster-admin +owns them from that point on. Fresh installs need no such step. + +`rbac.clusterScoped.create` is independent of the workspace mode. `managed` and +`operator` modes change what the `ClusterRole` grants, but they never require a +namespace-admin to apply it. Re-run the cluster-admin step after changing any +value that affects the `ClusterRole` rules. + +Set `rbac.clusterScoped.clusterRoleName` and +`rbac.clusterScoped.clusterRoleBindingName` when the cluster-admin owns the +naming. + +#### Which flag the installer needs + +In the default `shared` workspace mode the release also creates a namespaced +sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. +Kubernetes forbids granting permissions you do not hold, and the built-in +`admin` ClusterRole does not cover that CRD, so an installer holding only +`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not +enough, and the install fails with `attempting to grant RBAC permissions not +currently held`. + +| Workspace mode | Installer holds | Use | +|---|---|---| +| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | +| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | + +`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant +is needed there. + +With `rbac.create=false`, the cluster-admin applies the namespaced RBAC in the +same step by adding it to the render: + +```shell +helm template openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + -f my-values.yaml \ + --set rbac.create=true \ + --set rbac.clusterScoped.create=true \ + --set agentSandbox.preflight.enabled=false \ + --show-only templates/clusterrole.yaml \ + --show-only templates/clusterrolebinding.yaml \ + --show-only templates/role.yaml \ + --show-only templates/rolebinding.yaml \ + --show-only templates/peer-role.yaml | kubectl apply -f - +``` + +To grant the installer the sandbox permissions instead, bind it to a Role +carrying the same rules as the chart's `openshell-sandbox` Role. + ## Probes The gateway exposes `/healthz` for process liveness and `/readyz` for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes: