Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,114 @@ namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount, rendered from the same values the release uses:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:

```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.clusterScoped.create=false
```

The gateway ServiceAccount name and namespace do not change, so the
pre-created `ClusterRoleBinding` keeps matching the release. This works with
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
`serviceAccount.name`, so render the admin step with the same values.

### Which flag the installer needs

In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough and the install fails with `attempting to grant RBAC permissions not
currently held`.

| Workspace mode | Installer holds | Use |
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.

With `rbac.create=false` the cluster-admin applies the namespaced RBAC too,
adding it to the same render:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml \
--show-only templates/role.yaml \
--show-only templates/rolebinding.yaml \
--show-only templates/peer-role.yaml | kubectl apply -f -
```

To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.

The certgen hook and credential driver RBAC keep their own flags
(`pkiInitJob.enabled` and
`server.credentialDrivers.kubernetesSecrets.rbac.create`).

### Migrating an existing release

Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:

```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
what the `ClusterRole` contains, but they never force the namespaced release to
apply it. Re-run the cluster-admin step after changing values that affect the
`ClusterRole` rules.

## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
Expand Down Expand Up @@ -260,6 +368,10 @@ discovery endpoint or its TLS CA.
| probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. |
| probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. |
| probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. |
| rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `<fullname>-node-reader-<release namespace>` default. |
| rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `<fullname>-node-reader-<release namespace>` default. |
| rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. |
| rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox and peer Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. |
| replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. |
| resources | object | `{}` | Gateway pod resource requests and limits. |
| sandbox.image.digest | string | `""` | Sandbox image digest. When set, this takes precedence over tag. |
Expand Down
109 changes: 109 additions & 0 deletions deploy/helm/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,115 @@ namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount, rendered from the same values the release uses:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:

```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.clusterScoped.create=false
```

The gateway ServiceAccount name and namespace do not change, so the
pre-created `ClusterRoleBinding` keeps matching the release. This works with
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
`serviceAccount.name`, so render the admin step with the same values.

### Which flag the installer needs

In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough and the install fails with `attempting to grant RBAC permissions not
currently held`.

| Workspace mode | Installer holds | Use |
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.

With `rbac.create=false` the cluster-admin applies the namespaced RBAC too,
adding it to the same render:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml \
--show-only templates/role.yaml \
--show-only templates/rolebinding.yaml \
--show-only templates/peer-role.yaml | kubectl apply -f -
```

To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.

The certgen hook and credential driver RBAC keep their own flags
(`pkiInitJob.enabled` and
`server.credentialDrivers.kubernetesSecrets.rbac.create`).

### Migrating an existing release

Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:

```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
what the `ClusterRole` contains, but they never force the namespaced release to
apply it. Re-run the cluster-admin step after changing values that affect the
`ClusterRole` rules.


## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
Expand Down
25 changes: 25 additions & 0 deletions deploy/helm/openshell/ci/values-namespace-admin.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Namespace-admin overlay — renders only namespaced objects.
#
# Use this when a cluster-admin applies the gateway ClusterRole and
# ClusterRoleBinding once, out of band, and the OpenShell release is installed
# and upgraded by an installer that holds no cluster-scoped permissions.
#
# Generate the cluster-scoped objects for the cluster-admin step from the same
# release values, then apply them as cluster-admin:
# helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \
# --version <version> --namespace openshell -f my-values.yaml \
# --set rbac.create=true \
# --set rbac.clusterScoped.create=true \
# --set agentSandbox.preflight.enabled=false \
# --show-only templates/clusterrole.yaml \
# --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
#
# The gateway ServiceAccount name and release namespace are unchanged, so the
# pre-created ClusterRoleBinding still matches this release.

rbac:
clusterScoped:
create: false
60 changes: 60 additions & 0 deletions deploy/helm/openshell/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,66 @@ default to enabled so upgrades with --reuse-values preserve the old topology.
{{- if $enabled -}}true{{- end -}}
{{- end }}

{{/*
Whether this chart owns gateway RBAC objects. Missing legacy values default to
enabled so upgrades with --reuse-values preserve the old topology.
*/}}
{{- define "openshell.rbacCreate" -}}
{{- $rbac := .Values.rbac | default dict -}}
{{- $create := true -}}
{{- if hasKey $rbac "create" -}}
{{- $create = get $rbac "create" -}}
{{- end -}}
{{- if $create -}}true{{- end -}}
{{- end }}

{{/*
The rbac.clusterScoped values map, tolerating missing legacy values.
*/}}
{{- define "openshell.clusterScopedRbacValues" -}}
{{- $rbac := .Values.rbac | default dict -}}
{{- $clusterScoped := dict -}}
{{- if hasKey $rbac "clusterScoped" -}}
{{- $clusterScoped = get $rbac "clusterScoped" | default dict -}}
{{- end -}}
{{- toYaml $clusterScoped -}}
{{- end }}

{{/*
Whether this chart owns the cluster-scoped ClusterRole and ClusterRoleBinding.
Disable for a namespace-admin install where a cluster-admin applies them
separately. Missing legacy values default to enabled.
*/}}
{{- define "openshell.clusterRbacCreate" -}}
{{- if include "openshell.rbacCreate" . -}}
{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}}
{{- $create := true -}}
{{- if hasKey $clusterScoped "create" -}}
{{- $create = get $clusterScoped "create" -}}
{{- end -}}
{{- if $create -}}true{{- end -}}
{{- end -}}
{{- end }}

{{/*
Name of the gateway ClusterRole. The release namespace is part of the default
name so multiple releases on one cluster do not collide.
*/}}
{{- define "openshell.clusterRoleName" -}}
{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}}
{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}}
{{- default $default (get $clusterScoped "clusterRoleName") -}}
{{- end }}

{{/*
Name of the gateway ClusterRoleBinding.
*/}}
{{- define "openshell.clusterRoleBindingName" -}}
{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}}
{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}}
{{- default $default (get $clusterScoped "clusterRoleBindingName") -}}
{{- end }}

{{/* Gateway image reference. A digest takes precedence over a tag. */}}
{{- define "openshell.image" -}}
{{- $image := .Values.gateway.image -}}
Expand Down
4 changes: 3 additions & 1 deletion deploy/helm/openshell/templates/clusterrole.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
{{- if include "openshell.clusterRbacCreate" . }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" }}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }}
name: {{ include "openshell.clusterRoleName" . }}
labels:
{{- include "openshell.labels" . | nindent 4 }}
rules:
Expand Down Expand Up @@ -128,3 +129,4 @@ rules:
- update
{{- end }}
{{- end }}
{{- end }}
6 changes: 4 additions & 2 deletions deploy/helm/openshell/templates/clusterrolebinding.yaml
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
{{- if include "openshell.clusterRbacCreate" . }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }}
name: {{ include "openshell.clusterRoleBindingName" . }}
labels:
{{- include "openshell.labels" . | nindent 4 }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }}
name: {{ include "openshell.clusterRoleName" . }}
subjects:
- kind: ServiceAccount
name: {{ include "openshell.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end }}
2 changes: 2 additions & 0 deletions deploy/helm/openshell/templates/peer-role.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{{- if include "openshell.rbacCreate" . }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: rbac.authorization.k8s.io/v1
Expand Down Expand Up @@ -33,3 +34,4 @@ subjects:
- kind: ServiceAccount
name: {{ include "openshell.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end }}
2 changes: 1 addition & 1 deletion deploy/helm/openshell/templates/role.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}}
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }}
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: rbac.authorization.k8s.io/v1
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/openshell/templates/rolebinding.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}}
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }}
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: rbac.authorization.k8s.io/v1
Expand Down
Loading
Loading