feat: Profile for AKS by AdminTurnedDevOps · Pull Request #161 · kagent-dev/substrate

AdminTurnedDevOps · 2026-11-08T21:59:09Z

Resolves #160

The goal is a universal installer.

As people use the universal installer, however, they will need the ability to specify certain configurations for where they are installing. For example, if someone is installing on Azure Kubernetes Service (AKS), they may want to point to Azure Container Registry (ACR) or managed grafana (it's the primary monitoring option when you're creating an AKS cluster).

Therefore, the Profiles feature works if you want to have specific configurations for certain cluster types (like AKS).

Right now, this is just for AKS. However, if the proposed design/architecture is approved, we can use it as a "standard" to create profiles for other types of clusters as well (GKE, EKS, etc.).

2026-11-08_16-53-25.mp4

This PR also contains a fix for the installer.

Tests pass
Docs updated

EItanya · 2026-11-08T22:54:42Z

+  # Other host files the providers read, mounted read-only at the same path
+  # (AKS: /etc/kubernetes/azure.json).
+  imageCredentialProviderHostFiles: []


Can we just expose extra volumes + extra volume mounts for this?

EItanya · 2026-11-08T22:55:41Z

+limitations under the License.
+*/}}
+
+{{- /* atelet's PodMonitor is in atelet.yaml, so substrate-atelet renders it too. */}}


Atelet should be installed via substrate-atelet chart. this is just here for compatibility for one release

The substrate chart's atelet template is back to upstream's version, and extraVolumes/extraVolumeMounts and the atelet PodMonitor are now only in substrate-atelet.

EItanya · 2026-11-08T22:56:24Z

+  # Other host files the providers read, mounted read-only at the same path
+  # (AKS: /etc/kubernetes/azure.json).
+  imageCredentialProviderHostFiles: []


Same as above

EItanya · 2026-11-08T23:40:16Z

Thanks for this. Direction I'd like: a profile is just a well-known install config file that we embed, and profile: aks / --profile aks layers it under the user's config. No profile-specific code paths.

So for AKS:

profiles/aks.yaml sets the static AKS bits: the credential provider paths plus an extraVolumes/extraVolumeMounts entry for azure.json on the atelet release (commented out, with the service principal caveat), podMonitors.apiVersion: azmonitoring.coreos.com/v1, and storageClass: managed-csi-premium, with comments for the az setup steps.
ACR and node pool are per-cluster, so users set global.imageRegistry and atelet.nodeSelector themselves.
Drop aks.go, the AKS config struct, the ProfileAKS branches in install/upgrade/wizard, and the AKS help text. If we want KVM or NetworkPolicy checks, let's add them as generic checks separately.
The profile list comes from the embedded directory. Releases values from the profile deep-merge under the user's. Add a test that every embedded profile parses and validates.

Could you also split out the generic pieces as their own PRs first: the PodCertificateRequest v1 fix, storageClass/rustfs.storageClassName, a top-level imageRegistry (postgres needs it), the PodMonitor templates, and atelet extraVolumes/extraVolumeMounts?

Comment left by Claude on behalf of @EItanya

AdminTurnedDevOps · 2026-11-08T23:52:05Z

@EItanya Gotcha, makes sense! So you're thinking more of like a config file/YAML that someone points to with the specific configs they need for the cluster they're targeting vs having it has a flag/sub-command like --profile?

EItanya · 2026-11-09T13:34:20Z

Sort of, i was imagining profiles are just shortcuts for a well known set of yaml configs that we bake in at build time. Like istioctl

AdminTurnedDevOps · 2026-11-10T00:07:15Z

Sort of, i was imagining profiles are just shortcuts for a well known set of yaml configs that we bake in at build time. Like istioctl

@EItanya - let me know if you like this approach to simplifying the profiles:

Profiles are embedded files. pkg/kagent/install/profiles/aks.yaml is the only one, and it sets storageClass: managed-csi-premium. Adding a profile means adding a YAML file there.
Layering happens at read time. --profile aks deep-merges the user's config over the profile, then decodes it over the defaults. The config file has no profile key.
Config has no Profile field. The profile is consumed while reading, so installer code cannot branch on it. The wizard's printed config is therefore fully resolved and doesn't mention the profile.
aks.acr becomes a generic imageRegistry. It sets global.imageRegistry on the releases and renames the bundled PostgreSQL image on any cluster.
The KVM check is generic. With microvm.enabled, install now waits for a node to advertise ate.dev/kvm on every cluster. A microvm install on nodes without /dev/kvm (for example a local kind cluster) now fails after two minutes where it used to succeed.
AKS code is gone. aks.go, the AKS config section, the NetworkPolicy, node pool and Managed Prometheus checks, the "Check AKS" upgrade step and the wizard's AKS questions are all removed. upgrade.go matches upstream again.
The wizard asks for a profile first, then pre-fills the remaining questions from it.

EItanya · 2026-11-10T00:14:59Z

We shouldn't need profile: aks. Profile is a hint a layer above the config

AdminTurnedDevOps · 2026-11-10T00:24:43Z

We shouldn't need profile: aks. Profile is a hint a layer above the config

Are you saying it shouldn't be a key inside the config file? If so, the workflow would be that the profile is chosen by --profile aks or the wizard, and the file holds only settings. Or are you saying it shouldn't be specified at all?

Add a profile field to the install config and a --profile flag on install, upgrade and uninstall. generic keeps the current behavior. aks adds an aks section. Its acr field pulls every image install deploys from an Azure Container Registry: it sets global.imageRegistry on the charts and renames the bundled PostgreSQL image. The wizard asks for the cluster type unless --profile is given. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

An AKS cluster created without --network-policy accepts NetworkPolicy objects and ignores them, which leaves the ate-system policies and the per-WorkerPool policies without effect. With profile: aks, install and upgrade now fail unless a cilium, azure-npm or calico-node DaemonSet exists. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

storageClass sets the StorageClass of the bundled PostgreSQL and RustFS volumes, which otherwise use the cluster's default. The substrate chart gets a rustfs.storageClassName value for it. On AKS the wizard suggests managed-csi-premium. An existing PostgreSQL keeps its StorageClass, since a StatefulSet's volumeClaimTemplates can't change. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

aks.nodePool restricts atelet to one node pool through its kubernetes.azure.com/agentpool label. Install checks that the pool has nodes, and with microvm enabled, that one of them advertises ate.dev/kvm, which needs a VM size with nested virtualization. WorkerPools need the same node selector, and install prints a note saying so. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

aks.acrCredentialProvider lets atelet pull actor images from Azure Container Registry. atelet runs the node's acr-credential-provider with the kubelet's own config. The provider reads /etc/kubernetes/azure.json, which the charts now mount through a new atelet.imageCredentialProviderHostFiles value. It is off by default, since atelet can then pull from every registry the kubelet identity can. With profile aks, turning it off resets those values, so an earlier install's access goes away. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

otelEndpoint points atelet and the control plane at an existing OTLP collector by setting otel.endpoint on the atelet and substrate releases. The wizard asks for it. Install doesn't deploy a collector. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

aks.managedPrometheus has Azure Managed Prometheus scrape atelet and the control plane. It enables a new podMonitors chart value with the azmonitoring.coreos.com/v1 API. The charts then render PodMonitors for the /metrics endpoints of atelet, ate-api-server, ate-controller, atenet-router and atenet-egress. Preflight fails if the cluster doesn't serve that API. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

Install's cluster check only looked for podcertificaterequests in certificates.k8s.io/v1beta1. Kubernetes 1.37 clusters like AKS serve it only at v1, so install failed there, though the certificate controller already prefers v1. The check now accepts either version. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

atelet.extraVolumes and atelet.extraVolumeMounts add any volume to the atelet pod, so the charts don't need a value just for credential provider files. With aks.acrCredentialProvider on, install now mounts /etc/kubernetes/azure.json through them, after any the config sets. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

The substrate chart's atelet template stays only for one release, for compatibility. It goes back to upstream, and atelet.extraVolumes, atelet.extraVolumeMounts and the atelet PodMonitor stay in the substrate-atelet chart. The installer already installs atelet with that chart. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

Install waited for a node to advertise ate.dev/kvm only with profile aks. MicroVM workers can't schedule without it on any cluster, so the check now runs whenever microvm.enabled is set. It no longer filters nodes by aks.nodePool, since atelet advertises ate.dev/kvm only on the nodes it runs on. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

imageRegistry pulls every image install deploys from one registry, on any cluster. Like aks.acr, it sets global.imageRegistry on the charts and renames the bundled PostgreSQL image. A registry host with a port, like localhost:5001, is valid. The wizard no longer asks for a registry. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

A profile is becoming a built-in config file, which can't hold a value that differs per cluster or run a check. This removes aks.nodePool, aks.acrCredentialProvider and aks.managedPrometheus. It also removes the NetworkPolicy, node pool and Managed Prometheus checks that install and upgrade ran with profile aks. The chart values those options set are unchanged: atelet.nodeSelector, the atelet.imageCredentialProvider values, atelet.extraVolumes and podMonitors. Set them under releases. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

--profile names a config file embedded in kubectl ate, and the config from --config is layered over it. Maps merge and lists replace, as in Helm values. Adding a profile is adding a file to pkg/kagent/install/profiles. The aks profile sets storageClass to managed-csi-premium, which the wizard used to suggest. The config file no longer has a profile field, and there is no generic profile. Without --profile nothing is layered. The wizard asks for the profile first and fills its other questions from it. storageClass without a bundled volume is no longer an error, so the aks profile works with an external PostgreSQL and object store. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

Upgrade no longer reuses a release's values, so the note that turning an option off doesn't undo it is wrong. Upgrade also needs the --profile that install ran with, since the profile's settings are part of the config. Signed-off-by: adminturneddevops <mlevan1992@gmail.com>

AdminTurnedDevOps marked this pull request as draft November 8, 2026 22:22

AdminTurnedDevOps force-pushed the feat/installer-profiles branch from 1ad05c9 to e8a1d5d Compare November 8, 2026 22:39

AdminTurnedDevOps marked this pull request as ready for review November 8, 2026 22:39

EItanya reviewed Nov 8, 2026

View reviewed changes

AdminTurnedDevOps added 15 commits November 12, 2026 14:29

AdminTurnedDevOps force-pushed the feat/installer-profiles branch from e72cf84 to 29049e2 Compare November 12, 2026 20:52

Navigation Menu

Repository navigation

feat: Profile for AKS - #161

feat: Profile for AKS#161
AdminTurnedDevOps wants to merge 15 commits into
kagent-dev:mainfrom
AdminTurnedDevOps:feat/installer-profiles

AdminTurnedDevOps commented Nov 8, 2026 •

edited

Loading

Uh oh!

EItanya Nov 8, 2026

Uh oh!

AdminTurnedDevOps Nov 9, 2026

Uh oh!

EItanya Nov 8, 2026

Uh oh!

AdminTurnedDevOps Nov 9, 2026

Uh oh!

EItanya Nov 8, 2026

Uh oh!

AdminTurnedDevOps Nov 9, 2026

Uh oh!

EItanya commented Nov 8, 2026

Uh oh!

AdminTurnedDevOps commented Nov 8, 2026

Uh oh!

EItanya commented Nov 9, 2026

Uh oh!

AdminTurnedDevOps commented Nov 10, 2026 •

edited

Loading

Uh oh!

EItanya commented Nov 10, 2026

Uh oh!

AdminTurnedDevOps commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AdminTurnedDevOps commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EItanya Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

AdminTurnedDevOps Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

EItanya Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

AdminTurnedDevOps Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

EItanya Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

AdminTurnedDevOps Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

EItanya commented Nov 8, 2026

Uh oh!

AdminTurnedDevOps commented Nov 8, 2026

Uh oh!

EItanya commented Nov 9, 2026

Uh oh!

AdminTurnedDevOps commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EItanya commented Nov 10, 2026

Uh oh!

AdminTurnedDevOps commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AdminTurnedDevOps commented Nov 8, 2026 •

edited

Loading

AdminTurnedDevOps commented Nov 10, 2026 •

edited

Loading