Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/concepts/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_label: Architecture

# Architecture overview

schedkit has two tools for running OCI-packaged `sched_ext` CPU schedulers: schedctl on a host, and sked in Kubernetes. Both run scheduler images, but they manage them independently.
schedkit runs OCI-packaged `sched_ext` CPU schedulers with two tools: schedctl on a host and sked in Kubernetes.

## On one host: schedctl

Expand All @@ -16,11 +16,11 @@ schedctl talks to a local Podman or containerd runtime. It verifies scheduler im
user -> schedctl -> Podman/containerd -> scheduler binary -> sched_ext
```

The CLI does not require Kubernetes. Use it for a workstation, a standalone server, or host management driven by scripts. See [schedctl](../schedctl/overview).
Use [schedctl](../schedctl/overview) on a workstation or server, directly or from scripts. It does not require Kubernetes.

## In Kubernetes: sked

sked watches cluster-scoped `Scheduler` resources. Each names a scheduler image and can select a node pool, tolerate taints, and configure container arguments, environment, pull secrets, and resources.
sked watches cluster-scoped `Scheduler` resources that specify an image, node selection, and container settings.

```text
cluster administrator
Expand Down Expand Up @@ -49,7 +49,7 @@ See [sked architecture](../sked/architecture) for the reconcile loop and failure

## What the tools share

The common input is an OCI-packaged scheduler, not a controller-to-agent protocol. sked neither imports schedctl nor shells out to it. Both implement signature verification, but their trust-policy formats and defaults differ. Configuring one tool does not configure the other.
Both tools accept OCI scheduler images. sked does not import or invoke schedctl; each tool has its own signature verifier, trust-policy format, and defaults.

Do not let both manage a scheduler on the same node. sked's conflict handling only sees Scheduler resources; it cannot account for a scheduler started by schedctl or another host process.

Expand Down
20 changes: 7 additions & 13 deletions docs/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,20 @@ slug: /intro

# Introduction

schedkit is a small ecosystem of tools for distributing, running, and orchestrating Linux CPU schedulers built on top of `sched_ext`.
schedkit distributes and runs Linux `sched_ext` CPU schedulers as OCI images.

The kernel side of pluggable schedulers is solved: recent Linux kernels let you attach a BPF-based scheduler at runtime without rebooting, recompiling, or shipping a custom kernel module. The pieces that aren't solved by the kernel are the boring-but-important ones around it: how do you ship a scheduler binary, how do you decide which one is running on a given machine, how do you roll back when something misbehaves, and how do you do any of this across more than a single host. That's the problem space schedkit lives in.
On a kernel with `sched_ext` support, a BPF-based scheduler can attach at runtime without a reboot or a custom kernel module. schedkit provides image verification, container lifecycle management, and tools to check scheduler status.

## What's in this site

The documentation is split into three sections.

**Concepts** is for readers who want to understand the design before touching anything. It covers what `sched_ext` actually is and isn't, why we settled on OCI as the distribution format for schedulers, and how the various components fit together.

**schedctl** documents the host-side command-line tool. If you have a single machine and you want to try out a different scheduler for the duration of a build, a game session, or a test run, this is what you're looking for.

**sked** documents the Kubernetes operator. It uses cluster-scoped Scheduler resources to select node pools, verifies scheduler images, and reports DaemonSet rollout status. Start with the [overview](/docs/sked/overview) or [install it with Helm](./sked/installation).

You can read the sections in order or jump straight to whichever tool you came here for. The Concepts pages are prerequisite-free; the project pages assume you've at least skimmed Concepts but try not to lean on it too hard.
- **[Concepts](./concepts/sched-ext)** explains `sched_ext`, the OCI image format, and how the tools fit together.
- **[schedctl](./schedctl/overview)** is the host CLI. Use it to check a Linux host, run a scheduler image with Podman or containerd, and inspect or stop the scheduler.
- **[sked](./sked/overview)** is the Kubernetes operator. Use cluster-scoped Scheduler resources to select node pools, configure scheduler containers, and track DaemonSet rollouts. [Install it with Helm](./sked/installation).

## Audience

We assume you're comfortable on a Linux command line, you've used containers (any runtime is fine), and for the sked sections that you've operated a Kubernetes cluster before. We don't assume you've written a scheduler. We don't even assume you know what `sched_ext` is — that's the very next page.
The guides assume familiarity with Linux and containers. The sked guides also assume experience with Kubernetes. You do not need to have written a scheduler.

## Where to file things

Documentation issues belong in the [docs repo](https://github.com/schedkit/docs/issues). Tool issues belong in the relevant project repo: [schedctl](https://github.com/schedkit/schedctl/issues) for the CLI, [sked](https://github.com/schedkit/sked/issues) for the operator. If you're not sure which it is, open it against schedctl and we'll move it if needed.
Report documentation problems in the [docs repository](https://github.com/schedkit/docs/issues). Report tool bugs in [schedctl](https://github.com/schedkit/schedctl/issues) or [sked](https://github.com/schedkit/sked/issues).
14 changes: 7 additions & 7 deletions docs/sked/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ title: Architecture

# Architecture

sked is one manager binary built with controller-runtime. The process hosts a Scheduler reconciler, a validating admission webhook, a trust-policy refresher, metrics, and health endpoints. There is no node agent or controller-to-node protocol.
sked runs a single manager binary built with controller-runtime. It contains the Scheduler reconciler, validating webhook, trust-policy refresher, metrics, and health endpoints. It communicates with nodes through Kubernetes, without a separate node agent.

```text
kubectl create/update Scheduler
Expand All @@ -32,9 +32,9 @@ The kubelet runs the scheduler image directly. sked does not install or invoke s

## Processes and availability

Helm and Kustomize deploy two manager replicas by default. The lease `b4c95cb3.schedkit.io` elects one reconciler; both replicas serve admission requests and refresh their trust policy. Extra replicas provide failover and webhook capacity, not parallel controller leaders.
Helm and Kustomize deploy two manager replicas by default. They use the lease `b4c95cb3.schedkit.io` to elect a leader. Only the leader reconciles; both replicas serve admission requests and refresh their trust policy.

The default PodDisruptionBudget keeps one controller available during voluntary evictions. A soft topology-spread rule prefers different nodes but permits co-location. Neither setting protects against all replicas sharing a failed node.
The default PodDisruptionBudget requires one controller to remain available during voluntary evictions. A soft topology-spread rule prefers different nodes but allows replicas to share a node. If that node fails, both replicas can be lost.

If all controllers stop, existing DaemonSets and scheduler pods keep running. Reconciliation and status updates stop, and Scheduler creates and updates fail because the validating webhook has `failurePolicy: Fail`. Deletion is not handled by the webhook; Kubernetes owner-reference garbage collection does not require sked to be running.

Expand All @@ -61,15 +61,15 @@ For each Scheduler, the reconciler:

Conflict arbitration does not consider taints, tolerations, image validity, pod health, or schedulers started outside sked. A blocked older Scheduler does not automatically yield to a healthy newer one. See [overlapping selectors](./crds#overlapping-selectors).

The controller replaces the generated DaemonSet spec on reconciliation. Manual changes to its pod template are not a supported configuration path. Owner references connect Scheduler to DaemonSet to pods; there is no custom cleanup finalizer.
Reconciliation replaces the DaemonSet spec, overwriting manual changes to its pod template. Owner references connect Scheduler to DaemonSet to pods; there is no custom cleanup finalizer.

### Rollouts and status

A change to the generated pod template triggers Kubernetes' default DaemonSet rolling update. sked observes that rollout; it does not implement a separate rollout engine, atomic scheduler handoff, or automatic rollback.

`status.nodes` and the `Ready`, `Progressing`, and `Degraded` conditions come from DaemonSet status. `Active` records whether the Scheduler is fully blocked by conflicts. Partial conflicts can produce both `Ready=True` and `Degraded=True`: the remaining workload is ready, but some requested nodes belong to another Scheduler.

There is no direct kernel observation. A container that stays alive after failing to attach can appear ready. Conversely, image-pull failures and crashes appear through pod availability, without a kernel-specific diagnosis. Use pod logs and node kernel state to establish the cause.
sked does not inspect the kernel. A container that stays alive after failing to attach can appear ready. Image-pull failures and crashes affect pod availability, but status does not identify kernel errors. Check pod logs and node kernel state for the cause.

Events for cluster-scoped Schedulers are stored in `default`, not the workload namespace. Normal reasons include `RolloutInProgress`, `DaemonSetReady`, and `NoNodesScheduled`; warnings include conflicts, verification and reconcile failures, unavailable nodes, and `StatusUpdateFailed`. State-change events are not repeated just because a pod count changes. Events expire; conditions describe the last recorded state.

Expand All @@ -88,7 +88,7 @@ The installed defaults differ from some standalone binary defaults:
| HTTP/2 | Disabled | Disabled |
| Logging | Slog JSON at info on stderr | Same |

Both health endpoints use a ping check; they do not certify registry access, image verification, or node scheduler health.
Both health endpoints use a ping check. They do not test registry access, image verification, or node scheduler health.

Reconcile controls are exposed as flags and corresponding `controller` chart values:

Expand All @@ -101,7 +101,7 @@ Reconcile controls are exposed as flags and corresponding `controller` chart val
| `--reconcile-retry-base-delay` | `1s` |
| `--reconcile-retry-max-delay` | `5m` |

The timeout bounds a reconcile attempt. Failed reconciles use exponential backoff plus a shared retry rate limit. These QPS and burst settings limit failed-reconcile requeues, not all Kubernetes API traffic.
The timeout limits the duration of one reconcile attempt. Failed reconciles use exponential backoff plus a shared retry rate limit. These QPS and burst settings limit failed-reconcile requeues, not all Kubernetes API traffic.

## Failure handling

Expand Down
8 changes: 4 additions & 4 deletions docs/sked/crds.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ This creates a DaemonSet named `scx-rusty` in the workload namespace, targeting
| `spec.imagePullSecrets` | `LocalObjectReference[]` | No | Named pull secrets in the **workload namespace**. Used by the kubelet, not by the controller's signature verifier. |
| `spec.resources` | Kubernetes `ResourceRequirements` | No | Compute resources for the scheduler container, including requests and limits. These do not set CPU policy for other workloads. |

No other scheduler configuration fields are defined. In particular, there is no `priority`, `imagePullPolicy`, `command`, `affinity`, `volumes`, `serviceAccountName`, probe configuration, or update strategy. Fields inherited from Kubernetes types may depend on cluster feature gates; sked does not provide a general-purpose pod template.
The CRD does not expose `priority`, `imagePullPolicy`, `command`, `affinity`, `volumes`, `serviceAccountName`, probes, or an update strategy. Fields inherited from Kubernetes types may depend on cluster feature gates. There is no general-purpose pod template.

### Container configuration example

Expand Down Expand Up @@ -75,7 +75,7 @@ spec:
memory: 128Mi
```

The resource request is an example, not a sizing recommendation. Check the scheduler's own flags and memory requirements. Treat this as an alternative to the first example, not a second Scheduler for the same pool.
Choose CPU and memory requests for your scheduler, and check its supported flags. Use this manifest instead of the first example; applying both targets the same nodes.

### Validation

Expand All @@ -100,7 +100,7 @@ The webhook handles creates and updates with `failurePolicy: Fail`. If it cannot
| `status.nodes.unavailable` | DaemonSet `numberUnavailable`. |
| `status.conditions` | Conditions keyed by `type`, with `status`, `reason`, `message`, `observedGeneration`, and `lastTransitionTime`. |

There is no phase or per-node status list. Counts come from the DaemonSet, not a count of kernels with an attached scheduler. After a reconcile failure, image and node counts may still describe the last successful reconciliation; read the conditions too.
There is no phase or per-node status list. The counts describe DaemonSet pods, not kernel attachment. After a reconcile failure, the image and counts may still reflect the last successful reconciliation. Read the conditions too.

### Conditions

Expand Down Expand Up @@ -140,7 +140,7 @@ For each node, the oldest matching Scheduler wins. Equal creation timestamps are
- **Partial overlap:** the losing Scheduler excludes conflict nodes through required node affinity on `kubernetes.io/hostname`, and runs on the remaining nodes.
- **Complete overlap:** the losing Scheduler's DaemonSet is deleted. Its node counts become zero, `resolvedImage` is cleared, and `Active=False`, `Ready=False`, and `Degraded=True` report `SchedulerConflict`.

Arbitration compares node labels only. It does not consider tolerations, kernel support, image validity, or pod health. An older Scheduler that cannot run can still block a newer one. Use non-overlapping selectors rather than relying on age as a priority mechanism. Label changes trigger reassignment, but pod termination and startup are asynchronous; this is not an atomic handoff.
Arbitration compares node labels only. It does not consider tolerations, kernel support, image validity, or pod health. An older Scheduler that cannot run can still block a newer one. Use non-overlapping selectors. Label changes trigger reassignment, but pods stop and start asynchronously; a scheduler handoff is not atomic.

## Migrating from `SchedExt`

Expand Down
16 changes: 11 additions & 5 deletions docs/sked/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ title: Development

Run these commands from the [sked repository](https://github.com/schedkit/sked), not this documentation repository.

The module is `github.com/schedkit/sked`. The current checkout uses Go 1.27 and controller-runtime v0.25.2, with Kubebuilder v4 layout. Check `go.mod` and the Makefile for the versions required by the revision you are building.
The module is `github.com/schedkit/sked`, built with Go 1.27 and controller-runtime v0.25.2. Check `go.mod` and the Makefile when building a different revision.

## Build

Expand Down Expand Up @@ -42,9 +42,15 @@ helm install sked ./charts/sked \

Alternatively, use `make deploy IMG=ghcr.io/your-org/sked:dev` for Kustomize. Do not run both installations in one cluster.

For host-side debugging, `make install` installs only the CRD, and `make run` starts the manager with your kubeconfig. This is not a complete installation: the always-on webhook server needs `tls.crt` and `tls.key` under `/tmp/k8s-webhook-server/serving-certs`, and admission requests need a webhook endpoint reachable by the API server with the corresponding CA configuration. Set the workload and trust-policy namespaces explicitly when running outside a pod. Avoid running an unelected local reconciler alongside an installed controller.
For host-side debugging, `make install` installs only the CRD, and `make run` starts the manager with your kubeconfig. You must also:

Kind is useful for API, reconciliation, and chart tests. Its nodes share the host kernel; Kind does not supply `sched_ext` support. Tests that establish actual attachment need a compatible host kernel, scheduler image, and runtime permissions. A passing chart test alone does not establish kernel compatibility.
- Provide `tls.crt` and `tls.key` under `/tmp/k8s-webhook-server/serving-certs`.
- Configure an admission webhook endpoint the API server can reach, with the matching CA.
- Set the workload and trust-policy namespaces explicitly.

Do not run an unelected local reconciler alongside an installed controller.

Kind nodes share the host kernel. API and chart tests can run without `sched_ext`, but attachment tests need a compatible host kernel, scheduler image, and runtime permissions.

## Generate manifests

Expand Down Expand Up @@ -81,7 +87,7 @@ make test-helm
- `make lint` runs the pinned golangci-lint version; `make lint-fix` applies supported automatic fixes.
- `make test-helm` checks generated chart files, lints the chart, and tests rendered resources against Kustomize. CI tests Helm 3 and 4.

Cluster tests have separate entrypoints:
Run cluster tests separately:

```bash
make test-e2e
Expand Down Expand Up @@ -113,6 +119,6 @@ make build-installer IMG=ghcr.io/your-org/sked:dev
bash hack/package-helm.sh v1.2.3 /tmp/sked-chart
```

The first command renders `dist/install.yaml`. The second illustrates chart packaging for a release tag: `v1.2.3` produces chart version `1.2.3` and `appVersion: v1.2.3` without changing the checked-in `Chart.yaml`.
The first command writes `dist/install.yaml`. The second packages chart version `1.2.3` with `appVersion: v1.2.3`, without changing the checked-in `Chart.yaml`. Replace the example tag with your release tag.

The release workflow publishes and signs the controller image, attaches the chart to the GitHub Release, and publishes it to `https://charts.schedkit.io`. Chart and controller versions are released together. Follow the [release instructions](https://github.com/schedkit/sked#releases-and-helm-repository) for tag rules and repository setup.
Loading
Loading