Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 4 additions & 10 deletions docs/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ Nothing open. The 0.54.0 release PR's changelog was checked against what ships o
discard or race an open commit window or pending live edits
([render-fidelity.md](design/support-boundary/render-fidelity.md) §6b).
- [ ] **Watch the held applier**, so a switch-on or a `flux resume` is undone at once instead of
within the one-minute `heldClaimSteadyInterval`. Also makes ask #31 below exact.
within the one-minute `heldClaimSteadyInterval`. Also makes `claimed_resource_reconciled_total`
(ask #31) count at the moment of the resume.
[applier-claim-held-applier-watch.md](design/applier-claim-held-applier-watch.md).
- [ ] **F3: a Flux fetch is made before the hold is checked**, so a `Kustomization` resumed by hand
applies the fetched artifact at once (`internal/applier/flux.go`, `Request`).
Expand Down Expand Up @@ -51,18 +52,11 @@ Nothing open. The 0.54.0 release PR's changelog was checked against what ships o

Their argument for each is in [gitops-api-open-asks.md](future/gitops-api-open-asks.md).

- [ ] **#32**: a troubleshooting row in [bi-directional-flux.md](bi-directional-flux.md) for a
controller that replaces the `Kustomization`'s whole spec with an Update. Docs only.
- [ ] **#31**: a metric for an apply the claim did not start; today it is only the
`ClaimedResourceReconciled` Event, which the API server aggregates and expires.
- [ ] **#34**: an e2e with an in-cluster applier and a remote source cluster, provoking one apply
window collision.
- [ ] **#23**: verify with their reproduction that #431 fixed deletion-as-intent naming the
finalizing controller instead of the deleter.
- [ ] **#11**: one YAML encoder for `[CREATE]` and `[UPDATE]` bodies (indent churn).
- [ ] **#28**: `--watch-namespaces`, then an owner selector, so two instances can share one
config-plane cluster. It is the same mechanism HA needs.
- [ ] **#10**: consider the namespace when placement follows siblings. Cosmetic.
- **#28** (two instances sharing one cluster) is deferred, not open work: one instance serving many
namespaces is the tenancy model, and the restart cost behind the ask is the durable queue's.

## Attribution

Expand Down
4 changes: 2 additions & 2 deletions docs/bi-directional-argocd.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ for each of these, and for the settings that look like a fix and are not.
server-side apply under its own field manager, `gitops-reverser`, and touches nothing else of
`automated`, so `prune` and `selfHeal` stay as they are. Switching automated sync on again (Argo
CD's *Enable Auto-Sync*, `kubectl edit`) is undone within a minute, with a
`ClaimedResourceReconciled` Event; a sync Argo CD started in that minute runs to its end,
unchecked against the branch.
`ClaimedResourceReconciled` Event, and counted in `claimed_resource_reconciled_total`; a sync Argo CD
started in that minute runs to its end, unchecked against the branch.
Deleting the applier claim, or no longer naming it from any `GitTarget`, hands the switch back as it
was: syncing on its own if it did before, manual if it was. The applier claim records which in
`status.previousSuspendState` when it first switches it off: `NotSuspended` for automated sync on,
Expand Down
6 changes: 4 additions & 2 deletions docs/bi-directional-flux.md
Original file line number Diff line number Diff line change
Expand Up @@ -477,8 +477,9 @@ tip, the `Kustomization`s GitOps Reverser holds ignore the new artifact, and one
## Overriding by hand

- **`flux resume ks checkout`** applies the `Kustomization` at once, unchecked. Within a minute the
applier claim suspends it again, records a `ClaimedResourceReconciled` Event, and shows the apply as an
`Unresolved` operation that completes once nothing it started is running. It does so every time.
applier claim suspends it again, records a `ClaimedResourceReconciled` Event, counts it in
`claimed_resource_reconciled_total`, and shows the apply as an `Unresolved` operation that completes
once nothing it started is running. It does so every time.
- **`flux reconcile ks checkout`** does nothing visible while the `Kustomization` is suspended: Flux
marks the request handled without applying.
- **`flux suspend ks checkout`** is taken back with a token of GitOps Reverser's.
Expand Down Expand Up @@ -513,6 +514,7 @@ flux get ks checkout -n flux-system
| --- | --- | --- |
| `FluxClaim` `Configured=False`, `ApplierNotFound` | No `Kustomization` with that namespace and name in the applier claim's cluster, or Flux is not installed there | Create it, or recreate the applier claim with the right `kustomizationRef`; an omitted namespace is the applier claim's |
| `FluxClaim` `Configured=False`, `SuspendOwnedElsewhere` | The manifest that applies the `Kustomization` sets `spec.suspend` | Remove the field from that manifest |
| Repeated `ClaimedResourceReconciled` Events (a rising `claimed_resource_reconciled_total`), each followed by applier claim `OperationUnresolved` | A controller writes the `Kustomization` with an Update of its whole spec that omits `spec.suspend`, such as `controllerutil.CreateOrUpdate` with a spec it builds from scratch. An Update takes no ownership of a field it removes, so `SuspendOwnedElsewhere` cannot report it, and Flux applies unchecked on each of that controller's writes until the applier claim suspends it again | Make that controller copy `spec.suspend` from the object it read into the spec it writes, or apply with server-side apply and leave the field out |
| `ApplierClaimReady=False`, `SourceMismatch`, or applier claim `UnsupportedSource` | The message names the broken [`GitRepository` rule](#the-gitrepository) | Fix the `GitRepository`; the applier claim then adopts the `Kustomization`, and the apply that was waiting is asked for |
| `FluxClaim` `Configured=False`, `ApplierNamespaceNotAllowed` | The `Kustomization` is in a namespace the grants do not give, such as `flux-system` | Add it to `applierNamespaces` in the `KubeProvider`'s `spec.allowedApplierClaims` (Helm: `kubeProvider.allowedApplierClaims`) |
| `FluxClaim` `Configured=Unknown`, `NotBound` | A `GitTarget` names the applier claim, but cannot be judged yet: its `GitProvider` or `KubeProvider` does not exist | Create the missing object; the message names it |
Expand Down
110 changes: 40 additions & 70 deletions docs/future/gitops-api-open-asks.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,39 +5,18 @@ Status: **open asks from a production user**, condensed on 2026-10-24 from their
left out: #30 shipped as `GitReceiver` (#432), #33 went with the prefetch in #425, #22 is fixed
(`pkg/manifestanalyzer/repo.go`), #26 and #27 shipped (`git_branch_targets`,
`commit_requests_total`), #25 shipped in 0.44.0 (#343), and #6 is answered by immutability that
fails at admission with "delete and recreate". Kind and field names are updated to `main`
(`FluxClaim` for `FluxTrigger`, `KubeProvider` for `ClusterProvider`, `ClaimedResourceReconciled`
for `ResumedElsewhere`).

Their suggested order, as of revision 15: #32, then #31, then #34 when the e2e budget allows;
then #23, #11 and #10; #28 when two owners share a cluster.

## 32. A troubleshooting row: something replaces the `Kustomization`'s whole spec

**Ask.** One row in [the Flux guide](../bi-directional-flux.md)'s troubleshooting table: repeated
`ClaimedResourceReconciled` Events mean a controller writes the `Kustomization` with a full-spec
Update that omits `spec.suspend`; make it carry the field across.

**Why.** The guide covers a manifest that *sets* `spec.suspend` (`SuspendOwnedElsewhere`). The more
common case in practice is a controller that never sets the field and replaces the whole spec
(`controllerutil.CreateOrUpdate` with a spec it builds from scratch). An Update that removes a field
another manager owns takes no ownership of it, so `SuspendOwnedElsewhere` can never fire: the
symptom is a resume Event plus an `OperationUnresolved` on every reconcile of that controller, and
Flux applying unchecked until the claim suspends it again. They reproduced it on purpose. The fix on
their side was three lines; finding it from the guide alone would not have been obvious.

## 31. Count "resumed elsewhere" in a metric, not only in an `Event`

**Ask.** A counter for an apply the claim did not start (the resume, and the `OperationUnresolved`
that follows it), labeled like `apply_requests_total`. Possibly also a timestamp in the claim's
status.

**Why.** A resume behind the operator's back is the one way left for Flux to apply unchecked, so it
is what they most want to alarm on. Today it is only an Event, and an Event is a poor alarm source:
the API server aggregates identical ones (three resumes in three minutes showed as one Event with
`count: 3`) and drops them after an hour. They also saw the detection time vary from 2s to about
75s, which is the claim's steady poll; a watch on the held applier would close that window. That
watch is designed in [watching the held applier](../design/applier-claim-held-applier-watch.md).
fails at admission with "delete and recreate". Since revision 15: #32 is a row in
[the Flux guide](../bi-directional-flux.md#troubleshooting); #31 is
`claimed_resource_reconciled_total`, labeled by applier kind and claim; ask #23 is pinned at the
fact index, for a finalizer cleared by a PATCH or an UPDATE, in either order
(`TestFactIndex_AFinalizerClearedByAControllerNamesTheDeleter`); and #11 shipped in 0.41.0 (#290):
create and update write through one encoder, `internal/yamlstyle`, and a test fails on a second
one. A file GitOps Reverser did not write is re-indented once, on its first edit. Kind and field
names are updated to `main` (`FluxClaim` for `FluxTrigger`, `KubeProvider` for `ClusterProvider`,
`ClaimedResourceReconciled` for `ResumedElsewhere`).

Their suggested order, as of revision 15: #34 when the e2e budget allows; then #10; #28 when two
owners share a cluster. #28 is deferred: see below for why.

## 34. An e2e for an in-cluster applier with a remote source cluster

Expand All @@ -51,10 +30,11 @@ in it, but never saw the apply window withhold anything (`apply_window_collision
zero), so the part that reads `managedFields` on the remote object is untested in this topology.
An e2e that provokes one collision would settle it.

Two smaller things they noted: `OperationUnresolved` is recorded three times for one operation (on
the claim, the `GitTarget` and the `WatchRule`), and `apply_requests_total{outcome="sent"}` was
said to count both "the applier was asked" and "nothing needed asking". (On `main` an apply that is
not needed is never sent, so it is not counted; check whether that answers them.)
Two smaller things they noted, both answered. `OperationUnresolved` on the `GitTarget` and the
`WatchRule` is not recorded again: each object's readiness takes the reason of the one it depends
on (`ApplierClaimReady` on the `GitTarget`, `GitTargetReady` on the rule), so one operation shows on
three objects by design. And `apply_requests_total{outcome="sent"}` counts only an applier that was
asked: an apply that is not needed is never sent, so it is not counted.

## 28. Two operator instances cannot share one config-plane cluster

Expand All @@ -66,46 +46,36 @@ exactly one instance reconciles a given `GitTarget`. Smallest useful first, and
2. **An explicit owner selector**, a label selector or a `spec.instance`-style field paired with an
`--instance-name` flag, which covers the cluster-scoped kinds too.

**Why.** It is the same mechanism the chart's replica guard
(`charts/gitops-reverser/templates/validate-replica-count.yaml`) says is missing for HA: "Standby
replicas need ownership coordination first, so that two instances never write the same GitTarget."
They hit it moving onto a cluster that already runs an instance for its owner's own workloads: two
populations, different source clusters, providers, credentials and namespaces, and still both
instances would reconcile both sets. RBAC is no workaround: controller-runtime issues a cluster-wide
List, so a narrowed ClusterRole fails the cache sync and the manager exits.
**Why.** They hit it moving onto a cluster that already runs an instance for its owner's own
workloads: two populations, different source clusters, providers, credentials and namespaces, and
still both instances would reconcile both sets. RBAC is no workaround: controller-runtime issues a
cluster-wide List, so a narrowed ClusterRole fails the cache sync and the manager exits.

**What sharing costs them**, for calibration: one Valkey holding both populations' attribution
facts; one upgrade cadence; one RBAC posture that is the union of both; and, customer-visible, a
restart for one owner's reasons costs the other's users their commit message, because
`RenderMatchesLive` starts `Unknown` after a restart and an open window is dropped. They are not
blocked: one shared instance works.

## 23. Deletion-as-intent picks the wrong actor when a controller clears the finalizer

**Probably fixed by #431; verify with their reproduction.** #431 credits a terminating object's
removal to its deletion evidence first.

**Ask.** Attribute a deletion-as-intent `DELETE` to the actor who **set** `deletionTimestamp`, not
to whoever makes the next modification while it is set.

**Why.** Measured on `v0.40.1`: a human deletes a CR that their operator finalizes, and the commit
names the operator's service account. The operator's finalizer-removal `UPDATE` is itself a
modification of a terminating object, so it renders as the `DELETE` and wins the commit; the human's
own delete arrives a moment later and folds to a no-op because the file is already gone. The more
finalizers an object has, the more likely a controller's cleanup write is observed first.

**Reproduction** (about 90 seconds): create a namespaced CR with a finalizer, mirror it with a
`ClusterWatchRule`, delete it as a human, let the controller clear the finalizer, read the commit.

## 11. Serialize `[CREATE]` and `[UPDATE]` bodies identically

**Ask.** One encoder for both paths.

**Why.** A create rendered list items at 2-space indent, an update at 4, so every install churned
all the lines of a list to carry one field. Unverified since `v0.35.0`, but the two paths still
reach different encoders (`pkg/manifestanalyzer` renders via JSON to YAML; `internal/git/manifestedit`
uses `yaml.v3` with `SetIndent(2)`). Cosmetic, but it makes `git diff` on the mirror useless for
seeing what changed.
**Deferred, not built.** Step 1 was built and dropped before merge (#445). It removed two of the
four costs above, the shared Valkey and the restart reaching the other owner's users. It could not
remove the other two:

- **One upgrade cadence stays.** The CRDs are cluster-scoped, so both instances run against one
schema, and an API break (we make them without migrations) breaks whichever instance lags.
- **One RBAC posture stays.** The ClusterRole is still cluster-wide.

It is also not the HA mechanism the chart's replica guard
(`charts/gitops-reverser/templates/validate-replica-count.yaml`) asks for. Two instances serving
disjoint namespaces never contend for a `GitTarget`; two replicas of one instance would, and need
leader or ownership coordination instead. Against that, it adds a lasting cost: a second tenancy
model beside one instance serving many namespaces through grants (`consumerNamespaces`,
cross-namespace provider references). Every cross-namespace reference gains a "namespace this
instance does not serve" failure, and every cluster-scoped kind needs its own ownership check.

The cost their users see is the restart dropping an open window. That is better fixed inside one
instance, for every installation: the durable pending writes in [the TODO](../TODO.md). A second
owner who needs its own upgrade cadence or RBAC needs its own cluster; #28 would give it neither.

## 10. Sibling inference can place an object in another namespace's directory

Expand Down
Loading
Loading