Skip to content

refactor(control): hold leases in the agent's store, pin releases in the generation and add openmapx control - #976

Open
Medformatik wants to merge 44 commits into
mainfrom
refactor/control-agent-leases
Open

Medformatik wants to merge 44 commits into
mainfrom
refactor/control-agent-leases

Conversation

@Medformatik

@Medformatik Medformatik commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

What changed and why

This is the third of four PRs that replace the update system's signed machinery. Until now every lease lived in a directory on /data that every role could write, protected by role keys, signatures and mount masks, and several Docker effects (stop, restart, down, backups) took no lease at all, so services restart could run in the middle of an update. Leases now live in one store owned by the operations agent, every leased Docker effect goes through one admission, and release image pins move into the configuration generation. Production code changes by +6,045 / −9,185 lines (net −3,140).

  • One lease store. Leases live in infra/docker/state/ops-agent/leases.json, guarded by a lock file that is reclaimed only when its holder is provably gone (same boot, PID namespace, process start time and container). Data-manager asks the agent over HTTP with typed lease.acquire, lease.read and lease.update operations and its own token; the host CLI, the agent's fixed CLI children and the configuration helper write the store under the lock. .admin-target-control is gone, and check-ops-authority keeps it out.
  • Every leased effect admits the same way. Stop, restart, down, remove and backup create/restore now take a lease, like updates and configuration applies; pull, build, exec and probes do not. A busy target is refused before anything is written, and the refusal names the holder's operation key and points to openmapx control status. A Docker command that returned an error releases its lease with a failed result; only an unknown outcome (crash, timeout, lost daemon connection) keeps the lease for an operator.
  • openmapx control. status lists held leases, locks and unfinished operations; resolve settles a lease whose outcome the operator has checked (for a maintenance it also settles the data fence); clear releases a lease or a lock explicitly and warns about what may still be live. Every resolve and clear is appended to an audit file in the agent's state. services recover-update and its recovery modules are gone. Data-manager settles its own jobs at startup and recovery, so resolve refuses its leases and clear is the explicit release for them.
  • Release pins in the generation. A configuration generation renders the release's image pins into its own compose file and keeps the release manifest beside it. The release overlay, selections.json, current.json, latest.json and the release-store lock are gone; checked manifests are cached by digest. Clear selected release pin renders a new generation without pins. The backup-store and update-receipt locks use the new lock file.
  • openmapx control rollback. Returns to the configuration that was applied with the previous release: it renders that generation's inputs again as a new generation and recreates the services in the self-update order. It refuses when the database's newest migration is newer than the target release's migration head, which CI now writes into the release manifest from the migration journal. Rolling back also reverts configuration made since (service selection, service settings, secrets); the command and the docs say so.
  • Data-manager's API token. Data-manager answers 403 to the API's route-restricted token on any path outside its allowlist, including unknown paths, so the token's scope is checked before routing.

Moved to the next PR: removing the signing layer (records in each owner's private state keep their signatures until then), and letting compose up and data link apply hardlink plans that have real work (they still call data-manager's /link, which refuses).

Verification

  • In disposable Linux containers with a fresh PostGIS database: configuration-recovery, coordinator-recovery (57/57), export-restore (22/22) and ingestion. Configuration-recovery passed 98/98 before the review fixes and 97/98 after them: one mode of the "default CLI Update failure" test failed in 2 of 7 runs, a different mode each time. The ingestion failures are limited to the container's Python version. api-privacy passed before the review fixes, with one file passing only on rerun; it was not rerun after them.
  • Crash-point group as a non-root user: 22/22 before the review fixes. After them, every file passed, five of them only when rerun alone after the host slept during the full run. The scenario files are unchanged. The services restart known failure is gone because a restart during an update is now refused. The "agent killed during start" known failure now fails in a different way (the agent restarts, but the job stays running), and its recorded signature was updated.
  • The native update smoke (self-update and unhealthy image) in a disposable docker-in-docker container: both cases pass.
  • New tests cover lock reclaim (stale holder, reused inode, short reads, a dead reclaimer's guard), the lease store, the leased stop/restart/down/backup paths, community service ids as targets, a refused admission leaving no lease, a lost lease.acquire reply, the control commands and their audit file, pins rendered and cleared, and rollback (target selection, the migration refusal, the service order).
  • Type checks, Biome, pnpm check:policy, pnpm check-dead-code, pnpm check-unused-exports, the data-manager OpenAPI and SDK checks, actionlint, and the release-manifest, docker.yml and product-release contract tests pass.
  • Two independent reviews and a scoped re-review.

Release and upgrade impact

Product impact: breaking, migration, operator-action

@openmapx/extension-cli is covered by a patch Changeset because its bundled core changes.

Upgrade only while no update, configuration apply or recovery is in progress; leases in the old shared directory are not converted. Migration 0065 drops a fence column that is no longer used. The release overlay is no longer read, so the next configuration apply pins the release in the generation. Operators use openmapx control instead of services recover-update and the old rmdir instruction. See the control-leases and release-pins-in-generation fragments.

…ably gone

The lock records the holder's process, start time, boot, PID namespace and
container. Another writer reclaims it when the machine booted since, when the
holder shared its PID namespace and its process ended, or when the holder's
container is no longer running. Anything else is refused with the holder named
and the operator exit to clear it.
…px control

Every holder of a target lease now goes through the operations agent's own
lease store (state/ops-agent/leases.json), written under the recoverable
lock. Data-manager uses typed lease operations over its agent token; the
host CLI runs the fixed CLI in the agent's container with a host-CLI token
the agent creates, and writes the store itself only while the agent is
stopped. The fixed CLI, the configuration helper and the catalog bootstrap
use the store directly.

Stop, restart, compose down, removal and backup create and restore take a
lease on their targets, through the same admission path in the agent and in
the host CLI; a held target answers busy and names the holder. Community
service ids are leased like built-in ones.

The shared admission directory on the data volume is gone, and a policy
gate keeps it gone. Execution witnesses, data-manager terminals and the
feed-proxy terminal move to the observing role's private state;
data-manager trusts the caller its token names instead of adopting leases,
and transit candidates travel with the child's grant. Lifecycle receipts
move to state/ops-agent/lifecycle. The configuration fence names the
agent's lease.

openmapx control status, clear and resolve show and end held operations;
every change is appended to state/ops-agent/control-audit.jsonl.
Resolving a maintenance also settles the data-manager fence.
services recover-update and the in-process lifecycle exclusion are removed.
… removed

A reclaimer that judged an old holder gone could move aside a file a new
writer had just created, leaving two writers inside the lock. Reclaimers now
hold a short guard file beside the lock and remove the lock only when it is
still the file they judged; a reclaimer that died holding the guard is
reclaimed the same way. `openmapx control clear --lock` also removes a left
guard.
…g, and release known failures

The host CLI now writes the agent's lease store directly under its lock,
always. The `docker compose exec` lease transport, the agent's host-CLI role
and token, and the hidden `control lease` command are gone, so a `compose
down` that removes the agent no longer leaves its lease held.

Data-manager settles the hardlink child it ran under the agent's or the host
CLI's lease through a new `lease.settleChild` agent operation (or the store,
where it reads it), after its terminal, at startup repair and in recovery.
A known outcome is recorded even when the issuer's own observation was
interrupted; the same result recorded twice changes nothing.

The agent claims its targets once, before it records the job: a held target
answers busy with nothing written. The busy reply and the CLI message name
the holding operation and point to `openmapx control status`.

A leased stop, restart, remove, stack stop or backup whose Docker command
ran to its end and reported a failure records a failed result and releases
its lease, in the agent and in the host CLI. A command killed by a signal,
timed out, cancelled or cut off from the Docker daemon stays unknown and
keeps its lease for `openmapx control resolve`. A non-zero compose exit is a
failure, not a success.

Data-manager claims its lease before it begins: a busy target or an agent
that cannot answer ends the execution as failed with its publication
unchanged, so it never fences a configuration maintenance.

`openmapx control resolve` of a maintenance settles the data-manager fence
with the CLI's own data-manager token and the inputs the agent recorded on
the fence child in the lease store; data-manager accepts that token's
settle as the operator's resolution. `control clear` of a maintenance warns
that its fence stays prepared until `control resolve` settles it, which
still works after the clear.

The fence row no longer stores the maintenance lease id; migration 0064
only drops the old certificate column.
…the lease interfaces

The store's delegate rule now checks which child an issuer may record, from
the lease's owner or the parent child; rows no longer carry the permitted
further edges or their depth, and the agent no longer repeats the check for
data-manager. The configuration-fence settle edge had no issuer and is gone.

Roles that read the store use a local lease backend that also reads, lists
and adopts without waiting; data-manager's client over the agent's routes
has only the calls it makes.

The transit candidate on a child's row is just the candidate: the row
already names the child, its operation and its targets.
…trol clear must settle

The host CLI writes the lease store under the same lock as the agent and its helper. Holders in another PID namespace without a container ID, and holders on macOS, cannot be proved gone, so their locks need `openmapx control clear --lock`, which also removes a left reclaim guard. Clearing a maintenance leaves its data-manager fence prepared until `control resolve` settles it.
Since the leases moved into the agent's store, a data-manager hardlink leaf
ran with data-manager's own default targets instead of the consumer services
of the lease it runs under, so data-manager refused every non-empty plan and
the issuer waited until its deadline. The leaf now takes the targets the
delegation names.

The agent records data-manager's terminal status even when data-manager
already settled the leaf itself; later steps of the owner build on it.

A configuration maintenance whose root was released no longer compares its
final against the released lease tree, which the store does not keep; the
comparison ran before the release.
Fastify's default error reply carried the refusal reason in `message`, while
the data-manager client reads `error`, so the agent could not tell a busy
fence refusal, which wrote nothing, from another conflict. Its fence child
then stayed unsettled and the maintenance root was never released. The
prepare route now answers a busy refusal with that name.
…ease

Recording the terminal checks how the execution's proxy children settled,
which the lease shows only while it is held; the store keeps no released
lease. The terminal is now recorded after the settlement and before the
release, so a catalog validation no longer leaves its terminal unrecorded.

The catalog workflow test no longer reads a released lease's children,
which the store does not keep.
…Docker runtime

Its only other user, the in-process lifecycle exclusion, is gone.
…te the killed-start one

A host `services restart` during an update now takes a lease and exits busy,
so that scenario passes and leaves the registry. An agent killed during a
service start no longer leaves its configuration writer held; the restarted
agent keeps the target held with an unknown outcome while the API job stays
running, which is what that entry now describes.
…ting

The API's route-restricted token is checked against its allowlist before
routing, so a path outside that list is refused with 403 even when no route
matches it. Unknown paths still reach the not-found reply for the agent and
the host CLI, and unauthenticated requests are still refused with 401.
…recoverable lock files

Backup create, restore, delete, prune and a retention change now serialise on
one lock file in the backups directory under the recoverable lock's rules: a
holder that died is reclaimed when it is provably gone, a live one is waited
for up to a minute and then named in a busy error. The CLI's system update
receipt uses the same lock and refuses a second writer at once. Neither lock
expires by age any more, and inventories skip the lock and its reclaim files.
… add control rollback

A configuration generation now renders the release image pins into its own
compose file and keeps the release manifest beside it, so the generation names
the release it runs. A render that names no release keeps the applied
generation's pins; a release update's maintenance commits the generation with
the release it froze, and the self-update helper only verifies the images and
recreates the services. Clearing the selected pin, from the admin route or
compose release --clear, renders the applied configuration again without pins.

The release overlay, its plumbing through the stack Compose arguments, the
release store's pointers and selection receipts and the release-store lock are
gone. The agent keeps a cache of the manifests it checked, named by digest; the
newest is the release an update can apply. The generation also records its
domain, so the agent can render it again, and the configuration directory
records the last pointer move, which names the previous generation.

openmapx control rollback renders the previous generation's configuration and
release again and recreates the services in the self-update order. It refuses
when the database's newest applied migration is newer than the target
release's migration head, which CI now writes into the release manifest from
the API's migration journal, and it says that rolling back also reverts the
configuration made since. Without --yes it only prints the plan.

The producer and build metadata of a manifest are display metadata, and two
operator messages that still named the removed services recover-update now
point to openmapx control.
…ins and control rollback

The control-plane reference moves the lease store, the recoverable locks, the
release pins in the generation and the operator exit from the target design
into the current sections. The upgrading guide replaces the services
recover-update procedure with openmapx control and the old note that rollback
is not promised with the rollback command, its database refusal and the
configuration it reverts. Services administration replaces the manual rmdir
recovery with openmapx control and says which actions refuse while their
services are held; the CLI reference gains the control commands; and the
remaining mentions of the release overlay now describe the pins in the
applied configuration.
…pins and rollback

One fragment covers the agent-owned lease store, the recoverable locks and
openmapx control, superseding the services recover-update note; another covers
the release pins in the generation and openmapx control rollback. The pending
release channel note no longer speaks of hand-written overlays.
…manifest's migration head

The promotion step now reads the release's migration journal, so the
fixture provides one; the test checks the head written into the manifest and
that a journal without a valid newest entry stops the promotion.
…node

Linux reuses a freed inode number at once, so a holder whose lock was cleared
and a late reclaimer could each judge a new holder's file to be the one they
knew and remove it. Every holder record now carries a random nonce, and a lock
file counts as the same only when its inode and its bytes both match. The
existing tests for a cleared holder's release and for a replaced stale file
failed on Linux and pass now.
…its data-manager

Since target leases moved into the agent's store, the agent reads a
data-manager execution's terminal status from data-manager over HTTP. The
catalog workflow test started its agent process without a data-manager address,
so activation waited forever after the approval. The test now picks
data-manager's port before starting the agent and hands it the address and the
agent's token; the native data-manager modes take over that port.
…ny effect

A request that took its targets' lease and then failed while its plan was built, for example an update whose configuration revision is no longer current, kept every target held until an operator cleared it. Nothing has started at that point, so the lease is dropped.
A timeout or transport failure can arrive after the agent recorded the lease. Data-manager treated it as a refusal and never released that lease, so every later data-manager job and host restart of its targets was refused as busy. Only a held target and the agent's definite pre-effect refusals now end the job before any effect; other failures repeat the acquisition, which returns the lease already recorded.
…k clear accepts

A reclaimer that died while holding the reclaim guard made later acquisitions spin past their wait instead of refusing. A failed reclaim now ends in a busy error naming the guard once the wait is over, and a guard whose holder is gone is removed at once. An unreadable holder record is reclaimed only after it stayed the same file for the wait, or a second for a lock that does not wait. control clear --lock accepts every name a busy message prints, including a guard and an update receipt's lock, which is named by its path.
…s release

Rollback took its target from the last pointer move, so any render after an update, such as compose up or an admin save, made it target a generation with the same release and the pre-update configuration became unreachable. The target is now the newest retained generation whose pinned release differs from the applied one, and the pointer-move record is gone. Rollback refuses when no such generation is retained, always changes the release, and says that configuration made since is reverted and that services the target does not select keep running. A partial failure tells the operator to finish with compose up rather than roll back again, and a rollback that throws after its commit is still audited.
A busy refusal waited five seconds inside the agent's admission lock, so every other admission and job transition stalled meanwhile, including the progress of the operation holding the target. The agent now tries once inside the lock, waits for the holder outside it and tries again. The host CLI admits through the same rule, so a host effect started right after a data-manager job is not refused during its release.
…d check the store's directory

When the issuer had already settled a data-manager hardlink child from its status, data-manager's own report with a differently mapped outcome threw, aborted its release and repeated at every startup. The first known outcome now wins on both sides. The lease store refuses a directory that is not the agent's own, as the configuration writer lock does, so a CLI run as another user cannot leave a store the agent cannot read.
…ded operations

control resolve of a data-manager-owned lease removed the store row but left data-manager's job unknown, so its fence kept refusing maintenance and its recovery kept settling a lease that was gone. Resolve now refuses such a lease and points to control clear as the explicit release. control status no longer lists agent operations the audit file records as resolved or cleared.
…ay text

Nothing consumes the docker-compose.release.yml asset of a product release any more; release-manifest.json beside it carries the same data. Its INSTALL text, and comments and notes that still described the removed release overlay and release-store lock, now describe the pins in the generation and control rollback. The lease docs now say which failures release a lease in the agent and on the host, and that a busy target is waited for briefly.
…s exec support

heldTargets moves to a core test fixture, tests verify a root release with the synchronous local check, and the lease-store busy alias gives way to the lock's own error. The unused lease reason field, the remember helper and the separate admitLease step, folded into the lease rules' acquire, are gone. The crash-point engine no longer emulates compose exec and ps --services, which only the removed exec lease transport used.
…core directly

The one-call wrapper around the checked release reader is gone, and the system inspection takes the pinned service images from the release manifest helper instead of building the map by hand.
…lease on every service

The host-authority gate refuses the removed shared admission directory in source, tests and scripts and skips installed dependencies. A service start holds every registered service, so removing another service while it runs is refused as busy.
…t-leases

# Conflicts:
#	apps/ops-agent/src/docker-runtime.test.ts
The root release check these modes use is synchronous, so the assertion expects a throw rather than a rejection.
…Docker

A recreation that fails with an exit code releases the lease only while the Docker daemon still answers. The test asked the real daemon, so it failed on a machine where Docker was not running.
A lock file is identified by its inode and bytes. A single read could return part of the record, so a holder's release saw other bytes and refused as if the lock had been taken from it.
@Medformatik Medformatik self-assigned this Nov 16, 2026
…ck target

Clearing the release pin leaves a generation without pins, which rollback counted as a release of its own: after an update, a pin clear and compose up, it targeted the unpinned generation and refused although the previous release's configuration was still retained. Rollback now skips generations without pins and refuses while the applied generation pins no release.
…agent restart

Three attempts over 600 ms did not cover an agent restart, so a lease the agent recorded before it died stayed held until control clear. Data-manager now repeats the acquisition for about 30 seconds within the job's deadline. The agent answers a busy store lock or a file system error as a runtime error rather than a conflict, so data-manager repeats those too instead of treating them as a refusal that wrote nothing.
The store was checked by path and then read by path, so a file swapped in between would be read unchecked. It is now opened without following links and checked and read through the same descriptor. The coordinator lifecycle test's state snapshot reads its files the same way.
compose up claimed every service, so it was refused whenever a data job held Data Manager, including the notable-places build Data Manager starts on a fresh install. It takes no lease again, like services start and like compose up before; it only starts containers, and the commands that stop or replace them keep their leases.
Clearing the release pin leaves a generation without pins, which rollback now refuses to roll back from. The two clear tests checked that the cleared generation pointed back at the pinned one; they now check that it has no rollback target and that the pinned generation still pins its release.
…y inventory

The inventory maps every typed operation kind to its role and a business effect. The lease.acquire, lease.read, lease.update and lease.settleChild kinds data-manager uses to claim targets in the agent's store were missing from it.
…ocker daemon

An interrupted backup stops its dump and removes its partial archive before it reports the interruption, so its outcome is known. The host lease wrapper still asked the Docker daemon whether it was reachable before releasing, which delayed the CLI's exit after SIGTERM past what the backup cancellation test allows. An effect can now name the errors it raised after undoing its own work; those release at once.
Two assertions on data-manager's admission during a native helper launch lost their rejects when the neighbouring root release check became synchronous, so they asserted a promise to be a function and left the refusal unhandled.
…t-leases

Main added migration 0064 for privacy notifications, so the fence column drop is regenerated as 0065 on top of it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants