Repository navigation
refactor(control): hold leases in the agent's store, pin releases in the generation and add openmapx control - #976
Open
Medformatik wants to merge 44 commits into
Open
Medformatik wants to merge 44 commits into
Medformatik wants to merge 44 commits into
Conversation
…ably gone The lock records the holder's process, start time, boot, PID namespace and container. Another writer reclaims it when the machine booted since, when the holder shared its PID namespace and its process ended, or when the holder's container is no longer running. Anything else is refused with the holder named and the operator exit to clear it.
…px control Every holder of a target lease now goes through the operations agent's own lease store (state/ops-agent/leases.json), written under the recoverable lock. Data-manager uses typed lease operations over its agent token; the host CLI runs the fixed CLI in the agent's container with a host-CLI token the agent creates, and writes the store itself only while the agent is stopped. The fixed CLI, the configuration helper and the catalog bootstrap use the store directly. Stop, restart, compose down, removal and backup create and restore take a lease on their targets, through the same admission path in the agent and in the host CLI; a held target answers busy and names the holder. Community service ids are leased like built-in ones. The shared admission directory on the data volume is gone, and a policy gate keeps it gone. Execution witnesses, data-manager terminals and the feed-proxy terminal move to the observing role's private state; data-manager trusts the caller its token names instead of adopting leases, and transit candidates travel with the child's grant. Lifecycle receipts move to state/ops-agent/lifecycle. The configuration fence names the agent's lease. openmapx control status, clear and resolve show and end held operations; every change is appended to state/ops-agent/control-audit.jsonl. Resolving a maintenance also settles the data-manager fence. services recover-update and the in-process lifecycle exclusion are removed.
… removed A reclaimer that judged an old holder gone could move aside a file a new writer had just created, leaving two writers inside the lock. Reclaimers now hold a short guard file beside the lock and remove the lock only when it is still the file they judged; a reclaimer that died holding the guard is reclaimed the same way. `openmapx control clear --lock` also removes a left guard.
…g, and release known failures The host CLI now writes the agent's lease store directly under its lock, always. The `docker compose exec` lease transport, the agent's host-CLI role and token, and the hidden `control lease` command are gone, so a `compose down` that removes the agent no longer leaves its lease held. Data-manager settles the hardlink child it ran under the agent's or the host CLI's lease through a new `lease.settleChild` agent operation (or the store, where it reads it), after its terminal, at startup repair and in recovery. A known outcome is recorded even when the issuer's own observation was interrupted; the same result recorded twice changes nothing. The agent claims its targets once, before it records the job: a held target answers busy with nothing written. The busy reply and the CLI message name the holding operation and point to `openmapx control status`. A leased stop, restart, remove, stack stop or backup whose Docker command ran to its end and reported a failure records a failed result and releases its lease, in the agent and in the host CLI. A command killed by a signal, timed out, cancelled or cut off from the Docker daemon stays unknown and keeps its lease for `openmapx control resolve`. A non-zero compose exit is a failure, not a success. Data-manager claims its lease before it begins: a busy target or an agent that cannot answer ends the execution as failed with its publication unchanged, so it never fences a configuration maintenance. `openmapx control resolve` of a maintenance settles the data-manager fence with the CLI's own data-manager token and the inputs the agent recorded on the fence child in the lease store; data-manager accepts that token's settle as the operator's resolution. `control clear` of a maintenance warns that its fence stays prepared until `control resolve` settles it, which still works after the clear. The fence row no longer stores the maintenance lease id; migration 0064 only drops the old certificate column.
…the lease interfaces The store's delegate rule now checks which child an issuer may record, from the lease's owner or the parent child; rows no longer carry the permitted further edges or their depth, and the agent no longer repeats the check for data-manager. The configuration-fence settle edge had no issuer and is gone. Roles that read the store use a local lease backend that also reads, lists and adopts without waiting; data-manager's client over the agent's routes has only the calls it makes. The transit candidate on a child's row is just the candidate: the row already names the child, its operation and its targets.
…trol clear must settle The host CLI writes the lease store under the same lock as the agent and its helper. Holders in another PID namespace without a container ID, and holders on macOS, cannot be proved gone, so their locks need `openmapx control clear --lock`, which also removes a left reclaim guard. Clearing a maintenance leaves its data-manager fence prepared until `control resolve` settles it.
Since the leases moved into the agent's store, a data-manager hardlink leaf ran with data-manager's own default targets instead of the consumer services of the lease it runs under, so data-manager refused every non-empty plan and the issuer waited until its deadline. The leaf now takes the targets the delegation names. The agent records data-manager's terminal status even when data-manager already settled the leaf itself; later steps of the owner build on it. A configuration maintenance whose root was released no longer compares its final against the released lease tree, which the store does not keep; the comparison ran before the release.
Fastify's default error reply carried the refusal reason in `message`, while the data-manager client reads `error`, so the agent could not tell a busy fence refusal, which wrote nothing, from another conflict. Its fence child then stayed unsettled and the maintenance root was never released. The prepare route now answers a busy refusal with that name.
…ease Recording the terminal checks how the execution's proxy children settled, which the lease shows only while it is held; the store keeps no released lease. The terminal is now recorded after the settlement and before the release, so a catalog validation no longer leaves its terminal unrecorded. The catalog workflow test no longer reads a released lease's children, which the store does not keep.
…Docker runtime Its only other user, the in-process lifecycle exclusion, is gone.
…te the killed-start one A host `services restart` during an update now takes a lease and exits busy, so that scenario passes and leaves the registry. An agent killed during a service start no longer leaves its configuration writer held; the restarted agent keeps the target held with an unknown outcome while the API job stays running, which is what that entry now describes.
…ting The API's route-restricted token is checked against its allowlist before routing, so a path outside that list is refused with 403 even when no route matches it. Unknown paths still reach the not-found reply for the agent and the host CLI, and unauthenticated requests are still refused with 401.
…recoverable lock files Backup create, restore, delete, prune and a retention change now serialise on one lock file in the backups directory under the recoverable lock's rules: a holder that died is reclaimed when it is provably gone, a live one is waited for up to a minute and then named in a busy error. The CLI's system update receipt uses the same lock and refuses a second writer at once. Neither lock expires by age any more, and inventories skip the lock and its reclaim files.
… add control rollback A configuration generation now renders the release image pins into its own compose file and keeps the release manifest beside it, so the generation names the release it runs. A render that names no release keeps the applied generation's pins; a release update's maintenance commits the generation with the release it froze, and the self-update helper only verifies the images and recreates the services. Clearing the selected pin, from the admin route or compose release --clear, renders the applied configuration again without pins. The release overlay, its plumbing through the stack Compose arguments, the release store's pointers and selection receipts and the release-store lock are gone. The agent keeps a cache of the manifests it checked, named by digest; the newest is the release an update can apply. The generation also records its domain, so the agent can render it again, and the configuration directory records the last pointer move, which names the previous generation. openmapx control rollback renders the previous generation's configuration and release again and recreates the services in the self-update order. It refuses when the database's newest applied migration is newer than the target release's migration head, which CI now writes into the release manifest from the API's migration journal, and it says that rolling back also reverts the configuration made since. Without --yes it only prints the plan. The producer and build metadata of a manifest are display metadata, and two operator messages that still named the removed services recover-update now point to openmapx control.
…ins and control rollback The control-plane reference moves the lease store, the recoverable locks, the release pins in the generation and the operator exit from the target design into the current sections. The upgrading guide replaces the services recover-update procedure with openmapx control and the old note that rollback is not promised with the rollback command, its database refusal and the configuration it reverts. Services administration replaces the manual rmdir recovery with openmapx control and says which actions refuse while their services are held; the CLI reference gains the control commands; and the remaining mentions of the release overlay now describe the pins in the applied configuration.
…pins and rollback One fragment covers the agent-owned lease store, the recoverable locks and openmapx control, superseding the services recover-update note; another covers the release pins in the generation and openmapx control rollback. The pending release channel note no longer speaks of hand-written overlays.
…manifest's migration head The promotion step now reads the release's migration journal, so the fixture provides one; the test checks the head written into the manifest and that a journal without a valid newest entry stops the promotion.
…node Linux reuses a freed inode number at once, so a holder whose lock was cleared and a late reclaimer could each judge a new holder's file to be the one they knew and remove it. Every holder record now carries a random nonce, and a lock file counts as the same only when its inode and its bytes both match. The existing tests for a cleared holder's release and for a replaced stale file failed on Linux and pass now.
…its data-manager Since target leases moved into the agent's store, the agent reads a data-manager execution's terminal status from data-manager over HTTP. The catalog workflow test started its agent process without a data-manager address, so activation waited forever after the approval. The test now picks data-manager's port before starting the agent and hands it the address and the agent's token; the native data-manager modes take over that port.
…ny effect A request that took its targets' lease and then failed while its plan was built, for example an update whose configuration revision is no longer current, kept every target held until an operator cleared it. Nothing has started at that point, so the lease is dropped.
A timeout or transport failure can arrive after the agent recorded the lease. Data-manager treated it as a refusal and never released that lease, so every later data-manager job and host restart of its targets was refused as busy. Only a held target and the agent's definite pre-effect refusals now end the job before any effect; other failures repeat the acquisition, which returns the lease already recorded.
…k clear accepts A reclaimer that died while holding the reclaim guard made later acquisitions spin past their wait instead of refusing. A failed reclaim now ends in a busy error naming the guard once the wait is over, and a guard whose holder is gone is removed at once. An unreadable holder record is reclaimed only after it stayed the same file for the wait, or a second for a lock that does not wait. control clear --lock accepts every name a busy message prints, including a guard and an update receipt's lock, which is named by its path.
…s release Rollback took its target from the last pointer move, so any render after an update, such as compose up or an admin save, made it target a generation with the same release and the pre-update configuration became unreachable. The target is now the newest retained generation whose pinned release differs from the applied one, and the pointer-move record is gone. Rollback refuses when no such generation is retained, always changes the release, and says that configuration made since is reverted and that services the target does not select keep running. A partial failure tells the operator to finish with compose up rather than roll back again, and a rollback that throws after its commit is still audited.
A busy refusal waited five seconds inside the agent's admission lock, so every other admission and job transition stalled meanwhile, including the progress of the operation holding the target. The agent now tries once inside the lock, waits for the holder outside it and tries again. The host CLI admits through the same rule, so a host effect started right after a data-manager job is not refused during its release.
…d check the store's directory When the issuer had already settled a data-manager hardlink child from its status, data-manager's own report with a differently mapped outcome threw, aborted its release and repeated at every startup. The first known outcome now wins on both sides. The lease store refuses a directory that is not the agent's own, as the configuration writer lock does, so a CLI run as another user cannot leave a store the agent cannot read.
…ded operations control resolve of a data-manager-owned lease removed the store row but left data-manager's job unknown, so its fence kept refusing maintenance and its recovery kept settling a lease that was gone. Resolve now refuses such a lease and points to control clear as the explicit release. control status no longer lists agent operations the audit file records as resolved or cleared.
…ay text Nothing consumes the docker-compose.release.yml asset of a product release any more; release-manifest.json beside it carries the same data. Its INSTALL text, and comments and notes that still described the removed release overlay and release-store lock, now describe the pins in the generation and control rollback. The lease docs now say which failures release a lease in the agent and on the host, and that a busy target is waited for briefly.
…s exec support heldTargets moves to a core test fixture, tests verify a root release with the synchronous local check, and the lease-store busy alias gives way to the lock's own error. The unused lease reason field, the remember helper and the separate admitLease step, folded into the lease rules' acquire, are gone. The crash-point engine no longer emulates compose exec and ps --services, which only the removed exec lease transport used.
…core directly The one-call wrapper around the checked release reader is gone, and the system inspection takes the pinned service images from the release manifest helper instead of building the map by hand.
…lease on every service The host-authority gate refuses the removed shared admission directory in source, tests and scripts and skips installed dependencies. A service start holds every registered service, so removing another service while it runs is refused as busy.
…t-leases # Conflicts: # apps/ops-agent/src/docker-runtime.test.ts
The root release check these modes use is synchronous, so the assertion expects a throw rather than a rejection.
…Docker A recreation that fails with an exit code releases the lease only while the Docker daemon still answers. The test asked the real daemon, so it failed on a machine where Docker was not running.
A lock file is identified by its inode and bytes. A single read could return part of the record, so a holder's release saw other bytes and refused as if the lock had been taken from it.
…ck target Clearing the release pin leaves a generation without pins, which rollback counted as a release of its own: after an update, a pin clear and compose up, it targeted the unpinned generation and refused although the previous release's configuration was still retained. Rollback now skips generations without pins and refuses while the applied generation pins no release.
…agent restart Three attempts over 600 ms did not cover an agent restart, so a lease the agent recorded before it died stayed held until control clear. Data-manager now repeats the acquisition for about 30 seconds within the job's deadline. The agent answers a busy store lock or a file system error as a runtime error rather than a conflict, so data-manager repeats those too instead of treating them as a refusal that wrote nothing.
The store was checked by path and then read by path, so a file swapped in between would be read unchecked. It is now opened without following links and checked and read through the same descriptor. The coordinator lifecycle test's state snapshot reads its files the same way.
compose up claimed every service, so it was refused whenever a data job held Data Manager, including the notable-places build Data Manager starts on a fresh install. It takes no lease again, like services start and like compose up before; it only starts containers, and the commands that stop or replace them keep their leases.
Clearing the release pin leaves a generation without pins, which rollback now refuses to roll back from. The two clear tests checked that the cleared generation pointed back at the pinned one; they now check that it has no rollback target and that the pinned generation still pins its release.
…y inventory The inventory maps every typed operation kind to its role and a business effect. The lease.acquire, lease.read, lease.update and lease.settleChild kinds data-manager uses to claim targets in the agent's store were missing from it.
…ocker daemon An interrupted backup stops its dump and removes its partial archive before it reports the interruption, so its outcome is known. The host lease wrapper still asked the Docker daemon whether it was reachable before releasing, which delayed the CLI's exit after SIGTERM past what the backup cancellation test allows. An effect can now name the errors it raised after undoing its own work; those release at once.
Two assertions on data-manager's admission during a native helper launch lost their rejects when the neighbouring root release check became synchronous, so they asserted a promise to be a function and left the refusal unhandled.
…t-leases Main added migration 0064 for privacy notifications, so the fence column drop is regenerated as 0065 on top of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
This is the third of four PRs that replace the update system's signed machinery. Until now every lease lived in a directory on
/datathat every role could write, protected by role keys, signatures and mount masks, and several Docker effects (stop, restart, down, backups) took no lease at all, soservices restartcould run in the middle of an update. Leases now live in one store owned by the operations agent, every leased Docker effect goes through one admission, and release image pins move into the configuration generation. Production code changes by +6,045 / −9,185 lines (net −3,140).infra/docker/state/ops-agent/leases.json, guarded by a lock file that is reclaimed only when its holder is provably gone (same boot, PID namespace, process start time and container). Data-manager asks the agent over HTTP with typedlease.acquire,lease.readandlease.updateoperations and its own token; the host CLI, the agent's fixed CLI children and the configuration helper write the store under the lock..admin-target-controlis gone, andcheck-ops-authoritykeeps it out.openmapx control status. A Docker command that returned an error releases its lease with a failed result; only an unknown outcome (crash, timeout, lost daemon connection) keeps the lease for an operator.openmapx control.statuslists held leases, locks and unfinished operations;resolvesettles a lease whose outcome the operator has checked (for a maintenance it also settles the data fence);clearreleases a lease or a lock explicitly and warns about what may still be live. Everyresolveandclearis appended to an audit file in the agent's state.services recover-updateand its recovery modules are gone. Data-manager settles its own jobs at startup and recovery, soresolverefuses its leases andclearis the explicit release for them.selections.json,current.json,latest.jsonand the release-store lock are gone; checked manifests are cached by digest. Clear selected release pin renders a new generation without pins. The backup-store and update-receipt locks use the new lock file.openmapx control rollback. Returns to the configuration that was applied with the previous release: it renders that generation's inputs again as a new generation and recreates the services in the self-update order. It refuses when the database's newest migration is newer than the target release's migration head, which CI now writes into the release manifest from the migration journal. Rolling back also reverts configuration made since (service selection, service settings, secrets); the command and the docs say so.Moved to the next PR: removing the signing layer (records in each owner's private state keep their signatures until then), and letting
compose upanddata linkapply hardlink plans that have real work (they still call data-manager's/link, which refuses).Verification
services restartknown failure is gone because a restart during an update is now refused. The "agent killed during start" known failure now fails in a different way (the agent restarts, but the job stays running), and its recorded signature was updated.lease.acquirereply, thecontrolcommands and their audit file, pins rendered and cleared, and rollback (target selection, the migration refusal, the service order).pnpm check:policy,pnpm check-dead-code,pnpm check-unused-exports, the data-manager OpenAPI and SDK checks, actionlint, and the release-manifest,docker.ymland product-release contract tests pass.Release and upgrade impact
Product impact: breaking, migration, operator-action
@openmapx/extension-cliis covered by a patch Changeset because its bundled core changes.Upgrade only while no update, configuration apply or recovery is in progress; leases in the old shared directory are not converted. Migration
0065drops a fence column that is no longer used. The release overlay is no longer read, so the next configuration apply pins the release in the generation. Operators useopenmapx controlinstead ofservices recover-updateand the oldrmdirinstruction. See thecontrol-leasesandrelease-pins-in-generationfragments.