diff --git a/docs/en/antalya/cas/architecture/backend.md b/docs/en/antalya/cas/architecture/backend.md index 44847af56030b..789661f6999ae 100644 --- a/docs/en/antalya/cas/architecture/backend.md +++ b/docs/en/antalya/cas/architecture/backend.md @@ -125,7 +125,8 @@ battery, skipping the battery used to skip it too, which is exactly why the thir writable generation-token mount; every other backend still skips only its permitted access-check I/O. This is what makes the exact-token delete check below unskippable on an *ordinary* writable mount. Decommissioning a pool member is the deliberate exception — it opens writable with the - battery skipped, because the fail-closed tradeoff inverts there: refusing an ordinary mount costs + battery skipped and logs one line naming what went unverified (versioning, conditional writes, + list-after-write), because the fail-closed tradeoff inverts there: refusing an ordinary mount costs availability and protects data, whereas refusing a decommission strands a pool with a dead replica in it and leaves the operator no way forward. - `checkConditionalWriteSingleAttemptSupport` — inside the battery. Refuses to mount writable unless the underlying diff --git a/docs/en/antalya/cas/architecture/garbage-collection.md b/docs/en/antalya/cas/architecture/garbage-collection.md index 16729202760ef..fca0af81a0c65 100644 --- a/docs/en/antalya/cas/architecture/garbage-collection.md +++ b/docs/en/antalya/cas/architecture/garbage-collection.md @@ -194,7 +194,8 @@ token of each `mount` object. deletes - **Safety:** fences only after this leader's *own* monotonic clock has watched the mount's write-token hold unchanged for `L + 5% + cas_mount_renew_period_ms`, where `L` is the longer of this server's - `cas_mount_lease_ttl_ms` and the lease the slot's body declares (defaults 30 s and 10 s); the write is + `cas_mount_lease_ttl_ms` and the lease the slot's body declares (defaults 30 s and 10 s), and only at the + start of a round, so a killed member becomes `fenced` up to one GC round later than that; the write is guarded by that exact token. A completed fence-out stays valid even if this GC later loses leadership. `expires_at_ms` (another host's wall clock) is never trusted. - **Fails the round if:** nothing — a per-mount `PUT` conflict is re-observed and re-classified @@ -1009,12 +1010,12 @@ checksum disagrees with its seal). | Command | Effect | |---|---| -| `SYSTEM CAS GC RUN []` | One synchronous round execution on the contacted node; only the lease holder makes progress. The disk is optional: without it every content-addressed disk on the node runs one round. It runs even while the scheduler is stopped | +| `SYSTEM CAS GC RUN []` | One synchronous round execution on the contacted node; only the lease holder (the GC leader, `is_leader = 1` in `system.cas_mounts`) makes progress, and on any other node the command does nothing. The disk is optional: without it every content-addressed disk on the node runs one round. It runs even while the scheduler is stopped | | `SYSTEM CAS GC STOP ` / `SYSTEM CAS GC START ` | Stop or resume future background rounds on that disk's scheduler, preserving its identity. The disk is required | | `SYSTEM CAS GC REBUILD [FORCE] ` (`clickhouse-disks cas-gc-rebuild`) | Fail-closed disaster-recovery path for a lost or corrupt `GC` baseline — the `CORRUPTED_DATA` errors that name it in their message (missing adopted seal, snapshot without a surviving log, cursor/apply mismatch). An `ABORTED` commit conflict or a `LOGICAL_ERROR` delete marker is not a reason to rebuild. It deliberately over-protects — it prefers bounded leaks over risking an under-count — and cannot delete live data directly: deletions it produces still flow through the normal round's condemn, graduate, exact-token path. The disk is required | | `clickhouse-disks cas-gc-dryrun` | Opens the disk read-only, constructs a non-leader `GC`, and prints what would be deleted with a reason per entry. Write-free, resolves runs through the seal's references. Documented caveat: it does not fold new owner events, so away from quiescence it can **over-report** — the subset guarantee holds only at quiescence, and its output must never feed a real delete | -`SYSTEM CAS DROP POOL MEMBER '' FROM DISK ''` — permanent removal of a dead +`SYSTEM CAS DROP POOL MEMBER '' FROM DISK ''` — removal of a dead replica, distinct from ordinary `GC` — is covered on the [mounts-and-leases page](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle). `SYSTEM CAS FSCK ` (`clickhouse-disks cas-fsck`) and its `dangling`/`unreachable` vocabulary diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index ee3fda10126ba..b294782c89f5d 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -45,31 +45,74 @@ decommission of `a`. A server whose id contains `/` refuses to open the disk; se ## The owner claim {#owner-claim} -`claimOwnerOrThrow` binds `server_root_id` ↔ `server_uuid` **permanently**. The owner object is never deleted -and never reassigned — decommission only tombstones it in place. +`claimOwnerOrThrow` binds `server_root_id` to one `server_uuid` per owner life. The owner object is never +deleted. Decommission tombstones it in place; a server start over a verified, drained retirement takes it over +at the next life ([rejoin](#rejoin)). | Observed at `gc/server-roots//owner` | Action | |---|---| | present, same `server_uuid`, not tombstoned | proceed | -| present, `retired_at_ms` set | `CORRUPTED_DATA` — explicitly decommissioned; the refusal says to configure a new `cas_server_root_id` | -| present, different `server_uuid` | `CORRUPTED_DATA` — names the regenerated-uuid-file cause and, for a dead owner, `SYSTEM CAS DROP POOL MEMBER` | +| present, `retired_at_ms` set, `retirement_verified` | a server start rejoins when its preconditions hold ([below](#rejoin)); a remount fails with `CORRUPTED_DATA`; the decommission admin accepts its victim's tombstone | +| present, `retired_at_ms` set, not verified | `CORRUPTED_DATA` for a server start or remount: the decommission is not finished; the decommission admin accepts it | +| present, different `server_uuid`, not tombstoned | `CORRUPTED_DATA` — names the regenerated-uuid-file cause and, for a dead owner, `SYSTEM CAS DROP POOL MEMBER` | +| absent, subtree provably empty | `putIfAbsent` the owner (claim) | | absent, subtree non-empty | `CORRUPTED_DATA` — identity lost over existing data | | absent, subtree provably empty | `putIfAbsent` the owner (claim) | | lost the `putIfAbsent` race | re-read; equal uuid proceeds, else `CORRUPTED_DATA` | +The owner carries `owner_life`, a counter every owner write sets to the read value + 1 by `If-Match`, so +two owner writes never repeat their bytes and a delayed `If-Match` against an older incarnation cannot +land. A process records the life it started under; after every mount claim (start, self-remount, +decommission) it reads the owner once more and requires its own `server_uuid` and that life +(`recheckOwnerAfterClaim`); otherwise it fences its own mount body and fails, so a successor does not wait +the refused lease out. A self-remount also compares the life before it allocates an epoch: an owner at +another life, not retired, moves the pool to `IdentityLost` instead of retrying. `retirement_verified` +marks a decommission whose claim, tombstone and farewell all held. + "Provably empty" requires both an authoritative decoded catalog naming no life owned by `server_root_id` and a 1-key `LIST` probe finding nothing under `cas/manifests//` or `roots//`. Two failure modes this closes: -- A **second server with a different `server_uuid`** is refused at this gate and can never take - over, regardless of lease expiry. +- A **second server with a different `server_uuid`** is refused at this gate, regardless of lease expiry, + and takes over only by a rejoin after the id's decommission is verified and drained. - A **same-uuid live twin** (two processes sharing one uuid file and `server_root_id`) is caught separately, by the mount claim's token-stability observation, and aborts with an operator-facing message rather than corrupting the pool — this is the default behavior, with `cas_unsafe_remount_no_delay` off. With it on, a same-uuid claim over such a slot reclaims at once instead of observing (see `cas_unsafe_remount_no_delay` in the configuration reference). +### Rejoin at start {#rejoin} + +A server whose `server_root_id` was decommissioned rejoins on its next start, with the same or a new +`server_uuid`. There is no command. It comes back empty: the decommission deleted the data of the +previous life. + +`claimOwnerOrThrow` rejoins only from `Pool::open`, never from a self-remount, and only when: + +- the process has a non-zero `server_uuid` (`clickhouse disks` without a uuid file never rejoins and is refused); +- `epoch` and `mount` exist (checked first: an id retired by 26.6.4 has neither); +- the owner carries `retirement_verified`; +- the slot body is terminal (clean farewell or `gc_fenced`); +- the subtree is empty, after deleting the old life's access-probe key `roots//clickhouse_access_check_`. + +Then a new `server_uuid` first rewrites the terminal `mount` body under its own uuid, and the owner becomes +`{server_uuid, owner_life + 1}` by `If-Match`. A lost `If-Match` re-reads the owner and decides again. The +kept counter gives the new life an epoch above every earlier one. + +| Refusal | Code | Cause | +|---|---|---| +| `no server uuid` | `CORRUPTED_DATA` | the process has no server uuid | +| `cannot rejoin` | `CORRUPTED_DATA` | `epoch` or `mount` is missing: the id was retired by 26.6.4, or the object was deleted by hand | +| `decommission is not finished` | `CORRUPTED_DATA` | the tombstone is not verified; `system.cas_mounts` shows the id as `retired` with `retirement_verified = 0` once its slot is terminal | +| `still running` | `ABORTED` | a decommission run holds the slot, or the lease of a killed run or an interrupted restart has not lapsed and been fenced by GC (up to one GC round after the lease threshold) | +| `previous life remains` | `ABORTED` | the message names the first part left: catalog rows, `cas/manifests//` or `roots//` | +| `kept changing` | `ABORTED` | the owner or the slot changed under every rejoin attempt of this start; restart | + +For `previous life remains`, catalog rows and manifests are removed by GC over several rounds: run `SYSTEM CAS GC RUN ''` on the GC leader +(`is_leader = 1` in `system.cas_mounts`) and restart. A late object under `roots//` is not reclaimed by GC and blocks the rejoin until it is deleted by hand. +The operator rule for rerunning a decommission is in [bringing a member back](/antalya/cas/operations/migration#rejoin-rerun-rule). + ## The mount lease {#mount-lease} One object, `gc/server-roots//mount`, carries **both** the liveness lease and the build @@ -145,8 +188,7 @@ lease the slot's body declares (`expires_at_ms − started_at_ms`). A body witho non-positive interval (a farewell), declares nothing. A re-mounting server waits out a predecessor the same way with the step `max(1, floor(period/2))`. `period` is always the observer's own. The two stamps are compared only with each other, never with the observer's clock. `expires_at_ms` alone is a -writer-stamped diagnostic used by `system.cas_mounts` and the non-authoritative decommission -epoch-recovery precheck. Local fencing is derived from the confirmed request's pre-I/O `BOOTTIME` +writer-stamped diagnostic used by `system.cas_mounts`. Local fencing is derived from the confirmed request's pre-I/O `BOOTTIME` anchor plus the TTL, and wall-clock `now` stays audit-only. `cas_mount_lease_ttl_ms` is local to each server. It is how long this server may write after its last @@ -161,13 +203,33 @@ the first renewal attempt of a period can begin, where `envelope = attempt_timeo `cap` is `attempt_timeout` when the disk's `connect_timeout_ms` is `0`, else `min(connect_timeout_ms, attempt_timeout)` (7 s with defaults). +### Cloning a running server is unsupported {#clone-unsupported} + +Do not start a second server from a copy of a running server's data directory or volume, or restore one +snapshot of a running VM twice. The copy carries the `uuid` file, so both processes claim one mount slot +under one `server_uuid`. + +- A second start that finds the first one's slot is refused as `LiveDoubleStart` (same `server_uuid`, no + certificate of death). +- A clone of a process that is already running is worse. It inherits the same `server_uuid`, `writer_epoch` + and lease state, so both processes renew one slot as one incarnation. The lease is not designed to tell them + apart and cannot be relied on to fence the clone. + +To add a server, give it its own `cas_server_root_id` and a fresh `uuid` file. + ## The two monotone counters {#counters} | Counter | Storage | Scope | Protects against | |---|---|---|---| -| `writer_epoch` | durable, `gc/server-roots//epoch` (`ServerEpoch::next_writer_epoch`, CAS-bumped by `allocateWriterEpoch`) | across crashes and restarts | a same-`(uuid, epoch)` twin: a present mount under a normal claim attempt is `CORRUPTED_DATA` | +| `writer_epoch` | durable, `gc/server-roots//epoch` (`ServerEpoch::next_writer_epoch`, CAS-bumped by `allocateWriterEpoch`); never deleted: a retired `server_root_id` keeps its counter | across crashes and restarts | a same-`(uuid, epoch)` twin: a present mount under a normal claim attempt is `CORRUPTED_DATA` | | `build_seq` | in-memory only, `CasMountRuntime::next_build_seq`, reset to 1 on every process start | one process incarnation | orders builds *within* an epoch; combined with `writer_epoch` it gives GC a total order | +Each bump of the `epoch` counter also writes a fresh random 128-bit `allocation_nonce`. When a +counter write's outcome is unclear and the counter has since moved, the allocator counts the write +as landed only if the stored bytes match its own exactly. Two starts of the same `server_uuid` running this allocator that bump +to the same value write different nonces, so they never both adopt one `writer_epoch`. Allocators +of 26.6.4 write no nonce, so two of them can still collide. 26.6.4 readers skip the field. + The absent-epoch branch of `allocateWriterEpoch` is deliberately paranoid: absent with a non-empty subtree is `CORRUPTED_DATA` (reset hazard); absent with an empty subtree decides by an authoritative probe, never by plain-`get` absence, because a transport fault must not be flattened @@ -189,7 +251,7 @@ a `MountClaimResult::Kind` together with a `MountPriorState` describing which ce |---|---| | `Claimed` | fresh claim (absent slot), or a certified reclaim of an older `writer_epoch` | | `LiveDoubleStart` | same `server_uuid` with an older `writer_epoch` and no certificate of death yet — a live twin, wait it out; or the same `(uuid, epoch)` already in the slot — two starts got one epoch, refused at once | -| `ForeignOwner` | different `server_uuid` — refused unconditionally | +| `ForeignOwner` | different `server_uuid` — refused, except a terminal body below the claimer's epoch when a fresh owner read shows the claimer's uuid and life (a server start or self-remount after a rejoin) | | `FencedSelf` | same `(uuid, epoch)`, but `gc_fenced` — terminal for *this* epoch; the caller must mint a fresh one | A claim never installs a `writer_epoch` at or below the one in the slot. When the slot holds a newer @@ -221,7 +283,9 @@ stateDiagram-v2 Live --> Live: same-uuid claim, proven-dead token via UncleanObserved Live --> Live: same-uuid claim under cas_unsafe_remount_no_delay, no observation Fenced --> Fenced: same uuid and epoch claim, FencedSelf, no write - Live --> Absent: decommission tail, mount then epoch then owner tombstone + Terminated --> Terminated: decommission farewell, owner tombstoned and verified + Terminated --> Live: rejoin of a verified, drained retirement, any server_uuid + Fenced --> Live: rejoin of a verified, drained retirement, any server_uuid Terminated --> [*] ``` @@ -234,7 +298,7 @@ stateDiagram-v2 Live --> TransientNotLive: renewal failure, tripMountLost, lost=true TransientNotLive --> Live: self-remount succeeds with a fresh epoch TransientNotLive --> TransientNotLive: probe inconclusive, retry with backoff - TransientNotLive --> IdentityLost: pool meta and owner both authoritatively absent + TransientNotLive --> IdentityLost: pool meta and owner both absent, or the owner at another life TransientNotLive --> VanishedReplaced: foreign pool_id observed Live --> VanishedForgotten: SYSTEM CAS FORGET IdentityLost --> [*] @@ -280,28 +344,49 @@ while a stale conditional request from the predecessor is still in flight. and waiting out the token-stability observation, or the GC leader fencing the slot first, after which any reclaim is instant. -**Permanent removal** of a dead replica (`Cas::decommissionPoolMember`, driven by -`SYSTEM CAS DROP POOL MEMBER '' FROM DISK ''`) claims the victim's mount slot -as an administrative writer with a no-wait policy (refuses unless the slot is terminal: -`terminated` or `fenced`; an `expired` slot is refused until the disk's GC leader fences it), -drops every ref-bearing namespace, sweeps manifest debris before the slot (deleting the mount -removes the watermark authority), drains staging and roots, then — only with zero warnings — -retires in order: mount, epoch, a final liveness re-check, owner tombstone. +**Removal** of a dead replica (`Cas::decommissionPoolMember`, `SYSTEM CAS DROP POOL MEMBER '' FROM DISK ''`) +keeps `owner`, `epoch` and `mount` and removes only data, in six steps: + +1. Claim the victim's slot as an administrative writer with a no-wait policy. The claim needs a terminal + slot (`terminated` or `fenced`); an `expired` slot is refused until the disk's GC leader fences it. A + member without an epoch counter is refused. +2. Tombstone the owner at the next `owner_life`, on the admin's fenced mount plane: no attempt is sent + after the admin's lease deadline, but the store may apply a sent one later. When the outcome is unknown and + the owner does not read back as the tombstone, the run fails without a farewell: the victim restarts only + after GC fences the admin's slot or its own start sees the lease hold still for the observation threshold. + A tombstone applied later than that is the case the + [rerun rule](/antalya/cas/operations/migration#rejoin-rerun-rule) repairs by hand. +3. Confirm the slot still holds the admin's claim; otherwise roll the tombstone back, unless the owner was + already retired before this run. Every exit between (2) and (5) runs this confirm. +4. Drop every namespace, sweep manifest debris, drain `staging/` and `roots/`, all on the admin's fenced + mount plane. +5. Write the admin's clean farewell. +6. Read the farewell and the tombstone back and set `retirement_verified`. + +`member_retired = 1` only when the owner carries a verified retirement. An owner that moved on to a later, +unretired life after the tombstone (a rejoin, or another run's rollback) reports `member_retired = 0` with +the warning `the owner moved on to life N`. Catalog rows GC still has to delete are reported as +`namespaces_pending_gc`, not as a failure. The operator reruns only after `member_retired = 0`, under the +[rerun rule](/antalya/cas/operations/migration#rejoin-rerun-rule); a rerun refused as alive or contended +waits until GC fences the previous run's slot: about one mount lease TTL plus one GC round, with a GC leader +running. A later start under that `server_root_id` rejoins it as an empty member once the retirement is +verified and drained ([rejoin](#rejoin)). ## `system.cas_mounts` {#mounts-table} A read-only view of the same heartbeat-floor computation GC uses: one `LIST` of -`gc/server-roots/` plus one `GET` per slot, zero writes, per-row fail-open (an undecodable body +`gc/server-roots/` plus one `GET` per slot, plus one owner `GET` per terminated or fenced slot, zero writes, per-row fail-open (an undecodable body, or an undecodable owner of a terminated or fenced slot, becomes `state = 'corrupt'`, never an exception). Shows every `server_root_id` in the pool, including peers. | Column | Notes | |---|---| | `disk`, `server_root_id`, `server_uuid`, `hostname`, `process_id` | identity | | `writer_epoch`, `renewal_sequence`, `started_at`, `expires_at`, `min_active_build_sequence`, `gc_fenced` | lease state (`DateTime64(3)` columns; the millisecond-integer field names live only in the internal `MountLease` struct and the on-disk body) | -| `state` | one of `live`, `expired`, `terminated`, `fenced`, `corrupt` | +| `state` | one of `live`, `expired`, `terminated`, `fenced`, `retired`, `corrupt`; `retired` = terminated or fenced with a tombstoned owner (one more `GET` per terminated or fenced row) | | `is_leader`, `pending_reclaim`, `last_success_age_seconds`, `wedged_namespace_count` | GC health, process-local; **`NULL` on every peer row** — a process-local fact must never be stamped onto another server's row | | `lifecycle`, `lifecycle_reason`, `lifecycle_detail`, `lifecycle_since` | the SQL surface for the in-process `PoolLifecycle` runtime above: `lifecycle` is one of `live`, `not_live`, `identity_lost`, `vanished`, `constructing`, `shutdown`; `lifecycle_reason` distinguishes `replaced` from `forgotten` for a `vanished` disk; `lifecycle_detail` carries the full diagnosis text; `lifecycle_since` is when the current non-live state began (`NULL` while live) | | `renewal_in_progress`, `renewal_attempts`, `last_renewal_error` | the running (or last) lease renewal: conditional `PUT`s sent so far and the last failed request, updated per attempt; **`NULL` on every peer row** | +| `retirement_verified` | `1` on a `retired` row whose decommission verified the retirement, else `0`; a rejoin needs `1`, a rerun is for `0` only | The lifecycle snapshot is I/O-free and ungated, so a not-live, never-started, or vanished disk still produces a row instead of silently disappearing from the table. diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 07e134a9deb55..aef120ec00ff0 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -117,7 +117,8 @@ token holding stable on the observer's own `CLOCK_BOOTTIME` for `L + floor(L / 2 longer of the observer's `cas_mount_lease_ttl_ms` and the lease the holder's body declares (`expires_at_ms − started_at_ms`; a body without stamps or with a non-positive interval declares nothing). Startup's step is `max(1, floor(cas_mount_renew_period_ms / 2))` and GC's is -`cas_mount_renew_period_ms`, both the observer's own. Mixed values are therefore safe in any order; +`cas_mount_renew_period_ms`, both the observer's own. GC checks only at the start of a round, so a +fence-out can land up to one GC round after that. Mixed values are therefore safe in any order; a longer lease on one server delays recovery after that server's crash. A server built without this rule waits only its own lease, so keep the values equal until every server of the pool runs a build with it. @@ -132,7 +133,7 @@ where `envelope = cas_attempt_timeout_ms + 2 × cap` (7000 ms with defaults) and retrying until `confirmed deadline − cas_lease_safety_margin_ms`. The `expires_at_ms` stamped into the mount object is a writer-stamped diagnostic used by -`system.cas_mounts` and by the non-authoritative decommission epoch-recovery precheck; it never +`system.cas_mounts`; it never authorizes a reclaim or a GC fence-out. Local fencing is derived instead from the confirmed request's pre-I/O `CLOCK_BOOTTIME` anchor plus the TTL. diff --git a/docs/en/antalya/cas/operations/debugging.md b/docs/en/antalya/cas/operations/debugging.md index e8c6665a166a0..7313e72cbfb28 100644 --- a/docs/en/antalya/cas/operations/debugging.md +++ b/docs/en/antalya/cas/operations/debugging.md @@ -144,8 +144,7 @@ row followed by a `recovered` one — and the older `unresolved_reason`, `deadli closed admission while the operation was active), `deterministic_failure` (a local failure, or, for a bounded renewal, a store answer proving the write never applied; the background renewal sends a refused write again), and `vanished` (an exact resolve read proved the mount - slot absent — the pool directory was removed or renamed out of band, or a decommission raced the - renewal). `terminal_unclassified` means the renewal terminated through a path that assigned no + slot absent — the pool directory was removed or renamed out of band, or a 26.6.4 decommission deleted the slot; a decommission of this version keeps it). `terminal_unclassified` means the renewal terminated through a path that assigned no classification; that is a defect to report together with the surrounding rows, not an operator condition. Do not collapse these into a generic timeout — the action differs by classification, and only `external_lease_deadline` and `request_deadline` are about a @@ -193,6 +192,9 @@ available: SYSTEM CAS GC RUN cas; ``` +Run it on the GC leader (`is_leader = 1` in `system.cas_mounts`). On any other node it does nothing and +returns no error. + One row per disk it ran on: `disk`, `acquired_lease`, `deferred`, `round`, `candidates_marked`, `objects_deleted`, `objects_absent`, `objects_replaced`, `objects_spared`, `manifests_deleted`, `entries_condemned`, `entries_graduated`, `entries_redeleted`, `entries_redelete_failed`, `fence_outs`, @@ -227,8 +229,8 @@ SYSTEM CAS FORGET cas; It is an assertion, not a proof of erasure: the disk stays registered and answers further store-class access with a typed error, and a server restart re-registers the name. This is different from -[`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission), which permanently -retires one pool *member*'s identity across the whole shared pool — `FORGET` only affects this node's +[`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission), which decommissions +one pool *member* across the whole shared pool — `FORGET` only affects this node's own local view of one disk. ## Offline tools {#offline-tools} diff --git a/docs/en/antalya/cas/operations/migration.md b/docs/en/antalya/cas/operations/migration.md index 88927d4cfc996..99f3d08539741 100644 --- a/docs/en/antalya/cas/operations/migration.md +++ b/docs/en/antalya/cas/operations/migration.md @@ -1,5 +1,5 @@ --- -description: 'Adding a content-addressed disk to an existing deployment, moving a partition onto it with ALTER TABLE MOVE PARTITION, rolling back, and permanently decommissioning a pool member.' +description: 'Adding a content-addressed disk to an existing deployment, moving a partition onto it with ALTER TABLE MOVE PARTITION, rolling back, and decommissioning a pool member.' sidebar_label: 'Migration' sidebar_position: 1 slug: /antalya/cas/operations/migration @@ -164,17 +164,20 @@ directly (the raw disk, not the cache) was refused with `All parts of partition already on disk 'cas_cache'. (UNKNOWN_DISK)` — a real error message from the run, kept here because it is exactly what an operator sees after guessing the wrong disk name. -## Permanently removing a pool member {#decommission} +## Removing a pool member {#decommission} A `CAS` pool can be shared by several servers (see [`server_root_id`](/antalya/cas/architecture/mounts-and-leases#server-root-id)). -Scaling down — permanently removing a server that will never rejoin the pool — is a distinct, -irreversible operation from an ordinary restart or a temporary outage: it fences the member's -`server_root_id` and reclaims the storage attributable only to it. +Scaling down — removing a server and its data from the pool — is a distinct operation from an ordinary +restart or a temporary outage: it fences the member's current life and reclaims the storage attributable +only to it. The data is gone for good; the `server_root_id` can rejoin later as an empty member +([rejoin](#rejoin)). `SYSTEM CAS DROP POOL MEMBER` claims the victim's mount slot as an administrative writer (refusing -immediately if the member is still alive), drops every table namespace the member owned, sweeps -manifest debris, drains its staging and mountpoint objects, and — only once every drain is -confirmed — retires the mount slot itself. It emits ordinary ref-edge deltas rather than a GC +immediately if the member is still alive), tombstones its owner, drops every table namespace the member +owned, drains its debris and marks the retirement verified; the steps are in +[mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle). The member's control +objects (`owner`, `epoch`, `mount`) are kept: the epoch counter keeps epochs from being reused when the +`server_root_id` rejoins. It emits ordinary ref-edge deltas rather than a GC transition: it does not synchronously reclaim shared blob content, it only makes the now-unreferenced blobs eligible for an ordinary GC round to reclaim later. @@ -192,17 +195,18 @@ clickhouse-disks -C config.xml --disk cas cas-drop-member 'replica-2' ``` The command returns one row (or, offline, one line per field) with `namespaces_removed`, -`namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`, -`manifest_debris_removed`, `staging_objects_removed`, `mountpoint_objects_removed`, and -`slot_removed`. It is resumable: a rerun skips namespaces already marked removed and reports them -under `namespaces_already_removed` rather than redoing the work. A per-object drain failure is -recorded as a `warning` rather than raised as an exception, leaving the slot terminated but not -fully drained so a later invocation can resume; a non-empty `warnings` means exactly that, and the -mount slot stays in place as a resume anchor rather than being fully retired. - -**Preconditions.** Confirm the member is actually and permanently dead before running this: the -operation fences that `server_root_id` out even if the server comes back online, and it deletes -namespace and drain state that cannot be recovered. The command claims the slot without waiting +`namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`, `manifest_debris_removed`, +`staging_objects_removed`, `mountpoint_objects_removed`, `member_retired`, `warnings` and +`namespaces_pending_gc`; the SQL row ends with `namespaces_pending_gc`. It is resumable: a rerun skips namespaces already marked removed and reports them +under `namespaces_already_removed` rather than redoing the work. A per-object drain failure is recorded as +a `warning`; the owner keeps an unverified tombstone and the next run resumes over it. `member_retired = 1` +means the owner carries a verified retirement; rows still listed under `namespaces_pending_gc` are deleted +by GC and need no rerun. See [troubleshooting](/antalya/cas/operations/troubleshooting) for +`member_retired = 0`. + +**Preconditions.** Confirm the member is actually dead before running this: the +operation deletes namespace and drain state that cannot be recovered, and a server that comes back under +that id after the run starts empty. The command claims the slot without waiting and trusts no wall clock, so it accepts only a terminal slot in `system.cas_mounts`: `state = 'terminated'` (clean stop) or `state = 'fenced'` (GC fenced the lease). A `live` or `expired` row is refused with `alive or contended`; a hard-killed member stays `expired` until the disk's GC leader fences it at the start of a GC round, once it has seen the lease unchanged for about one @@ -210,12 +214,74 @@ row is refused with `alive or contended`; a hard-killed member stays `expired` u (`is_leader = 1` in `system.cas_mounts`) can trigger the fence, but the run that fences must come at least that long after the first run that saw the lease; two back-to-back runs do not fence. -**Verification.** After the command reports `slot_removed = true` with no warnings, the member's -`server_root_id` no longer appears as a row in `system.cas_mounts` on any peer, and a subsequent -`SYSTEM CAS GC RUN` on the pool will no longer wait on or fence its heartbeat. See +**Verification.** After `member_retired = 1`, `system.cas_mounts` shows the member's +row as `retired` with `retirement_verified = 1` (an unfinished run leaves `retirement_verified = 0`); a `SYSTEM CAS GC RUN` no longer waits on its heartbeat. Run it on the GC leader (`is_leader = 1` in `system.cas_mounts`); on another node it does nothing. See [mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle) for how the claim, drain, and retirement steps fit into the mount-slot lifecycle. +### Bringing a member back {#rejoin} + +Restart the server. A server whose `server_root_id` was decommissioned rejoins the pool on its next start, +with the same or a new `server_uuid`. It comes back empty: its local table definitions attach with no data. + +The start rejoins only after the decommission finished: + +- the last run reported `member_retired = 1`, and `system.cas_mounts` shows the member as `retired` with + `retirement_verified = 1`; +- GC removed the catalog rows and manifests of the old life: run `SYSTEM CAS GC RUN ''` on the GC leader + (`is_leader = 1` in `system.cas_mounts`) after the verified run; it can take several rounds. + +| Start fails with | Action | +|---|---| +| `cannot rejoin` | `epoch` or `mount` is missing: the id was retired by 26.6.4 or the object was deleted by hand. Configure a new `cas_server_root_id` | +| `decommission is not finished` | finish the decommission under the [rerun rule](#rejoin-rerun-rule) until `member_retired = 1`, then restart | +| `previous life remains` | the message names the part left. Catalog rows or manifests: run `SYSTEM CAS GC RUN ''` on the GC leader, `is_leader = 1` (it can take several rounds) and restart. An object under `roots//`: delete it by hand and restart | +| `still running` | a decommission run, a killed run or an interrupted restart holds the slot: wait for the run to finish, or for the lease to lapse and GC to fence it (up to one GC round after the lease threshold), then restart | +| `kept changing` | other starts or a decommission run kept rewriting the owner or the slot during this start: restart | +| `no server uuid` | the process (for example `clickhouse disks` before a uuid file exists) has no server uuid and never rejoins: start the server that should own the id | + +Keep `/var/lib/clickhouse/uuid` (the server uuid file) on a persistent volume. A lost uuid file makes the +server start as a different server: a `server_root_id` that is not decommissioned refuses it as owned by a +different server, and a decommissioned one is rejoined under the new `server_uuid`. + +#### Rerun rule {#rejoin-rerun-rule} + +Rerun `SYSTEM CAS DROP POOL MEMBER` to finish a decommission only while both hold: + +- `system.cas_mounts` shows the id as `retired` with `retirement_verified = 0`; +- no decommission of that id is running. + +A run killed after its tombstone leaves the row `live`, then `expired`, then `retired` with +`retirement_verified = 0` once GC fences its lease, up to one GC round after the lease threshold. A run killed +before its tombstone ends `fenced` with the owner not retired. Do not rerun on any other state. `retired` with +`retirement_verified = 1` is finished: restart the member to rejoin. A `live`, `expired`, `terminated` or +`fenced` row belongs to whoever owns the id now, a rejoined member included, and a rerun would delete its data. +A run that warns `the tombstone was rolled back` lost its claim and left the id unretired: running the command +again is a new decommission of the id. + +A tombstone can land late: the store may apply it after the run reported an error. If a server of the id ran +after the tombstone time printed in its start error, do not rerun: the rerun deletes what that server wrote. +Instead keep the owner object's header line, replace its body line with +`{"server_uuid":"","owner_life":""}` (the same `server_uuid`, the life from the error plus one) +and restart that member. + +### Upgrading and downgrading {#decommission-compatibility} + +| Case | Behaviour | +|---|---| +| A `server_root_id` retired by 26.6.4 (its `epoch` and `mount` deleted) | not repaired: a decommission rerun refuses the absent counter and it cannot rejoin; use a new `server_root_id` | +| A 26.6.4 decommission that stopped midway | the same refusal; use a new `server_root_id` | +| Flat 26.6.4 staging keys `staging//.tmp` | swept as epoch 0 at the next mount | +| Downgrade to 26.6.4 with a retired `server_root_id` kept by this version | 26.6.4 refuses to start it and its GC treats the farewell as terminal | +| Downgrade: `owner_life`, `retirement_verified`, epoch staging keys | skipped by 26.6.4 readers; staging swept recursively | +| Mixed versions: a 26.6.4 decommission tool rewrites an owner | the life drops to 0; servers of this version fail their owner check and stop (fail-closed) | +| Downgrade: a `server_root_id` rejoined under a new `server_uuid` | start it once on this version before downgrading; 26.6.4 then starts it normally | +| Downgrade: a `server_root_id` rejoined under the same `server_uuid` | 26.6.4 starts it and continues the counter | +| Mixed versions: a 26.6.4 server starts with a retired `server_root_id` | refused, as in 26.6.4: it cannot rejoin | +| Mixed versions: a 26.6.4 member restarts during a decommission | its owner check refuses after the tombstone; before it there is no recheck, as in 26.6.4 | + +Rolling upgrades keep the old races on the old binaries; decommission from an upgraded server. + ## Recreating a member whose id contains `/` {#recreate-id-with-slash} A `cas_server_root_id` that contains `/` is refused: the disk fails to open with `BAD_ARGUMENTS` and a diff --git a/docs/en/antalya/cas/operations/troubleshooting.md b/docs/en/antalya/cas/operations/troubleshooting.md index 264dc59c95455..94d8c41b23e22 100644 --- a/docs/en/antalya/cas/operations/troubleshooting.md +++ b/docs/en/antalya/cas/operations/troubleshooting.md @@ -18,17 +18,18 @@ tools. |---|---|---| | A server keeps losing its mount lease and self-remounting | Check `system.cas_mounts` for the server's own `state`/`expires_at`, then correlate `watermark_renew` and `mount_remount` in `system.cas_log`; losing the lease trips a local fence and latches a remount generation | Read the failed renewal's `classification` before changing anything — it alone now says why (see [the decision flow](#mount-renewal-remount-flow)). Look for object-store latency consuming the confirmed lease or BOOTTIME advancement; see [the mount lease](/antalya/cas/architecture/mounts-and-leases#mount-lease) | | Writes slow down or stall under load, with no exception reaching the client | S3 `SlowDown`/`ServiceUnavailable`/`RequestTimeout`/`InternalError` (5xx) responses are not on the request engine's `isDefinitelyRefusedWrite` definite-failure list (only malformed-request, entity-too-large, and access-denied that no credential refresh can fix are), so they classify as ambiguous and are retried automatically. Confirm with `sum(ProfileEvents['CASConditionalWriteUnresolved'])` rising alongside `sum(ProfileEvents['CASConditionalWriteAttempts'])` over `system.query_log` for the affected window (or `ProfileEvent_CASConditionalWriteUnresolved` in `system.metric_log` for a cumulative view across queries), and check `system.blob_storage_log` for `disk_name = ''` rows with a nonzero `error_code` around the same window | Nothing to configure per-request: the request engine retries the same `(key, bytes)` with capped-exponential backoff (200ms initial, capped at 5s, full jitter) until the 90-second operation deadline — for these conditional writes there is no separate attempts ceiling, only the deadline. A blob body upload is different: it is never resent as is after an ambiguous outcome, and each retry publishes a fresh envelope, up to 8 attempts, and no new attempt starts after the 90-second deadline. An attempt already started can run longer: each creating request waits up to the disk's `request_timeout_ms`, and multipart part uploads keep their own retries. `CASBlobPublishAttemptFailed` counts failed attempts; `CASBlobPublishGaveUp` counts uploads that failed at least once and then ended with an exception, at the attempt cap or the deadline. Meanwhile the mount-lease renewer keeps extending the fence across the disruption — this is the "blips, throttling, partial outages" case the write path is built to survive. Confirm the mount lease itself is still renewing (`system.cas_mounts.expires_at` moving forward, `last_success_age_seconds` not climbing) — if it is, this is expected and self-resolving. If `SlowDown` responses are sustained rather than transient, check the bucket's request-rate limits against the pool's actual PUT/GET rate (see [bucket requirements](/antalya/cas/bucket-requirements)) and consider lowering `cas_blob_upload_pool_size` to reduce concurrent upload traffic; a write only surfaces a client-visible `NETWORK_ERROR` if the 90-second deadline passes before the store recovers (an attempt started before the deadline can finish after it), and that error is retried by the ordinary merge/insert backoff, not silently dropped | -| `GC` never seems to reclaim space after tables are dropped | `SELECT * FROM system.cas_gc_log WHERE event_type='Finish' ORDER BY event_time DESC LIMIT 5` — check `outcome`; also `SELECT is_leader FROM system.cas_mounts` on this node | If `outcome != 'Success'`/`'Deferred'`, see [reading GC health](/antalya/cas/operations/monitoring#gc-health); if this node is not the leader (`is_leader = 0`), it never reclaims for this disk — check the peer holding leadership. Reclamation also needs at least two full rounds past condemnation by design (the grace period is rounds, not acks) — a single manual `SYSTEM CAS GC RUN` will not finish it | +| `GC` never seems to reclaim space after tables are dropped | `SELECT * FROM system.cas_gc_log WHERE event_type='Finish' ORDER BY event_time DESC LIMIT 5` — check `outcome`; also `SELECT is_leader FROM system.cas_mounts` on this node | If `outcome != 'Success'`/`'Deferred'`, see [reading GC health](/antalya/cas/operations/monitoring#gc-health); if this node is not the leader (`is_leader = 0`), it never reclaims for this disk — check the peer holding leadership. Reclamation also needs at least two full rounds past condemnation by design (the grace period is rounds, not acks) — a single manual `SYSTEM CAS GC RUN` will not finish it, and it does nothing on a node that is not the leader | | A dangling-access exception or `CORRUPTED_DATA` on read | Run `clickhouse-disks cas-fsck --detail` and check `dangling` specifically — it is the one class that means data loss, distinct from `unreachable`/`awaiting-gc`, which are just waiting for graduation | A nonzero `dangling` count is a real incident: collect the `--detail` output (see [what to collect before filing a bug](/antalya/cas/operations/debugging#filing-a-bug)) before taking any destructive action | | `SYSTEM CAS FSCK` or `clickhouse-disks cas-fsck` times out on a large pool | The scan is bounded by `--timeout` (default 600s / the `SYSTEM` form has no override); a large `roots/` prefix can make the scan slow | Retry with `--partial` to see the counts accumulated so far instead of aborting empty-handed, or `--namespace ` to scope the scan to a subset of namespaces | -| `SYSTEM CAS DROP POOL MEMBER` returns a non-empty `warnings` column | A per-object drain step could not confirm emptiness; the mount slot is left terminated but not fully drained, as a resume anchor | Rerun the same command — it is resumable and skips namespaces already marked removed, reporting them under `namespaces_already_removed` | +| `SYSTEM CAS DROP POOL MEMBER` returns `member_retired = 0` or a non-empty `warnings` column | The run stopped before its verified mark: a drain step failed, the admin lost its claim, or the farewell could not be confirmed; the owner keeps an unverified tombstone as the resume anchor. A warning `the owner moved on to life N` means the owner was un-retired after this run's tombstone, by a rejoin or by another run's rollback | Rerun the same command only under the [rerun rule](/antalya/cas/operations/migration#rejoin-rerun-rule); it resumes over its own tombstone. A rerun refused as alive or contended waits until GC fences the previous run's slot: about one mount lease TTL (`cas_mount_lease_ttl_ms`) plus one GC round, with a GC leader running. `namespaces_pending_gc` alone is not a reason to rerun | | `SYSTEM CAS DROP POOL MEMBER` refuses with `alive or contended` although the member is down | The claim trusts no wall clock: `expired` in `system.cas_mounts` is not a certificate of death; only a terminal slot (`terminated` or `fenced`) is reclaimable | Wait until the member's row shows `state = 'terminated'` (clean stop) or `state = 'fenced'` (the disk's GC leader fences the stale lease at the start of a GC round, once it has seen it unchanged for about one `cas_mount_lease_ttl_ms` plus one `cas_mount_renew_period_ms`, so the wait can be one GC round longer; `SYSTEM CAS GC RUN` on the leader, `is_leader = 1` in `system.cas_mounts`, can trigger the fence, but the run that fences must come at least that long after the first run that saw the lease), then rerun | | A server refuses to start with `owned by a different server` | The owner object binds the `server_root_id` to another `server_uuid`: the local uuid file was regenerated, or two servers share one `cas_server_root_id` | Restore the uuid file (keep it on a persistent volume); or start this server under a new `cas_server_root_id`. To reclaim the old id's data once its owning server is permanently gone, decommission it with `SYSTEM CAS DROP POOL MEMBER` from a surviving member, once its slot is `terminated` or `fenced`; the two steps are independent, and in a single-member pool starting under the new id comes first, because a hard-killed old id is fenced only by a running GC leader | | A `CAS` disk refuses to open with `contains '/'` | Its `cas_server_root_id` contains `/`, directly or through a macro; such ids are refused, including ids that already own their owner object in the pool | Configure a new `cas_server_root_id` without `/` and restart; see [recreating a member whose id contains `/`](/antalya/cas/operations/migration#recreate-id-with-slash) | -| A server refuses to start with `explicitly decommissioned` | Its `server_root_id` was retired by `SYSTEM CAS DROP POOL MEMBER`; the tombstone is permanent so writer epochs never repeat | Configure a new `cas_server_root_id` for the disk; do not clear the tombstone by hand | +| A server fails to start with `decommission is not finished`, `previous life remains`, `still running`, `kept changing`, `no server uuid` or `cannot rejoin` | Its `server_root_id` was decommissioned and the rejoin cannot proceed: the retirement is unverified or not drained, a run still holds the slot, concurrent writers kept changing the owner or slot, the process has no server uuid, or the id cannot be rejoined | See [bringing a member back](/antalya/cas/operations/migration#rejoin) | +| `system.cas_mounts.lifecycle = 'identity_lost'` after a remount, with `the owner moved from life N to life M` in the log | A rejoin or a decommission rollback rewrote the owner while this process was fenced; it cannot resume under a life it did not start under | Restart the server; it starts under the current life | | Writes or `ALTER`s on a `CAS` disk fail with a `READONLY`-class error | The disk's metadata storage rejects every mutating entry point; this is deliberate for a disk opened with `true`, used by every offline `clickhouse-disks` tool | Confirm whether the disk was intentionally configured read-only (offline inspection, `cas-fsck`, `cas-gc-dryrun`, `cas-gc-rebuild`, `cas-drop-member` all require it); a production disk serving writes must not carry `true` | | A table stays unavailable after a transient network error during startup | `AsyncLoader` has no retry/requeue path for a failed table load job: a transient S3 `NETWORK_ERROR` during `CAS` ref-table startup recovery can leave the job permanently `FAILED` | Restart the server, or issue a fresh load for the table; this is a one-shot job design, not a `CAS`-specific bug | -| A mounted pool directory was removed or renamed out of band | Renewal observes an absent, foreign, successor, or otherwise conflicting mount body and terminates the keeper with a typed fail-closed exception; the runtime closes the local write fence and requests remount rather than adopting the body | Never remove or rename a live pool's storage path. To retire a member permanently use [`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission) instead of raw filesystem operations; collect the `watermark_renew` classification (`vanished` for an absent body, `conflict` for a foreign, successor, or otherwise conflicting one) and the subsequent `mount_remount` step | +| A mounted pool directory was removed or renamed out of band | Renewal observes an absent, foreign, successor, or otherwise conflicting mount body and terminates the keeper with a typed fail-closed exception; the runtime closes the local write fence and requests remount rather than adopting the body | Never remove or rename a live pool's storage path. To remove a member, use [`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission) instead of raw filesystem operations; collect the `watermark_renew` classification (`vanished` for an absent body, `conflict` for a foreign, successor, or otherwise conflicting one) and the subsequent `mount_remount` step | | Stale-looking part metadata after an out-of-band change to the pool | The part-folder view cache may be serving a retained (not re-validated) view | Set the disk-level `cas_part_folder_cache_bytes = 0` to disable view retention and `cas_manifest_decode_cache_bytes = 0` to make every manifest read fetch the body, then run `fsck`; both are diagnostic kill switches, not steady-state settings | | A wide merge (many thousands of columns) fails with a port-exhaustion error from the network layer | Each column in a wide part can cost a separate object-store operation in one merge, and a very wide part can issue on the order of the column count in requests, exhausting local ephemeral TCP ports under load | Reduce concurrent merge parallelism on that table, or increase the host's ephemeral port range; this is a general high-fan-out-merge limit, not specific to content addressing | diff --git a/docs/en/antalya/metrics_and_events.md b/docs/en/antalya/metrics_and_events.md index 14af72f826b8a..6617978787072 100644 --- a/docs/en/antalya/metrics_and_events.md +++ b/docs/en/antalya/metrics_and_events.md @@ -63,7 +63,7 @@ Pool lifecycle and teardown. A non-zero value here is an incident signal. | `CASDataRootVanished` | `Number` | Counts CAS pools that entered a terminal Vanished lifecycle state (replaced / forgotten): the pool's data root was proven foreign (pool_id mismatch) or was decommissioned by SYSTEM CAS FORGET. The disk stays registered but store-class access fails loud with a typed error naming the sub-state; restart re-registers the name. | | `CASDetachedWorkDrainTimeouts` | `Number` | Counts CAS storage teardowns whose bounded wait for detached background work expired with work still in flight. The teardown proceeds, but for that teardown it could not be established that no tracked task still holds the pool. Expected to stay at zero. | | `CASEventDroppedContextExpired` | `Number` | Number of CAS system-log events dropped because the storage's `Context` reference expired before delivery. A non-zero value indicates event production outlived the owning server context. | -| `CASIdentityLost` | `Number` | Counts CAS pools that entered the IdentityLost lifecycle state: the pool sentinels (_pool_meta and the owner anchor) were observed authoritatively absent (both KeyAbsent). IdentityLost is a fail-loud TERMINAL state -- store-class access fails loud from this point and this pool's background threads (remount + GC) self-exit; there is no observer. Recover by restart or SYSTEM CAS FORGET; a matching-sentinel restore does not auto-revive. | +| `CASIdentityLost` | `Number` | Counts CAS pools that entered the IdentityLost lifecycle state: the pool sentinels (_pool_meta and the owner anchor) were observed authoritatively absent (both KeyAbsent), or a self-remount found the owner at another life. IdentityLost is a fail-loud TERMINAL state -- store-class access fails loud from this point and this pool's background threads (remount + GC) self-exit; there is no observer. Recover by restart or SYSTEM CAS FORGET; a matching-sentinel restore does not auto-revive. | #### Blob requests {#cas-blob-requests} diff --git a/docs/en/operations/system-tables/cas_mounts.md b/docs/en/operations/system-tables/cas_mounts.md index 0e6eba43cc8e4..0aa8aad3ce473 100644 --- a/docs/en/operations/system-tables/cas_mounts.md +++ b/docs/en/operations/system-tables/cas_mounts.md @@ -32,7 +32,7 @@ rows. - `expires_at` ([DateTime64(3)](/sql-reference/data-types/datetime64)) — Time when the lease expires. - `min_active_build_sequence` ([UInt64](/sql-reference/data-types/int-uint)) — Oldest in-flight build sequence (`UINT64_MAX` means the mount said farewell). - `gc_fenced` ([UInt8](/sql-reference/data-types/int-uint)) — `1` if GC fenced this slot out (terminal). -- `state` ([String](/sql-reference/data-types/string)) — One of `live`, `expired`, `terminated`, `fenced`, `corrupt`. +- `state` ([String](/sql-reference/data-types/string)) — One of `live`, `expired`, `terminated`, `fenced`, `retired`, `corrupt`. `retired` is a terminated or fenced slot of a decommissioned member; `retirement_verified` says whether the decommission finished. `corrupt` covers an undecodable mount body, and a terminated or fenced slot whose owner object cannot be decoded. Each terminated or fenced slot costs one more `GET` for its owner; a `corrupt` row from an undecodable mount body costs none. - `is_leader` ([Nullable(UInt8)](/sql-reference/data-types/nullable)) — `1` if this server's GC scheduler currently holds this disk's leadership lease. - `pending_reclaim` ([Nullable(Int64)](/sql-reference/data-types/nullable)) — Cumulative two-phase deletion backlog observed by this process's GC on this disk (condemned entries minus executed exact-token deletes). - `last_success_age_seconds` ([Nullable(UInt64)](/sql-reference/data-types/nullable)) — Seconds since this disk's GC last led a round (`0` if it has never led or GC is not running here). @@ -44,6 +44,7 @@ rows. - `renewal_in_progress` ([Nullable(UInt8)](/sql-reference/data-types/nullable)) — `1` while this server renews its mount lease. - `renewal_attempts` ([Nullable(UInt64)](/sql-reference/data-types/nullable)) — Conditional `PUT`s sent by the current renewal, or by the last one once it ended. Updated per attempt, so a renewal stuck in retries is visible before it ends. - `last_renewal_error` ([Nullable(String)](/sql-reference/data-types/nullable)) — The last failed request of that renewal, as ` : `. `class` is `refused` when the store definitely refused the request (access denied, malformed request, entity too large), `transport` otherwise; `code` is the store's error name, such as `SlowDown`. `NULL` when no request failed. +- `retirement_verified` ([UInt8](/sql-reference/data-types/int-uint)) — `1` on a `retired` row whose decommission verified the retirement, `0` otherwise. A server started under that `server_root_id` rejoins only from `retired` with `retirement_verified = 1`. Rerun `SYSTEM CAS DROP POOL MEMBER` only on `retired` with `retirement_verified = 0`; see the [rerun rule](/antalya/cas/operations/migration#rejoin-rerun-rule). `lifecycle`/`lifecycle_reason`/`lifecycle_detail`/`lifecycle_since` are the SQL surface for diagnosing an identity-lost or forgotten disk without reading server logs — see the @@ -82,4 +83,4 @@ FORMAT Vertical; - [`system.cas_gc_log`](/operations/system-tables/cas_gc_log) — per-round GC event log. - [`system.cas_log`](/operations/system-tables/cas_log) — per-decision event log for the CAS garbage collector and writer. - [`SYSTEM CAS GC RUN`](/sql-reference/statements/system#system-cas-gc-run) — run one GC round synchronously. -- [`SYSTEM CAS DROP POOL MEMBER`](/sql-reference/statements/system#system-cas-drop-pool-member) — permanently decommission a dead pool member's `server_root_id`. +- [`SYSTEM CAS DROP POOL MEMBER`](/sql-reference/statements/system#system-cas-drop-pool-member) — decommission a dead pool member's `server_root_id`. diff --git a/docs/en/sql-reference/statements/system.md b/docs/en/sql-reference/statements/system.md index f82f6310f9d0e..24777a75b1336 100644 --- a/docs/en/sql-reference/statements/system.md +++ b/docs/en/sql-reference/statements/system.md @@ -580,19 +580,17 @@ SYSTEM CAS FORGET [ON CLUSTER cluster_name] disk_name `disk_name` is **required**. It is an assertion, not a proof of erasure: the disk stays registered and answers further store-class access with a typed error, and a server restart re-registers the name. Returns no result set. This is different from -[`SYSTEM CAS DROP POOL MEMBER`](#system-cas-drop-pool-member), which permanently retires one pool -*member's* identity across the whole shared pool — `FORGET` only affects this node's own local view +[`SYSTEM CAS DROP POOL MEMBER`](#system-cas-drop-pool-member), which decommissions one pool +*member* across the whole shared pool — `FORGET` only affects this node's own local view of one disk. ### SYSTEM CAS DROP POOL MEMBER {#system-cas-drop-pool-member} -Permanently decommissions a dead member (`server_root_id`) of a content-addressed disk -pool. It claims the member's mount slot as an administrative writer — fencing that `server_root_id` from ever -writing again — then drops every table namespace the member owned, drains eligible manifest debris, -staging objects, and mountpoint objects belonging to it, and retires the mount slot once all drains -are confirmed. This is a **destructive, irreversible** operation: only run it once the `server_root_id` is -confirmed permanently dead, since it fences the member out even if it later comes back online, and -it deletes namespace and drain state that cannot be recovered. +Decommissions a dead member (`server_root_id`) of a content-addressed disk +pool. It claims the member's mount slot as an administrative writer — fencing the member's current life — tombstones the member's owner, drops every table namespace the member owned, drains +eligible manifest debris, staging objects, and mountpoint objects belonging to it, writes the member's +farewell and marks the retirement verified. The control objects (`owner`, `epoch`, `mount`) stay. This is a **destructive** operation: run it only once the member is confirmed dead, since the +namespace and drain state it deletes cannot be recovered. The claim accepts only a terminal mount slot (`state` `terminated` or `fenced` in `system.cas_mounts`). A hard-killed member shows `expired` until the disk's GC leader fences it at the start of a GC round, once it has @@ -615,13 +613,19 @@ plain identifier, so it cannot be written unquoted). A `server_root_id` that con The operation is resumable: a rerun skips namespaces already marked `Removed` and reports them separately from namespaces newly removed by this invocation. Per-object drain failures are recorded -as warnings and leave the slot in a terminated-but-not-fully-drained state that a later invocation -can resume from, rather than raising an exception. +as warnings; the owner keeps an unverified tombstone that a later invocation resumes over, rather than +raising an exception. The command returns one row with columns `server_root_id`, `namespaces_removed`, `namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`, `manifest_debris_removed`, `staging_objects_removed`, -`mountpoint_objects_removed`, `slot_removed`, and `warnings`. A non-empty `warnings` means some -drain was not confirmed and the mount slot was left in place as a resume anchor. +`mountpoint_objects_removed`, `member_retired`, `warnings`, and `namespaces_pending_gc`. +`member_retired = 1` means the member's retirement is verified. `member_retired = 0` means the run stopped +before its verified mark: rerun it only under the +[rerun rule](/antalya/cas/operations/migration#rejoin-rerun-rule). Rows under `namespaces_pending_gc` +are deleted by GC and need no rerun. + +A server started later under the same `server_root_id` rejoins as an empty member once the retirement is +verified and drained ([details](/antalya/cas/operations/migration#rejoin)). ## Managing ReplicatedMergeTree Tables {#managing-replicatedmergetree-tables} diff --git a/programs/disks/CommandCaDropMember.cpp b/programs/disks/CommandCaDropMember.cpp index f95693c2a02f3..bceab1b1f3258 100644 --- a/programs/disks/CommandCaDropMember.cpp +++ b/programs/disks/CommandCaDropMember.cpp @@ -56,12 +56,13 @@ class CommandCaDropMember final : public ICommand std::cout << "server_root_id=" << report.srid << "\n" << "namespaces_removed=" << report.namespaces_removed << "\n" << "namespaces_already_removed=" << report.namespaces_already_removed << "\n" + << "namespaces_pending_gc=" << report.namespaces_pending_gc << "\n" << "committed_refs_removed=" << report.committed_refs_removed << "\n" << "precommits_removed=" << report.precommits_removed << "\n" << "manifest_debris_removed=" << report.manifest_debris_removed << "\n" << "staging_objects_removed=" << report.staging_objects_removed << "\n" << "mountpoint_objects_removed=" << report.mountpoint_objects_removed << "\n" - << "slot_removed=" << (report.slot_removed ? "true" : "false") << "\n"; + << "member_retired=" << (report.member_retired ? "true" : "false") << "\n"; for (const auto & w : report.warnings) std::cout << "warning=" << w << "\n"; } diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index e3df44f1c7aa1..2063eac7383f3 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -966,7 +966,7 @@ The server successfully detected this situation and will download merged part fr M(CASMountReleaseSkippedForeignOccupant, "Counts conclusive CAS mount-renewal observations that found a FOREIGN successor in the slot. This is the EXPECTED end state of a failover: renewal fences the deposed runtime, terminal teardown skips the farewell without backend I/O, and the successor's slot is left byte-for-byte untouched. Steady non-zero values on a cluster that is not failing over are worth investigating; a value that tracks failovers is normal.", ValueType::Number) \ M(CASMountExclusivityViolation, "Counts CAS mount releases where a runtime that still BELIEVED IT OWNED the mount (no deposition ever observed) found a FOREIGN occupant in the slot. This is the single-writer guarantee being broken, not a failover: the release refuses, the slot is left untouched, and the write fence is latched so the runtime stops trusting itself. This must always be zero.", ValueType::Number) \ M(CASRemountHeldTransient, "Counts CAS remount attempts that could not decide the pool's identity and were held transient: the `_pool_meta` probe was inconclusive or its body could not be decoded (a partially written object, an unreadable store, or a pool whose format this build no longer reads). The mount stays fenced closed and the remount loop retries; a value that keeps growing means the pool will never remount without operator action -- read the accompanying warning for the probe's own reason.", ValueType::Number) \ - M(CASIdentityLost, "Counts CAS pools that entered the IdentityLost lifecycle state: the pool sentinels (_pool_meta and the owner anchor) were observed authoritatively absent (both KeyAbsent). IdentityLost is a fail-loud TERMINAL state -- store-class access fails loud from this point and this pool's background threads (remount + GC) self-exit; there is no observer. Recover by restart or SYSTEM CAS FORGET; a matching-sentinel restore does not auto-revive.", ValueType::Number) \ + M(CASIdentityLost, "Counts CAS pools that entered the IdentityLost lifecycle state: the pool sentinels (_pool_meta and the owner anchor) were observed authoritatively absent (both KeyAbsent), or a self-remount found the owner at another life. IdentityLost is a fail-loud TERMINAL state -- store-class access fails loud from this point and this pool's background threads (remount + GC) self-exit; there is no observer. Recover by restart or SYSTEM CAS FORGET; a matching-sentinel restore does not auto-revive.", ValueType::Number) \ M(CASDataRootVanished, "Counts CAS pools that entered a terminal Vanished lifecycle state (replaced / forgotten): the pool's data root was proven foreign (pool_id mismatch) or was decommissioned by SYSTEM CAS FORGET. The disk stays registered but store-class access fails loud with a typed error naming the sub-state; restart re-registers the name.", ValueType::Number) \ M(CASRefRecoveryEpochSealed, "Number of CAS ref epoch-seal transactions MINTED by a recovery compare-and-swap walk, one per dead writer epoch it closed. Each seal occupies the exact log key a dying predecessor's in-flight PUT would take, so growth tracks epoch transitions over touched namespaces (including burned, never-written epochs), not anomalies.", ValueType::Number) \ M(CASRefRecoveryEpochSealAdopted, "Number of dead CAS ref writer epochs a recovery compare-and-swap walk found ALREADY closed by a concurrent recoverer's seal and adopted. A peer that got there first is the designed outcome, not contention to alarm on.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp index 625c04af8b41d..481910f1d91f7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp @@ -36,8 +36,8 @@ Removal removeOrReportDeleteMarker(CasOperation & op, const String & key, const "object VERSIONING enabled, and a content-addressed pool cannot run on a versioned bucket: " "every GC delete would archive a noncurrent version instead of reclaiming storage (the bucket " "grows forever), and the constantly-rewritten ref objects would pile up versions on every " - "commit. This is NOT ignorable and has no override. Use a bucket where versioning was NEVER " - "enabled — note that merely SUSPENDING versioning is not enough (deletes on a " + "commit. Use a bucket where versioning was NEVER enabled; skip_access_check skips this probe " + "but does not make a versioned bucket usable — note that merely SUSPENDING versioning is not enough (deletes on a " "versioning-suspended bucket still mint delete markers, so this probe will refuse again)"); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index addb9f4992d51..015780d809fc4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -893,7 +893,8 @@ void ContentAddressedMetadataStorage::startup() { Cas::sweepOwnMountStaging( *object_storage, - physicalKey(view.pool_prefix + "/staging/" + server_root_id) + "/"); + physicalKey(Cas::Layout(view.pool_prefix).stagingPrefix(server_root_id)), + pool->liveWriterEpoch()); } /// The background GC scheduler runs only on the disk-factory path (context non-null) and when @@ -1354,7 +1355,12 @@ String ContentAddressedMetadataStorage::stagingKeyPrefix() const { /// This is the writer-owned `staging//` subtree. `store` throws `INVALID_STATE` /// when no pool is published; every writer caller runs after `startup`. - return physicalKey(store()->poolConfig().pool_prefix + "/staging/" + server_root_id); + return physicalKey(store()->layout().stagingPrefix(server_root_id)); +} + +String ContentAddressedMetadataStorage::stagingKey(uint64_t writer_epoch, const String & leaf) const +{ + return physicalKey(store()->layout().stagingKey(server_root_id, writer_epoch, leaf)); } /// ==== namespace mapping ==== diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 28bf1020a083f..d5078dbef80d9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -402,9 +402,10 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// Returns the underlying object storage for an S3 staging writer. It is meaningful only when /// `stagingBackend` is `S3`. const ObjectStoragePtr & objectStorage() const { return object_storage; } - /// Returns the physical `pool_prefix/staging/server_root_id` prefix for this mount's staging - /// area. Callers append a unique leaf. + /// The physical `Cas::Layout::stagingPrefix` of this mount's staging area, with a trailing `/`. String stagingKeyPrefix() const; + /// The physical `Cas::Layout::stagingKey` of one staged upload of this mount. + String stagingKey(uint64_t writer_epoch, const String & leaf) const; /// Bytes that live INSIDE pool metadata rather than as their own object: an Inline-placement /// manifest tree entry, or a verbatim namespace file. nullopt = the path is blob-backed (a real diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp index fb1c7fc926461..4ed8742db4fda 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp @@ -880,8 +880,6 @@ std::unique_ptr ContentAddressedTransaction::writeFile( if (metadata_storage.stagingBackend() == Cas::StagingBackend::S3) { - const std::string staging_key = metadata_storage.stagingKeyPrefix() + "/" + getRandomASCIIString(32) + ".tmp"; - auto object_sink = metadata_storage.objectStorage()->writeObject(StoredObject(staging_key), WriteMode::Rewrite); /// Build the fixed-length CABL envelope header now (before /// the payload is streamed) so the staging object holds `[header][payload]` and the promote /// stays a verbatim server-side copy. The header carries a FRESH `incarnation_tag`; `build_id` @@ -891,11 +889,12 @@ std::unique_ptr ContentAddressedTransaction::writeFile( std::string envelope_header = buildS3StagingBlobHeader(*r); /// The staged upload becomes durable in `sink->finalize()`, outside this request contract by /// design, so its admission is re-checked there through the callback below, against the - /// generation captured HERE. `admit().generation()` reads a plain value off a temporary - /// `CasOperation` that does not outlive this statement -- it names the fence generation at - /// THIS instant, not a live operation the two calls share. + /// generation captured HERE. The epoch in the key and the generation the finalize rechecks are + /// one capture, so a remount between them cannot stage new-generation bytes under an old epoch. const Cas::PoolPtr pool = metadata_storage.store(); - const uint64_t admitted_generation = pool->mountRequests().admit().generation(); + const Cas::MountIncarnation incarnation = pool->captureMountIncarnation(); + const std::string staging_key = metadata_storage.stagingKey(incarnation.writer_epoch, getRandomASCIIString(32) + ".tmp"); + auto object_sink = metadata_storage.objectStorage()->writeObject(StoredObject(staging_key), WriteMode::Rewrite); return std::make_unique( std::move(object_sink), staging_key, @@ -909,7 +908,7 @@ std::unique_ptr ContentAddressedTransaction::writeFile( const Cas::BlobRef ref{hash_algo, Cas::codecFor(hash_algo).fromHex(hash_hex)}; stageBlobPartFile(route, ref, size, key, Cas::StagingBackend::S3); }, - [pool, admitted_generation] { pool->checkFenceOrThrow(admitted_generation); }); + [pool, generation = incarnation.fence_generation] { pool->checkFenceOrThrow(generation); }); } return std::make_unique( diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h index 849a5ff06d2df..385db8964a180 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h @@ -44,7 +44,7 @@ enum class FormatId : uint16_t /// Value 15 is retired: the fold seal is the sole per-generation coverage record after the /// one-pass GC round. The following three classes are per-server-root mount-safety objects. Owner = 16, /// `cas_owner` anchor from server-root ID to server UUID. - ServerEpoch = 17, /// `cas_epoch` writer-epoch fence carrying `next_writer_epoch`. + ServerEpoch = 17, /// `cas_epoch` writer-epoch fence carrying `next_writer_epoch` and an optional `allocation_nonce`. MountLease = 18, /// Live `cas_mount_lease` object. /// These identifiers cover objects that were added to the registry after their initial codecs: /// the ref transaction log, complete ref snapshot, blob freshness sidecar, and GC heartbeat. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp index d3c92e767c12b..3263e6095e156 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp @@ -345,4 +345,20 @@ void Layout::checkNamespace(const RootNamespace & ns) const } } +std::optional Layout::stagedEpochOf(std::string_view rest) +{ + const size_t slash = rest.find('/'); + if (slash == std::string_view::npos) + return rest.empty() ? std::nullopt : std::optional(0); + const std::string_view dir = rest.substr(0, slash); + const std::string_view leaf = rest.substr(slash + 1); + if (dir.empty() || leaf.empty() || leaf.find('/') != std::string_view::npos) + return std::nullopt; + uint64_t epoch = 0; + const auto [end, ec] = std::from_chars(dir.data(), dir.data() + dir.size(), epoch); + if (ec != std::errc{} || end != dir.data() + dir.size()) + return std::nullopt; + return epoch; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h index 5ab57bb7854b0..17ae145efa77e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h @@ -403,6 +403,19 @@ class Layout /// the sole reclaimer of `staging/` debris. String blobsPrefix() const { return prefix + "/blobs/"; } + /// One mount's S3 staging area: `/staging//`. + String stagingPrefix(const String & server_root_id) const { return prefix + "/staging/" + server_root_id + "/"; } + /// A staged upload: `/`. The epoch lets a successor's sweep tell a + /// predecessor's leftovers from its own in-flight uploads. + String stagingKey(const String & server_root_id, uint64_t writer_epoch, const String & leaf) const + { + return stagingPrefix(server_root_id) + std::to_string(writer_epoch) + "/" + leaf; + } + /// The writer epoch of a key under `stagingPrefix`, given the part after the prefix: `/` parses, + /// a flat `` is epoch 0 (written before keys carried one), anything else belongs to another member + /// nested under this prefix. + static std::optional stagedEpochOf(std::string_view rest); + /// Per-server-root control subtree, keyed by the configured `server_root_id` /// (validated by `DB::Cas::validateServerRootId`). All four control objects live together under /// `/gc/server-roots//` so a server's mount-safety state is one subtree. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp index b692a82c5811f..31773bf010e9b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp @@ -18,11 +18,14 @@ namespace OwnerWire { constexpr WireKey server_uuid{"server_uuid"}; constexpr WireKey retired_at_ms{"retired_at_ms"}; + constexpr WireKey owner_life{"owner_life"}; + constexpr WireKey retirement_verified{"retirement_verified"}; } namespace ServerEpochWire { constexpr WireKey next_writer_epoch{"next_writer_epoch"}; + constexpr WireKey allocation_nonce{"allocation_nonce"}; } namespace MountLeaseWire @@ -60,6 +63,10 @@ String encodeOwner(const OwnerObject & o) writeHex128Field(out, OwnerWire::server_uuid, o.server_uuid, first); if (o.retired_at_ms) writeNumberField(out, OwnerWire::retired_at_ms, *o.retired_at_ms, first); + if (o.owner_life != 0) + writeU64StringField(out, OwnerWire::owner_life, o.owner_life, first); + if (o.retirement_verified) + writeBoolField(out, OwnerWire::retirement_verified, true, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -86,6 +93,10 @@ OwnerObject decodeOwner(std::string_view data) } else if (key == OwnerWire::retired_at_ms) rt = r.readU64Number(); + else if (key == OwnerWire::owner_life) + o.owner_life = r.readU64String(); + else if (key == OwnerWire::retirement_verified) + o.retirement_verified = r.readBool(); else r.skipUnknown(key); } @@ -103,6 +114,8 @@ String encodeServerEpoch(const ServerEpoch & e) writeHeaderLine(out, FormatId::ServerEpoch); bool first = true; writeU64StringField(out, ServerEpochWire::next_writer_epoch, e.next_writer_epoch, first); + if (e.allocation_nonce) + writeHex128Field(out, ServerEpochWire::allocation_nonce, *e.allocation_nonce, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -126,6 +139,8 @@ ServerEpoch decodeServerEpoch(std::string_view data) e.next_writer_epoch = r.readU64String(); saw = true; } + else if (key == ServerEpochWire::allocation_nonce) + e.allocation_nonce = r.readHex128(); else r.skipUnknown(key); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h index 776288c96906b..0647817580c79 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -26,18 +27,36 @@ namespace DB::Cas struct OwnerObject { UInt128 server_uuid{}; - /// Set when this identity was explicitly decommissioned by an operator (`CasDecommission`'s - /// final step tombstones the owner object in place rather than deleting it, since its content + /// Set when this identity was explicitly decommissioned by an operator (`CasDecommission` + /// tombstones the owner object in place rather than deleting it, since its content /// alone cannot distinguish decommissioned debris from a legitimate successor's live anchor). /// A normal claim (`claimOwnerOrThrow`) must refuse to silently resume a tombstoned identity. std::optional retired_at_ms; + /// Every owner write sets this to the read value + 1, so two writes never carry the same bytes and an + /// `If-Match` against an older incarnation cannot land late. 0 = written before the field existed. + uint64_t owner_life = 0; + /// Set by the decommission run whose claim, tombstone and farewell all held; a later tombstone refresh + /// keeps it, a rollback drops it. + bool retirement_verified = false; }; +/// The owner of the next life after `current`. Every owner write after the first claim goes through it, so +/// the life only grows and no write repeats earlier bytes. +inline OwnerObject nextOwnerLife(const OwnerObject & current, std::optional retired_at_ms, bool retirement_verified) +{ + return OwnerObject{.server_uuid = current.server_uuid, .retired_at_ms = retired_at_ms, + .owner_life = current.owner_life + 1, .retirement_verified = retirement_verified}; +} + /// Durable counter state used to allocate unique writer epochs. The stored value is the next epoch /// to allocate; the caller CAS-writes a larger value and adopts the previous value for its mount. struct ServerEpoch { uint64_t next_writer_epoch = 0; + /// Random per allocation decision, so an unclear write is ours only if the stored bytes match + /// exactly: a twin of the same uuid bumping to the same value writes a different nonce. Absent in + /// bodies written by 26.6.4. + std::optional allocation_nonce = std::nullopt; }; /// The current liveness lease for one `(server_uuid, writer_epoch)` writer incarnation. The pool @@ -58,6 +77,18 @@ struct MountLease /// One holder-originated body identity. Every physical retry of that one logical write reuses /// it; a GC fence copies the observed value while every successor holder body mints a new one. UInt128 write_attempt_id{}; + + /// The holder's clean farewell (release): `min_active_build_sequence == UINT64_MAX`. + bool isFarewell() const { return min_active_build_sequence == std::numeric_limits::max(); } + bool isHeldBy(UInt128 uuid, uint64_t epoch) const { return server_uuid == uuid && writer_epoch == epoch; } + /// What a fence writes over this body: the same holder, `gc_fenced`, one `seq` later. + MountLease fencedCopy() const + { + MountLease fenced = *this; + fenced.gc_fenced = true; + fenced.seq += 1; + return fenced; + } }; /// Encode the owner anchor as canonical text with the `cas_owner` header and a final newline. The @@ -66,8 +97,8 @@ struct MountLease /// decisions belong to the caller that coordinates the server-root object. String encodeOwner(const OwnerObject & o); -/// Decode an owner anchor, requiring its `server_uuid` field, tolerating an absent optional `retired_at_ms` retirement -/// timestamp, and rejecting bytes after the body line. Unknown JSON fields are skipped for +/// Decode an owner anchor, requiring its `server_uuid` field, tolerating absent `retired_at_ms`, `owner_life` and +/// `retirement_verified`, and rejecting bytes after the body line. Unknown JSON fields are skipped for /// forward-compatible reads; malformed input, a missing `server_uuid`, and trailing data throw /// `CORRUPTED_DATA`. OwnerObject decodeOwner(std::string_view data); @@ -77,7 +108,8 @@ OwnerObject decodeOwner(std::string_view data); /// codec. String encodeServerEpoch(const ServerEpoch & e); -/// Decode the epoch counter, requiring its `next_writer_epoch` field and rejecting bytes after the body line. +/// Decode the epoch counter, requiring its `next_writer_epoch` field, tolerating an absent `allocation_nonce`, and +/// rejecting bytes after the body line. /// Unknown JSON fields are skipped for forward-compatible reads; malformed input, a missing or zero `next_writer_epoch`, /// and trailing data throw `CORRUPTED_DATA`. ServerEpoch decodeServerEpoch(std::string_view data); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index f56b3d0c90281..cb4fb749c3633 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -32,7 +32,7 @@ trailer, followed by a banner-framed raw payload zone for inline file bytes. | `gc/gen//attempt//outcomes/…​.zst` | outcome log (`kind`, `outcome`) | `CasGcOutcomesFormat` (`.zst`) | GC | | `gc/gen//attempt//fold_seal` | fold seal (deterministic; `generation`/`parent_generation`, `kind`: `ref_life`/`blob_run`/`condemned`) | `CasFoldSealFormat` | GC | | `gc/gen//…​/runs` | GC source-edge record-stream runs (`ref`, `src`, `mark`; condemned: `pending`, `size`, `condemn_round`, `confirmed`) | `CasRecordStreamFormat` | GC | -| `gc/server-roots//{owner,epoch,mount}` | server-root singletons (`server_uuid`, optional `retired_at_ms`; `next_writer_epoch`; `server_uuid`, `writer_epoch`, `hostname`, `pid`, `started_at_ms`, `seq`, `expires_at_ms`, `min_active_build_sequence`, `gc_fenced`, `write_attempt_id`) | `CasServerRootFormats` | mount | +| `gc/server-roots//{owner,epoch,mount}` | server-root singletons (`server_uuid`, optional `retired_at_ms`, optional `owner_life` (u64 decimal string; absent in 26.6.4 bodies, read as 0; additive, no `v` bump), optional `retirement_verified` (bool; absent = false); `next_writer_epoch`, optional `allocation_nonce` (32 lowercase hex, drawn once per allocation decision; absent in 26.6.4 bodies, which still decode; additive ordinary key, no `v` bump); `server_uuid`, `writer_epoch`, `hostname`, `pid`, `started_at_ms`, `seq`, `expires_at_ms`, `min_active_build_sequence`, `gc_fenced`, `write_attempt_id`) | `CasServerRootFormats` | mount | | `roots/…` | raw passthrough (verbatim) | — (never interpreted) | upper layers | ## Codec table diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp index 28a471b673ff6..822cb25303515 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp @@ -546,7 +546,7 @@ bool prefixEligibleUnder(const std::optional & floor, const BuildPre return true; if (prefix.writer_epoch > w.writer_epoch) return false; - if (w.min_active_build_sequence == std::numeric_limits::max()) + if (w.isFarewell()) return true; /// farewell/retired sentinel: every seq is retired return w.min_active_build_sequence > prefix.build_sequence; } @@ -561,7 +561,8 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi std::vector * warnings) { const Layout & layout = store.layout(); - CasOperation op = store.openRequests().admit(); + /// The decommission drain runs under the admin's lease; a fenced admin must stop deleting. + CasOperation op = store.mountRequests().admit(); if (!prefixEligibleOn(op, layout, ns, prefix)) return 0; /// not eligible by the durable watermark fact — delete nothing (controls #8/#9) @@ -658,6 +659,7 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi throw; warnings->push_back("CAS orphan sweep: " + listed.key + " delete failed: " + getCurrentExceptionMessage(/*with_stacktrace=*/false)); + return op.admitted(); } return true; }, Retry::standard(), 1000, onGcEnumerationPage); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h index d4da7e38f5a5a..e2d63f5053904 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h @@ -177,8 +177,8 @@ struct ManifestSweepResult /// /// `warnings`, when non-null, opts in to the decommission drain's tolerate-and-continue contract: a /// per-key transient failure (a thrown backend exception on `head`/`remove`) -/// is pushed onto `*warnings` and the sweep continues with the next key, instead of throwing out of -/// this call; likewise a protection-view-unavailable namespace (the pre-existing corrupt-snapshot skip +/// is pushed onto `*warnings` and the sweep continues with the next key while `store`'s mount still admits +/// requests, instead of throwing out of this call; likewise a protection-view-unavailable namespace (the pre-existing corrupt-snapshot skip /// below) also pushes a "cannot confirm emptiness" warning, not just a `LOG_WARNING`. `warnings == /// nullptr` (the default, every pre-existing caller) preserves the original behaviour exactly: a /// per-key failure propagates as an exception (fail-close default), and the protection-view skip is @@ -186,6 +186,8 @@ struct ManifestSweepResult /// normal "a fresh owner reclaimed it" race the periodic sweep expects, not a failure to warn about. /// This direct decommission path relies on the caller's held server-root claim/fence: while that claim /// is held, a same-server-root rebirth cannot become live between its catalog cut and exact-token delete. +/// Every request runs on `store`'s mount plane, so `store` must hold a writable mount; once its fence trips, +/// the remaining requests are refused. /// It still rejects every catalog cut with an ambiguous current life id before making any deletion decision. uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix, std::vector * warnings = nullptr); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp index 7028e4f657215..d12a45f290683 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp @@ -1028,7 +1028,7 @@ std::unique_lock CasMountRuntime::lockTerminalPublication() return lock; } -void CasMountRuntime::enterIdentityLost() +void CasMountRuntime::enterIdentityLost(const String & reason) { /// `TransientNotLive -> IdentityLost`, one way. The compare-exchange FROM `TransientNotLive` gives /// the brief's "from TransientNotLive only" precondition, idempotency (a second call finds the state @@ -1063,12 +1063,11 @@ void CasMountRuntime::enterIdentityLost() ProfileEvents::increment(ProfileEvents::CASIdentityLost); LOG_WARNING(getLogger("CasPool"), - "Content-addressed pool '{}' entered IdentityLost: the pool sentinels (_pool_meta and the owner " - "anchor) are authoritatively absent (both KeyAbsent). This is a fail-loud TERMINAL state: " + "Content-addressed pool '{}' entered IdentityLost: {}. This is a fail-loud TERMINAL state: " "store-class access now fails loud and this pool's remount + GC threads self-exit. " "Recover by restart or SYSTEM CAS FORGET — a matching-sentinel restore does NOT " "auto-revive this disk.", - server_root_id); + server_root_id, reason); } void CasMountRuntime::enterVanished(PoolLifecycle which, const String & reason) @@ -1241,7 +1240,7 @@ void CasMountRuntime::finishTeardown(bool drained) { LOG_WARNING( getLogger("CasPool"), - "CAS store shutdown with an unresolved ref-log PUT: skipping the clean-release marker; " + "CAS store shutdown with an unresolved write: skipping the clean-release marker; " "the next mount will treat this end as unclean"); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h index 1b69c885d092f..e5bb19a2c1e74 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h @@ -32,8 +32,8 @@ using PartWriteTxnPtr = std::shared_ptr; /// - `Live` — the steady state; the mount lease is (or was last) held. /// - `TransientNotLive` — the lease was lost; access is uncertain and a self-remount retries. The §2 /// `Present`+identity-match recovery rule fires only from here (or `Live`). -/// - `IdentityLost` — the pool sentinels are authoritatively absent (both KeyAbsent): -/// fail-loud and TERMINAL (rev.8). The remount/GC workers self-exit; +/// - `IdentityLost` — the pool sentinels are authoritatively absent (both KeyAbsent), or a +/// remount found the owner at another life: fail-loud and TERMINAL (rev.8). The remount/GC workers self-exit; /// matching-sentinel reappearance does NOT auto-revive it ([D3]); recovery is a /// restart or `SYSTEM CAS FORGET`. /// - `Vanished*` — fully terminal truth: the data root was replaced by a foreign pool, or the @@ -263,8 +263,9 @@ class CasMountRuntime /// (and the GC scheduler self-exits, through `Pool`) at its next boundary; there is no demoted observer. /// It deliberately does NOT publish the `vanished_intent` latch (which is reserved for the `Vanished*` /// idempotency/FORGET protocol); `remountTerminal` widens the worker-exit boundary to include it. + /// `reason` names the cause in that WARN: the pool sentinels are gone, or the owner moved to another life. /// Must be called under the caller's remount serialization (Pool::remount_mutex). - void enterIdentityLost(); + void enterIdentityLost(const String & reason); /// Test seam: force the lifecycle condition directly to `lc`, bypassing the natural transition /// preconditions (used by the operation-gate tests to pin each class × state cell without driving a /// full remount/erase sequence). For a `Vanished*` value it also latches `vanished_intent`, so the @@ -477,8 +478,8 @@ class CasMountRuntime /// The renewal's liveness: facts the mount fence cannot see -- a shutdown request, a parked or /// park-requested driver, a pool that left `Live`. FALSE ends the renewal. bool renewalLive(bool worker_call) const; - /// Whether this node has already been asked to stop. Sampled ONCE, before the write, so a refusal - /// caused by the stop cannot be mistaken for one that preceded it. + /// Whether this node has been asked to stop; read after a refused renewal, so a stop that lands + /// mid-renewal is not mistaken for a lost lease. bool renewalCancelled() const; void tripFenceWithoutOperationalLoss(); std::unique_lock lockTerminalPublication(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index c9308e83ab402..d297503fa1b61 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -77,6 +77,16 @@ namespace std::atomic remount_attempt_sequence{0}; +/// What a mount that skipped the capability battery did not prove about its store. Said once per open at +/// the default level, because nothing else tells an operator the checks did not run. +void logSkippedCapabilityProbe(const String & srid, const String & pool_prefix, std::string_view reason) +{ + LOG_INFO(getLogger("CasPool"), + "CAS pool '{}' (prefix '{}'): the capability probe was skipped ({}); unverified on this mount: bucket " + "versioning, conditional-write support, list-after-write visibility", + srid, pool_prefix, reason); +} + /// Validate every writable factory's lease/request/cadence relationship before it can publish /// owner, epoch, mount, or probe authority. Decommission forces background renewal before calling /// this helper, so it is held to the same cadence window as an ordinary production mount. @@ -338,8 +348,8 @@ String Pool::lifecycleReasonDetail(PoolLifecycle lc) const case PoolLifecycle::TransientNotLive: return {}; case PoolLifecycle::IdentityLost: - return "identity lost — the pool sentinels are absent; access fails loud. Recover by restart or " - "SYSTEM CAS FORGET (a matching-sentinel restore does not auto-revive it)."; + return "identity lost — the pool sentinels are absent, or the owner moved to another life; access fails " + "loud. Recover by restart or SYSTEM CAS FORGET (a matching-sentinel restore does not auto-revive it)."; case PoolLifecycle::VanishedReplaced: return "data root replaced by a foreign pool (pool_id mismatch) — our generation is gone; " "restart re-registers the name."; @@ -531,6 +541,7 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// working single-attempt client still fails closed at open instead of silently /// corrupting CAS state under blind retries later. backend->checkConditionalWriteSingleAttemptSupport(); + logSkippedCapabilityProbe(config.server_root_id, config.pool_prefix, "skip_access_check"); } } /// The catalog is mandatory for every minted pool. Make it durable before `_pool_meta`: otherwise @@ -580,13 +591,26 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) return store; } +MountIncarnation Pool::captureMountIncarnation() const +{ + /// Admission first, then the epoch, then the admission again: a remount publishes the new epoch + /// before it bumps the generation, so a generation that held across the epoch read dates the epoch. + const uint64_t generation = mount_requests.admit().generation(); + if (incarnation_capture_hook_for_test) + incarnation_capture_hook_for_test(); + const uint64_t writer_epoch = liveWriterEpoch(); + checkFenceOrThrow(generation); + return MountIncarnation{.writer_epoch = writer_epoch, .fence_generation = generation}; +} + void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy policy) { /// === Mount-safety startup protocol === /// STRICT ORDER: validate id → claim owner (identity) → allocate durable writer_epoch → claim - /// the mount lease (liveness) + arm the local write fence → anchor the watermark. owner / epoch - /// / mount / watermark are BOOTSTRAP-CONTROL writes: they establish the very right to write and - /// run BEFORE the write fence gates ordinary data/ref/manifest mutations. Fail closed throughout. + /// the mount lease (liveness) → re-read the owner → install the renewer + arm the local write fence + /// → anchor the watermark. owner / epoch / mount / watermark are BOOTSTRAP-CONTROL writes: they + /// establish the very right to write and run BEFORE the write fence gates ordinary data/ref/manifest + /// mutations. Fail closed throughout. /// Shared by `open` (`policy = WaitForExpiry`, `our_uuid = config.server_id`) and /// `openForDecommission` (`policy = NoWait`, `our_uuid` = the impersonated victim owner uuid; /// -- the two differ only in WHO they mount @@ -610,17 +634,22 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// can reach a slot mutation; the callback remains available for fresh conflict rechecks below. (void)observe_catalog(); - /// 2. Owner anchor — IDENTITY (clock-free). A foreign uuid fails closed; an absent owner over a - /// non-empty subtree is CORRUPTED_DATA; a fresh empty root is claimed. + /// 2. Owner anchor — IDENTITY (clock-free). A foreign uuid fails closed unless it is a verified, drained + /// retirement (rejoin); an absent owner over a non-empty subtree is CORRUPTED_DATA; a fresh empty root + /// is claimed. /// Every step of this protocol runs on the OPEN plane. These are bootstrap-control writes: they /// establish the very right to write, and the mount fence they would otherwise be gated on is /// either unarmed (a first open) or latched lost (a self-remount, which could then never reclaim). + const bool is_decommission = policy == MountClaimPolicy::NoWait; + const RetiredOwnerPolicy claim_retired_policy + = is_decommission ? RetiredOwnerPolicy::AcceptForDecommission : RetiredOwnerPolicy::RejoinIfVerified; + const RetiredOwnerPolicy recheck_retired_policy + = is_decommission ? RetiredOwnerPolicy::AcceptForDecommission : RetiredOwnerPolicy::Refuse; CasOperation owner_op = store->gc_requests.admit(); - claimOwnerOrThrow(owner_op, store->pool_layout, srid, our_uuid, observe_catalog); + const uint64_t owner_life = claimOwnerOrThrow( + owner_op, store->pool_layout, srid, our_uuid, observe_catalog, claim_retired_policy); + store->recorded_owner_life = owner_life; - /// Wall-clock `now_ms`, hoisted above the writer_epoch allocation below: the absent-epoch - /// branch's `DecommissionRecovery` policy needs it to judge a surviving mount's liveness before - /// the mount-lease claim (step 4) gets its own use of the same clock. const auto now_ms = []() -> uint64_t { return static_cast(std::chrono::duration_cast( @@ -634,16 +663,9 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// Mutable: a GC fence of our fresh lease during open (expiry mid-open racing a GC round) is /// recoverable — a fence costs an epoch, so the fence-recovery loop below re-allocates a fresh /// writer_epoch and re-claims (the TLA+-checked `NoPermanentWedge` invariant). - /// `epoch_policy`: `openForDecommission`'s `NoWait` gates the absent-epoch branch's mount-probe - /// on a TERMINAL (not live) surviving mount instead of authoritative absence (Phase C) — passed - /// uniformly to every call below, including the fence-recovery re-allocations, where it is inert - /// because those run with the epoch object already present. - const EpochMintPolicy epoch_policy = (policy == MountClaimPolicy::NoWait) - ? EpochMintPolicy::DecommissionRecovery - : EpochMintPolicy::NormalMount; CasOperation epoch_op = store->gc_requests.admit(); uint64_t writer_epoch = allocateWriterEpoch( - epoch_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); + epoch_op, store->pool_layout, srid, observe_catalog); store->mount_runtime.setProcessEpoch(writer_epoch, std::memory_order_relaxed); /// 4. Mount lease — LIVENESS. Decide over the current mount object using the wall-clock `now_ms` @@ -714,6 +736,8 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol if (policy == MountClaimPolicy::WaitForExpiry) { CasOperation claim_op = store->gc_requests.admit(); + const OwnerExpectation foreign_reclaim_owner{ + .server_uuid = our_uuid, .owner_life = owner_life, .retired = RetiredOwnerPolicy::Refuse}; const bool unsafe = store->config.unsafe_remount_no_delay; if (unsafe) { @@ -723,11 +747,11 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// freshly minted) falls through to the ordinary observed path /// below. claim = claimMount(claim_op, store->pool_layout, srid, our_uuid, writer_epoch, now_ms(), ttl_ms, - /*proven_dead_incarnation=*/{}, emit_mount_event); + /*proven_dead_incarnation=*/{}, emit_mount_event, /*unsafe_reclaim_authorization=*/{}, foreign_reclaim_owner); if (claim.kind == MountClaimResult::LiveDoubleStart && claim.etag && claim.body && claim.body->server_uuid == our_uuid) claim = claimMount(claim_op, store->pool_layout, srid, our_uuid, writer_epoch, now_ms(), ttl_ms, - {}, emit_mount_event, /*unsafe_reclaim_authorization=*/claim.etag); + {}, emit_mount_event, /*unsafe_reclaim_authorization=*/claim.etag, foreign_reclaim_owner); } /// A `FencedSelf`, a foreign-uuid `LiveDoubleStart`/`ForeignOwner`, or a raced /// `LiveDoubleStart` the authorization above did not cover falls through here and re-runs @@ -739,13 +763,15 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol claim = claimMountAwaitingExpiry( claim_op, store->pool_layout, srid, our_uuid, writer_epoch, [&now_ms]() { return now_ms(); }, [raw] { return raw->bootMsNow(); }, - ttl_ms, poll_interval_ms, sleep_ms, on_wait_start, emit_mount_event); + ttl_ms, poll_interval_ms, sleep_ms, on_wait_start, emit_mount_event, foreign_reclaim_owner); } else { /// NoWait (decommission gate): a single unobserved attempt -- no bounded wait-and-retry /// for a stale-looking lease to lapse. Anything but Claimed/FencedSelf below is refused /// immediately. + /// No owner expectation: the admin impersonates the owner it retires, so whose life counts as + /// "ours" is ambiguous, and it never reclaims another uuid's body. CasOperation claim_op = store->gc_requests.admit(); claim = claimMount(claim_op, store->pool_layout, srid, our_uuid, writer_epoch, now_ms(), ttl_ms, /*proven_dead_incarnation=*/{}, emit_mount_event); @@ -760,7 +786,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol srid, max_fence_recoveries); CasOperation reallocate_op = store->gc_requests.admit(); writer_epoch = allocateWriterEpoch( - reallocate_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); + reallocate_op, store->pool_layout, srid, observe_catalog); store->mount_runtime.setProcessEpoch(writer_epoch, std::memory_order_relaxed); continue; } @@ -797,6 +823,13 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol } claimed_prior = claim.prior; + /// The owner may have moved since the claim began: a tombstone, a rollback or a rejoin. Nothing + /// below runs under an owner this process did not start under. + CasOperation recheck_op = store->gc_requests.admit(); + recheckOwnerAfterClaim(recheck_op, store->pool_layout, srid, + OwnerExpectation{.server_uuid = our_uuid, .owner_life = owner_life, .retired = recheck_retired_policy}, + writer_epoch, emit_mount_event); + /// The mount object now holds OUR live (uuid, epoch) body. `installRenewer` constructs the renewer /// -- which ADOPTS that very (uuid, epoch) slot rather than self-tripping the double-start guard -- /// AND wires its `minActive` build-watermark reader, its event sink, and the fence-coupling @@ -815,7 +848,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol store->mount_runtime.renewerReset(); CasOperation refenced_epoch_op = store->gc_requests.admit(); writer_epoch = allocateWriterEpoch( - refenced_epoch_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); + refenced_epoch_op, store->pool_layout, srid, observe_catalog); store->mount_runtime.setProcessEpoch(writer_epoch, std::memory_order_relaxed); continue; } @@ -938,6 +971,8 @@ PoolPtr Pool::openForDecommission(BackendPtr backend, PoolConfig config, const S /// refusing an ordinary mount costs availability and protects data, whereas refusing a /// decommission strands a pool with a dead replica in it and leaves the operator no way forward. config.skip_access_check = true; + logSkippedCapabilityProbe( + victim_srid, config.pool_prefix, fmt::format("decommission of pool member '{}'", victim_srid)); /// The admin claim must be RENEWED like any writable mount: the host disk may be observe-only /// (background_watermark=false), but an unrenewed claim (TTL ~30s) aborts any long drain midway. config.background_watermark = true; @@ -952,7 +987,7 @@ PoolPtr Pool::openForDecommission(BackendPtr backend, PoolConfig config, const S /// below is then EXACTLY the crash-recovery reclaim semantics (`MountClaimPolicy::NoWait`): a /// fenced/terminated/clean-farewell lease reclaims; a live lease refuses immediately (no bounded /// observation wait -- see `mountWritable`). Owner anchor absent + mount absent = nothing to - /// decommission. + /// decommission. A tombstoned owner of the victim is accepted: a rerun resumes over its own tombstone. /// The open plane: this factory impersonates the victim to take its mount, so there is no lease of /// ours to be gated on until the claim below establishes one. `config.retry_sleep_fn` must travel /// with `config.boot_ms_fn`: a retry loop bound to a frozen test clock that only a fake sleep @@ -973,6 +1008,15 @@ PoolPtr Pool::openForDecommission(BackendPtr backend, PoolConfig config, const S } config.server_id = *victim_uuid; + /// A member of this build always has a counter next to its owner. An absent counter is a state left by + /// 26.6.4 or by a process that died between the owner create and the counter create; minting one here + /// would hand out epoch 1 again over whatever that member wrote. + if (!owner_op.read(layout.epochKey(victim_srid), Retry::standard())) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS decommission '{}': no epoch counter '{}'; a decommission never creates one. Start the server with a new " + "server_root_id; see decommission compatibility in the CAS migration docs", + victim_srid, layout.epochKey(victim_srid)); + backend->checkConditionalWriteSingleAttemptSupport(); /// Decommission operates on an EXISTING pool member (an owner anchor / mount lease was just found), so /// `_pool_meta` must already be present. It never bootstraps: `allow_mint=false` so an absent meta @@ -1051,7 +1095,7 @@ Pool::~Pool() config.teardown_phase2_throw_for_test(); const bool ref_lanes_drained = ref_ledger.drainRefLanesForShutdown( config.cas_request_budget.attemptEnvelopeMs() + config.cas_request_budget.lease_safety_margin_ms); - drained = ref_lanes_drained && !writerCleanupDutiesPending(); + drained = ref_lanes_drained && !writerCleanupDutiesPending() && !farewell_withheld; }, "CAS pool teardown: draining ref lanes"); /// 3. Retire the merged heartbeat. This always runs after phase 2, even when that phase failed, and @@ -1418,7 +1462,7 @@ bool Pool::tryRemountOnce() /// fail-loud TERMINAL state — the remount worker self-exits at its next boundary (see /// `CasMountRuntime::remountTerminal`), so there is no demoted observer. if (mount_runtime.lifecycle() != PoolLifecycle::IdentityLost) - mount_runtime.enterIdentityLost(); + mount_runtime.enterIdentityLost(gate.reason); return false; case LifecycleGateVerdict::StayTransient: /// Uncertain — remain transient and let the recovery loop retry. The probe's own reason is @@ -1459,11 +1503,20 @@ bool Pool::tryRemountOnce() /// not clear it), so an operation admitted under the fence could never make the claim that /// re-establishes it. CasOperation owner_op = gc_requests.admit(); - claimOwnerOrThrow(owner_op, pool_layout, srid, our_uuid, observe_catalog); + const uint64_t owner_life = claimOwnerOrThrow(owner_op, pool_layout, srid, our_uuid, observe_catalog); + if (owner_life != recorded_owner_life) + { + /// A rejoin or a rollback moved the owner since this process started. No retry can bring the old + /// life back, so stop for good before an epoch is allocated; a restart runs under the current life. + error = fmt::format("the owner moved from life {} to life {} since this process started; restart the server", + recorded_owner_life, owner_life); + mount_runtime.enterIdentityLost(error); + return false; + } step = "writer_epoch_allocate"; CasOperation epoch_op = gc_requests.admit(); const uint64_t writer_epoch = allocateWriterEpoch( - epoch_op, pool_layout, srid, EpochMintPolicy::NormalMount, 0, observe_catalog); + epoch_op, pool_layout, srid, observe_catalog); result_writer_epoch = writer_epoch; /// Mount-slot writer audit: `this` is already fully open (setEventSink ran long ago), so @@ -1487,16 +1540,23 @@ bool Pool::tryRemountOnce() "epoch={} expires_at_ms={}) for up to ~{} ms before reclaiming", srid, u128ToHex(held.server_uuid), held.writer_epoch, held.expires_at_ms, threshold_ms); }, - emit_mount_event); + emit_mount_event, + OwnerExpectation{.server_uuid = our_uuid, .owner_life = recorded_owner_life, .retired = RetiredOwnerPolicy::Refuse}); if (claim.kind != MountClaimResult::Claimed) { LOG_WARNING(getLogger("CasPool"), "CAS self-remount '{}': mount not claimable ({}); will retry", srid, - claim.kind == MountClaimResult::ForeignOwner ? "foreign owner — never taking over" + claim.kind == MountClaimResult::ForeignOwner ? "foreign live body, or the owner did not match — not taking over" : "a live twin holds the lease"); return false; } + step = "owner_recheck"; + CasOperation recheck_op = gc_requests.admit(); + recheckOwnerAfterClaim(recheck_op, pool_layout, srid, + OwnerExpectation{.server_uuid = our_uuid, .owner_life = recorded_owner_life, .retired = RetiredOwnerPolicy::Refuse}, + writer_epoch, emit_mount_event); + /// NO WAIT HERE, AND THE ASSERT IS WHAT REPLACES IT. This used to pay `materialization_grace_ms` /// whenever this incarnation's own ref lanes had not provably settled before the fence tripped: /// an unresolved (still-wedged) ref-log conditional `PUT` from the dying epoch could otherwise diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index ca9415369eeae..badf118364480 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -362,6 +362,12 @@ struct PoolConfig } }; +/// The live incarnation at one instant: the writer epoch and the fence generation it was admitted under. +struct MountIncarnation +{ + uint64_t writer_epoch = 0; + uint64_t fence_generation = 0; +}; struct PartWriteInfo { @@ -424,19 +430,21 @@ class Pool : public std::enable_shared_from_this /// Construct a pool after backend capabilities and durable pool metadata have been validated. /// For a writable configuration, `open` also claims the configured mount before returning; a /// read-only configuration returns an instance that performs no mutating startup operations. + /// A start over a verified, drained retirement of its `server_root_id` rejoins it as an empty member + /// (`claimOwnerOrThrow`). static PoolPtr open(BackendPtr backend, PoolConfig config); - /// Admin writer mount of the VICTIM `server_root_id`, for `SYSTEM CAS DROP POOL - /// MEMBER`. Impersonates the victim's - /// owner uuid (`readOwnerUuid`, or -- when the owner anchor itself is missing -- recovered from a - /// lingering mount lease) and mounts writable under `MountClaimPolicy::NoWait`: a live victim - /// lease is an immediate `ABORTED` refusal (no wait-and-observe, no FORCE variant), unlike the - /// bounded reclaim wait a normal `open` pays. Throws `BAD_ARGUMENTS` when there is nothing to - /// decommission (no owner anchor and no mount lease for `victim_srid`). + /// Admin writer mount of the VICTIM `server_root_id`, for `SYSTEM CAS DROP POOL MEMBER`. + /// Impersonates the victim's owner uuid (`readOwnerUuid`, or, when the owner was deleted by hand, the uuid + /// of the surviving lease) and mounts writable under `MountClaimPolicy::NoWait`: a live victim lease is an + /// immediate `ABORTED` refusal; a tombstoned owner of the victim is accepted; a member without an epoch + /// counter is refused (`CORRUPTED_DATA`). Throws `BAD_ARGUMENTS` when neither owner nor lease exists. static PoolPtr openForDecommission(BackendPtr backend, PoolConfig config, const String & victim_srid); /// Stop the mount-renewal and remount activity, drain ref-log work, and release the owned /// backend-facing components in their dependency order. Destruction is also the clean-farewell /// path for a writable mount, so it must complete before the owning backend is released. ~Pool(); + /// The teardown writes no clean farewell: a successor then waits for this lease to be fenced or observed dead. + void withholdFarewell() noexcept { farewell_withheld = true; } bool tryDispatchDetached(std::function task); /// Marks this pool as being torn down. The open request plane refuses every further admission and @@ -514,6 +522,9 @@ class Pool : public std::enable_shared_from_this /// Throws the typed transient refusal (`throwCasTransientUnavailable`) unless the fence is currently /// held AND still at `admitted_generation`. void checkFenceOrThrow(uint64_t admitted_generation) const { mount_runtime.checkFenceOrThrow(admitted_generation); } + /// Throws the transient refusal when the fence moved between the two reads, so the pair is never mixed. + MountIncarnation captureMountIncarnation() const; + void setIncarnationCaptureHookForTest(std::function hook) { incarnation_capture_hook_for_test = std::move(hook); } /// ---- pool lifecycle condition (rev.7 §1; owned by `mount_runtime`) ---- /// Atomic read of the current lifecycle condition. Thin forwarder; safe to call from any thread. @@ -773,10 +784,10 @@ class Pool : public std::enable_shared_from_this /// on the fence would make a self-remount, which runs with the fence latched lost, unable ever to /// reclaim. CasRequests & openRequests() { return gc_requests; } - /// The owning `BackendPtr` itself (not just a reference into it): the decommission slot-retirement - /// decommission step (`CasDecommission.cpp`) must keep the backend alive across `admin.reset()` -- the graceful - /// close that stamps the mount's farewell -- to physically delete the control objects afterward. A - /// bare `Backend &` from `backend()` would dangle the instant the owning `Pool` is destroyed. + /// The owning `BackendPtr` itself (not just a reference into it): decommission (`CasDecommission.cpp`) + /// keeps the backend alive across `admin.reset()` -- the graceful close that stamps the mount's farewell -- + /// to read the farewell back and mark the retirement verified. A bare `Backend &` from `backend()` would + /// dangle the instant the owning `Pool` is destroyed. BackendPtr poolBackendPtr() const { return pool_backend; } /// Staging write surface for `PartWriteTxn`: thin delegate onto the ref ledger, so a staging write @@ -1211,6 +1222,7 @@ class Pool : public std::enable_shared_from_this /// config they take their clock from. `mutable` because issuing a request is not a change to the /// pool: a `const` observer still has to read the store. mutable CasRequests mount_requests; + std::function incarnation_capture_hook_for_test; mutable CasRequests lease_requests; mutable CasRequests gc_requests; @@ -1222,6 +1234,7 @@ class Pool : public std::enable_shared_from_this /// Sticky fail-close bit for the destructor's allocation-failure path. If a duty could not enter /// the queue, no mount teardown may claim a clean farewell even though the guarded map is empty. std::atomic writer_cleanup_queue_failed{false}; + std::atomic farewell_withheld{false}; /// CAS mixed-algo pools: monotone in-memory cache of `algos_used`, seeded from /// `meta.algos_used` at open. Guards `isAlgoAdmitted`/`refreshAdmittedAlgos` -- ITS OWN mutex, @@ -1279,6 +1292,9 @@ class Pool : public std::enable_shared_from_this /// the Pool-side orchestration, so it stays on Pool. std::mutex remount_mutex; + /// Set once by the owner claim in `mountWritable`; a remount checks against it and never updates it. + uint64_t recorded_owner_life = 0; + /// The single home of the [D5] per-lifecycle reason detail (spec §1) — the human-readable text that /// names the ACTUAL sub-state, WITHOUT the `content-addressed pool '' ` prefix. Both /// `throwIfLifecycleTerminal` (which prefixes it and throws) and the non-gated `lifecycleSnapshot` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index ac75f757e1754..b65df65fec516 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -429,10 +429,6 @@ void deliverMountRenewObservability( } } -/// Forward declaration: defined below (same TU-unique anonymous namespace) — `allocateWriterEpoch` -/// names the current mount holder in its DecommissionRecovery live-refusal message. -String describeMountHolder(const MountLease & m); - std::optional readOwnerObject(CasOperation & op, const Layout & l, const String & server_root_id) { const auto got = op.read(l.ownerKey(server_root_id), Retry::standard()); @@ -441,17 +437,22 @@ std::optional readOwnerObject(CasOperation & op, const Layout & l, return decodeOwner(got->bytes); } -void throwIfOwnerRetired(const OwnerObject & owner, const String & srid) +void throwIfOwnerRetired(const OwnerObject & owner, const String & srid, const String & owner_key) { if (!owner.retired_at_ms) return; + if (owner.retirement_verified) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS server-root '{}' was decommissioned (owner tombstoned at {} ms, life {}); this process cannot run " + "under the retired owner. Restart the server: a server start rejoins a finished decommission as an " + "empty member", + srid, *owner.retired_at_ms, owner.owner_life); throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS server-root '{}' was explicitly decommissioned (owner tombstoned at {} ms by " - "SYSTEM CAS DROP POOL MEMBER) and refuses to start again: writer epochs of a retired id must never " - "repeat. To run a server on this pool again, configure a new for this disk and " - "keep the local uuid file. Do not clear the tombstone by hand.", - srid, *owner.retired_at_ms); + "CAS server-root '{}' was decommissioned (owner object '{}' retired at {} ms, owner life {}) and the " + "decommission is not finished: its retirement is not verified. Finish it with SYSTEM CAS DROP POOL MEMBER under " + "the rerun rule of the CAS docs (bringing a member back), then restart this server", + srid, owner_key, *owner.retired_at_ms, owner.owner_life); } } @@ -500,21 +501,27 @@ void deliverDeferredMountRenewObservability(uint64_t remount_attempt_no) noexcep deliverMountRenewObservability(completed, remount_attempt_no); } -bool serverRootSubtreeEmpty( +ServerRootLeftover serverRootLeftover( CasOperation & op, const Layout & l, const String & srid, const RefCatalog & catalog_observation) { const String owned_prefix = srid + "/"; for (const CatalogEntry & entry : catalog_observation.entries) if (entry.ns.string() == srid || entry.ns.string().starts_with(owned_prefix)) - return false; + return ServerRootLeftover::CatalogRows; /// Manifests and loose roots retain logical path identity. Opaque namespace stream/state debris /// alone is not evidence that this server root owns live work. if (prefixHasAnyKey(op, l.casManifestsServerPrefix(srid))) - return false; + return ServerRootLeftover::Manifests; if (prefixHasAnyKey(op, l.serverRootDataPrefix(srid))) - return false; - return true; + return ServerRootLeftover::RootsObjects; + return ServerRootLeftover::None; +} + +bool serverRootSubtreeEmpty( + CasOperation & op, const Layout & l, const String & srid, const RefCatalog & catalog_observation) +{ + return serverRootLeftover(op, l, srid, catalog_observation) == ServerRootLeftover::None; } std::optional readOwnerUuid(CasOperation & op, const Layout & l, const String & server_root_id) @@ -525,22 +532,150 @@ std::optional readOwnerUuid(CasOperation & op, const Layout & l, const return owner->server_uuid; } -void claimOwnerOrThrow( +namespace +{ +/// Every holder-originated mount body gets a random durable identity. UUIDv4 cannot be zero, but +/// keep the postcondition explicit because zero is reserved as an uninitialized in-memory value. +UInt128 newMountWriteAttemptId() +{ + const UInt128 id = UUIDHelpers::generateV4().toUnderType(); + chassert(id != UInt128{}); + return id; +} + +/// The key `IDisk::checkAccess` writes under the disk root, `clickhouse_access_check_`. +String accessProbeKey(const Layout & l, const String & srid, UInt128 server_uuid) +{ + const String hex = u128ToHex(server_uuid); + return fmt::format("{}clickhouse_access_check_{}-{}-{}-{}-{}", l.serverRootDataPrefix(srid), + hex.substr(0, 8), hex.substr(8, 4), hex.substr(12, 4), hex.substr(16, 4), hex.substr(20)); +} + +/// Take over the retired `owner` as `our_uuid` at its next life. nullopt: an `If-Match` lost to another +/// write, and the caller decides again from a fresh owner read. +std::optional rejoinRetiredOwner( + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, const OwnerObject & owner, + const Etag & owner_etag, const ObserveRefCatalog & observe_catalog) +{ + /// Checked before the retirement, so an id retired by 26.6.4 (no `epoch`, no `mount`) is never sent to a + /// rerun, which would refuse the absent counter. + const std::optional mount = op.read(l.mountKey(srid), Retry::standard()); + if (!mount || !op.head(l.epochKey(srid), Retry::standard())) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS server-root '{}' is retired but its {} object is missing: it was retired by an older version or the " + "object was deleted by hand, so it cannot rejoin. Configure a new for this disk", + srid, mount ? "epoch" : "mount"); + /// A rerun of an unfinished decommission drains whoever owns the id when it starts. + if (!owner.retirement_verified) + throwIfOwnerRetired(owner, srid, l.ownerKey(srid)); + const MountLease slot = decodeMountLease(mount->bytes); + if (!slot.gc_fenced && !slot.isFarewell()) + throw Exception(ErrorCodes::ABORTED, + "CAS server-root '{}' is retired and its mount slot is held (server_uuid={}, writer_epoch={}): a " + "decommission run is still running, or its lease has not yet lapsed and been fenced by GC. Restart this " + "server later", srid, u128ToHex(slot.server_uuid), slot.writer_epoch); + + /// A late `IDisk::checkAccess` of the old life can land after the drain; nothing else of it may remain. + const String probe = accessProbeKey(l, srid, owner.server_uuid); + if (const std::optional probe_meta = op.head(probe, Retry::standard())) + /// A lost `If-Match` leaves the probe in place, and the leftover check below refuses on it. + (void)op.remove(probe, probe_meta->etag, Retry::standard()); + switch (serverRootLeftover(op, l, srid, observe_catalog())) + { + case ServerRootLeftover::None: + break; + case ServerRootLeftover::CatalogRows: + throw Exception(ErrorCodes::ABORTED, + "CAS server-root '{}' is retired but data of its previous life remains: catalog rows of the old life are not " + "yet removed by GC. Run SYSTEM CAS GC RUN '' on the GC leader (is_leader = 1 in system.cas_mounts; " + "it can take several rounds) and restart this server", srid); + case ServerRootLeftover::Manifests: + throw Exception(ErrorCodes::ABORTED, + "CAS server-root '{}' is retired but data of its previous life remains: manifests under cas/manifests/{}/ are " + "not yet removed by GC. Run SYSTEM CAS GC RUN '' on the GC leader (is_leader = 1 in system.cas_mounts; " + "it can take several rounds) and restart this server", srid, srid); + case ServerRootLeftover::RootsObjects: + throw Exception(ErrorCodes::ABORTED, + "CAS server-root '{}' is retired but data of its previous life remains: an object under roots/{}/ appeared " + "after the drain. Delete it by hand and restart this server", srid, srid); + } + + /// A uuid other than the retired owner's replaces the terminal body before the owner write, so a 26.6.4 + /// starter of that owner's uuid loses its `If-Match` and has no claim recheck to fall back on. The owner's + /// own uuid writes nothing here: if its owner write then lost, its body would sit under another owner, + /// where such a starter could reclaim it. + if (our_uuid != owner.server_uuid) + { + MountLease ours = slot; + ours.server_uuid = our_uuid; + ours.hostname = getFQDNOrHostName(); + ours.pid = static_cast(::getpid()); + ours.write_attempt_id = newMountWriteAttemptId(); + ours.gc_fenced = false; + ours.min_active_build_sequence = std::numeric_limits::max(); + ours.seq += 1; + if (conflictOrThrow( + op.replace(l.mountKey(srid), encodeMountLease(ours), mount->etag, Retry::standard()), + fmt::format("CAS server-root '{}' rejoin mount write", srid))) + return std::nullopt; + } + + const uint64_t life = owner.owner_life + 1; + if (conflictOrThrow( + op.replace( + l.ownerKey(srid), + encodeOwner(OwnerObject{.server_uuid = our_uuid, .retired_at_ms = std::nullopt, .owner_life = life}), + owner_etag, Retry::standard()), + fmt::format("CAS server-root '{}' rejoin owner write", srid))) + return std::nullopt; + LOG_WARNING(getLogger("CasMountLease"), + "CAS server-root '{}': rejoined as an empty member under server_uuid {} at owner life {} (retired life {} of {})", + srid, u128ToHex(our_uuid), life, owner.owner_life, u128ToHex(owner.server_uuid)); + return life; +} +} + +uint64_t claimOwnerOrThrow( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, - const ObserveRefCatalog & observe_catalog) + const ObserveRefCatalog & observe_catalog, RetiredOwnerPolicy retired_policy) { if (!observe_catalog) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS server-root '{}': catalog observer is required", srid); const String key = l.ownerKey(srid); - /// Owner present → it is identity: equal UUID is ok, a different UUID fails closed regardless - /// of any lease/clock state. - if (const std::optional owner = readOwnerObject(op, l, srid)) + /// A lost rejoin write decides again from a fresh read; an owner or slot that changes on every decision fails + /// the start instead of spinning. + constexpr size_t max_owner_decisions = 4; + const bool may_rejoin = retired_policy == RetiredOwnerPolicy::RejoinIfVerified; + for (size_t decision = 1;; ++decision) { - if (owner->server_uuid == our_uuid) + const std::optional current = op.read(key, Retry::standard()); + if (!current) + break; + const OwnerObject owner = decodeOwner(current->bytes); + if (owner.retired_at_ms && may_rejoin) { - throwIfOwnerRetired(*owner, srid); - return; + if (our_uuid == UInt128{}) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS server-root '{}' is retired (owner tombstoned at {} ms, life {}) and this process has no " + "server uuid, so it does not rejoin it. Only a server start with a server uuid rejoins a retired " + "id: start the server that should own it, or configure a new for this disk", + srid, *owner.retired_at_ms, owner.owner_life); + if (const std::optional life = rejoinRetiredOwner(op, l, srid, our_uuid, owner, current->etag, observe_catalog)) + return *life; + if (decision == max_owner_decisions) + throw Exception(ErrorCodes::ABORTED, + "CAS server-root '{}': the owner or the mount slot kept changing during {} rejoin attempts; " + "restart to try again", srid, max_owner_decisions); + continue; + } + /// Owner present → it is identity: equal UUID is ok, a different UUID fails closed regardless + /// of any lease/clock state. + if (owner.server_uuid == our_uuid) + { + if (retired_policy != RetiredOwnerPolicy::AcceptForDecommission) + throwIfOwnerRetired(owner, srid, key); + return owner.owner_life; } /// The common cause is a regenerated local uuid file (wiped /var/lib/clickhouse, a pod recreated /// without a persistent volume) while the pool kept the old identity; name it and the supported @@ -556,7 +691,7 @@ void claimOwnerOrThrow( "'terminated' or 'fenced'. " "Deleting the owner object '{}' by hand is a last resort, only after verifying that NO server uses this " "root; it is refused while the subtree still holds data.", - srid, u128ToHex(owner->server_uuid), u128ToHex(our_uuid), srid, key); + srid, u128ToHex(owner.server_uuid), u128ToHex(our_uuid), srid, key); } /// Owner absent. Claiming is allowed ONLY over a provably-empty subtree; an absent owner over @@ -564,15 +699,16 @@ void claimOwnerOrThrow( if (!serverRootSubtreeEmpty(op, l, srid, observe_catalog())) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-root '{}' has no owner anchor but its data subtree is non-empty " - "(identity lost over existing data) — refusing to re-claim", + "(identity lost over existing data) — refusing to re-claim. Restoring the local uuid file does not help, " + "since no owner records a uuid to match. Configure a new for this disk", srid); const std::optional occupant = conflictOrThrow( - op.create(key, encodeOwner(OwnerObject{.server_uuid = our_uuid, .retired_at_ms = std::nullopt}), + op.create(key, encodeOwner(OwnerObject{.server_uuid = our_uuid, .retired_at_ms = std::nullopt, .owner_life = 1}), Retry::standard()), fmt::format("CAS server-root '{}' owner claim", srid)); if (!occupant) - return; + return 1; /// The conditional create conflicted. Recompute the whole catalog + manifest + roots bundle; /// no stale emptiness result is carried across the conflict. @@ -593,8 +729,9 @@ void claimOwnerOrThrow( "CAS server-root '{}' owner anchor vanished during claim", srid); if (reread->server_uuid == our_uuid) { - throwIfOwnerRetired(*reread, srid); - return; + if (retired_policy != RetiredOwnerPolicy::AcceptForDecommission) + throwIfOwnerRetired(*reread, srid, key); + return reread->owner_life; } throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-root '{}' was claimed by a different server during our claim (foreign owner) " @@ -603,7 +740,7 @@ void claimOwnerOrThrow( } uint64_t allocateWriterEpoch( - CasOperation & op, const Layout & l, const String & srid, EpochMintPolicy policy, uint64_t now_ms, + CasOperation & op, const Layout & l, const String & srid, const ObserveRefCatalog & observe_catalog) { if (!observe_catalog) @@ -653,40 +790,12 @@ uint64_t allocateWriterEpoch( break; /// authoritative absence — fresh-root bootstrap proceeds below case ProbeOutcome::Present: { - if (policy == EpochMintPolicy::DecommissionRecovery) - { - chassert(now_ms != 0); /// the decommission caller must pass its clock - const MountLease surviving = decodeMountLease(*mount_probe.body); - /// Deliberately weaker than claimMount's reclaim gate, which never trusts a - /// bare wall-clock comparison alone (only gc_fenced / the clean-farewell - /// min_active_build_sequence==UINT64_MAX marker / a caller-proven-dead - /// incarnation justify a reclaim there, because clock skew can misjudge - /// liveness). This is still safe: (a) the mint below is DISTINCT from the - /// survivor's epoch by construction, so no same-(uuid, epoch) pair is ever - /// representable even if this liveness read is wrong; (b) claimMount right - /// after this still applies its own STRONG liveness gate and refuses a - /// genuinely live member regardless of what happens here. So a clock-skewed - /// "terminal" misread can only burn one epoch number on a doomed - /// decommission attempt that aborts at claimMount — it can never admit a - /// claim over a live member. - const bool live = !surviving.gc_fenced && surviving.expires_at_ms > now_ms; - if (live) - throw Exception(ErrorCodes::ABORTED, - "CAS decommission '{}': epoch object missing but a LIVE mount lease " - "exists ({}) — refusing to re-mint an epoch under a live member " - "(stop the server or wait for its lease to lapse)", - srid, describeMountHolder(surviving)); - /// Terminal mount: proceed, but mint an epoch DISTINCT from the survivor's - /// by construction — the same-pair state is unrepresentable on this path. - current.next_writer_epoch = std::max(1, surviving.writer_epoch + 1); - break; - } throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-root '{}' has no durable epoch object but a mount lease exists — " "durable epoch state was lost while a mount is live or recently live; " - "refusing to re-mint epoch 1. If no server is live on this root, " - "decommission it or manually remove the stale mount object '{}'.", - srid, l.mountKey(srid)); + "refusing to re-mint epoch 1. If no server uses this root, start with a new " + "server_root_id; a decommission never recreates a counter", + srid); } case ProbeOutcome::ContainerAbsent: case ProbeOutcome::AccessDenied: @@ -702,7 +811,8 @@ uint64_t allocateWriterEpoch( } allocated = current.next_writer_epoch; - return encodeServerEpoch(ServerEpoch{.next_writer_epoch = allocated + 1}); + return encodeServerEpoch( + ServerEpoch{.next_writer_epoch = allocated + 1, .allocation_nonce = UUIDHelpers::generateV4().toUnderType()}); }, Retry::standard()); @@ -716,15 +826,6 @@ uint64_t allocateWriterEpoch( namespace { -/// Every holder-originated mount body gets a random durable identity. UUIDv4 cannot be zero, but -/// keep the postcondition explicit because zero is reserved as an uninitialized in-memory value. -UInt128 newMountWriteAttemptId() -{ - const UInt128 id = UUIDHelpers::generateV4().toUnderType(); - chassert(id != UInt128{}); - return id; -} - /// Build a fresh mount-lease body for (uuid, epoch) with the given seq, stamped from `now_ms`. MountLease makeMountBody(UInt128 uuid, uint64_t epoch, uint64_t seq, uint64_t now_ms, uint64_t ttl_ms) { @@ -789,12 +890,35 @@ void emitMountEvent(const CasEventSink & sink, CasEventType type, const String & /// allocate or recurse through the same diagnostic machinery at a protocol-critical site. } } + +/// Why `owner` is not the one `expected` names; nullopt when it is. +std::optional ownerRefusal(const std::optional & owner, const String & srid, const OwnerExpectation & expected) +{ + if (!owner) + return fmt::format("CAS server-root '{}': the owner anchor vanished after the mount claim", srid); + if (owner->server_uuid != expected.server_uuid) + return fmt::format( + "CAS server-root '{}' is owned by a different server since this claim began (owner server_uuid={}, " + "ours={}); the mount slot was handed to another server and this claim is withdrawn", + srid, u128ToHex(owner->server_uuid), u128ToHex(expected.server_uuid)); + if (owner->retired_at_ms && expected.retired != RetiredOwnerPolicy::AcceptForDecommission) + return fmt::format( + "CAS server-root '{}' was decommissioned (tombstoned at {} ms) while this claim was in progress; " + "refusing to run under a retired owner", srid, *owner->retired_at_ms); + if (owner->owner_life != expected.owner_life) + return fmt::format( + "CAS server-root '{}': the owner moved from life {} to life {} since this process started (a " + "retirement, a rollback or a rejoin happened in between); restart to continue under the current life", + srid, expected.owner_life, owner->owner_life); + return std::nullopt; +} } MountClaimResult claimMount( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation, - const CasEventSink & sink, const std::optional & unsafe_reclaim_authorization) + const CasEventSink & sink, const std::optional & unsafe_reclaim_authorization, + const std::optional & foreign_reclaim_owner) { const String key = l.mountKey(srid); const auto got = op.read(key, Retry::standard()); @@ -815,13 +939,21 @@ MountClaimResult claimMount( const MountLease existing = decodeMountLease(got->bytes); - /// Foreign owner → fail closed regardless of expiry. (This runs after the owner gate, so a foreign - /// mount should not normally exist, but the lease must never be taken across UUIDs.) + const bool clean_marker = existing.isFarewell(); + + /// A live body of another uuid is never taken. A terminal one below our epoch is what a rejoin under a new + /// uuid finds; it is reclaimed only once the owner reads as this claimer's own life. The owner is read before + /// the write, not only rechecked after it, so a reclaim never lands under somebody else's owner. if (existing.server_uuid != our_uuid) { - emitMountEvent(sink, CasEventType::MountConflict, srid, "foreign_owner", &existing, - "mount slot is held by a foreign server_uuid — refusing to take over across identities"); - return {.kind = MountClaimResult::ForeignOwner, .body = existing, .etag = std::nullopt}; + const bool reclaimable = (existing.gc_fenced || clean_marker) && existing.writer_epoch < our_epoch + && foreign_reclaim_owner && !ownerRefusal(readOwnerObject(op, l, srid), srid, *foreign_reclaim_owner); + if (!reclaimable) + { + emitMountEvent(sink, CasEventType::MountConflict, srid, "foreign_owner", &existing, + "mount slot is held by a foreign server_uuid — refusing to take over across identities"); + return {.kind = MountClaimResult::ForeignOwner, .body = existing, .etag = std::nullopt}; + } } /// An older allocation that reached the slot late must not replace a newer incarnation of this @@ -875,7 +1007,6 @@ MountClaimResult claimMount( /// clean-marked, not (yet) proven-dead, not unsafe-authorized lease may simply be a live twin, and /// `expires_at_ms` alone can never distinguish that from a dead predecessor across two different /// clocks. - const bool clean_marker = existing.min_active_build_sequence == std::numeric_limits::max(); const bool proven_dead = proven_dead_incarnation && *proven_dead_incarnation == got->etag; const bool unsafe_authorized = unsafe_reclaim_authorization && *unsafe_reclaim_authorization == got->etag; if (existing.gc_fenced || clean_marker || proven_dead || unsafe_authorized) @@ -893,7 +1024,8 @@ MountClaimResult claimMount( : proven_dead ? MountPriorState::UncleanObserved : MountPriorState::UncleanUnsafe; emitMountEvent(sink, CasEventType::MountClaim, srid, "reclaim", &existing, - existing.gc_fenced ? "same server_uuid, different writer_epoch, GC-fenced — reclaimed" + existing.server_uuid != our_uuid ? "terminal body of another server_uuid, the owner is ours — reclaimed" + : existing.gc_fenced ? "same server_uuid, different writer_epoch, GC-fenced — reclaimed" : clean_marker ? "same server_uuid, different writer_epoch, clean farewell — reclaimed" : proven_dead ? "same server_uuid, different writer_epoch, observed dead by " "token-stability observation — reclaimed" @@ -912,6 +1044,55 @@ MountClaimResult claimMount( return {.kind = MountClaimResult::LiveDoubleStart, .body = existing, .etag = got->etag}; } +namespace +{ +/// Rewrite the claimer's own live body `gc_fenced`, preserving everything else as the GC fence does. Only +/// `(our_uuid, our_epoch)`, not yet fenced or farewelled, is touched; anything else in the slot is left alone. +void fenceOwnClaim(CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, + const CasEventSink & sink) +{ + const String key = l.mountKey(srid); + WriteResult fenced = op.readModifyWrite(key, + [&](const std::optional & observed) -> std::optional + { + if (!observed) + return std::nullopt; + const MountLease body = decodeMountLease(observed->bytes); + if (!body.isHeldBy(our_uuid, our_epoch) || body.gc_fenced || body.isFarewell()) + return std::nullopt; + return encodeMountLease(body.fencedCopy()); + }, + Retry::standard()); + if (std::holds_alternative(fenced)) + emitMountEvent(sink, CasEventType::MountRelease, srid, "self_fenced", nullptr, + "the owner recheck refused this claim; the slot is fenced so a successor need not wait"); + else if (!std::holds_alternative(fenced)) + LOG_WARNING(getLogger("CasMountLease"), + "CAS server-root '{}': the refused claim's mount body could not be fenced; its lease expires on its own", srid); +} +} + +void recheckOwnerAfterClaim( + CasOperation & op, const Layout & l, const String & srid, const OwnerExpectation & expected, + uint64_t our_epoch, const CasEventSink & sink) +{ + const std::optional refusal = ownerRefusal(readOwnerObject(op, l, srid), srid, expected); + if (!refusal) + return; + + /// The fence only spares a successor the lease wait; its failure must not replace the owner's reason. + try + { + fenceOwnClaim(op, l, srid, expected.server_uuid, our_epoch, sink); + } + catch (...) + { + tryLogCurrentException(getLogger("CasMountLease"), + fmt::format("CAS server-root '{}': the refused claim's mount body could not be fenced", srid)); + } + throw Exception(ErrorCodes::CORRUPTED_DATA, "{}", *refusal); +} + String mountDoubleStartMessage(const String & srid, const std::optional & existing) { const String identity = existing @@ -968,7 +1149,8 @@ MountClaimResult claimMountAwaitingExpiry( uint64_t ttl_ms, uint64_t poll_interval_ms, const std::function & sleep_ms_fn, const std::function & on_wait_start, - const CasEventSink & sink) + const CasEventSink & sink, + const std::optional & foreign_reclaim_owner) { /// A zero poll interval would spin; a single-ms floor keeps the loop a real (bounded) wait. const uint64_t poll = poll_interval_ms == 0 ? 1 : poll_interval_ms; @@ -985,16 +1167,16 @@ MountClaimResult claimMountAwaitingExpiry( { const bool threshold_met = observed && mono_ms_fn() - observed_since >= threshold_ms; MountClaimResult r = claimMount(op, l, srid, our_uuid, our_epoch, now_ms_fn(), ttl_ms, - threshold_met ? observed : std::nullopt, sink, /*unsafe_reclaim_authorization=*/{}); + threshold_met ? observed : std::nullopt, sink, /*unsafe_reclaim_authorization=*/{}, foreign_reclaim_owner); if (r.kind != MountClaimResult::LiveDoubleStart) return r; /// Observation certifies only an older epoch dead; a body of our own pair is never replaced, /// so waiting on it could not end. - if (r.body && r.body->server_uuid == our_uuid && r.body->writer_epoch == our_epoch) + if (r.body && r.body->isHeldBy(our_uuid, our_epoch)) return r; /// A raced write observed a newer epoch of ours; the re-read refuses it with the newer-epoch error. if (r.body && r.body->server_uuid == our_uuid && r.body->writer_epoch > our_epoch) - return claimMount(op, l, srid, our_uuid, our_epoch, now_ms_fn(), ttl_ms, std::nullopt, sink, {}); + return claimMount(op, l, srid, our_uuid, our_epoch, now_ms_fn(), ttl_ms, std::nullopt, sink, {}, foreign_reclaim_owner); /// `claimMount` already read the current body, and a raced write reports whatever its own /// resolve read observed. Reuse `r.etag` whenever it is set instead of re-reading the SAME key @@ -1007,14 +1189,9 @@ MountClaimResult claimMountAwaitingExpiry( const auto got = op.read(l.mountKey(srid), Retry::standard()); if (!got) { - /// The slot vanished between claimMount's own GET and ours — normally self-resolving - /// within one more `claimMount` call (which re-mints fresh on an absent slot), but under - /// slot churn (something else concurrently removing/re-minting it) that resolution could - /// keep losing the same race. Pace this like every other iteration and - /// count it toward the SAME bounded restart budget the incarnation-churn case below - /// uses, instead of spinning read/claim/write at backend RTT with no sleep and no bound - /// — a persistently vanishing slot is exactly as "alive and contended" as a persistently - /// renewing holder. + /// The slot vanished between `claimMount`'s read and ours (a 26.6.4 decommission or a hand delete): + /// the next `claimMount` re-mints it. Paced and counted like a renewing holder, so a slot that + /// keeps vanishing cannot spin read, claim and write at backend RTT. if (++restarts > kMaxObservationRestarts) return r; sleep_ms_fn(poll); @@ -1079,15 +1256,7 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 /// Taken before any read: it can confirm that a token held, never date a sighting. const uint64_t round_start_ms = mono_ms_fn(); - /// `obs` is keyed by every srid this leader has EVER observed, but a - /// srid removed from the LIST entirely (its `/mount` key gone -- e.g. `SYSTEM CAS - /// DROP POOL MEMBER`) is never visited by the walk below again, so its entry would otherwise linger - /// forever (~150-250 B/srid, worse on a long-lived leader across many decommissions). Track every - /// srid actually seen THIS pass and prune anything else out of `obs` at the end -- disjoint from the - /// mid-walk `obs.erase(srid)` calls below (those fire for a srid seen but now terminal/fenced/gone - /// this pass; this is for a srid not seen AT ALL). std::set seen_srids; - const String prefix = l.serverRootsPrefix(); op.forEachListedKey(prefix, [&](const ListedKey & listed) { @@ -1126,7 +1295,7 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 obs.erase(srid); /// terminal — no further observation needed return std::nullopt; } - if (m.min_active_build_sequence == std::numeric_limits::max()) + if (m.isFarewell()) { ++floor.terminated; obs.erase(srid); /// terminal — no further observation needed @@ -1156,10 +1325,7 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 /// Stable past the threshold, not yet fenced → fence-out preserving the whole body /// (gc_fenced = true, seq + 1) against the incarnation this decision observed. - MountLease fenced = m; - fenced.gc_fenced = true; - fenced.seq = m.seq + 1; - return encodeMountLease(fenced); + return encodeMountLease(m.fencedCopy()); }, Retry::standard()); @@ -1183,9 +1349,9 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 return true; }, Retry::standard()); - /// Prune every `obs` and `lease_mismatch_warned` entry for a srid this pass's walk never saw at all. - for (auto it = obs.begin(); it != obs.end(); ) - it = seen_srids.contains(it->first) ? std::next(it) : obs.erase(it); + /// A slot deleted by a 26.6.4 decommission or by hand is never listed again; without this its srid would + /// stay in both maps forever on a long-lived leader. + std::erase_if(obs, [&](const auto & entry) { return !seen_srids.contains(entry.first); }); std::erase_if(lease_mismatch_warned, [&](const String & s) { return !seen_srids.contains(s); }); return floor; @@ -1228,7 +1394,7 @@ std::vector probeNonTerminalMountSlots(CasOperation & op, return true; } - if (m.gc_fenced || m.min_active_build_sequence == std::numeric_limits::max()) + if (m.gc_fenced || m.isFarewell()) return true; /// terminal: fenced out by GC, or the holder's own graceful farewell. slots.push_back(NonTerminalMountSlot{srid, fmt::format( @@ -1270,12 +1436,33 @@ std::vector listMounts(CasOperation & op, const Layout & layout, uint } if (info.lease.gc_fenced) info.state = "fenced"; - else if (info.lease.min_active_build_sequence == std::numeric_limits::max()) + else if (info.lease.isFarewell()) info.state = "terminated"; else if (now_ms <= info.lease.expires_at_ms + skew_margin_ms) info.state = "live"; else info.state = "expired"; + /// A decommission keeps the slot; the owner says whether the member behind a terminal row is retired, and + /// whether that retirement is verified: a rerun is for an unverified one only. + if (info.state == "terminated" || info.state == "fenced") + { + if (const auto owner_got = op.read(layout.ownerKey(info.srid), Retry::standard())) + { + try + { + const OwnerObject owner = decodeOwner(owner_got->bytes); + if (owner.retired_at_ms) + { + info.state = "retired"; + info.retirement_verified = owner.retirement_verified; + } + } + catch (...) + { + info.state = "corrupt"; + } + } + } out.push_back(std::move(info)); return true; }, Retry::standard()); @@ -1300,7 +1487,7 @@ FenceCertificate classifyFenceCertificate(const MountLease & lease, uint64_t fen { if (lease.gc_fenced) return FenceCertificate::GcFenced; - if (lease.min_active_build_sequence == std::numeric_limits::max()) + if (lease.isFarewell()) return FenceCertificate::CleanFarewell; if (lease.writer_epoch != fence_writer_epoch) return FenceCertificate::SupersededEpoch; @@ -1678,12 +1865,6 @@ MountRenewResult MountLeaseRenewer::renewOn( static_cast(renewer_state)); const auto boot_clock = environment.boot_ms ? environment.boot_ms : boot_ms_fn; - /// Sampled BEFORE each engine call. A refused admission is reported as "never attempted" only when - /// this node had already been asked to stop, and reading the flag afterwards could not tell that - /// apart from a flag the refusal itself set. - const auto sample_cancelled = [&] { return environment.cancelled && environment.cancelled(); }; - bool cancelled = sample_cancelled(); - const uint64_t wall_ms = now_ms_fn(); const uint64_t attempt_start_boot_ms = boot_clock(); const uint64_t next_seq = seq + 1; @@ -1760,7 +1941,6 @@ MountRenewResult MountLeaseRenewer::renewOn( break; } op.pause(pause_ms); - cancelled = sample_cancelled(); written = op.replace(key, body, expected, policy); } } @@ -1824,9 +2004,12 @@ MountRenewResult MountLeaseRenewer::renewOn( if (gave_up->why == GaveUp::Why::Deadline) result.deadline_source = gave_up->deadline_source; + /// Read after the refusal, because a stop that lands mid-renewal is what refused it. + const bool cancelled = environment.cancelled && environment.cancelled(); + /// The call that gave up sent nothing, every earlier call of a series ended in a refusal or in a - /// read that found the slot as our last commit left it, and the node was already stopping: the - /// renewal never ran and the farewell may still hand the slot back. + /// read that found the slot as our last commit left it, and the node is stopping: the renewal + /// never ran and the farewell may still hand the slot back. if (gave_up->why == GaveUp::Why::FenceLost && !gave_up->sent_any && cancelled) { markMountRenewTermination(MountRenewTerminalClassification::Cancelled); @@ -1838,8 +2021,7 @@ MountRenewResult MountLeaseRenewer::renewOn( switch (gave_up->why) { case GaveUp::Why::FenceLost: - /// Sampled again: a stop that arrived during the call still names why it ended. - classification = sample_cancelled() + classification = cancelled ? MountRenewTerminalClassification::Cancelled : MountRenewTerminalClassification::FenceOrLifecycleLost; break; @@ -1956,42 +2138,40 @@ void MountLeaseRenewer::release() terminate(op); } -void sweepOwnMountStaging(IObjectStorage & object_storage, const String & mount_staging_prefix) noexcept +void sweepOwnMountStaging(IObjectStorage & object_storage, const String & mount_staging_prefix, uint64_t live_writer_epoch) noexcept { try { - /// max_keys=0 asks `listObjects` for the FULL listing under the prefix (it paginates until - /// exhausted rather than capping at some default page size) — see `IObjectStorage::listObjects`. - /// A mount's own staging debris is bounded (one mount's in-flight + leaked uploads), so a single - /// unbounded LIST at startup is acceptable; unlike GC's per-round budgets, this runs once per - /// mount, not on a recurring schedule. + /// `max_keys=0` lists the whole prefix. One mount's staging debris is bounded and this runs once per + /// mount, not on GC's recurring schedule, so an unbounded LIST is acceptable here. RelativePathsWithMetadata children; object_storage.listObjects(mount_staging_prefix, children, /*max_keys=*/0); - size_t removed = 0; for (const auto & child : children) { + const String & key = child->relative_path; + if (!key.starts_with(mount_staging_prefix)) + continue; + const std::optional epoch = Layout::stagedEpochOf(std::string_view(key).substr(mount_staging_prefix.size())); + if (!epoch || *epoch >= live_writer_epoch) + continue; /// the live epoch's own in-flight uploads, or a nested member's keys try { - object_storage.removeObjectIfExists(StoredObject(child->relative_path)); + object_storage.removeObjectIfExists(StoredObject(key)); ++removed; } catch (...) // NOLINT(bugprone-empty-catch) { - /// Best-effort: one stubborn key must not abort the sweep of the rest — it is retried - /// by a later mount's sweep. + /// One stubborn key must not abort the sweep; a later mount retries it. } } - if (removed) - LOG_INFO(getLogger("CasStagingSweeper"), - "Reclaimed {} leaked S3 staging object(s) under '{}' at mount start", - removed, mount_staging_prefix); + LOG_INFO(getLogger("CasStagingSweeper"), "Reclaimed {} staging object(s) below writer epoch {} under '{}' at mount start", + removed, live_writer_epoch, mount_staging_prefix); } catch (...) // NOLINT(bugprone-empty-catch) { - /// Best-effort: a LIST failure (a transient backend hiccup) at mount time must never fail the - /// mount — any leaked staging objects are bounded debris, reclaimed by a later mount's sweep. + /// A LIST failure at mount time must never fail the mount; the debris is bounded and retried later. } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index 09c6307fcbc3e..88509bedbcfe9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -68,9 +68,8 @@ struct MountRenewOperationEnvironment /// Facts the mount fence cannot see (park requested, pool no longer live, shutdown). FALSE ends /// the renewal exactly as a lost fence does; the engine does not need to know which refused. std::function live; - /// Sampled before each engine call. A call refused before its first attempt ends the renewal as - /// `NotAttempted` rather than terminal only when this node had already been asked to stop -- - /// sampling it afterwards would read a flag that the refusal itself may have set. + /// Read once the write has given up. A renewal refused before its first attempt is reported as + /// `NotAttempted` rather than terminal when this node has been asked to stop by then. std::function cancelled; RequestObserver requests; /// The background renewal's wait after the store refused a request: the store said it cannot serve @@ -129,6 +128,18 @@ class Layout; bool serverRootSubtreeEmpty( CasOperation & op, const Layout & l, const String & srid, const RefCatalog & catalog_observation); +/// The first part of a server root's subtree that is not empty, checked in this order. +enum class ServerRootLeftover +{ + None, + CatalogRows, + Manifests, + RootsObjects, +}; + +ServerRootLeftover serverRootLeftover( + CasOperation & op, const Layout & l, const String & srid, const RefCatalog & catalog_observation); + /// Supplied by the pool layer so the low-level server-root protocol always observes the mandatory /// catalog. Every absent-control retry obtains a fresh, successfully decoded observation. using ObserveRefCatalog = std::function; @@ -139,27 +150,49 @@ using ObserveRefCatalog = std::function; /// counterpart used by normal opens and reuses this GET+decode path. std::optional readOwnerUuid(CasOperation & op, const Layout & l, const String & server_root_id); -/// Claim (or validate) the sticky owner anchor that binds `srid` to a server UUID (identity). -/// - owner present, equal `our_uuid`, and not tombstoned → ok (return); -/// - owner present and tombstoned → throw `CORRUPTED_DATA` (explicitly retired — fail closed); -/// - owner present, different → throw `CORRUPTED_DATA` (foreign owner — fail closed); -/// - owner absent BUT the subtree is non-empty → throw `CORRUPTED_DATA` (identity lost over -/// existing data — never silently re-claim); -/// - owner absent, subtree provably empty → `create` the owner (claim). -/// The owner object is never deleted and never reassigned to a different UUID. -void claimOwnerOrThrow( +enum class RetiredOwnerPolicy : uint8_t +{ + Refuse, /// a remount, and every check after the owner claim, never accept a tombstoned owner + AcceptForDecommission, /// the decommission admin impersonates a retired victim to drain or rerun + RejoinIfVerified, /// a server start takes over a verified, drained retirement as an empty member +}; + +/// Claim or validate the owner anchor that binds `srid` to a server uuid, and return the owner life. +/// - present, equal `our_uuid`, not tombstoned → the stored life; +/// - present and tombstoned, under `RejoinIfVerified` (a server start), any non-zero uuid → rejoin as an +/// empty member: the owner becomes `{our_uuid, owner_life + 1}` and that life is returned. Refused with +/// `CORRUPTED_DATA` (zero uuid, `epoch` or `mount` missing, retirement not verified) or `ABORTED` (slot held, +/// data of the old life left, owner or slot changing on every decision); +/// - present, equal `our_uuid`, tombstoned → `CORRUPTED_DATA` under `Refuse`; the stored life under +/// `AcceptForDecommission`; +/// - present, different uuid, outside a rejoin → `CORRUPTED_DATA` (foreign owner), tombstoned or not; +/// - absent, subtree provably empty → create `{our_uuid, owner_life = 1}` and return 1; +/// - absent, subtree non-empty → `CORRUPTED_DATA` (identity lost over data). +/// The returned life is 0 for a body written before the field existed. +/// The owner object is never deleted; a rejoin is the one supported change of its uuid. +uint64_t claimOwnerOrThrow( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, - const ObserveRefCatalog & observe_catalog); + const ObserveRefCatalog & observe_catalog, RetiredOwnerPolicy retired_policy = RetiredOwnerPolicy::Refuse); -/// Which mint policy governs `allocateWriterEpoch`'s absent-epoch branch (see below). -enum class EpochMintPolicy : uint8_t +/// What a claimer requires of the owner: after its body is in the slot and before it installs the renewer, +/// and, for a foreign terminal body, before the reclaim write. +struct OwnerExpectation { - NormalMount, /// absent-epoch re-mint requires authoritative mount absence - DecommissionRecovery, /// absent-epoch re-mint requires a TERMINAL mount; mints a distinct epoch + UInt128 server_uuid; + uint64_t owner_life; + RetiredOwnerPolicy retired; }; +/// One `GET owner` after `claimMount` returned `Claimed`. Throws `CORRUPTED_DATA` unless the owner is exactly +/// `(expected.server_uuid, expected.owner_life)` and, under `Refuse`, not retired. Before throwing it rewrites +/// the claimer's own `(uuid, our_epoch)` body `gc_fenced`, so a successor need not wait the lease out; a +/// failed fence write is logged and the refusal is still thrown. +void recheckOwnerAfterClaim( + CasOperation & op, const Layout & l, const String & srid, const OwnerExpectation & expected, + uint64_t our_epoch, const CasEventSink & sink = {}); + /// Allocate the next durable-monotone `writer_epoch` by CAS-bumping the sticky `epoch` object -/// (`ServerEpoch{next_writer_epoch}`), returning the value the caller adopts as its writer_epoch. +/// (`ServerEpoch{next_writer_epoch, allocation_nonce}`), returning the value the caller adopts as its writer_epoch. /// - `epoch` absent AND the subtree is non-empty → throw `CORRUPTED_DATA` (missing epoch over /// data is a reset hazard); /// - `epoch` absent AND the subtree is empty → the absent-epoch branch is a LIFECYCLE decision, @@ -167,23 +200,21 @@ enum class EpochMintPolicy : uint8_t /// flattens transport faults into "not found"), to decide whether the mount object is really /// gone: /// - `KeyAbsent` → authoritative absence, mint epoch 1 (fresh-root bootstrap); -/// - `Present`, `policy == NormalMount` → throw `CORRUPTED_DATA` (durable epoch state was -/// lost while a mount is live or recently live — refusing to re-mint epoch 1 there is how a -/// same-(uuid, epoch) twin is avoided); -/// - `Present`, `policy == DecommissionRecovery` → the surviving mount must be TERMINAL (not -/// live); a live member throws `ABORTED`, otherwise mint `surviving.writer_epoch + 1` — -/// distinct from the survivor's by construction (`now_ms` is required, nonzero, here); +/// - `Present` → throw `CORRUPTED_DATA` (durable epoch state was lost while a mount is live or +/// recently live; refusing to re-mint epoch 1 is how a same-(uuid, epoch) twin is avoided. If no +/// server uses this root, start with a new server_root_id; a decommission never recreates a counter); /// - anything else (`ContainerAbsent`/`AccessDenied`/`Indeterminate`) → throw /// `CORRUPTED_DATA` (absence was never proven; fail closed); /// - otherwise read `next = current.next_writer_epoch`, write `{next + 1}` against the observed /// incarnation, re-deciding on conflict, and return `next`. +/// Every decision writes a fresh `allocation_nonce`, so an unclear write counts as ours only when the +/// stored bytes are exactly ours; a same-uuid twin bumping to the same value is a conflict. uint64_t allocateWriterEpoch(CasOperation & op, const Layout & l, const String & srid, - EpochMintPolicy policy, uint64_t now_ms, const ObserveRefCatalog & observe_catalog); /// Which certificate of death, or the operator's explicit unsafe authorization, justified a -/// same-uuid, older-epoch mount reclaim. `None` when no reclaim of that kind happened (a fresh -/// claim, `LiveDoubleStart`, `ForeignOwner`, `FencedSelf`). +/// older-epoch mount reclaim (`Fenced` and `Clean` also cover another uuid's terminal body). `None` when no +/// reclaim of that kind happened (a fresh claim, `LiveDoubleStart`, `ForeignOwner`, `FencedSelf`). enum class MountPriorState { None, @@ -222,7 +253,10 @@ enum class MountPriorState /// now_ms` ALONE is never sufficient — comparing a predecessor's stamp against OUR wall clock /// is unsafe because a clock-skewed or merely late-observing /// caller must never conclude death from a bare timestamp read); -/// - different `server_uuid` → `ForeignOwner` (do NOT write, regardless of expiry or prior state). +/// - different `server_uuid` → `ForeignOwner`, no write, unless the body is terminal (`gc_fenced` or the +/// clean marker), its `writer_epoch` is below `our_epoch`, and a fresh owner read matches +/// `foreign_reclaim_owner`: then it is reclaimed like an older epoch of our own (`prior = Fenced` or +/// `Clean`). A rejoin under a new uuid leaves such a body. struct MountClaimResult { /// Plain (unscoped) enum: callers compare with `MountClaimResult::Claimed` directly. @@ -284,7 +318,8 @@ class MountFencedException : public DB::Exception MountClaimResult claimMount( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation = {}, - const CasEventSink & sink = {}, const std::optional & unsafe_reclaim_authorization = {}); + const CasEventSink & sink = {}, const std::optional & unsafe_reclaim_authorization = {}, + const std::optional & foreign_reclaim_owner = {}); /// Format the operator-actionable startup error shown when the mount lease is held by a genuinely /// live second server (the same `server_root_id` is mounted twice). Produced only AFTER this server @@ -297,6 +332,8 @@ String mountDoubleStartMessage(const String & srid, const std::optional & sleep_ms_fn, const std::function & on_wait_start = {}, - const CasEventSink & sink = {}); + const CasEventSink & sink = {}, + const std::optional & foreign_reclaim_owner = {}); /// One `server_root_id`'s cross-round incarnation-stability observation, /// owned by the GC leader instance (`Cas::Gc::mount_obs`) and threaded through consecutive @@ -397,8 +435,8 @@ using MountObservationMap = std::map; /// leader instance, `Cas::Gc::mount_obs`) — a fresh leader starts with an empty map (safe: delays /// fencing one round, never fences early). /// A body whose declared lease is non-zero and differs from `observer_lease_ms` is named in one `WARNING` -/// per srid; `lease_mismatch_warned` holds the srids already named and is pruned like `obs`. The check -/// never changes a classification or a write. +/// per srid; `lease_mismatch_warned` holds the srids already named. The check never changes a classification +/// or a write. `obs` and `lease_mismatch_warned` drop every srid this pass did not list. /// /// The fence-out is BOTH safety and liveness. Safety: a sleeper's later renewal permanently fails /// (its `replace` now mismatches the fenced incarnation → `tripMountLost`), so it can never re-arm @@ -449,16 +487,20 @@ std::vector probeNonTerminalMountSlots(CasOperation & op, /// A read-only snapshot of one server's mount slot, for introspection (`system.cas_mounts`). /// state: `live` (lease within TTL+skew), `expired` (lease ran out; the next GC round's heartbeat floor /// will fence it), `terminated` (clean farewell: `min_active_build_sequence == UINT64_MAX`), `fenced` (`gc_fenced`), +/// `retired` (terminated or fenced, owner tombstoned; one more `GET` per terminated or fenced row), /// `corrupt` (body failed to decode — surfaced as a row, never an exception). struct MountInfo { String srid; MountLease lease; String state; + /// For a `retired` row: whether a decommission verified the retirement. A rerun is for an unverified one only. + bool retirement_verified = false; }; /// Enumerate every mount slot under `gc/server-roots/`, decoded and classified — the read-only sibling -/// of `computeHeartbeatFloor`: ZERO writes (no fence-out), per-row fail-open. One LIST + one GET per slot. +/// of `computeHeartbeatFloor`: ZERO writes (no fence-out); an undecodable mount body, or an undecodable owner of a +/// terminal row, is a `corrupt` row. One LIST, one GET per slot, and one more per terminated or fenced slot. std::vector listMounts(CasOperation & op, const Layout & layout, uint64_t now_ms, uint64_t skew_margin_ms); /// Whether the mounted writer identified by `(server_root_id, writer_epoch)` (the two fields of a @@ -620,31 +662,18 @@ class MountLeaseRenewer /// Mount-lease-scoped staging sweeper for objects left behind by S3-native staging. /// -/// A leaked S3 staging object is one whose transaction never ran `cleanupPendingTempFiles` (the process -/// died, or an exception hit between `sink->finalize()` and `on_finalized` registering the key in -/// `CaContentWriteBuffer::finalizeImpl`) or whose best-effort delete on commit or abort was lost. While -/// the member lives, this sweeper is its only reclaimer. Debris from either case is -/// bounded to `staging//` — the ONE mount that could ever have written under that prefix, since -/// every staging key this mount ever mints comes from `ContentAddressedMetadataStorage::stagingKeyPrefix()` -/// (`physicalKey(pool_prefix + "/staging/" + server_root_id)`), keyed by THIS mount's own `server_root_id`. -/// -/// LEASE-FENCE (fail-closed, never fail-open): `sweepOwnMountStaging` removes ONLY objects whose key -/// starts with the given `mount_staging_prefix` — pass your OWN mount's prefix, never another mount's. -/// The caller (`ContentAddressedMetadataStorage::startup()`) invokes this exactly once, at mount start, -/// with `stagingKeyPrefix() + "/"` — the SAME prefix construction the writer uses to mint staging keys, so -/// this sweep can never reach a different mount's `staging//` subtree: no other writer -/// ever stages a key under THIS mount's own `server_root_id` prefix, and this function never lists or -/// touches anything outside the prefix it is given. +/// Staging keys are `staging///.tmp`, minted from +/// `ContentAddressedMetadataStorage::stagingKeyPrefix()`. A key leaks when its transaction never ran +/// `cleanupPendingTempFiles` (the process died, or an exception hit between `sink->finalize()` and `on_finalized` +/// registering the key in `CaContentWriteBuffer::finalizeImpl`) or when its best-effort delete on commit or abort +/// was lost. While the member lives, this sweep is the only reclaimer. GC never lists `staging/`. /// -/// Best-effort and NEVER THROWS: one stubborn key (or a LIST failure) must never abort the sweep of the -/// rest, and must never fail the mount (mirrors `feedback_ca_gc_never_throw_on_404` — a throw here would -/// only wedge startup, not GC, but the same fail-open-on-error discipline applies to any best-effort -/// reclaim of debris). +/// Deletes `/` with `epoch < live_writer_epoch`, and flat +/// `` keys (epoch 0). The cutoff is the live epoch fixed at the start of the +/// sweep, so a successor's in-flight uploads (its epoch is not below the cutoff) survive. Every other path +/// is left alone, including another member's keys nested under this prefix. Pass your OWN mount's prefix. /// -/// GC excludes `staging/` entirely: GC blob discovery LISTs `Layout::blobsPrefix()` -/// (`/blobs/`) — a distinct top-level prefix from `staging/`, `cas/ns/`, and -/// `cas/manifests/` (see `CasLayout.h`) — so a `staging/` object is never listed, HEAD'd, or condemned by -/// GC's fold. This sweeper is the ONLY reclaimer of `staging/` debris. -void sweepOwnMountStaging(IObjectStorage & object_storage, const String & mount_staging_prefix) noexcept; +/// Best-effort and never throws: a LIST failure or one stubborn key must not fail the mount. +void sweepOwnMountStaging(IObjectStorage & object_storage, const String & mount_staging_prefix, uint64_t live_writer_epoch) noexcept; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h index 05b9b75a89679..2fe285d596c16 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h @@ -32,10 +32,9 @@ enum class CasEventType GateRevalidate, GateResurrect, WatermarkRenew, MountRemount, MountClaim, MountRelease, MountConflict, /// Operator-driven erasure of a dead pool member's namespace. `decommissionPoolMember` runs as - /// a writer, never as GC: it claims the member's mount, drains its namespaces and debris, and - /// deletes the mount slot only after the drain is confirmed. The slot is the interrupted-operation - /// resume anchor, so a failed drain leaves it terminated for a later retry. `outcome` is one of - /// "begin", "namespace_removed", or "end". + /// a writer, never as GC: it claims the member's mount, tombstones its owner, drains its namespaces + /// and debris, and keeps the owner, epoch and mount objects. `outcome` is one of "begin", + /// "namespace_removed", or "end". MemberDecommission, /// Incidental-detection reaction to foreign bytes at a ref-log wedge key owned by this mount. /// The mount lease makes the key exclusive, so this is impossible under legitimate diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index 19339638ecc59..dc85d9422bff2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -10,9 +10,9 @@ #include #include #include +#include #include #include -#include #include #include @@ -20,7 +20,9 @@ namespace DB { namespace ErrorCodes { + extern const int ABORTED; extern const int CORRUPTED_DATA; + extern const int LOGICAL_ERROR; } } @@ -52,11 +54,13 @@ std::string_view removalName(Removal r) /// fenced by the decommission claim (`Pool::openForDecommission`), so nothing should be racing these /// deletes, and a plain exact-incarnation delete of every listed object is race-free. /// +/// Runs on the admin's mount plane, so a fenced admin's deletes are refused locally; only attempts already +/// sent can land late. +/// /// A per-object failure — a backend exception, a `Mismatch` or `Gone` outcome, or an object /// disappearing between `LIST` and `HEAD` — is recorded as a warning and does not prevent the remaining -/// objects from being attempted. The caller keeps the pool slot whenever warnings are present, so the -/// terminated slot remains available as a resume anchor instead of being deleted after an unconfirmed -/// drain. Returns only the objects whose exact-incarnation delete was reported as `Removed`. +/// objects from being attempted while `op` is still admitted. Returns only the objects whose +/// exact-incarnation delete was reported as `Removed`. uint64_t deleteListedPrefix(CasOperation & op, const String & prefix, std::vector & warnings) { uint64_t deleted = 0; @@ -87,116 +91,254 @@ uint64_t deleteListedPrefix(CasOperation & op, const String & prefix, std::vecto { warnings.push_back("decommission drain: " + listed.key + " delete failed: " + getCurrentExceptionMessage(/*with_stacktrace=*/false)); + return op.admitted(); } return true; }, Retry::standard()); return deleted; } -/// Delete one slot control object by an incarnation captured at the protocol-defined fence point. Slot -/// retirement is fail-closed: unlike the debris drains above, any non-`Removed` outcome or exception -/// stops the tail before it can touch the next control object. -bool deleteSlotObject(CasOperation & op, const String & key, const Etag & etag, std::vector & warnings) +String whyNotWritten(const WriteResult & result) { - try + return std::visit(detail::Overload{ + [](const Committed &) -> String { return "committed"; }, + [](const Declined &) -> String { return "declined"; }, + [](const Conflict &) -> String { return "the object changed since it was read"; }, + [](const Refused & refused) -> String { return "the store refused it: " + refused.message; }, + [](const GaveUp & gave_up) -> String + { + return gave_up.sent_any ? "gave up after sending, it may have landed" : "gave up before sending"; + }}, result); +} + +/// A mark counts only on a retired owner: clearing `retired_at_ms` by hand to restart a member leaves the mark +/// of the earlier life behind. +bool holdsVerifiedRetirement(const OwnerObject & owner) +{ + return owner.retirement_verified && owner.retired_at_ms.has_value(); +} + +/// This run's tombstone: what a later exit needs to decide whether to keep it or roll it back. +struct Tombstone +{ + OwnerObject owner; /// the tombstone's body + bool previously_retired = false; /// the owner was already retired; a rollback never undoes that + Etag etag; /// the tombstone's incarnation, the rollback's precondition +}; + +void logWarnings(const String & srid, const std::vector & warnings) +{ + for (const String & warning : warnings) + LOG_WARNING(getLogger("CasDecommission"), "CAS decommission '{}': {}", srid, warning); +} + +/// Undo this run's tombstone after a lost claim: the next life, not retired, no mark. An earlier retirement +/// stays: its tombstone landed while its run held the claim, so no member runs under it. A rollback whose +/// precondition moved was overtaken by a later owner write, which then owns the outcome. +void rollBackTombstone(CasOperation & op, const Layout & layout, const String & srid, const Tombstone & tombstone, + std::vector & warnings) +{ + if (tombstone.previously_retired) { - const Removal outcome = op.remove(key, etag, Retry::standard()); - if (outcome == Removal::Removed) - return true; + warnings.push_back("the admin claim was lost after the tombstone; the owner keeps its earlier retirement"); + return; + } + const OwnerObject restored = nextOwnerLife(tombstone.owner, std::nullopt, false); + const WriteResult written = op.replace(layout.ownerKey(srid), encodeOwner(restored), tombstone.etag, Retry::standard()); + if (std::holds_alternative(written)) + warnings.push_back("the admin claim was lost after the tombstone; the tombstone was rolled back (owner life " + + std::to_string(restored.owner_life) + "); rerun to decommission"); + else if (std::holds_alternative(written)) + warnings.push_back("the admin claim was lost after the tombstone and the rollback did not land: the owner " + "changed after this run's tombstone; read the owner object '" + layout.ownerKey(srid) + + "' before rerunning"); + else + warnings.push_back("the admin claim was lost after the tombstone and the rollback did not complete (" + + whyNotWritten(written) + "); read the owner object '" + layout.ownerKey(srid) + + "' before rerunning"); +} - warnings.push_back("slot delete failed: " + key + ": delete outcome " - + String(removalName(outcome))); +/// Every exit after the tombstone and before the farewell: the tombstone stays while the claim holds. A failed +/// confirm is a warning, so an exit that already carries an error still propagates its own. +void confirmClaimOrRollBack(CasOperation & op, const Layout & layout, const String & srid, UInt128 victim_uuid, + uint64_t admin_epoch, const Tombstone & tombstone, std::vector & warnings) +{ + try + { + if (!adminClaimStillHeld(op, layout, srid, victim_uuid, admin_epoch)) + rollBackTombstone(op, layout, srid, tombstone, warnings); } catch (...) { - warnings.push_back("slot delete failed: " + key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); + warnings.push_back("could not confirm the admin claim: " + getCurrentExceptionMessage(/*with_stacktrace=*/false) + + "; read the owner object '" + layout.ownerKey(srid) + "' before rerunning"); } - return false; -} - } -DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, - const String & victim_srid, const CasEventSink & sink, - const std::function & request_gc_round, - const std::function & drain_now_fn, - const std::function & drain_sleep_fn) +/// Reads the owner on `control`, requires the victim's uuid, and replaces it on the admin's mount plane by the +/// next life, retired, keeping an earlier verified mark. The mount plane sends no attempt after the admin's +/// lease deadline, but the store may apply one later. A paused earlier run cannot roll this tombstone back: +/// its precondition is older. A write with an unknown outcome is still an exit after the tombstone when it +/// landed: if the owner reads back as exactly these bytes, the claim is confirmed (or the tombstone rolled +/// back) before the error propagates. Otherwise the admin withholds its clean farewell: the victim then runs +/// only after its slot is fenced or observed dead. +Tombstone writeTombstone(CasOperation & control, Pool & admin, const Layout & layout, const String & srid, + UInt128 victim_uuid, uint64_t admin_epoch, std::vector & warnings) { - DecommissionReport report; - report.srid = victim_srid; - bool gc_round_needed = false; - /// A namespace may have reached `Removing` before a later namespace fails closed. Preserve the - /// already-earned liveness signal on every exit: the callback only wakes the existing serialized - /// GC worker and cannot perform catalog work itself. - SCOPE_EXIT({ - if (gc_round_needed && request_gc_round) - request_gc_round(); - }); - - /// Decommission is administrative and non-hot-path: it has no mount-lease fence of its own -- the - /// exact-incarnation compare on every write is the safety mechanism -- so it opens its own - /// always-admitted request engine rather than one of `Pool`'s fenced planes. The pre-impersonation - /// cut below runs against the raw backend passed in, because `Pool::openForDecommission` has not - /// yet wrapped it for instrumentation and no `Pool` exists yet to route through. - CasRequests preflight_requests(backend, Fence::open()); - CasOperation preflight_op = preflight_requests.admit(); - - /// Validate one required immutable ownership cut before impersonating the victim. The admin open - /// performs its own fresh catalog observation for mount safety, but namespace selection below - /// must reuse this exact pre-mutation decision rather than read a later authority set. - const Layout catalog_layout(config.pool_prefix); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(preflight_op, catalog_layout); - catalog_cut.life_index.throwIfAmbiguous("CAS decommission"); - - /// `drain_now_fn`/`drain_sleep_fn` below replace the clock and sleep the STANDALONE `requests` - /// engine (opened further down) paces its own retries on, but `config.boot_ms_fn`/ - /// `config.retry_sleep_fn` -- the seams `Pool::openForDecommission` itself constructs its - /// mount/farewell/GC planes with -- are distinct. Left unset, those planes retry on the real boot - /// clock and a real sleep, and a caller that fakes only the standalone engine's clock ends up - /// comparing it against an unrelated one at the mount lease's farewell bound (or, worse, against a - /// clock that only the standalone engine's sleep advances: a retry loop on `mount_requests`/ - /// `gc_requests` bound to that frozen clock while actually sleeping for real would never see its - /// own deadline elapse). Fold BOTH into `config` here, together, unless the caller already asked - /// for a specific clock or sleep of its own -- installing only one of the two is exactly the - /// half-fix that leaves the other seam retrying forever. - if (drain_now_fn && drain_sleep_fn) + CasOperation fenced = admin.mountRequests().admit(); + const String key = layout.ownerKey(srid); + const std::optional current = control.read(key, Retry::standard()); + if (!current) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS decommission '{}': the owner anchor vanished after the admin claim", srid); + const OwnerObject read = decodeOwner(current->bytes); + if (read.server_uuid != victim_uuid) + throw Exception(ErrorCodes::ABORTED, + "CAS decommission '{}': the owner is now server_uuid={}, the claim was made for {}; another server " + "took this server_root_id since the claim, nothing was removed", + srid, u128ToHex(read.server_uuid), u128ToHex(victim_uuid)); + + const bool previously_retired = read.retired_at_ms.has_value(); + const OwnerObject owner = nextOwnerLife(read, nowMs(), holdsVerifiedRetirement(read)); + const String bytes = encodeOwner(owner); + const String what = fmt::format("CAS decommission '{}' owner tombstone", srid); + + std::optional written; + std::exception_ptr unknown_outcome; + try { - if (!config.boot_ms_fn) - config.boot_ms_fn = drain_now_fn; - if (!config.retry_sleep_fn) - config.retry_sleep_fn = drain_sleep_fn; + written = fenced.replace(key, bytes, current->etag, Retry::standard()); + } + catch (...) + { + /// The engine rethrows a local failure even after an attempt was sent. + unknown_outcome = std::current_exception(); + } + if (written) + { + if (const Committed * committed = std::get_if(&*written)) + return Tombstone{.owner = owner, .previously_retired = previously_retired, .etag = committed->etag}; + if (std::holds_alternative(*written)) + throw Exception(ErrorCodes::ABORTED, + "CAS decommission '{}': the owner was rewritten between the claim and the tombstone (a rerun or a " + "rejoin); nothing was removed, rerun to decide against the current owner", srid); + const GaveUp * gave_up = std::get_if(&*written); + if (!gave_up || !gave_up->sent_any) + { + orThrow(std::move(*written), what); + throw Exception(ErrorCodes::LOGICAL_ERROR, "{}: a conditional replace was declined", what); + } + try + { + orThrow(std::move(*written), what); + } + catch (...) + { + unknown_outcome = std::current_exception(); + } } - config.event_sink = sink; - PoolPtr admin = Pool::openForDecommission(std::move(backend), std::move(config), victim_srid); - if (drain_now_fn && drain_sleep_fn) + std::optional landed; + try { - /// Re-affirms the same values `config` above already installed on `mount_requests`/ - /// `lease_requests`/`gc_requests` at construction, and additionally wires `ref_ledger`'s own - /// retry sleep, which has no construction-time seam of its own. `sweepNamespace` below issues - /// its deletes on `admin`'s own GC plane. - admin->setCasRequestNowFnForTest(drain_now_fn); - admin->setCasRetrySleepForTest(drain_sleep_fn); + landed = control.read(key, Retry::standard()); } + catch (...) + { + warnings.push_back("could not read the owner back after the tombstone write: " + + getCurrentExceptionMessage(/*with_stacktrace=*/false)); + } + if (landed && landed->bytes == bytes) + { + const Tombstone tombstone{.owner = owner, .previously_retired = previously_retired, .etag = landed->etag}; + confirmClaimOrRollBack(control, layout, srid, victim_uuid, admin_epoch, tombstone, warnings); + } + else + admin.withholdFarewell(); + logWarnings(srid, warnings); + std::rethrow_exception(unknown_outcome); +} - /// A second engine over the pool's own (now instrumented) backend: `CasRequests` keeps its own - /// shared_ptr to it, so `op` stays usable after `admin.reset()` retires the `Pool` below. - CasRequests requests(admin->poolBackendPtr(), Fence::open()); - if (drain_now_fn && drain_sleep_fn) +/// Whether the owner carries a verified retirement once this run can no longer act. Otherwise says why not: +/// an unretired owner past this run's tombstone moved on (a rejoin, or another run's rollback), and a rerun +/// there could drain a rejoined member; anything else, a later run's unverified tombstone included, means +/// rerun. +bool ownerHoldsVerifiedRetirement(CasOperation & op, const Layout & layout, const String & srid, const Tombstone & tombstone, + std::vector & warnings) +{ + try { - requests.setNowFnForTest(drain_now_fn); - requests.setSleepFnForTest(drain_sleep_fn); + if (const std::optional current = op.read(layout.ownerKey(srid), Retry::standard())) + { + const OwnerObject owner = decodeOwner(current->bytes); + if (holdsVerifiedRetirement(owner)) + return true; + if (!owner.retired_at_ms && owner.owner_life > tombstone.owner.owner_life) + { + warnings.push_back(fmt::format( + "the owner moved on to life {} after this run's tombstone (a rejoin or another run's rollback); " + "read system.cas_mounts and the owner object '{}' before rerunning", + owner.owner_life, layout.ownerKey(srid))); + return false; + } + } + } + catch (...) + { + warnings.push_back("could not re-read the owner: " + getCurrentExceptionMessage(/*with_stacktrace=*/false)); } - CasOperation op = requests.admit(); + warnings.push_back("the retirement is not verified; rerun SYSTEM CAS DROP POOL MEMBER"); + return false; +} - EventEmitter{*admin}.emit([&](CasEvent & e) +/// The mark records that this run's claim, tombstone and farewell all held; it does not wait for GC. +bool verifyAndMark(CasOperation & op, const Layout & layout, const String & srid, UInt128 victim_uuid, + uint64_t admin_epoch, const Tombstone & tombstone, std::vector & warnings) +{ + const size_t warnings_before = warnings.size(); + const String owner_key = layout.ownerKey(srid); + try { - e.type = CasEventType::MemberDecommission; - e.outcome = "begin"; - e.reason = "operator decommission of pool member"; - e.detail = {{"server_root_id", victim_srid}}; - }); + const std::optional slot = op.read(layout.mountKey(srid), Retry::standard()); + const std::optional current = op.read(owner_key, Retry::standard()); + if (slot && current) + { + const MountLease body = decodeMountLease(slot->bytes); + const OwnerObject owner = decodeOwner(current->bytes); + const bool farewell_held = body.isHeldBy(victim_uuid, admin_epoch) && !body.gc_fenced && body.isFarewell(); + const bool tombstone_held = owner.server_uuid == victim_uuid && owner.retired_at_ms + && owner.owner_life == tombstone.owner.owner_life; + if (farewell_held && tombstone_held) + { + const OwnerObject marked_owner = nextOwnerLife(owner, owner.retired_at_ms, true); + const WriteResult marked = op.replace(owner_key, encodeOwner(marked_owner), current->etag, Retry::standard()); + if (std::holds_alternative(marked)) + return true; + warnings.push_back("the verified mark was not written (" + whyNotWritten(marked) + ")"); + } + else + warnings.push_back(farewell_held ? "the owner is not this run's tombstone" : "the slot does not hold this run's farewell"); + } + } + catch (...) + { + warnings.push_back("verification failed: " + getCurrentExceptionMessage(/*with_stacktrace=*/false)); + } + if (!ownerHoldsVerifiedRetirement(op, layout, srid, tombstone, warnings)) + return false; + /// The owner re-read found the mark: the failure above, e.g. an unclear mark write, needs no rerun. + for (size_t i = warnings_before; i < warnings.size(); ++i) + LOG_INFO(getLogger("CasDecommission"), "CAS decommission '{}': settled by the owner re-read: {}", srid, warnings[i]); + warnings.resize(warnings_before); + return true; +} + +/// The drain on the admin's fenced mount plane: victim namespaces, manifest debris, staging, roots. Per-object +/// failures land in `report.warnings`; a namespace that changed or lost its checkpoint throws. +void drainVictim(Pool & admin, const CasRefCatalog::Snapshot & catalog_cut, const String & victim_srid, + DecommissionReport & report, bool & gc_round_needed) +{ + CasOperation op = admin.mountRequests().admit(); /// The pre-impersonation catalog cut is the complete ownership universe. Physical life keys carry /// no logical path, and raw string prefixes such as `victim` must not select the distinct owner @@ -222,7 +364,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// Refuse a same-name lifecycle move that landed after the immutable selection cut. The /// exact-life overloads below also pin recovery to `life`, closing the race after this check: /// a later replacement can never redirect a removal to its new incarnation. - const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(op, admin->layout()); + const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(op, admin.layout()); const auto current_entry = std::find_if( current_catalog.catalog.entries.begin(), current_catalog.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns.string() == ns_str; }); @@ -234,7 +376,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, if (selected_entry.state == NsState::Removing) { - if (!op.head(admin->layout().refCkptKey(life), Retry::standard()).has_value()) + if (!op.head(admin.layout().refCkptKey(life), Retry::standard()).has_value()) throw Exception(ErrorCodes::CORRUPTED_DATA, "ca-decommission: namespace '{}' is Removing but its exact checkpoint is absent; " "the catalog row remains owned and the victim slot cannot be retired", @@ -243,13 +385,13 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// `dropNamespace` is the sole terminal writer. On an already-complete removal this is an /// idempotent observation; on a pre-terminal `Removing` life it resumes the exact terminal /// append under the administrative writer fence. Catalog deletion remains GC's job. - (void)admin->dropNamespace(life); + (void)admin.dropNamespace(life); ++report.namespaces_already_removed; gc_round_needed = true; continue; } - const auto stats = admin->dropNamespace(life); + const auto stats = admin.dropNamespace(life); ++report.namespaces_removed; report.committed_refs_removed += stats.committed_refs; report.precommits_removed += stats.precommits; @@ -257,7 +399,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, if (selected_entry.state != NsState::Creating) gc_round_needed = true; - EventEmitter{*admin}.emit([&](CasEvent & e) + EventEmitter{admin}.emit([&](CasEvent & e) { e.type = CasEventType::MemberDecommission; e.outcome = "namespace_removed"; @@ -268,255 +410,167 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, }); } - /// Manifest debris must be removed before the mount slot: deleting the mount body removes the - /// watermark authority, after which `floorForNamespace` returns no value and the ordinary orphan - /// sweep cannot prove that old-epoch debris is eligible. The decommission claim has advanced the - /// writer epoch, so every build prefix with `prefix.writer_epoch < w.writer_epoch` is eligible here. - /// Group the listed keys by namespace and build prefix so each group can use the exact-token orphan - /// sweep while the mount body still supplies its authority. + /// The decommission claim has advanced the writer epoch, so every build prefix with + /// `prefix.writer_epoch < w.writer_epoch` is eligible here. Group the listed keys by namespace and + /// build prefix so each group can use the exact-token orphan sweep. { - const String debris_prefix = admin->layout().casManifestsServerPrefix(victim_srid); + const String debris_prefix = admin.layout().casManifestsServerPrefix(victim_srid); std::set> groups; /// (namespace, writer epoch, build sequence) op.forEachListedKey(debris_prefix, [&](const ListedKey & listed) { - if (const auto parsed = admin->layout().parseManifestKey(listed.key)) + if (const auto parsed = admin.layout().parseManifestKey(listed.key)) groups.emplace(parsed->root_namespace.string(), parsed->ref.writer_epoch, parsed->ref.build_sequence); return true; }, Retry::standard()); for (const auto & [ns_str, writer_epoch, build_sequence] : groups) report.manifest_debris_removed += sweepNamespace( - *admin, RootNamespace(ns_str), BuildPrefix{writer_epoch, build_sequence}, &report.warnings); + admin, RootNamespace(ns_str), BuildPrefix{writer_epoch, build_sequence}, &report.warnings); } /// Drain the victim's own `/staging//` area. The live-mount staging helper uses /// an `IObjectStorage`, while this command intentionally works at the `Backend` layer, so the same /// prefix is listed and deleted directly. The claim fences the victim's writers during this sweep. - report.staging_objects_removed += deleteListedPrefix( - op, admin->poolConfig().pool_prefix + "/staging/" + victim_srid + "/", report.warnings); + report.staging_objects_removed += deleteListedPrefix(op, admin.layout().stagingPrefix(victim_srid), report.warnings); /// Drain the victim's mountpoint objects. These are loose, non-content-addressed files under /// `Layout::serverRootDataPrefix`; they have no writer epoch of their own, so the claim is what /// prevents a returning victim from racing this deletion. report.mountpoint_objects_removed += deleteListedPrefix( - op, admin->layout().serverRootDataPrefix(victim_srid), report.warnings); - - /// The catalog, not physical debris, owns the slot-retirement decision. A terminal append only - /// moves a row to `Removing`; GC must fold/prune/delete it before the member's ownership anchor can - /// disappear. Capture one exact whole-catalog cut after every drain, then revalidate its incarnation - /// and canonical value immediately before entering the retirement tail. The administrative claim - /// fences the victim writer between those observations. - std::optional retirement_catalog_cut; - if (report.warnings.empty()) + op, admin.layout().serverRootDataPrefix(victim_srid), report.warnings); + + if (!op.admitted()) { - retirement_catalog_cut = CasRefCatalog::read(op, admin->layout()); - const uint64_t victim_owned_count = std::count_if( - retirement_catalog_cut->catalog.entries.begin(), retirement_catalog_cut->catalog.entries.end(), - [&](const CatalogEntry & entry) - { - return entry.ns.string() == victim_srid - || entry.ns.string().starts_with(victim_namespace_prefix); - }); - if (victim_owned_count > 0) - report.warnings.push_back( - "pool member decommission underway: " + std::to_string(victim_owned_count) - + " namespace(s) are still owned by this member; upcoming GC rounds perform the final " - "cleanup — re-run this command afterwards to retire the slot"); + report.warnings.push_back("the admin lost its mount lease during the drain; rerun"); + return; } - /// Retire the slot strictly last and only after a clean drain. Copy the layout before - /// `admin.reset()`: graceful close destroys the `Pool`, while `op` (holding its own shared_ptr to - /// the backend) remains usable to retire the slot objects afterwards. - const Layout layout = admin->layout(); - if (report.warnings.empty()) + /// Rows still owned by the victim after the drain. `Removing` rows are GC's: they count as pending, not + /// as a failure. Anything else appeared after the ownership cut and was not drained: rerun. + const CasRefCatalog::Snapshot after_drain = CasRefCatalog::read(op, admin.layout()); + for (const CatalogEntry & entry : after_drain.catalog.entries) { - const CasRefCatalog::Snapshot fresh_retirement_catalog = CasRefCatalog::read(op, admin->layout()); - if (!retirement_catalog_cut - || fresh_retirement_catalog.etag != retirement_catalog_cut->etag - || fresh_retirement_catalog.catalog != retirement_catalog_cut->catalog) - { - report.warnings.push_back( - "catalog changed after the victim ownership check; refusing slot retirement against a stale cut"); - } + if (entry.ns.string() != victim_srid && !entry.ns.string().starts_with(victim_namespace_prefix)) + continue; + if (entry.state == NsState::Removing) + ++report.namespaces_pending_gc; + else + report.warnings.push_back("namespace '" + entry.ns.string() + "' appeared after the ownership cut and was not drained; rerun"); } - if (report.warnings.empty()) +} + +} + +bool adminClaimStillHeld(CasOperation & op, const Layout & layout, const String & srid, UInt128 victim_uuid, uint64_t admin_epoch) +{ + const std::optional slot = op.read(layout.mountKey(srid), Retry::standard()); + if (!slot) + return false; + const MountLease body = decodeMountLease(slot->bytes); + return body.isHeldBy(victim_uuid, admin_epoch) && !body.gc_fenced && !body.isFarewell(); +} + +DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, + const String & victim_srid, const CasEventSink & sink, + const std::function & request_gc_round, + const std::function & drain_now_fn, + const std::function & drain_sleep_fn) +{ + DecommissionReport report; + report.srid = victim_srid; + bool gc_round_needed = false; + /// A namespace may have reached `Removing` before a later namespace fails closed: wake GC on every exit. + SCOPE_EXIT({ + if (gc_round_needed && request_gc_round) + request_gc_round(); + }); + + /// Validate one immutable ownership cut before impersonating the victim, on the raw backend: no `Pool` + /// exists yet. Namespace selection reuses this exact pre-mutation decision rather than a later read. + CasRequests preflight_requests(backend, Fence::open()); + CasOperation preflight_op = preflight_requests.admit(); + const Layout catalog_layout(config.pool_prefix); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(preflight_op, catalog_layout); + catalog_cut.life_index.throwIfAmbiguous("CAS decommission"); + + /// One fake clock and sleep for every plane: the farewell deadline is bound to the boot clock, and a + /// plane on a frozen clock that sleeps for real would never see its retry deadline elapse. + if (drain_now_fn && drain_sleep_fn) { - const String mount_key = layout.mountKey(victim_srid); - const String epoch_key = layout.epochKey(victim_srid); - const String owner_key = layout.ownerKey(victim_srid); - - /// Capture both the epoch value and its exact incarnation while the decommission claim still - /// fences the victim. A successor can only bump this object after the farewell below releases - /// the claim, so this incarnation is the epoch-side successor fence for the retirement tail. - std::optional claimed_epoch; - try - { - claimed_epoch = op.read(epoch_key, Retry::standard()); - if (!claimed_epoch) - report.warnings.push_back("slot capture failed: " + epoch_key + " is absent under the admin claim"); - } - catch (...) - { - report.warnings.push_back("slot capture failed: " + epoch_key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); - } + if (!config.boot_ms_fn) + config.boot_ms_fn = drain_now_fn; + if (!config.retry_sleep_fn) + config.retry_sleep_fn = drain_sleep_fn; + } + config.event_sink = sink; + + /// The admin claim. `NoWait` refuses a live victim and accepts a tombstoned owner. + PoolPtr admin = Pool::openForDecommission(std::move(backend), std::move(config), victim_srid); + if (drain_now_fn && drain_sleep_fn) + { + admin->setCasRequestNowFnForTest(drain_now_fn); + admin->setCasRetrySleepForTest(drain_sleep_fn); + } + const UInt128 victim_uuid = admin->poolConfig().server_id; + const uint64_t admin_epoch = admin->writerEpoch(); + const Layout layout = admin->layout(); - /// Graceful close stamps an already-expired lease and the watermark farewell - /// (`min_active_build_sequence = UINT64_MAX`), making the slot `terminated` before its mutable control objects - /// are removed and its owner anchor is tombstoned. - admin.reset(); + /// The control plane: always admitted, so a fenced admin can still confirm, roll back and verify. It keeps + /// its own shared_ptr to the backend and outlives `admin.reset()`. + CasRequests control_requests(admin->poolBackendPtr(), Fence::open()); + if (drain_now_fn && drain_sleep_fn) + { + control_requests.setNowFnForTest(drain_now_fn); + control_requests.setSleepFnForTest(drain_sleep_fn); + } + CasOperation control = control_requests.admit(); - /// Read the farewell immediately after `finishTeardown` wrote it. Its exact incarnation is the - /// mount-side fence: deleting by this incarnation can remove only THIS decommission's farewell, - /// not a successor reclaim. Validate the body against the epoch value captured under the claim - /// so a successor that completed before this read is also recognized and left untouched. - std::optional farewell_mount; - try - { - farewell_mount = op.read(mount_key, Retry::standard()); - if (!farewell_mount) - report.warnings.push_back("slot capture failed: " + mount_key + " farewell is absent"); - } - catch (...) - { - report.warnings.push_back("slot capture failed: " + mount_key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); - } + EventEmitter{*admin}.emit([&](CasEvent & e) + { + e.type = CasEventType::MemberDecommission; + e.outcome = "begin"; + e.reason = "operator decommission of pool member"; + e.detail = {{"server_root_id", victim_srid}, {"server_uuid", u128ToHex(victim_uuid)}}; + }); - bool captures_match = claimed_epoch && farewell_mount; - if (captures_match) + /// From here on every exit decides the tombstone's fate (`confirmClaimOrRollBack`). + const Tombstone tombstone = writeTombstone(control, *admin, layout, victim_srid, victim_uuid, admin_epoch, report.warnings); + + try + { + /// The claim must have held through the tombstone write. + if (!adminClaimStillHeld(control, layout, victim_srid, victim_uuid, admin_epoch)) { - try - { - const ServerEpoch epoch_value = decodeServerEpoch(claimed_epoch->bytes); - const MountLease mount_value = decodeMountLease(farewell_mount->bytes); - captures_match = epoch_value.next_writer_epoch != 0 - && mount_value.writer_epoch == epoch_value.next_writer_epoch - 1 - && mount_value.min_active_build_sequence == std::numeric_limits::max() - && !mount_value.gc_fenced; - if (!captures_match) - { - report.warnings.push_back( - "slot capture failed: " + mount_key - + " is not this decommission's farewell for the epoch captured under the admin claim"); - } - } - catch (...) - { - report.warnings.push_back("slot capture failed while validating " + mount_key + " and " + epoch_key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); - captures_match = false; - } + rollBackTombstone(control, layout, victim_srid, tombstone, report.warnings); } - - /// Mount first: if a successor reclaimed it after the farewell capture, the stale farewell - /// incarnation yields `Mismatch` and the tail stops before touching epoch or owner. Epoch - /// second: its under-claim incarnation similarly detects a successor allocation. Before - /// touching owner, re-read both mutable objects: a same-UUID successor can recreate them after - /// both deletes without rewriting the owner identity anchor. Mere presence proves that the slot - /// is live again. Every delete must be explicitly confirmed as `Removed`, and the final owner - /// tombstone rewrite must succeed against the exact incarnation read immediately before it. - /// - /// ACCEPTED RESIDUAL WINDOW (final review, not closed by this recheck): a same-UUID successor - /// can still recreate epoch/mount in the narrow gap strictly AFTER this liveness recheck but - /// BEFORE the owner CAS below reads its own incarnation -- the successor's owner anchor (same - /// server_uuid, not yet retired) then gets tombstoned by this decommission run. The successor's - /// live process is not deleted (only its owner anchor is marked retired), but a LATER restart of - /// that same identity would refuse to reclaim it (claimOwnerOrThrow's tombstone guard). This is - /// a narrow, low-probability window, deliberately not closed here: T5's owner-tombstone design - /// (finding #9) intentionally stopped short of making concurrent decommission-vs-recreate - /// airtight to the microsecond, since that was explicitly not the priority for this fix. - report.slot_removed = false; - if (captures_match && deleteSlotObject(op, mount_key, farewell_mount->etag, report.warnings) - && deleteSlotObject(op, epoch_key, claimed_epoch->etag, report.warnings)) + else { - std::optional current_mount; - std::optional current_epoch; - bool liveness_recheck_succeeded = true; - try - { - current_mount = op.read(mount_key, Retry::standard()); - } - catch (...) - { - report.warnings.push_back("slot liveness recheck failed: " + mount_key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); - liveness_recheck_succeeded = false; - } - try - { - current_epoch = op.read(epoch_key, Retry::standard()); - } - catch (...) - { - report.warnings.push_back("slot liveness recheck failed: " + epoch_key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); - liveness_recheck_succeeded = false; - } - - if (liveness_recheck_succeeded && (current_mount || current_epoch)) - { - report.warnings.push_back( - "slot delete aborted: successor reappeared after mutable control-object deletion; owner kept"); - } - else if (liveness_recheck_succeeded) - { - try - { - if (const auto owner = op.read(owner_key, Retry::standard())) - { - OwnerObject tombstoned = decodeOwner(owner->bytes); - tombstoned.retired_at_ms = nowMs(); - /// `op.replace` resolves an ambiguous attempt with a resolve read on its own: - /// a transient transport error here (or one whose response was simply lost) - /// must not be reported as a hard failure when the write actually landed. - /// Unchanged incarnation means the write never applied (legitimately retryable - /// within budget); matching bytes means this exact tombstone already landed - /// (`Committed`, not a failure); a genuine successor reclaim is `Conflict`; - /// `Refused` is the store's own definite answer (a denial, a malformed - /// request, an expired credential) and carries its own code and message, - /// which is worth more here than the generic retry advice below. - WriteResult result = op.replace(owner_key, encodeOwner(tombstoned), owner->etag, Retry::standard()); - if (std::holds_alternative(result)) - report.slot_removed = true; - else if (std::holds_alternative(result)) - report.warnings.push_back( - "slot tombstone failed: " + owner_key - + ": successor reclaimed the owner anchor before this decommission's tombstone write"); - else if (const Refused * refused = std::get_if(&result)) - report.warnings.push_back( - "slot tombstone failed: " + owner_key + ": the store refused the write (" - + std::to_string(refused->store_error) + "): " + refused->message); - else - report.warnings.push_back( - "slot tombstone failed: " + owner_key - + ": tombstone write outcome could not be resolved (retry budget exhausted " - "or the resolve read itself failed) -- rerun the command to retry"); - } - else - report.warnings.push_back( - "slot tombstone failed: " + owner_key + ": object absent before tombstone write"); - } - catch (...) - { - report.warnings.push_back("slot tombstone failed: " + owner_key + ": " - + getCurrentExceptionMessage(/*with_stacktrace=*/false)); - } - } + drainVictim(*admin, catalog_cut, victim_srid, report, gc_round_needed); + if (!report.warnings.empty()) + confirmClaimOrRollBack(control, layout, victim_srid, victim_uuid, admin_epoch, tombstone, report.warnings); } } - else + catch (...) + { + /// The farewell `admin`'s destructor writes on unwind is a clean body, which the confirm would read as + /// a lost claim, so the decision is made here, while the admin is alive. + confirmClaimOrRollBack(control, layout, victim_srid, victim_uuid, admin_epoch, tombstone, report.warnings); + logWarnings(victim_srid, report.warnings); + throw; + } + + /// The farewell: it lands while the claim holds and loses to the fence after a lost claim. + /// `MountLeaseRenewer::terminate` reports no outcome, so `verifyAndMark` reads it back. + admin.reset(); + if (report.warnings.empty()) { - report.slot_removed = false; - LOG_WARNING(getLogger("CasDecommission"), - "CAS decommission '{}': drain incomplete ({} warnings) — mount slot kept (terminated); " - "re-run the command to finish", victim_srid, report.warnings.size()); - admin.reset(); /// Graceful close still stamps the farewell, leaving the slot `terminated`. + report.member_retired = verifyAndMark(control, layout, victim_srid, victim_uuid, admin_epoch, tombstone, report.warnings); + if (report.member_retired) + LOG_INFO(getLogger("CasDecommission"), "CAS decommission '{}': retired server_uuid={} at owner life {}", + victim_srid, u128ToHex(victim_uuid), tombstone.owner.owner_life); } + logWarnings(victim_srid, report.warnings); - /// The `end` event is emitted via `sink` directly, not `EventEmitter{*admin}`: `admin` is gone by - /// now. This also means its `warnings` count reflects the FINAL total, including a slot-retirement - /// failure appended just above -- `EventEmitter`'s own zero-cost-when-absent guard is reproduced by - /// the `if (sink)` below. + /// Emitted through `sink` directly: `admin` is gone by now. if (sink) { CasEvent e; @@ -524,9 +578,12 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, e.outcome = "end"; e.reason = "decommission finished"; e.detail = {{"server_root_id", victim_srid}, + {"server_uuid", u128ToHex(victim_uuid)}, + {"owner_life", std::to_string(tombstone.owner.owner_life)}, {"namespaces_removed", std::to_string(report.namespaces_removed)}, + {"namespaces_pending_gc", std::to_string(report.namespaces_pending_gc)}, {"warnings", std::to_string(report.warnings.size())}, - {"slot_removed", report.slot_removed ? "1" : "0"}}; + {"member_retired", report.member_retired ? "1" : "0"}}; sink(std::move(e)); } return report; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h index 6832fd9cfe731..98adfbb014f29 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h @@ -16,51 +16,57 @@ namespace DB::Cas /// the manifest, staging, and mountpoint drains. Blob bytes are intentionally not reported: removing /// ref edges makes them eligible for ordinary GC, but this operation does not synchronously reclaim /// shared content. -/// -/// A decommission is resumable. A previous run may already have moved namespaces to `Removing`, and a -/// warning means that the corresponding drain was not confirmed. In either case the report lets the -/// caller distinguish work done by this invocation from work observed from an earlier invocation. struct DecommissionReport { String srid; /// The decommissioned member's `server_root_id`. uint64_t namespaces_removed = 0; /// Namespaces erased by this invocation. uint64_t namespaces_already_removed = 0; /// Namespaces already `Removing` on entry. + /// Victim rows still `Removing` when the run ended: GC deletes them; no rerun is needed for them. + uint64_t namespaces_pending_gc = 0; uint64_t committed_refs_removed = 0; /// Committed ref records removed by namespace drops. uint64_t precommits_removed = 0; /// Precommit records removed by namespace drops. uint64_t edge_deltas_emitted = 0; /// The sum of `committed_refs_removed` and `precommits_removed`. uint64_t manifest_debris_removed = 0; /// Eligible manifest objects deleted from old build prefixes. uint64_t staging_objects_removed = 0; /// Objects deleted from the member's staging prefix. uint64_t mountpoint_objects_removed = 0; /// Objects deleted from the member's roots/mountpoint prefix. - bool slot_removed = false; /// Whether mount and epoch were deleted and the owner was tombstoned. - std::vector warnings; /// Drain or slot-retirement failures; a non-empty list keeps the slot. + /// The owner carries a verified retirement: the member is retired, do not rerun. + bool member_retired = false; + /// Why the member is not retired: a drain failure, a lost claim, an unverified outcome (rerun), or an owner + /// that moved on to a later life (read the owner before rerunning). + std::vector warnings; }; -/// Erases all content owned by a permanently dead pool member. The operation first claims the member's -/// slot as an administrative writer; a live lease is refused, and the claim fences the dead member from -/// writing while cleanup runs. It then drops each table namespace through `Pool::dropNamespace`, drains -/// eligible manifest debris, staging objects, and mountpoint objects, and retires the slot only after all -/// drains are confirmed. Namespace drops are idempotent: a rerun resumes any missing terminal append -/// and leaves exact catalog-row deletion to GC. The member slot remains while any catalog entry still -/// belongs to the victim. +/// Whether the slot still holds the admin's claim: the victim's uuid, `admin_epoch`, neither fenced nor +/// farewelled. Renewals change the ETag and `write_attempt_id`, so neither is compared. +bool adminClaimStillHeld(CasOperation & op, const Layout & layout, const String & srid, UInt128 victim_uuid, uint64_t admin_epoch); + +/// Erases all content owned by a permanently dead pool member and retires its owner. Never deletes +/// `owner`, `epoch` or `mount`. The steps: +/// 1. Claim the member's slot as an administrative writer (`Pool::openForDecommission`). A live lease is +/// refused; an owner retired by an earlier run is accepted. +/// 2. Tombstone the owner on the admin's fenced mount plane: the next owner life, retired, keeping an earlier +/// verified mark. No attempt is sent after the admin's lease deadline. +/// 3. Confirm the claim still holds. A lost claim rolls the tombstone back unless the owner was already +/// retired; the same confirm runs on every exit before the farewell. +/// 4. Drain on the admin's fenced mount plane: each victim namespace through `Pool::dropNamespace`, then +/// eligible manifest debris, staging objects and mountpoint objects. +/// 5. Write the admin's clean farewell. +/// 6. Read the slot and the owner back and mark the retirement verified (the next owner life). Only a +/// verified mark reports `member_retired`. /// -/// This is a writer operation, not GC: it emits the normal ref-edge deltas and does not invent ref -/// transitions. Per-object drain failures are recorded in `DecommissionReport::warnings` and leave the -/// terminated slot as a resume anchor; other failures, including refusal to claim the member, propagate -/// as exceptions. When set, `sink` receives `MemberDecommission` audit events for the run's begin, -/// per-namespace, and end milestones. +/// A rerun resumes over the run's own unverified tombstone; a verified retirement is rerun-safe and +/// reports retired again. Catalog rows GC still has to delete count as `namespaces_pending_gc`, not as a +/// failure. Per-object drain failures are recorded in `DecommissionReport::warnings`; other failures, +/// including refusal to claim the member, propagate as exceptions. When set, `sink` receives +/// `MemberDecommission` audit events for the run's begin, per-namespace, and end milestones. /// -/// `drain_now_fn`/`drain_sleep_fn`, when both set, replace the clock the drain's own request engine -/// paces its retries on -- the engine this function opens is a standalone one over the instrumented -/// backend, not one of `Pool`'s planes, so `Pool::setCasRetrySleepForTest` cannot reach it. A test -/// driving a latched per-object fault to `Retry::standard()`'s own give-up needs this seam, or it pays -/// the real 90-second deadline. +/// This is a writer operation, not GC: it emits the normal ref-edge deltas and does not invent ref +/// transitions. /// -/// `drain_now_fn`/`drain_sleep_fn`, when BOTH set, also become the opened `Pool`'s own boot clock and -/// retry sleep (`PoolConfig::boot_ms_fn`/`retry_sleep_fn`) whenever the caller left those fields unset: -/// the mount lease's farewell deadline is bound to the boot clock, and `Pool::openForDecommission`'s own -/// mount/farewell/GC planes are constructed with it too, so a caller that fakes only the standalone -/// engine's clock must not end up comparing it against the real one, or -- worse -- against a plane that -/// shares the frozen clock but still sleeps for real between retries. +/// `drain_now_fn`/`drain_sleep_fn`, when both set, replace the clock and retry sleep of the control-plane +/// request engine this function opens and of the admin `Pool`'s planes, and become the `Pool`'s boot clock +/// and retry sleep (`PoolConfig::boot_ms_fn`/`retry_sleep_fn`) whenever the caller left those unset: the +/// farewell deadline is bound to the boot clock, so one fake clock must drive all of them. DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const String & victim_srid, const CasEventSink & sink = {}, const std::function & request_gc_round = {}, diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index d87e16d3b15df..ef8391034263b 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -28,13 +29,17 @@ #include #include #include +#include #include #include #include #include +#include + #include +#include #include #include #include @@ -85,6 +90,37 @@ namespace DB::ErrorCodes namespace DB::Cas::tests { +/// Redirects the named logger into a string for the scope, at the given level. +class ScopedLoggerCapture +{ +public: + ScopedLoggerCapture(const String & logger_name, const String & level) + : logger(getLogger(logger_name)) + , channel(new Poco::StreamChannel(stream)) + , old_channel(logger->getChannel(), /*shared=*/true) + , old_level(logger->getLevel()) + { + logger->setChannel(channel.get()); + logger->setLevel(level); + } + + ~ScopedLoggerCapture() + { + logger->setChannel(old_channel); + logger->setLevel(old_level); + } + + String captured() const { return stream.str(); } + +private: + LoggerPtr logger; + std::ostringstream stream; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + Poco::AutoPtr channel; + /// A real reference (shared=true), so the parked previous channel cannot die while ours is installed. + Poco::AutoPtr old_channel; + int old_level; +}; + /// A `CasRequests` over an always-open fence, for a fixture that has a backend but no mounted pool. /// Every operation admitted from it holds a reference to it, so it must be named and outlive them. inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::BackendPtr backend) @@ -2327,6 +2363,32 @@ class HintHoleBackendOn : public Base /// The plain form, over a bare `InMemoryBackend`. using HintHoleBackend = HintHoleBackendOn; +/// GC's fence-out applied directly to the mount lease: preserve the body, set `gc_fenced`, bump `seq`. +/// A following `tryRemountOnce` reclaims a fresh incarnation. +inline void fenceOutMountForTest(DB::Cas::Backend & backend, const String & mount_key) +{ + OperationForTest op(backend); + const auto got = (*op).read(mount_key, DB::Cas::Retry::once()); + ASSERT_TRUE(got.has_value()); + const DB::Cas::MountLease fenced = DB::Cas::decodeMountLease(got->bytes).fencedCopy(); + const auto put = (*op).replace(mount_key, DB::Cas::encodeMountLease(fenced), got->etag, DB::Cas::Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); +} + +/// A decommission tombstone by hand: the next owner life, retired, keeping a verified mark of a retired owner; +/// `verified` also marks it, as a completed run does. +inline void tombstoneOwnerForTest(DB::Cas::Backend & backend, const String & owner_key, bool verified = false) +{ + OperationForTest op(backend); + const auto current = (*op).read(owner_key, DB::Cas::Retry::once()); + ASSERT_TRUE(current.has_value()); + const DB::Cas::OwnerObject owner = DB::Cas::decodeOwner(current->bytes); + const DB::Cas::OwnerObject tombstone = DB::Cas::nextOwnerLife( + owner, 1, verified || (owner.retirement_verified && owner.retired_at_ms.has_value())); + const auto put = (*op).replace(owner_key, DB::Cas::encodeOwner(tombstone), current->etag, DB::Cas::Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); +} + /// Stand in for the self-remount that `Pool::reportImpossibleInterference` schedules. That reaction /// trips the local write fence closed AND schedules a remount; a unit-test Pool runs no background /// remount (`background_watermark` is off by design there), so without this the fence stays closed and diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index c06bc246b55a8..ea6f3c1cf7a0c 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -2,15 +2,34 @@ #include #include #include +#include #include +#include +#include +#include #include #include +#include +#include +#include +#include #include #include +#include #include +#include +#include +#include + +namespace ProfileEvents +{ + extern const Event CASRemountFailed; +} namespace DB::ErrorCodes { + extern const int ABORTED; + extern const int BAD_ARGUMENTS; extern const int CORRUPTED_DATA; } @@ -22,9 +41,27 @@ namespace { /// Open a store for the VICTIM srid over `backend` (the pool's future dead member). -PoolPtr openVictim(std::shared_ptr backend) +PoolPtr openVictim(BackendPtr backend) +{ + return Pool::open(std::move(backend), PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); +} + +OwnerObject readOwner(Backend & backend, const Layout & layout, const String & srid) +{ + OperationForTest op(backend); + const auto current = (*op).read(layout.ownerKey(srid), Retry::once()); + if (!current) + throw std::runtime_error("owner absent: " + layout.ownerKey(srid)); + return decodeOwner(current->bytes); +} + +MountLease readSlot(Backend & backend, const Layout & layout, const String & srid) { - return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); + OperationForTest op(backend); + const auto current = (*op).read(layout.mountKey(srid), Retry::once()); + if (!current) + throw std::runtime_error("mount absent: " + layout.mountKey(srid)); + return decodeMountLease(current->bytes); } void drainCompletedNamespaceRemovals(const std::shared_ptr & backend) @@ -40,27 +77,15 @@ void drainCompletedNamespaceRemovals(const std::shared_ptr & ba ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); } -/// Fails a delete for one or two designated keys -- either by throwing (a transient backend hiccup) or -/// by returning a synthetic `Mismatch` (a "listed but raced" outcome) -- delegating every other key to -/// the base `InMemoryBackend` untouched. Drives the drain phases' per-object fail-close path -/// (`deleteListedPrefix`/`sweepNamespace`, `CasDecommission.cpp`/`CasOrphanManifestSweep.cpp`): a -/// failure on one listed object must record a warning and let the rest of the sweep proceed, never -/// abort the whole phase. Injects on the `remove` PRIMITIVE, not the legacy `deleteExact`, so it -/// intercepts a caller on either surface. -/// -/// The thrown fault is a `Poco::TimeoutException`, the class the request engine classifies as a -/// transport failure and reissues (`Retry::standard()`); a `std::runtime_error` propagates immediately -/// and never exercises the retry path this test's own name claims to drive. `latch` keeps it armed -/// across every reissue of the SAME logical call, so the engine reaches its own retry deadline and -/// gives up rather than recovering on a later attempt -- a one-shot throw would be outlived by the -/// reissue and the delete would simply succeed. +/// Fails the delete of one key with a transient `Poco::TimeoutException` (the engine reissues it), and of +/// another with `Mismatch`. `latch` keeps the throw armed across every reissue, so the engine gives up. class FailingDeleteBackend : public InMemoryBackend { public: void failWithThrow(const String & key) { throw_key = key; } void failWithTokenMismatch(const String & key) { mismatch_key = key; } void latch() { latched = true; } - /// Clears every injected failure -- the resume half of a fail-then-retry test (Task 4). + /// Clears every injected failure. void disarm() { throw_key.clear(); mismatch_key.clear(); latched = false; } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override @@ -178,237 +203,81 @@ std::vector> snapshotPrefixObjects( } } -/// Installs a same-UUID successor deterministically in the retirement tail's read/delete window. -/// Once armed, the backend recognizes the admin's clean farewell `putOverwrite`. On the next read of -/// either mutable control object it first captures the value that read observed, then bumps `epoch` -/// and reclaims `mount` with fresh tokens before returning the captured result. Thus the caller holds -/// exactly the stale token it would have obtained immediately before a concurrent restart reclaimed -/// the slot, without threads or sleeps. -class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend +/// Once armed, swaps the owner to another uuid on the second owner read after the admin's slot claim +/// landed: the first is the claim recheck, the second is the tombstone's. Arm it after the victim closed, +/// or the victim's own lease writes count as the claim. +class SwapOwnerBeforeTombstoneBackend : public InMemoryBackend { public: - void armForSuccessorReclaim() { armed = true; } - - /// Injects on the `read` PRIMITIVE: the retirement tail's own reads of `mount_key`/`epoch_key` - /// (`CasDecommission.cpp`) go through `CasOperation::read`, not the legacy `get`. - std::optional read(const String & key, TransportAccess & access) override - { - std::optional result = InMemoryBackend::read(key, access); - if (farewell_seen && !successor_injected && (key == mount_key || key == epoch_key)) - injectSuccessor(access); - return result; - } + void arm() { armed = true; } + bool fired() const { return swapped; } - /// Injects on the `write` PRIMITIVE: `putOverwrite` is not one of the two verb-identity - /// exceptions (`putIfAbsent`/`casPut`), so both the legacy caller and `CasOperation::replace` - /// reach it here. std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { const auto result = InMemoryBackend::write(key, bytes, expected_value, access); - if (armed && result.has_value() && key == mount_key) - { - const MountLease mount = decodeMountLease(bytes); - if (mount.min_active_build_sequence == std::numeric_limits::max()) - farewell_seen = true; - } + if (armed && result && key == mount_key && expected_value && !decodeMountLease(bytes).gc_fenced + && !decodeMountLease(bytes).isFarewell()) + claimed = true; return result; } - bool successorInjected() const { return successor_injected; } - const String & successorMountValue() const { return successor_mount_value; } - const String & successorEpochValue() const { return successor_epoch_value; } - const String & successorMountBytes() const { return successor_mount_bytes; } - const String & successorEpochBytes() const { return successor_epoch_bytes; } - -private: - /// Every request here is issued on the PRIMITIVE `write`, using the `access` token the caller's - /// own in-flight request already holds -- no `CasRequests`/`CasOperation` exists inside this - /// reentrant hook to mint one. The captured `successor_*_value` fields are the raw wire values - /// `write` returned, which the tests compare against a real incarnation's rendered value - /// (`PersistedEtag::capture`) taken through their own `CasOperation`. - void injectSuccessor(TransportAccess & access) - { - const auto epoch = InMemoryBackend::read(epoch_key, access); - const auto mount = InMemoryBackend::read(mount_key, access); - if (!epoch || !mount) - throw std::runtime_error("successor-reclaim fixture: control object disappeared before reclaim"); - - ServerEpoch epoch_value = decodeServerEpoch(epoch->bytes); - const uint64_t successor_writer_epoch = epoch_value.next_writer_epoch; - ++epoch_value.next_writer_epoch; - successor_epoch_bytes = encodeServerEpoch(epoch_value); - const auto epoch_written = InMemoryBackend::write(epoch_key, successor_epoch_bytes, epoch->value, access); - if (!epoch_written) - throw std::runtime_error("successor-reclaim fixture: epoch bump conflicted"); - successor_epoch_value = *epoch_written; - - MountLease mount_value = decodeMountLease(mount->bytes); - mount_value.writer_epoch = successor_writer_epoch; - ++mount_value.seq; - ++mount_value.started_at_ms; - mount_value.expires_at_ms = mount_value.started_at_ms + 30'000; - mount_value.min_active_build_sequence = 0; - mount_value.gc_fenced = false; - successor_mount_bytes = encodeMountLease(mount_value); - const auto mount_written = InMemoryBackend::write(mount_key, successor_mount_bytes, mount->value, access); - if (!mount_written) - throw std::runtime_error("successor-reclaim fixture: mount reclaim conflicted"); - successor_mount_value = *mount_written; - successor_injected = true; - } - - inline static const String mount_key = "p/gc/server-roots/victim/mount"; - inline static const String epoch_key = "p/gc/server-roots/victim/epoch"; - bool armed = false; - bool farewell_seen = false; - bool successor_injected = false; - String successor_mount_value; - String successor_epoch_value; - String successor_mount_bytes; - String successor_epoch_bytes; -}; - -/// Recreates the mutable slot objects immediately after decommission successfully deletes `epoch`. -/// This models a same-UUID successor starting in the final retirement window: `owner` remains the -/// unchanged identity anchor, while the successor legitimately creates a fresh `epoch` and `mount`. -class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend -{ -public: - void armForSuccessorReclaim() { armed = true; } - - /// Injects on the `remove` PRIMITIVE: `deleteSlotObject`'s epoch delete (`CasDecommission.cpp`) - /// goes through `CasOperation::remove`, not the legacy `deleteExact`. - RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + std::optional read(const String & key, TransportAccess & access) override { - const RawRemoval result = InMemoryBackend::remove(key, expected_value, access); - if (armed && !successor_injected && key == epoch_key && result == RawRemoval::Removed) - injectSuccessor(access); - return result; + if (claimed && !swapped && key == owner_key && ++owner_reads == 2) + { + const auto current = InMemoryBackend::read(key, access); + OwnerObject foreign = nextOwnerLife(decodeOwner(current->bytes), std::nullopt, false); + foreign.server_uuid = UInt128(0xBEEF); + if (!InMemoryBackend::write(key, encodeOwner(foreign), current->value, access)) + throw std::runtime_error("owner swap fixture: the rewrite conflicted"); + swapped = true; + } + return InMemoryBackend::read(key, access); } - bool successorInjected() const { return successor_injected; } - uint64_t ownerRewriteAttempts() const { return owner_rewrite_attempts; } - const String & successorMountValue() const { return successor_mount_value; } - const String & successorEpochValue() const { return successor_epoch_value; } - const String & successorMountBytes() const { return successor_mount_bytes; } - const String & successorEpochBytes() const { return successor_epoch_bytes; } - private: - /// Counts on the `write` PRIMITIVE: the owner tombstone write this test asserts is never - /// attempted (`CasDecommission.cpp`'s `op.replace`) reaches the store through here, not through - /// the legacy `putOverwrite`. Guarded on `expected_value`: the primitive also sees the owner - /// anchor's own CREATE during this fixture's `openVictim`, which `putOverwrite` (a conditional - /// REPLACE only) never did -- counting it would start this counter at 1 before the interesting - /// part of the test begins. - std::expected write(const String & key, const String & bytes, - const std::optional & expected_value, TransportAccess & access) override - { - if (key == owner_key && expected_value) - ++owner_rewrite_attempts; - return InMemoryBackend::write(key, bytes, expected_value, access); - } - - /// On the `write` PRIMITIVE, using the caller's own in-flight `access`: this hook is reentrant - /// (called from inside another primitive override on the same object), so it reaches the store - /// directly through `InMemoryBackend::write` rather than admitting a fresh request of its own. - void injectSuccessor(TransportAccess & access) - { - successor_epoch_bytes = encodeServerEpoch(ServerEpoch{.next_writer_epoch = 102}); - const auto epoch_written = InMemoryBackend::write(epoch_key, successor_epoch_bytes, std::nullopt, access); - if (!epoch_written) - throw std::runtime_error("late-successor fixture: epoch recreation conflicted"); - successor_epoch_value = *epoch_written; - - successor_mount_bytes = encodeMountLease(MountLease{ - .server_uuid = UInt128(0x1234), - .writer_epoch = 101, - .hostname = "successor", - .pid = 42, - .started_at_ms = 1'000, - .seq = 1, - .expires_at_ms = 31'000, - .min_active_build_sequence = 0, - }); - const auto mount_written = InMemoryBackend::write(mount_key, successor_mount_bytes, std::nullopt, access); - if (!mount_written) - throw std::runtime_error("late-successor fixture: mount recreation conflicted"); - successor_mount_value = *mount_written; - successor_injected = true; - } - inline static const String mount_key = "p/gc/server-roots/victim/mount"; - inline static const String epoch_key = "p/gc/server-roots/victim/epoch"; inline static const String owner_key = "p/gc/server-roots/victim/owner"; bool armed = false; - bool successor_injected = false; - uint64_t owner_rewrite_attempts = 0; - String successor_mount_value; - String successor_epoch_value; - String successor_mount_bytes; - String successor_epoch_bytes; + bool claimed = false; + bool swapped = false; + int owner_reads = 0; }; -/// Rewrites the owner anchor after decommission reads it but before its conditional tombstone write. -/// Returning the captured result gives decommission a stale owner token, deterministically modeling -/// the successor race without threads or sleeps. -class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend +/// Once armed, fences the admin's body right before its farewell lands, so the farewell conflicts and +/// `MountLeaseRenewer::terminate` returns without an outcome. Arm it after the victim closed, or the +/// victim's own farewell is the one fenced. +class FenceBeforeFarewellBackend : public InMemoryBackend { public: - void armForSuccessorRewrite() { armed = true; } + void arm() { armed = true; } + bool fired() const { return fenced; } - /// Injects on the `read` PRIMITIVE: the tombstone tail's owner read (`CasDecommission.cpp`) - /// goes through `CasOperation::read`, not the legacy `get`. - std::optional read(const String & key, TransportAccess & access) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - std::optional result = InMemoryBackend::read(key, access); - if (armed && epoch_deleted && !successor_injected && key == owner_key && result) + if (armed && !fenced && key == mount_key && expected_value && decodeMountLease(bytes).isFarewell()) { - successor_owner_bytes = encodeOwner(OwnerObject{ - .server_uuid = decodeOwner(result->bytes).server_uuid, - .retired_at_ms = std::nullopt, - }); - const auto put = InMemoryBackend::write(owner_key, successor_owner_bytes, result->value, access); - if (!put.has_value()) - throw std::runtime_error("owner-successor fixture: owner rewrite conflicted"); - successor_owner_value = *put; - successor_injected = true; + const auto current = InMemoryBackend::read(key, access); + if (!InMemoryBackend::write(key, encodeMountLease(decodeMountLease(current->bytes).fencedCopy()), current->value, access)) + throw std::runtime_error("fence-before-farewell fixture: the fence conflicted"); + fenced = true; } - return result; - } - - /// Injects on the `remove` PRIMITIVE: `deleteSlotObject`'s epoch delete (`CasDecommission.cpp`) - /// goes through `CasOperation::remove`, not the legacy `deleteExact`. - RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override - { - const RawRemoval result = InMemoryBackend::remove(key, expected_value, access); - if (armed && key == epoch_key && result == RawRemoval::Removed) - epoch_deleted = true; - return result; + return InMemoryBackend::write(key, bytes, expected_value, access); } - bool successorInjected() const { return successor_injected; } - const String & successorOwnerValue() const { return successor_owner_value; } - const String & successorOwnerBytes() const { return successor_owner_bytes; } - private: - inline static const String epoch_key = "p/gc/server-roots/victim/epoch"; - inline static const String owner_key = "p/gc/server-roots/victim/owner"; + inline static const String mount_key = "p/gc/server-roots/victim/mount"; bool armed = false; - bool epoch_deleted = false; - bool successor_injected = false; - String successor_owner_value; - String successor_owner_bytes; + bool fenced = false; }; -/// Seed one victim table with `committed` committed refs and `precommits` dangling precommit bindings, -/// via the raw ref-log seeding helpers (fixture idiom of e.g. `gtest_cas_gc_fold.cpp`: `writeManifestRaw` -/// + `publishCommittedTransition`/`addPrecommitTransition` against `victim`'s own backend/layout) -- this -/// fixture only needs the ref-table SHAPE `dropNamespace` erases, not a real build. Precommit bindings -/// are seeded at an artificially high `writer_epoch` so the writer's own stale-precommit sweep (armed -/// unconditionally by this table's recovery, unrelated to decommission -- spec §Clean Up Old Precommits) -/// never reclaims them, in its OWN separate transaction, ahead of `dropNamespace`'s removal. +/// Seed one victim table with `committed` committed refs and `precommits` dangling precommit bindings through +/// the raw ref-log helpers: only the ref-table shape `dropNamespace` erases, not a real build. Precommits +/// carry a high `writer_epoch`, so the writer's stale-precommit sweep never reclaims them in its own +/// transaction ahead of `dropNamespace`. void makeTableWithRefs(Pool & victim, const String & ns_str, uint64_t committed, uint64_t precommits) { const RootNamespace ns(ns_str); @@ -464,37 +333,131 @@ void makeTableWithRefs(Pool & victim, const String & ns_str, uint64_t committed, ASSERT_EQ(victim.listRefs(ns).size(), committed); } -/// Pre-precommit manifest debris: a staged manifest body under `ns_str`, at the store's own -/// `writer_epoch`, named by NO owner event -- a build the writer staged and never finished (fixture -/// idiom of `gtest_cas_orphan_manifest_sweep.cpp`'s `EligibleAndUnownedIsDeleted`). `build_sequence = 99` -/// is picked well clear of `makeTableWithRefs`'s own committed/precommit build sequences so it can never -/// collide with a real owned manifest key. Returns the seeded body's `ManifestId` so a caller can target -/// it (e.g. its exact object key) for further fixture setup. +/// Manifest debris: a staged manifest body under `ns_str` at the store's own `writer_epoch` that no owner +/// event names, a build staged and never finished. `build_sequence = 99` stays clear of `makeTableWithRefs`. ManifestId seedOrphanManifestBody(Pool & victim, const String & ns_str) { const RootNamespace ns(ns_str); const ManifestRef ref{.writer_epoch = victim.writerEpoch(), .build_sequence = 99, .manifest_ordinal = 1}; const ManifestId id = writeManifestRaw(*victim.poolBackendPtr(), victim.layout(), ns, ref, {}); - /// EXPECT, not ASSERT: this function returns a value now, and ASSERT_* expands to a bare `return;` - /// -- invalid in a non-void function. OperationForTest op(*victim.poolBackendPtr()); EXPECT_TRUE((*op).head(victim.layout().manifestKey(id), Retry::once()).has_value()); return id; } -/// THE MANIFEST-DEBRIS DRAIN NO LONGER DELETES, AND THE FIXTURES BELOW SAY SO RATHER THAN WORKING -/// AROUND IT. The drain goes through `sweepNamespace`, which is subject to the §6 deletion premise: a -/// manifest of an epoch-`E` build is deletable only once the namespace's sealed fold cursor sits in an -/// epoch STRICTLY above `E`. Every object in these fixtures -- the table's ref stream, the debris, and -/// the removal transaction decommission itself appends -- lives in ONE writer epoch, and a single-epoch -/// pool cannot satisfy that: any cursor high enough to clear the debris's epoch also sits above the -/// removal record, which would strip the tail-removal protection off the table's real manifests. The -/// two facts are mutually exclusive here, so there is no honest seeding that restores the deletions; -/// the tests assert retention, and the drain's reclaim path returns with registers R2/R3 (Stage B). +/// The manifest-debris drain deletes an epoch-`E` manifest only once the namespace's sealed fold cursor sits +/// above `E`. In these single-epoch fixtures no cursor can clear the debris without also clearing the removal +/// record, so the tests assert retention. } -TEST(CASDecommission, RefusesLiveMember) + +/// The pool "p" with the member "victim" every test decommissions; the helpers work on `victim_backend`. +class CASDecommission : public ::testing::Test +{ +protected: + const Layout layout{"p"}; + BackendPtr victim_backend; + + /// Opens the victim over `backend`, runs `prepare` on it and closes it; `backend` then backs every helper + /// below. + template + std::shared_ptr startVictim(std::shared_ptr backend = std::make_shared(), const std::function & prepare = {}) + { + victim_backend = backend; + auto victim = openVictim(backend); + if (prepare) + prepare(*victim); + return backend; + } + + static void oneTable(Pool & victim) { makeTableWithRefs(victim, "victim/db/t1", 1, 0); } + /// `oneTable` plus a staged manifest no ref names; returns the manifest's key. + static String oneTableWithDebris(Pool & victim) + { + oneTable(victim); + return victim.layout().manifestKey(seedOrphanManifestBody(victim, "victim/db/t1")); + } + + /// The owner, epoch and mount objects, bytes and incarnation. + std::vector>> controlObjects() const + { + OperationForTest op(*victim_backend); + std::vector>> objects; + for (const String & key : {layout.ownerKey("victim"), layout.epochKey("victim"), layout.mountKey("victim")}) + { + const auto got = (*op).read(key, Retry::once()); + objects.push_back(got ? std::optional(std::pair(got->bytes, got->etag)) : std::nullopt); + } + return objects; + } + + DecommissionReport run(const String & admin, PoolConfig config = {}) + { + config.pool_prefix = "p"; + config.server_root_id = admin; + return decommissionPoolMember(victim_backend, std::move(config), "victim"); + } + + /// The run's clock and retry sleep, and the admin's lease, on `clock`. The closures own the clock: the + /// admin's renewer may outlive the call that installed them. + DecommissionReport runOnClock(const String & admin, const std::shared_ptr & clock, PoolConfig config = {}) + { + config.pool_prefix = "p"; + config.server_root_id = admin; + return decommissionPoolMember(victim_backend, std::move(config), "victim", /*sink=*/{}, /*request_gc_round=*/{}, + [clock] { return clock->nowFn()(); }, [clock](uint64_t ms) { clock->sleepFn()(ms); }); + } + + OwnerObject owner() const { return readOwner(*victim_backend, layout, "victim"); } + MountLease slot() const { return readSlot(*victim_backend, layout, "victim"); } + + /// The owner fields a decommission decides: its life, whether it is retired, whether the retirement is verified. + void expectOwner(uint64_t life, bool retired, bool verified) const + { + const OwnerObject current = owner(); + EXPECT_EQ(current.owner_life, life); + EXPECT_EQ(current.retired_at_ms.has_value(), retired); + EXPECT_EQ(current.retirement_verified, verified); + } + + void seed(const String & key) const + { + OperationForTest op(*victim_backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, "x", Retry::once()))); + } + + bool exists(const String & key) const + { + OperationForTest op(*victim_backend); + return (*op).head(key, Retry::once()).has_value(); + } + + /// A fake clock far past any real host's boot time, so a split between it and the real boot clock fails. + static std::shared_ptr farClock() + { + auto clock = std::make_shared(); + clock->now = 1'000'000'000'000'000ULL; + return clock; + } + + /// A pool on real time with a 1 s lease renewed every 200 ms. + static PoolConfig realTimeConfig() + { + return PoolConfig{ + .pool_prefix = "p", + .server_root_id = "", + .mount_lease_ttl_ms = std::chrono::milliseconds(1000), + .mount_renew_period = std::chrono::milliseconds(200), + .cas_request_budget = CasRequestBudget{ + .attempt_timeout_ms = 100, .lease_safety_margin_ms = 100, .connect_timeout_cap_ms = std::nullopt}}; + } + + void fenceSlot() const { fenceOutMountForTest(*victim_backend, layout.mountKey("victim")); } + void tombstoneOwner() const { tombstoneOwnerForTest(*victim_backend, layout.ownerKey("victim")); } +}; + +TEST_F(CASDecommission, RefusesLiveMember) { auto backend = std::make_shared(); auto victim = openVictim(backend); /// keeps its mount lease unexpired — the member is alive @@ -508,7 +471,7 @@ TEST(CASDecommission, RefusesLiveMember) /// The refusal names the state a hard-killed member has to reach. `expired` in `system.cas_mounts` is /// wall-clock arithmetic; the claim trusts only a terminal slot (`terminated` or `fenced`), and an /// expired, unfenced lease takes exactly the branch an unexpired one takes here. -TEST(CASDecommission, NoWaitRefusalNamesTheTerminalStates) +TEST_F(CASDecommission, NoWaitRefusalNamesTheTerminalStates) { auto backend = std::make_shared(); auto victim = openVictim(backend); @@ -528,7 +491,7 @@ TEST(CASDecommission, NoWaitRefusalNamesTheTerminalStates) } } -TEST(CASDecommission, ClaimsDeadMemberAndBumpsEpoch) +TEST_F(CASDecommission, ClaimsDeadMemberAndBumpsEpoch) { auto backend = std::make_shared(); uint64_t victim_epoch = 0; @@ -544,7 +507,7 @@ TEST(CASDecommission, ClaimsDeadMemberAndBumpsEpoch) EXPECT_EQ(admin->poolConfig().server_root_id, "victim"); } -TEST(CASDecommission, AlwaysRenewsAdminClaimEvenWhenHostDiskIsObserveOnly) +TEST_F(CASDecommission, AlwaysRenewsAdminClaimEvenWhenHostDiskIsObserveOnly) { auto backend = std::make_shared(); { @@ -562,7 +525,19 @@ TEST(CASDecommission, AlwaysRenewsAdminClaimEvenWhenHostDiskIsObserveOnly) EXPECT_TRUE(admin->poolConfig().bounded_background_renewal); } -TEST(CASDecommission, RefusesUnknownMember) +TEST_F(CASDecommission, OpensOverARetiredOwner) +{ + auto backend = startVictim(); + tombstoneOwner(); + auto admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + ASSERT_TRUE(admin != nullptr); + /// The claim only reads: the owner stays retired at the life it had, and the admin holds the slot. + EXPECT_TRUE(owner().retired_at_ms.has_value()); + EXPECT_EQ(owner().owner_life, 2u); + EXPECT_EQ(slot().writer_epoch, admin->liveWriterEpoch()); +} + +TEST_F(CASDecommission, RefusesUnknownMember) { auto backend = std::make_shared(); expectThrowsCode(ErrorCodes::BAD_ARGUMENTS, [&] @@ -573,7 +548,7 @@ TEST(CASDecommission, RefusesUnknownMember) /// `SYSTEM CAS DROP POOL MEMBER 'a/b'` is refused before it touches the member, even when `a/b` owns its /// owner object: the drain selects objects by key prefix, so it is defined only for ids without `/`. -TEST(CASDecommission, RefusesIdWithSlash) +TEST_F(CASDecommission, RefusesIdWithSlash) { auto backend = std::make_shared(); const Layout layout("p"); @@ -596,10 +571,9 @@ TEST(CASDecommission, RefusesIdWithSlash) EXPECT_FALSE((*raw_op).head(layout.epochKey("shard-01/replica-a"), Retry::once()).has_value()); } -TEST(CASDecommission, SecondConcurrentDecommissionRefused) +TEST_F(CASDecommission, SecondConcurrentDecommissionRefused) { - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } + auto backend = startVictim(); auto first = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); expectThrowsCode(ErrorCodes::ABORTED, [&] @@ -608,11 +582,9 @@ TEST(CASDecommission, SecondConcurrentDecommissionRefused) }); } -TEST(CASDecommission, DuplicateLifeIdRefusesBeforeAnyNamespaceOrSlotMutation) +TEST_F(CASDecommission, DuplicateLifeIdRefusesBeforeAnyNamespaceOrSlotMutation) { - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - const Layout layout("p"); + auto backend = startVictim(); RefCatalog catalog; catalog.entries = { CatalogEntry{.ns = RootNamespace{"victim/a"}, .state = NsState::Live, .incarnation = UInt128{77}}, @@ -627,34 +599,17 @@ TEST(CASDecommission, DuplicateLifeIdRefusesBeforeAnyNamespaceOrSlotMutation) ASSERT_TRUE(empty_catalog); ASSERT_TRUE(std::holds_alternative((*raw_op).replace( layout.refCatalogKey(), encodeRefCatalog(catalog), empty_catalog->etag, Retry::once()))); - const auto owner_before = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); - const auto epoch_before = (*raw_op).read(layout.epochKey("victim"), Retry::once()); - const auto mount_before = (*raw_op).read(layout.mountKey("victim"), Retry::once()); - ASSERT_TRUE(owner_before); - ASSERT_TRUE(epoch_before); - ASSERT_TRUE(mount_before); - - EXPECT_THROW(decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"), DB::Exception); - const auto owner_after = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); - const auto epoch_after = (*raw_op).read(layout.epochKey("victim"), Retry::once()); - const auto mount_after = (*raw_op).read(layout.mountKey("victim"), Retry::once()); - ASSERT_TRUE(owner_after); - ASSERT_TRUE(epoch_after); - ASSERT_TRUE(mount_after); - EXPECT_EQ(owner_after->bytes, owner_before->bytes); - EXPECT_EQ(owner_after->etag, owner_before->etag); - EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); - EXPECT_EQ(epoch_after->etag, epoch_before->etag); - EXPECT_EQ(mount_after->bytes, mount_before->bytes); - EXPECT_EQ(mount_after->etag, mount_before->etag); + const auto before = controlObjects(); + + EXPECT_THROW(run("admin"), DB::Exception); + EXPECT_EQ(controlObjects(), before); } -TEST(CASDecommission, CatalogCutIsValidatedBeforeImpersonationAndReusedForSelection) +TEST_F(CASDecommission, CatalogCutIsValidatedBeforeImpersonationAndReusedForSelection) { auto backend = std::make_shared(); + victim_backend = backend; { auto victim = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); } - const Layout layout("p"); RefCatalog ambiguous; ambiguous.entries = { @@ -666,38 +621,20 @@ TEST(CASDecommission, CatalogCutIsValidatedBeforeImpersonationAndReusedForSelect .removal_started_round = 1}, }; - OperationForTest raw_op(*backend); - const auto owner_before = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); - const auto epoch_before = (*raw_op).read(layout.epochKey("victim"), Retry::once()); - const auto mount_before = (*raw_op).read(layout.mountKey("victim"), Retry::once()); - ASSERT_TRUE(owner_before); - ASSERT_TRUE(epoch_before); - ASSERT_TRUE(mount_before); + const auto before = controlObjects(); backend->armCatalogReplacement(layout.refCatalogKey(), std::move(ambiguous)); - EXPECT_THROW(decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"), DB::Exception); + EXPECT_THROW(run("admin"), DB::Exception); ASSERT_TRUE(backend->fired()); - const auto owner_after = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); - const auto epoch_after = (*raw_op).read(layout.epochKey("victim"), Retry::once()); - const auto mount_after = (*raw_op).read(layout.mountKey("victim"), Retry::once()); - ASSERT_TRUE(owner_after); - ASSERT_TRUE(epoch_after); - ASSERT_TRUE(mount_after); - EXPECT_EQ(owner_after->bytes, owner_before->bytes); - EXPECT_EQ(owner_after->etag, owner_before->etag); - EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); - EXPECT_EQ(epoch_after->etag, epoch_before->etag); - EXPECT_EQ(mount_after->bytes, mount_before->bytes); - EXPECT_EQ(mount_after->etag, mount_before->etag); + EXPECT_EQ(controlObjects(), before); } -TEST(CASDecommission, NamespaceSelectionUsesThePreImpersonationCut) +TEST_F(CASDecommission, NamespaceSelectionUsesThePreImpersonationCut) { auto backend = std::make_shared(); + victim_backend = backend; { auto victim = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); } - const Layout layout("p"); RefCatalog later; later.entries = { @@ -705,19 +642,18 @@ TEST(CASDecommission, NamespaceSelectionUsesThePreImpersonationCut) }; backend->armCatalogReplacement(layout.refCatalogKey(), std::move(later)); - const DecommissionReport report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const DecommissionReport report = run("admin"); ASSERT_TRUE(backend->fired()); EXPECT_EQ(report.namespaces_removed, 0u) << "a namespace visible only to mount safety's later observation is outside the validated cut"; EXPECT_EQ(report.namespaces_already_removed, 0u); } -TEST(CASDecommission, SameNameRebirthAfterTheCutIsRefusedWithoutTouchingTheNewLife) +TEST_F(CASDecommission, SameNameRebirthAfterTheCutIsRefusedWithoutTouchingTheNewLife) { auto backend = std::make_shared(); + victim_backend = backend; { auto victim = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); } - const Layout layout("p"); const RootNamespace ns{"victim/same"}; const NamespaceLifeId old_life = NamespaceLifeId::fromCatalogEntry(ns, UInt128{70}); const NamespaceLifeId new_life = NamespaceLifeId::fromCatalogEntry(ns, UInt128{71}); @@ -761,8 +697,7 @@ TEST(CASDecommission, SameNameRebirthAfterTheCutIsRefusedWithoutTouchingTheNewLi String refusal; try { - (void)decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + (void)run("admin"); } catch (const DB::Exception & e) { @@ -775,10 +710,9 @@ TEST(CASDecommission, SameNameRebirthAfterTheCutIsRefusedWithoutTouchingTheNewLi << "decommission must not append a removal transaction to the post-cut incarnation"; } -TEST(CASDecommission, VictimNameMatchesOneCanonicalPathComponent) +TEST_F(CASDecommission, VictimNameMatchesOneCanonicalPathComponent) { - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } + auto backend = startVictim(); const RootNamespace neighbor_ns{"victim2/db/t1"}; { @@ -786,8 +720,7 @@ TEST(CASDecommission, VictimNameMatchesOneCanonicalPathComponent) makeTableWithRefs(*neighbor, neighbor_ns.string(), /*committed=*/1, /*precommits=*/0); } - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); EXPECT_EQ(report.namespaces_removed, 0u); auto neighbor = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim2"}); @@ -795,9 +728,10 @@ TEST(CASDecommission, VictimNameMatchesOneCanonicalPathComponent) << "decommissioning victim must not select victim2 by raw string prefix"; } -TEST(CASDecommission, ErasesAllVictimNamespaces) +TEST_F(CASDecommission, ErasesAllVictimNamespaces) { auto backend = std::make_shared(); + victim_backend = backend; { auto victim = openVictim(backend); /// Two tables: ns "victim/db/t1" with 2 committed refs, ns "victim/db/t2" with 1 committed @@ -806,8 +740,7 @@ TEST(CASDecommission, ErasesAllVictimNamespaces) makeTableWithRefs(*victim, "victim/db/t2", /*committed=*/1, /*precommits=*/1); } - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); EXPECT_EQ(report.srid, "victim"); EXPECT_EQ(report.namespaces_removed, 2u); @@ -816,48 +749,34 @@ TEST(CASDecommission, ErasesAllVictimNamespaces) EXPECT_EQ(report.precommits_removed, 1u); EXPECT_EQ(report.edge_deltas_emitted, 4u); - /// Terminal publication is writer work; exact catalog-row deletion remains GC work. The first - /// command therefore keeps the slot as an ownership anchor, and a retry may retire it only after - /// GC's next invocation drains the completed `Removing` rows. - EXPECT_FALSE(report.warnings.empty()); - EXPECT_FALSE(report.slot_removed); + /// Terminal publication is writer work; exact catalog-row deletion remains GC work, reported as + /// pending rather than as a failure. + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.member_retired); + EXPECT_EQ(report.namespaces_pending_gc, 2u); drainCompletedNamespaceRemovals(backend); - const auto retired = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); - EXPECT_TRUE(retired.warnings.empty()); - EXPECT_TRUE(retired.slot_removed); -} - -/// Task 2 review finding 1: `makeTableWithRefs`'s precommit seed uses an artificially high -/// `writer_epoch` (999999) specifically to dodge the writer's OWN stale-precommit sweep -- which -/// means it never exercised the path a REAL victim precommit takes. A genuine writer stamps -/// `manifest_ref.writer_epoch` from its OWN `liveWriterEpoch()` at precommit time -/// (`PartWriteTxn::precommitAdd`, CasPool.cpp:2087), i.e. the victim's era -- always LOWER than the admin -/// mount's freshly-minted epoch (`openForDecommission` always bumps strictly higher). `appendRefOps` -/// hoists `maybeSweepStalePrecommits` at its top (CasPool.cpp:1716), so without the -/// `skip_stale_precommit_sweep` fix that sweep would reclaim this realistic-epoch precommit in its -/// OWN transaction before `dropNamespace`'s removal transaction ever counts it, leaving -/// `precommits_removed` at 0 for exactly the case that matters. -TEST(CASDecommission, CountsRealisticEpochPrecommit) + const auto rerun = run("admin2"); + EXPECT_TRUE(rerun.warnings.empty()); + EXPECT_TRUE(rerun.member_retired); + EXPECT_EQ(rerun.namespaces_pending_gc, 0u); + EXPECT_EQ(rerun.namespaces_already_removed, 0u); +} + +/// A real precommit carries the victim's own epoch, below the admin's. The writer's stale-precommit sweep +/// must not reclaim it in its own transaction before the namespace drop counts it. +TEST_F(CASDecommission, CountsRealisticEpochPrecommit) { - auto backend = std::make_shared(); - uint64_t victim_epoch = 0; + startVictim(std::make_shared(), [](Pool & victim) { - auto victim = openVictim(backend); - victim_epoch = victim->writerEpoch(); - makeTableWithRefs(*victim, "victim/db/t1", /*committed=*/1, /*precommits=*/0); - + oneTable(victim); const RootNamespace ns("victim/db/t1"); - /// `build_sequence = 2`: distinct from `makeTableWithRefs`'s committed ref (`build_sequence = 1`) - /// -- a REAL build's `ManifestRef` is unique per build, and a colliding one would trip the ref - /// state machine's "manifest already has a conflicting owner" guard. - const ManifestRef ref{.writer_epoch = victim_epoch, .build_sequence = 2, .manifest_ordinal = 1}; - writeManifestRaw(*victim->poolBackendPtr(), victim->layout(), ns, ref, {}); - addPrecommitTransition(*victim->poolBackendPtr(), victim->layout(), ns, UInt128(1), "precommit_0", std::nullopt, ref); - } + /// `build_sequence = 2`: a real build's `ManifestRef` is unique; the table's ref uses 1. + const ManifestRef ref{.writer_epoch = victim.writerEpoch(), .build_sequence = 2, .manifest_ordinal = 1}; + writeManifestRaw(*victim.poolBackendPtr(), victim.layout(), ns, ref, {}); + addPrecommitTransition(*victim.poolBackendPtr(), victim.layout(), ns, UInt128(1), "precommit_0", std::nullopt, ref); + }); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); EXPECT_EQ(report.namespaces_removed, 1u); EXPECT_EQ(report.committed_refs_removed, 1u); @@ -865,16 +784,9 @@ TEST(CASDecommission, CountsRealisticEpochPrecommit) EXPECT_EQ(report.edge_deltas_emitted, 2u); } -/// Task 2 review finding 2: the `member_decommission` begin/namespace_removed/end events -/// (CasDecommission.cpp) had no assertion at all. Wire a capturing sink (the `gtest_cas_event_log.cpp` -/// idiom) into `decommissionPoolMember` and check the emitted sequence and its per-namespace detail. -TEST(CASDecommission, EmitsMemberDecommissionEvents) +TEST_F(CASDecommission, EmitsMemberDecommissionEvents) { - auto backend = std::make_shared(); - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", /*committed=*/1, /*precommits=*/0); - } + auto backend = startVictim(std::make_shared(), oneTable); std::vector seen; (void)decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", @@ -893,463 +805,144 @@ TEST(CASDecommission, EmitsMemberDecommissionEvents) EXPECT_EQ(member_events[1].detail.at("precommits"), "0"); EXPECT_EQ(member_events[2].outcome, "end"); EXPECT_EQ(member_events[2].detail.at("namespaces_removed"), "1"); + EXPECT_EQ(member_events[2].detail.at("member_retired"), "1"); + EXPECT_EQ(member_events[2].detail.at("owner_life"), "2"); } -/// Task 3: the manifest-debris / staging / roots drain phases fill their three `DecommissionReport` -/// counters and leave nothing of the victim behind under `staging/` or `roots/`. -TEST(CASDecommission, DrainsDebrisStagingAndRoots) +/// The staging and mountpoint drains empty their prefixes even when the manifest-debris drain retains. +TEST_F(CASDecommission, DrainsDebrisStagingAndRoots) { - auto backend = std::make_shared(); - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - seedOrphanManifestBody(*victim, "victim/db/t1"); - } - /// Foreign staging + mountpoint objects, written raw (no writer machinery needed): the victim's - /// writers are fenced by the claim before decommission ever gets here, so these are ordinary debris, - /// not a live in-flight write. - { - OperationForTest seed_op(*backend); - (*seed_op).create("p/staging/victim/upload1.tmp", "x", Retry::once()); - (*seed_op).create("p/staging/victim/upload2.tmp", "x", Retry::once()); - (*seed_op).create("p/roots/victim/clickhouse_access_check_abc", "x", Retry::once()); - } + auto backend = startVictim(std::make_shared(), [](Pool & victim) { oneTableWithDebris(victim); }); + seed("p/staging/victim/upload1.tmp"); + seed("p/staging/victim/upload2.tmp"); + seed("p/roots/victim/clickhouse_access_check_abc"); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); - /// The staging and mountpoint phases are unchanged and still drain completely. The manifest-debris - /// phase retains under the §6 premise (see the note on the helpers above) and reports why, which is - /// what keeps the slot; `RetainsDebrisWhoseEpochSealIsUnconsumed` is that outcome's own test. + /// The manifest-debris retention is `RetainsDebrisWhoseEpochSealIsUnconsumed`'s subject. EXPECT_EQ(report.manifest_debris_removed, 0u); EXPECT_EQ(report.staging_objects_removed, 2u); EXPECT_EQ(report.mountpoint_objects_removed, 1u); - EXPECT_FALSE(report.warnings.empty()) - << "the retained debris is reported, so the incomplete drain is visible"; - - /// Nothing of the victim remains under staging/ or roots/ (scoped LISTs are empty). Those two phases - /// run to completion even though the debris phase retained -- the drain is per-phase, not all-or-nothing. + EXPECT_FALSE(report.warnings.empty()); OperationForTest raw_op(*backend); EXPECT_TRUE((*raw_op).list("p/staging/victim/", "", 10, Retry::once()).keys.empty()); EXPECT_TRUE((*raw_op).list("p/roots/victim/", "", 10, Retry::once()).keys.empty()); } -/// The §6 deletion premise applies to the decommission drain too, and this pins what that COSTS. With no -/// sealed fold cursor for the victim's namespace — the state of a pool whose GC has not folded past the -/// victim's closed epoch — the drain cannot show the debris is unreferenced, so it RETAINS it, says why -/// in `warnings`, and therefore keeps the slot for a later re-run. Delay, not damage: the objects are -/// untouched and a re-run after GC catches up drains them (`DrainsDebrisStagingAndRoots`). -/// -/// This is the visible edge of a real Stage-A limitation, not a test-only artifact: debris under a -/// namespace GC never folds — the pure pre-precommit orphan, whose whole point is that no ref record was -/// ever appended for it — has no cursor to consume any seal, so the premise retains it indefinitely. -/// Reclaiming it needs the sweep's own rework (registers R2/R3, Stage B), which is why the premise ships -/// as the safety floor and not as the reclaim policy. -TEST(CASDecommission, RetainsDebrisWhoseEpochSealIsUnconsumed) +/// With no sealed fold cursor past the victim's epoch the drain cannot show the debris is unreferenced: it +/// retains it, names it in a warning, and leaves the unverified tombstone for a rerun after GC catches up. +TEST_F(CASDecommission, RetainsDebrisWhoseEpochSealIsUnconsumed) { - auto backend = std::make_shared(); String debris_key; - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - const ManifestId debris_id = seedOrphanManifestBody(*victim, "victim/db/t1"); - debris_key = victim->layout().manifestKey(debris_id); - /// Deliberately NO `seedFoldedPastVictimEpoch` here — that absence is the subject. - } + startVictim(std::make_shared(), [&](Pool & victim) { debris_key = oneTableWithDebris(victim); }); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); EXPECT_EQ(report.manifest_debris_removed, 0u); - OperationForTest raw_op(*backend); - EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()) - << "the body is retained untouched, not deleted and not corrupted"; - ASSERT_FALSE(report.warnings.empty()) - << "a retained manifest is a visible decision -- the operator must be able to see why the drain " - "did not complete"; - bool named = false; - for (const String & w : report.warnings) - if (w.find(debris_key) != String::npos && w.find("seal") != String::npos) - named = true; - EXPECT_TRUE(named) << "the warning names the object and the premise that retained it"; - EXPECT_FALSE(report.slot_removed) - << "an incomplete drain keeps the slot as the resume anchor, exactly as a per-key failure does"; -} - -/// Task 3 fail-close nuance (spec §core "Fail-close"): a per-object failure in the staging/roots drain -/// -- a thrown exception (a transient hiccup) or a `TokenMismatch` outcome (a "listed but raced" miss) -/// -- must record a warning and let the rest of the sweep proceed, never abort the whole phase or the -/// whole command. One staging object throws, the roots object comes back `TokenMismatch`; the OTHER -/// staging object must still be deleted and counted. -TEST(CASDecommission, PerObjectFailureWarnsAndContinuesDrain) + EXPECT_TRUE(exists(debris_key)); + ASSERT_EQ(report.warnings.size(), 1u) << fmt::format("{}", fmt::join(report.warnings, "; ")); + EXPECT_NE(report.warnings.front().find(debris_key), String::npos) << report.warnings.front(); + EXPECT_FALSE(report.member_retired); + expectOwner(2, true, false); +} + +/// A per-object failure in the staging or roots drain, a throw or a `Mismatch`, is a warning and the rest of +/// the drain proceeds. +TEST_F(CASDecommission, PerObjectFailureWarnsAndContinuesDrain) { - auto backend = std::make_shared(); - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - } - { - OperationForTest seed_op(*backend); - (*seed_op).create("p/staging/victim/upload_ok.tmp", "x", Retry::once()); - (*seed_op).create("p/staging/victim/upload_throws.tmp", "x", Retry::once()); - (*seed_op).create("p/roots/victim/clickhouse_access_check_abc", "x", Retry::once()); - } + auto backend = startVictim(std::make_shared(), oneTable); + seed("p/staging/victim/upload_ok.tmp"); + seed("p/staging/victim/upload_throws.tmp"); + seed("p/roots/victim/clickhouse_access_check_abc"); backend->failWithThrow("p/staging/victim/upload_throws.tmp"); backend->latch(); backend->failWithTokenMismatch("p/roots/victim/clickhouse_access_check_abc"); - /// The engine reissues an unresolved delete until its own retry window closes, measured on this - /// clock, so the latched fault reaches a genuine give-up with no real time passing. - /// Heap-owned, not a plain stack local: `decommissionPoolMember` installs this clock into the - /// Pool's `boot_ms_fn` background mount-lease renewer, which can still be running on a detached - /// thread after this function returns, so a by-reference capture of a local would dangle. Wrapped - /// (rather than passing `clock->nowFn()`/`clock->sleepFn()` directly) so the closures stored in - /// `PoolConfig` hold the shared_ptr itself, not just the raw `FakeClock*` those methods capture. - auto clock = std::make_shared(); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, - [clock] - { - return clock->nowFn()(); - }, - [clock](uint64_t ms) - { - clock->sleepFn()(ms); - }); + /// The latched fault reaches the engine's give-up on the fake clock, with no real time passing. + auto clock = std::make_shared(); + const auto report = runOnClock("admin", clock); - EXPECT_EQ(report.staging_objects_removed, 1u) - << "the OTHER staging object must still be deleted despite the injected failure on its sibling"; + EXPECT_EQ(report.staging_objects_removed, 1u); EXPECT_EQ(report.mountpoint_objects_removed, 0u); - EXPECT_EQ(report.warnings.size(), 2u) - << "one warning for the thrown exception, one for the TokenMismatch outcome"; - - OperationForTest raw_op(*backend); - EXPECT_FALSE((*raw_op).head("p/staging/victim/upload_ok.tmp", Retry::once()).has_value()) - << "the healthy staging object was actually deleted, not merely skipped"; - EXPECT_TRUE((*raw_op).head("p/staging/victim/upload_throws.tmp", Retry::once()).has_value()) - << "the failing object is left behind (untouched) so a re-run can retry it"; - EXPECT_TRUE((*raw_op).head("p/roots/victim/clickhouse_access_check_abc", Retry::once()).has_value()) - << "TokenMismatch means nothing was actually deleted -- the object survives"; + EXPECT_EQ(report.warnings.size(), 2u); + EXPECT_FALSE(exists("p/staging/victim/upload_ok.tmp")); + EXPECT_TRUE(exists("p/staging/victim/upload_throws.tmp")); + EXPECT_TRUE(exists("p/roots/victim/clickhouse_access_check_abc")); } -/// Opaque physical debris carries no logical owner and therefore cannot widen or redirect -/// decommission's catalog-derived victim set. Task 5's ownership-tree janitor owns that debris. -TEST(CASDecommission, LifelessPhysicalKeyCannotRedirectCatalogOwnedDecommission) +/// Opaque physical debris carries no logical owner, so it cannot widen or redirect the catalog-derived victim +/// set; the namespace janitor owns it. +TEST_F(CASDecommission, LifelessPhysicalKeyCannotRedirectCatalogOwnedDecommission) { - auto backend = std::make_shared(); - String lifeless; - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - /// Hand-built: no helper can mint the un-incarnated shape any more. - lifeless = victim->layout().casRefsPrefix() + String("victim/db/t1/_log/") - + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; - OperationForTest seed_op(*backend); - ASSERT_TRUE(std::holds_alternative((*seed_op).create(lifeless, "garbage", Retry::once()))); - } + startVictim(std::make_shared(), oneTable); + /// Hand-built: no helper can mint the un-incarnated shape any more. + const String lifeless = layout.casRefsPrefix() + String("victim/db/t1/_log/") + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; + seed(lifeless); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); EXPECT_EQ(report.namespaces_removed, 1u); - OperationForTest raw_op(*backend); - EXPECT_TRUE((*raw_op).head(lifeless, Retry::once()).has_value()) - << "decommission must neither adopt nor delete an unowned physical life key"; + EXPECT_TRUE(exists(lifeless)); } -/// The manifest-debris drain honors the same tolerate-and-continue contract as -/// `deleteListedPrefix`: a per-key `deleteExact` failure becomes a warning, while the namespace -/// erasure and subsequent staging drain continue. Protection reads now use opaque physical life -/// prefixes, so a logical-name substring can no longer target an otherwise unlisted namespace. -TEST(CASDecommission, ManifestDebrisDeleteFailureWarnsAndContinues) +/// A per-key failure in the manifest-debris drain is a warning; the namespace drop and the staging drain go on. +TEST_F(CASDecommission, ManifestDebrisDeleteFailureWarnsAndContinues) { - auto backend = std::make_shared(); String debris_key; - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - const ManifestId debris_id = seedOrphanManifestBody(*victim, "victim/db/t1"); - debris_key = victim->layout().manifestKey(debris_id); - } + auto backend = startVictim(std::make_shared(), [&](Pool & victim) { debris_key = oneTableWithDebris(victim); }); backend->failWithThrow(debris_key); backend->latch(); - { - OperationForTest seed_op(*backend); - (*seed_op).create("p/staging/victim/upload_ok.tmp", "x", Retry::once()); - } - - /// The engine reissues an unresolved delete until its own retry window closes, measured on this - /// clock, so the latched fault reaches a genuine give-up with no real time passing. - /// Heap-owned, not a plain stack local: `decommissionPoolMember` installs this clock into the - /// Pool's `boot_ms_fn` background mount-lease renewer, which can still be running on a detached - /// thread after this function returns, so a by-reference capture of a local would dangle. Wrapped - /// (rather than passing `clock->nowFn()`/`clock->sleepFn()` directly) so the closures stored in - /// `PoolConfig` hold the shared_ptr itself, not just the raw `FakeClock*` those methods capture. - auto clock = std::make_shared(); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, - [clock] - { - return clock->nowFn()(); - }, - [clock](uint64_t ms) - { - clock->sleepFn()(ms); - }); - - EXPECT_EQ(report.namespaces_removed, 1u) - << "victim/db/t1's namespace erasure (Task 2) is untouched by either injected failure"; - EXPECT_EQ(report.manifest_debris_removed, 0u); - EXPECT_EQ(report.warnings.size(), 1u) - << "the thrown per-key delete must keep the retirement tail fail-closed"; - EXPECT_EQ(report.staging_objects_removed, 1u) - << "the staging phase still ran to completion after the manifest-debris phase's failures -- " - "the whole command did not abort"; - - OperationForTest raw_op(*backend); - EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()) - << "the failing object is left behind (untouched) so a re-run can retry it"; -} - -/// GC owns the completed catalog-row deletion. Once it drains the row, a clean decommission retry -/// removes the mutable slot objects and tombstones the owner anchor. -TEST(CASDecommission, RemovesMutableSlotAndRefusesTombstonedRerun) -{ - auto backend = std::make_shared(); - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - } - const auto pending = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); - EXPECT_FALSE(pending.slot_removed); - EXPECT_FALSE(pending.warnings.empty()); - drainCompletedNamespaceRemovals(backend); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); - EXPECT_TRUE(report.slot_removed); - EXPECT_TRUE(report.warnings.empty()); - OperationForTest raw_op(*backend); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()); - const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/epoch", Retry::once()).has_value()); - - expectThrowsCode(ErrorCodes::CORRUPTED_DATA, [&] - { - decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a2"}, "victim"); - }); -} - -/// Triage #9: a successor may reclaim the same UUID immediately after the decommission admin writes -/// its farewell. The retirement tail must use the farewell/claimed-epoch tokens captured around that -/// release, delete `mount` first, and stop on its `TokenMismatch`; re-reading current tokens would -/// delete the live successor's control objects and falsely report the slot removed. -TEST(CASDecommission, SuccessorReclaimFencesSlotRetirementTail) -{ - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - backend->armForSuccessorReclaim(); + seed("p/staging/victim/upload_ok.tmp"); - std::vector seen; - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", - [&](const CasEvent & event) { seen.push_back(event); }); - - ASSERT_TRUE(backend->successorInjected()); - EXPECT_FALSE(report.slot_removed); - ASSERT_EQ(report.warnings.size(), 1u); - EXPECT_NE(report.warnings.front().find("p/gc/server-roots/victim/mount"), String::npos); - EXPECT_NE(report.warnings.front().find("mismatch"), String::npos); - - OperationForTest raw_op(*backend); - const auto mount = (*raw_op).read("p/gc/server-roots/victim/mount", Retry::once()); - ASSERT_TRUE(mount.has_value()); - EXPECT_EQ(PersistedEtag::capture(mount->etag).value, backend->successorMountValue()); - EXPECT_EQ(mount->bytes, backend->successorMountBytes()); - - const auto epoch = (*raw_op).read("p/gc/server-roots/victim/epoch", Retry::once()); - ASSERT_TRUE(epoch.has_value()); - EXPECT_EQ(PersistedEtag::capture(epoch->etag).value, backend->successorEpochValue()); - EXPECT_EQ(epoch->bytes, backend->successorEpochBytes()); - EXPECT_TRUE((*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()).has_value()); - - ASSERT_FALSE(seen.empty()); - EXPECT_EQ(seen.back().outcome, "end"); - EXPECT_EQ(seen.back().detail.at("slot_removed"), "0"); -} - -/// A successor can also restart after both stale mutable objects were deleted but before `owner` is -/// retired. Mere presence of either freshly recreated mutable object must stop owner retirement. -TEST(CASDecommission, SuccessorReclaimAfterEpochDeleteKeepsOwnerAnchor) -{ - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - - const String owner_key = "p/gc/server-roots/victim/owner"; - OperationForTest raw_op(*backend); - const auto original_owner = (*raw_op).read(owner_key, Retry::once()); - ASSERT_TRUE(original_owner.has_value()); - backend->armForSuccessorReclaim(); - - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); - - ASSERT_TRUE(backend->successorInjected()); - EXPECT_FALSE(report.slot_removed); - EXPECT_FALSE(report.warnings.empty()); - EXPECT_EQ(backend->ownerRewriteAttempts(), 0u); + /// The latched fault reaches the engine's give-up on the fake clock, with no real time passing. + auto clock = std::make_shared(); + const auto report = runOnClock("admin", clock); - const auto owner = (*raw_op).read(owner_key, Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_EQ(owner->etag, original_owner->etag); - EXPECT_EQ(owner->bytes, original_owner->bytes); - - const auto mount = (*raw_op).read("p/gc/server-roots/victim/mount", Retry::once()); - ASSERT_TRUE(mount.has_value()); - EXPECT_EQ(PersistedEtag::capture(mount->etag).value, backend->successorMountValue()); - EXPECT_EQ(mount->bytes, backend->successorMountBytes()); - - const auto epoch = (*raw_op).read("p/gc/server-roots/victim/epoch", Retry::once()); - ASSERT_TRUE(epoch.has_value()); - EXPECT_EQ(PersistedEtag::capture(epoch->etag).value, backend->successorEpochValue()); - EXPECT_EQ(epoch->bytes, backend->successorEpochBytes()); + EXPECT_EQ(report.namespaces_removed, 1u); + EXPECT_EQ(report.manifest_debris_removed, 0u); + EXPECT_EQ(report.warnings.size(), 1u); + EXPECT_EQ(report.staging_objects_removed, 1u); + EXPECT_TRUE(exists(debris_key)); } -/// Triage #9 control: absent a successor interleaving, the fenced tail removes both mutable control -/// objects, tombstones the owner anchor, and preserves the existing successful `slot_removed=1` result. -TEST(CASDecommission, FencedSlotRetirementTailRetiresUncontendedSlot) -{ - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); - - EXPECT_TRUE(report.warnings.empty()); - EXPECT_TRUE(report.slot_removed); - OperationForTest raw_op(*backend); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/epoch", Retry::once()).has_value()); - const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); -} - -/// One decommission command spans several requests on the SAME open-fence engine: the -/// pre-impersonation catalog cut (before any `Pool` exists), the namespace drop's own catalog re-read, -/// and -- once the row it dropped is no longer owned -- the full retirement tail's reads, deletes and -/// final owner tombstone, all issued after `admin.reset()` destroys the `Pool`. Catalog-row deletion -/// is GC's job (`dropNamespace` only reaches `Removing`), so this is necessarily two commands: the -/// first proves the drop and its catalog read landed, the second (after GC folds the row) proves the -/// retirement tail's requests landed past the `Pool`'s own lifetime. -TEST(CASDecommission, RunsOnAnOpenFence) +/// A rerun over a verified retirement writes a fresh tombstone and mark of its own each time; the control +/// objects stay. +TEST_F(CASDecommission, RerunOverAVerifiedRetirementRetiresAgain) { - auto backend = std::make_shared(); - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - } - - const auto pending = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); - EXPECT_EQ(pending.namespaces_removed, 1u); - EXPECT_FALSE(pending.slot_removed); - EXPECT_FALSE(pending.warnings.empty()); - + auto backend = startVictim(std::make_shared(), oneTable); + const auto first = run("admin"); + EXPECT_TRUE(first.member_retired); + EXPECT_EQ(first.namespaces_pending_gc, 1u); drainCompletedNamespaceRemovals(backend); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); - EXPECT_TRUE(report.warnings.empty()); - EXPECT_TRUE(report.slot_removed); - OperationForTest raw_op(*backend); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/epoch", Retry::once()).has_value()); - const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); -} - -TEST(CASDecommission, SuccessfulDecommissionLeavesTombstonedOwnerAnchor) -{ - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - - const String owner_key = "p/gc/server-roots/victim/owner"; - OperationForTest raw_op(*backend); - const auto before = (*raw_op).read(owner_key, Retry::once()); - ASSERT_TRUE(before.has_value()); - EXPECT_FALSE(decodeOwner(before->bytes).retired_at_ms.has_value()); - - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto second = run("admin2"); + EXPECT_TRUE(second.member_retired); + EXPECT_TRUE(second.warnings.empty()); + EXPECT_EQ(second.namespaces_pending_gc, 0u); + EXPECT_TRUE(exists(layout.mountKey("victim"))); + EXPECT_TRUE(exists(layout.epochKey("victim"))); + expectOwner(5, true, true); - EXPECT_TRUE(report.warnings.empty()); - EXPECT_TRUE(report.slot_removed); - const auto after = (*raw_op).read(owner_key, Retry::once()); - ASSERT_TRUE(after.has_value()); - EXPECT_NE(after->etag, before->etag); - EXPECT_EQ(decodeOwner(after->bytes).server_uuid, decodeOwner(before->bytes).server_uuid); - EXPECT_TRUE(decodeOwner(after->bytes).retired_at_ms.has_value()); + const auto third = run("a2"); + EXPECT_TRUE(third.member_retired); + EXPECT_TRUE(third.warnings.empty()); + expectOwner(7, true, true); } -TEST(CASDecommission, SuccessorOwnerRewriteWinsBeforeTombstone) -{ - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - backend->armForSuccessorRewrite(); - - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); - - ASSERT_TRUE(backend->successorInjected()); - EXPECT_FALSE(report.slot_removed); - ASSERT_EQ(report.warnings.size(), 1u); - EXPECT_NE(report.warnings.front().find("successor reclaimed"), String::npos); - - OperationForTest raw_op(*backend); - const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_EQ(PersistedEtag::capture(owner->etag).value, backend->successorOwnerValue()); - EXPECT_EQ(owner->bytes, backend->successorOwnerBytes()); - EXPECT_FALSE(decodeOwner(owner->bytes).retired_at_ms.has_value()); -} - -/// Final whole-branch review finding (Important): -/// an ambiguous outcome on the owner tombstone write -- the write lands but its response is lost, the -/// same shape as a real SDK timeout after a landed write -- must not be reported as a hard failure. -/// `op.replace`'s own resolve read settles this (current bytes already match the intended tombstone) -/// and reports `Committed`. `injectAmbiguousLandedWrite` throws `Poco::TimeoutException` after -/// applying the write: the engine's write loop treats a `Poco::Exception` as a transport fault (never -/// a caller bug), which is exactly the class this scenario models -- a bare `std::runtime_error` here -/// would propagate unchanged instead of being resolved. -TEST(CASDecommission, OwnerTombstoneAmbiguousSuccessResolvesToCommitted) +/// A resolve read settles an ambiguous tombstone write: the bytes already match, so the tombstone is committed. +TEST_F(CASDecommission, OwnerTombstoneAmbiguousSuccessResolvesToCommitted) { - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } - backend->injectAmbiguousLandedWrite("p/gc/server-roots/victim/owner"); + auto backend = startVictim(); + backend->injectAmbiguousLandedWrite(layout.ownerKey("victim")); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const auto report = run("admin"); - EXPECT_TRUE(report.slot_removed) << "the ambiguous write actually landed and must resolve to Committed"; + EXPECT_TRUE(report.member_retired); EXPECT_TRUE(report.warnings.empty()); - - OperationForTest raw_op(*backend); - const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); + expectOwner(3, true, true); } -/// Delegates every op to `inner`, except `remove`: while `armed`, any key starting with `fail_prefix` -/// throws an injected transient failure instead of deleting -- models a real backend transiently -/// failing to delete under one whole prefix. `disarm()` clears the failure (the resume half of -/// `FailedDrainKeepsSlotThenResumes`). Forwards every pure-virtual `Backend` member (the `CasBackend.h` -/// list) to `inner` untouched. Injects on the `remove` PRIMITIVE, not the legacy `deleteExact`: -/// `deleteListedPrefix`'s mountpoint drain (`CasDecommission.cpp`) goes through `CasOperation::remove`. +/// Delegates every op to `inner`, except `remove`: while armed, a key under `fail_prefix` throws a transient +/// failure the engine reissues until it gives up. `disarm()` clears it. class FailDeletesUnderPrefixBackend : public Backend { public: @@ -1362,18 +955,12 @@ class FailDeletesUnderPrefixBackend : public Backend bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The transport primitives forward to `inner`, except `remove`, which is what this double - /// injects through. Declared because `Backend` declares them pure. std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { if (armed && key.starts_with(fail_prefix)) - /// `Poco::TimeoutException`, the class the request engine classifies as a transport fault - /// and reissues (`Retry::standard()`): this fixture models a real backend transiently - /// failing, and `armed` stays set across every reissue of the same call, so the engine - /// reaches its own retry deadline and gives up rather than recovering on a later attempt. throw Poco::TimeoutException("injected transient delete failure for " + key); return inner->remove(key, expected_value, access); } @@ -1394,261 +981,1964 @@ class FailDeletesUnderPrefixBackend : public Backend bool armed = true; }; -/// Task 4 fail-close: a drain failure under the roots prefix keeps the slot terminated-but-present -/// (`report.slot_removed == false`, the mount object survives as the resume anchor). Once the fault is -/// cleared, a re-run finishes the job: the already-erased namespace is counted as -/// `namespaces_already_removed`, the leftover roots object is finally swept, and the slot is removed. -TEST(CASDecommission, FailedDrainKeepsSlotThenResumes) +/// A drain failure under the roots prefix leaves this run's unverified tombstone. Once the fault is +/// cleared, a rerun resumes over it: the already-erased namespace is counted as +/// `namespaces_already_removed`, the leftover roots object is swept and the retirement is verified. +TEST_F(CASDecommission, FailedDrainLeavesAnUnverifiedTombstoneThenResumes) { - auto inner = std::make_shared(); - { - auto victim = Pool::open(inner, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - } - OperationForTest raw_op(*inner); - (*raw_op).create("p/roots/victim/loose_file", "x", Retry::once()); - + auto inner = startVictim(std::make_shared(), oneTable); + seed("p/roots/victim/loose_file"); auto failing = std::make_shared(inner, "p/roots/victim/"); - /// The engine reissues an unresolved delete until its own retry window closes, measured on this - /// clock, so the fault (armed across every reissue) reaches a genuine give-up with no real time - /// passing -- `clock` fast-forwards through the whole 90 s `Retry::standard()` window in one call. - /// `drain_now_fn` is also this session's boot clock (decommissionPoolMember unifies the two), so the - /// admin's own mount lease needs a TTL well past that fast-forward or the farewell it attempts on - /// the way out would refuse against a deadline the retry exhaustion already ran past -- a real - /// decommission's background renewer would have kept the deadline current over 90 real seconds, but - /// nothing here advances real time to let it. - auto clock = std::make_shared(); - const auto first = decommissionPoolMember( - failing, - PoolConfig{.pool_prefix = "p", .server_root_id = "a1", .mount_lease_ttl_ms = std::chrono::milliseconds(300'000)}, - "victim", /*sink=*/{}, /*request_gc_round=*/{}, - [clock] - { - return clock->nowFn()(); - }, - [clock](uint64_t ms) - { - clock->sleepFn()(ms); - }); + victim_backend = failing; + + /// The admin's boot clock is the run's clock, so its lease must outlast the retries' fast-forward. + auto clock = std::make_shared(); + const auto first = runOnClock( + "a1", clock, PoolConfig{.pool_prefix = "p", .server_root_id = "", .mount_lease_ttl_ms = std::chrono::milliseconds(300'000)}); EXPECT_FALSE(first.warnings.empty()); - EXPECT_FALSE(first.slot_removed); - EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) - << "slot kept -- resume anchor"; + EXPECT_FALSE(first.member_retired); + expectOwner(2, true, false); + EXPECT_TRUE(exists(layout.mountKey("victim"))); failing->disarm(); - const auto second = decommissionPoolMember( - failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a2"}, "victim"); - EXPECT_FALSE(second.warnings.empty()); - EXPECT_FALSE(second.slot_removed); - EXPECT_EQ(second.namespaces_already_removed, 1u); + const auto second = run("a2"); + EXPECT_TRUE(second.warnings.empty()); + EXPECT_TRUE(second.member_retired); EXPECT_EQ(second.mountpoint_objects_removed, 1u); + EXPECT_EQ(second.namespaces_already_removed, 1u); + EXPECT_EQ(second.namespaces_pending_gc, 1u); drainCompletedNamespaceRemovals(inner); - const auto third = decommissionPoolMember( - failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a3"}, "victim"); + const auto third = run("a3"); EXPECT_TRUE(third.warnings.empty()); - EXPECT_TRUE(third.slot_removed); + EXPECT_TRUE(third.member_retired); + EXPECT_EQ(third.namespaces_pending_gc, 0u); } -/// Task 4 fail-close, manifest-debris variant (review follow-up: the plan's own example only exercises -/// a roots-phase failure). A per-key `deleteExact` throw inside the manifest-debris drain must ALSO -/// keep the slot: `report.slot_removed == false`, the mount object survives, and once the injected -/// failure is cleared a re-run drains the leftover debris and removes the slot. -TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) +/// A per-key throw inside the manifest-debris drain leaves the unverified tombstone too. The debris stays +/// retained once the fault is cleared (no sealed fold cursor), so the rerun is not verified either. +TEST_F(CASDecommission, ManifestDebrisFailureLeavesAnUnverifiedTombstone) { - auto backend = std::make_shared(); String debris_key; - { - auto victim = openVictim(backend); - makeTableWithRefs(*victim, "victim/db/t1", 1, 0); - const ManifestId debris_id = seedOrphanManifestBody(*victim, "victim/db/t1"); - debris_key = victim->layout().manifestKey(debris_id); - } + auto backend = startVictim(std::make_shared(), [&](Pool & victim) { debris_key = oneTableWithDebris(victim); }); backend->failWithThrow(debris_key); backend->latch(); - /// The engine reissues an unresolved delete until its own retry window closes, measured on this - /// clock, so the latched fault reaches a genuine give-up with no real time passing. - auto clock = std::make_shared(); - const auto first = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, - [clock] - { - return clock->nowFn()(); - }, - [clock](uint64_t ms) - { - clock->sleepFn()(ms); - }); + auto clock = std::make_shared(); + const auto first = runOnClock("a1", clock); EXPECT_FALSE(first.warnings.empty()); - EXPECT_FALSE(first.slot_removed); + EXPECT_FALSE(first.member_retired); EXPECT_EQ(first.manifest_debris_removed, 0u); - OperationForTest raw_op(*backend); - EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) - << "slot kept -- resume anchor"; - EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()) - << "the failing object is left behind (untouched) so a re-run can retry it"; - - /// COVERAGE LOST HERE, DELIBERATELY NAMED. Before the §6 premise, clearing the injected failure let - /// a re-run drain the debris and retire the slot, which is what proved the per-key fail-close path - /// RESUMES rather than merely refuses. Under the premise the sweep never reaches `deleteExact` for - /// this body at all (single-epoch pool -- see the note on the helpers above), so disarming changes - /// nothing and the resume half of this test is no longer expressible. What survives is the half that - /// still has a mechanism: the slot stays kept across the re-run, and the object stays untouched. - /// The resume assertion comes back with the drain's reclaim path (registers R2/R3, Stage B). + expectOwner(2, true, false); + EXPECT_TRUE(exists(debris_key)); + backend->disarm(); - const auto second = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a2"}, "victim"); + const auto second = run("a2"); EXPECT_EQ(second.namespaces_already_removed, 1u); EXPECT_EQ(second.manifest_debris_removed, 0u); - EXPECT_FALSE(second.slot_removed); - EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()); - EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) - << "the slot is still the resume anchor -- nothing was retired against unreclaimed debris"; -} - -/// CI run 7 (PR #2300): `drain_now_fn` fakes the drain's own request clock, but the mount lease's -/// farewell deadline is bound to `PoolConfig::boot_ms_fn`, a distinct clock that -- before this test's -/// fix -- stayed on the real boot clock regardless. On a freshly booted CI host (real boot time below -/// `FakeClock`'s starting instant) the two clocks disagreed enough that the farewell's own bound looked -/// already-expired, so `admin.reset()` released nothing and the slot was never retired -- passing locally -/// only because a long-lived dev box's real uptime dwarfs the fake clock. Pin the fake clock far beyond -/// ANY real host's boot time (rather than relying on the host actually being fresh) so the mismatch -- -/// and the fix -- are exercised deterministically on every machine. -TEST(CASDecommission, DrainClockUnifiesWithTheFarewellBootClockRegardlessOfHostUptime) + EXPECT_FALSE(second.member_retired); + expectOwner(3, true, false); + EXPECT_TRUE(exists(debris_key)); + EXPECT_TRUE(exists(layout.mountKey("victim"))); +} + +/// The farewell deadline is bound to the admin's boot clock, which the run's fake clock must drive too. A fake +/// clock far beyond any real host's boot time makes a split between the two clocks fail on every machine. +TEST_F(CASDecommission, DrainClockUnifiesWithTheFarewellBootClockRegardlessOfHostUptime) { - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } /// identity only -- no namespace, so retirement runs straight - /// to the farewell instead of stopping on an unrelated warning - - auto clock = std::make_shared(); - clock->now = 1'000'000'000'000'000ULL; /// dwarfs any real CLOCK_BOOTTIME on any host - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, - [clock] - { - return clock->nowFn()(); - }, - [clock](uint64_t ms) - { - clock->sleepFn()(ms); - }); + startVictim(); + auto clock = farClock(); + const auto report = runOnClock("a1", clock); EXPECT_TRUE(report.warnings.empty()); - EXPECT_TRUE(report.slot_removed); - OperationForTest raw_op(*backend); - EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) - << "one clock for both the request engine and the mount lease -- the farewell must commit and the " - "slot must retire in a single call, with no leftover mount to resume against"; -} - -/// Review round 9r2: folding `drain_now_fn` into `config.boot_ms_fn` alone left `Pool::openForDecommission`'s -/// own mount/farewell/GC planes -- constructed and used DURING opening, before `decommissionPoolMember` -/// gets a chance to call `setCasRequestNowFnForTest`/`setCasRetrySleepForTest` on the already-open `Pool` -/// -- retrying on a clock that only a fake SLEEP advances, while those planes still slept for REAL between -/// attempts. A transient failure during opening (the owner-object read on the open plane) would then never -/// see its own `Retry::standard()` deadline elapse, because the bound is measured against a clock frozen -/// at the value it had when the retry loop started: nothing calls the fake clock's `sleepFn` from a path -/// that still sleeps for real. `PoolConfig::retry_sleep_fn` closes this by installing the matching fake -/// sleep on those same planes AT CONSTRUCTION, together with `boot_ms_fn`. -/// -/// Failing-first without risking an actual hang: this fixture flakes the owner read 5 times, so a correct -/// fix paces exactly 5 retries on the fake clock and returns in well under a second of real time; the -/// pre-fix code either never returns (the bound never elapses) or, if it did return, would show an empty -/// `clock.sleeps` (nothing ever called the fake sleep) and real wall time consumed by 5 real backoffs. -TEST(CASDecommission, OpeningRetriesPaceOnTheSameFakeClockAndSleepAsTheDrain) -{ - auto backend = std::make_shared(); - { auto victim = openVictim(backend); } /// identity only + EXPECT_TRUE(report.member_retired); + EXPECT_TRUE(slot().isFarewell()); +} - const Layout layout("p"); +/// The planes `Pool::openForDecommission` builds retry during opening, before the run installs its clock on +/// the pool; they must pace on the fake sleep from construction, or a retry deadline on the frozen clock +/// never elapses. +TEST_F(CASDecommission, OpeningRetriesPaceOnTheSameFakeClockAndSleepAsTheDrain) +{ + auto backend = startVictim(std::make_shared()); backend->failReadNTimes(layout.ownerKey("victim"), /*times=*/5); - auto clock = std::make_shared(); - clock->now = 1'000'000'000'000'000ULL; + auto clock = farClock(); const auto started = std::chrono::steady_clock::now(); - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, - [clock] + const auto report = runOnClock("a1", clock); + const auto wall_elapsed = std::chrono::steady_clock::now() - started; + + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.member_retired); + EXPECT_TRUE(slot().isFarewell()); + EXPECT_FALSE(clock->sleeps.empty()); + EXPECT_LT(wall_elapsed, std::chrono::seconds(5)); +} + +/// The victim restarts during its own decommission: its owner check passes before the tombstone, its mount +/// claim waits on the admin's live lease, the admin tombstones and farewells during that wait, the victim +/// reclaims the farewell and its recheck sees the tombstone. +TEST_F(CASDecommission, VictimClaimingAfterTheTombstoneFencesItselfAndFails) +{ + auto backend = startVictim(); + const UInt128 victim_uuid = owner().server_uuid; + auto admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + const uint64_t admin_epoch = admin->writerEpoch(); + + auto fake_boot = std::make_shared>(0); + bool decommissioned = false; + PoolConfig victim_config{ + .pool_prefix = "p", .server_root_id = "victim", + .boot_ms_fn = [fake_boot] { return fake_boot->load(); }, + .wait_sleep_fn = [&, fake_boot](uint64_t ms) { - return clock->nowFn()(); - }, - [clock](uint64_t ms) + *fake_boot += ms; + if (std::exchange(decommissioned, true)) + return; + tombstoneOwner(); + admin.reset(); /// the clean farewell the victim's next poll reclaims + }}; + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "decommissioned", [&] { Pool::open(backend, victim_config); }); + ASSERT_TRUE(decommissioned); + const MountLease body = slot(); + EXPECT_EQ(body.server_uuid, victim_uuid); + EXPECT_GT(body.writer_epoch, admin_epoch); + EXPECT_TRUE(body.gc_fenced); + EXPECT_TRUE(owner().retired_at_ms.has_value()); +} + + +namespace +{ +/// While armed, `_pool_meta` cannot be probed: an admin self-remount stops at its identity probe and retries +/// on a backoff the teardown interrupts. +class PoolMetaUnreadableBackend : public InMemoryBackend +{ +public: + std::atomic armed = false; + + std::optional head(const String & key, TransportAccess & access) override + { + if (armed && key == Layout("p").poolMetaKey()) + throw Poco::TimeoutException("pool meta fixture: unreadable"); + return InMemoryBackend::head(key, access); + } +}; +} + +/// The tombstone's first attempt goes unanswered and the admin's lease deadline passes before the retry. +/// The retry is refused locally, so a victim restarted after the deadline runs under the life it reads. +TEST_F(CASDecommission, TombstoneRetryAfterTheLeaseDeadlineIsNotSent) +{ + auto backend = startVictim(std::make_shared()); + auto clock = farClock(); + bool stalled = false; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (std::exchange(stalled, true)) + return; + /// The lapsed lease sends the admin into a self-remount; keep it from racing the teardown. + backend->armed = true; + clock->now += 31'000; /// past the default 30 s TTL + backend->injectAmbiguousWrite(layout.ownerKey("victim")); + }); + + EXPECT_THROW(runOnClock("a1", clock), DB::Exception); + + ASSERT_TRUE(stalled); + backend->armed = false; + expectOwner(1, false, false); + auto restarted = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim", .unsafe_remount_no_delay = true}); + EXPECT_TRUE(restarted->mayMutate()); + expectOwner(1, false, false); +} + +/// A local fault after the tombstone went out leaves its fate unknown, and the read-back finds the old owner: +/// the store may still apply the tombstone. The admin leaves its live body instead of a clean farewell, so a +/// restarted victim waits for a fence before it runs. +TEST_F(CASDecommission, UnresolvedTombstoneLeavesNoFarewell) +{ + auto backend = startVictim(); + const UInt128 victim_uuid = owner().server_uuid; + bool sent = false; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (!std::exchange(sent, true)) + throw std::runtime_error("fixture: a local fault after the tombstone was sent"); + }); + + EXPECT_THROW(runOnClock("a1", farClock()), std::runtime_error); + + ASSERT_TRUE(sent); + expectOwner(1, false, false); + const MountLease left = slot(); + EXPECT_EQ(left.server_uuid, victim_uuid); + EXPECT_FALSE(left.isFarewell()); + EXPECT_FALSE(left.gc_fenced); + + auto fake_boot = std::make_shared>(0); + bool fenced = false; + PoolConfig victim_config{ + .pool_prefix = "p", .server_root_id = "victim", + .boot_ms_fn = [fake_boot] { return fake_boot->load(); }, + .wait_sleep_fn = [&, fake_boot](uint64_t ms) { - clock->sleepFn()(ms); - }); - const auto wall_elapsed = std::chrono::steady_clock::now() - started; + *fake_boot += ms; + if (!std::exchange(fenced, true)) + fenceSlot(); + }}; + auto restarted = Pool::open(backend, victim_config); + EXPECT_TRUE(fenced) << "the victim reclaimed the slot without waiting"; + EXPECT_TRUE(restarted->mayMutate()); + EXPECT_GT(restarted->writerEpoch(), left.writer_epoch); +} + +/// The owner is rewritten between the claim and the tombstone: the tombstone's precondition fails, the run +/// aborts, the rewrite stays and nothing is drained. +TEST_F(CASDecommission, OwnerRewrittenBeforeTheTombstoneAbortsTheRun) +{ + auto backend = startVictim(); + seed("p/roots/victim/loose_file"); + String rewritten; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (!rewritten.empty()) + return; + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + OwnerObject owner = decodeOwner(current->bytes); + owner.owner_life += 1; + rewritten = encodeOwner(owner); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.ownerKey("victim"), rewritten, current->etag, Retry::once()))); + }); + + expectThrowsCode(ErrorCodes::ABORTED, [&] { run("a1"); }); + + ASSERT_FALSE(rewritten.empty()); + expectBytes(*backend, layout.ownerKey("victim"), rewritten); + EXPECT_TRUE(exists("p/roots/victim/loose_file")); +} + +/// A member last written by 26.6.4: owner without a life, counter without a nonce, its clean farewell. A +/// decommission that loses its claim rolls back to life 2, and the member restarts under it. +TEST_F(CASDecommission, A2664MemberRollsBackAndRestarts) +{ + auto backend = startVictim(); + const String hex = u128ToHex(owner().server_uuid); + { + OperationForTest op(*backend); + const auto put = [&](const String & key, const String & bytes) + { + const auto current = (*op).read(key, Retry::once()); + ASSERT_TRUE(current.has_value()); + ASSERT_TRUE(std::holds_alternative((*op).replace(key, bytes, current->etag, Retry::once()))); + }; + put(layout.ownerKey("victim"), "{\"type\":\"cas_owner\",\"v\":1}\n{\"server_uuid\":\"" + hex + "\"}\n"); + put(layout.epochKey("victim"), "{\"type\":\"cas_epoch\",\"v\":1}\n{\"next_writer_epoch\":\"5\"}\n"); + put(layout.mountKey("victim"), "{\"type\":\"cas_mount_lease\",\"v\":1}\n{\"server_uuid\":\"" + hex + + "\",\"writer_epoch\":\"4\",\"hostname\":\"h\",\"pid\":1,\"started_at_ms\":1,\"seq\":\"3\",\"expires_at_ms\":1," + "\"min_active_build_sequence\":\"18446744073709551615\",\"gc_fenced\":false," + "\"write_attempt_id\":\"0000000000000000000000000000000a\"}\n"); + } + bool fenced = false; + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (!std::exchange(fenced, true)) + fenceSlot(); + }); + + const auto report = run("a1"); + + ASSERT_TRUE(fenced); + EXPECT_FALSE(report.member_retired); + expectOwner(2, false, false); /// 26.6.4 life 0, tombstone 1, rollback 2 + auto restarted = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim", .unsafe_remount_no_delay = true}); + EXPECT_GT(restarted->writerEpoch(), 5u); + expectOwner(2, false, false); +} + +TEST_F(CASDecommission, KeepsEpochAndFarewellMount) +{ + uint64_t victim_epoch = 0; + auto backend = startVictim(std::make_shared(), [&](Pool & victim) { victim_epoch = victim.writerEpoch(); }); + const UInt128 victim_uuid = owner().server_uuid; + + const auto report = run("admin"); EXPECT_TRUE(report.warnings.empty()); - EXPECT_TRUE(report.slot_removed); - EXPECT_FALSE(clock->sleeps.empty()) - << "the opening retries must have been paced on the injected clock, not a real sleep"; - EXPECT_LT(wall_elapsed, std::chrono::seconds(5)) - << "paced on the fake clock, five retries during opening should cost no real wall time at all"; -} - -/// Task 5 (Task-1 carry-forward, escalated by review): preserve recovery from the legacy partial -/// hand-cleanup shape where owner and epoch are absent but the mount lease remains. Triage #9 changed -/// new retirements to delete `mountKey`/`epochKey` and tombstone `ownerKey`, so the current tail no -/// longer creates this shape, but `openForDecommission`'s owner-anchor-absent + -/// mount-lease-present fallback ("partial hand-cleanup: adopt from the lease", `CasPool.cpp`) remains -/// compatibility-critical for slots left by older binaries or manual repair. -/// -/// `claimOwnerOrThrow` (`CasServerRoot.cpp`) gates the owner-absent path a SECOND, stricter way: the -/// same catalog cut must name no `Creating`, `Live` or `Removing` namespace under this canonical root, -/// and the name-bearing `cas/manifests//` and `roots//` families must be empty. Opaque -/// stream/state debris cannot be attributed to a server root and is deliberately inert. This test -/// therefore uses a victim with NO namespaces at all: identity persisted -/// (mount/owner/epoch exist from a real graceful close), data subtree genuinely empty -- the exact -/// precondition the fallback is designed for. Simulate the crash directly: claim the slot once (exactly -/// `decommissionPoolMember`'s own first step), let it close gracefully (the mount-lease renewer's -/// farewell stamp, same as a real `admin.reset()`), then manually strike `epochKey`+`ownerKey`, leaving -/// `mountKey`. A `decommissionPoolMember` re-run must resolve identity via the mount-lease fallback and -/// finish retiring the slot; a further re-run then sees the tombstone and refuses to resume it. -TEST(CASDecommission, MidRetirementCrashResumesViaMountLeaseFallback) + EXPECT_TRUE(report.member_retired); + const MountLease body = slot(); + EXPECT_EQ(body.writer_epoch, victim_epoch + 1) << "the admin's own clean farewell stays in the slot"; + EXPECT_TRUE(body.isFarewell()); + EXPECT_FALSE(body.gc_fenced); + OperationForTest raw_op(*backend); + const auto epoch = (*raw_op).read(layout.epochKey("victim"), Retry::once()); + ASSERT_TRUE(epoch.has_value()); + EXPECT_EQ(decodeServerEpoch(epoch->bytes).next_writer_epoch, victim_epoch + 2) << "the counter is never deleted"; + expectOwner(3, true, true); /// claim 1, tombstone 2, verified mark 3 + EXPECT_EQ(owner().server_uuid, victim_uuid); +} + +/// The admin is GC-fenced between its claim and its tombstone: the confirm sees the fence and the +/// tombstone is rolled back to a fresh, unretired life. +TEST_F(CASDecommission, FenceBeforeTombstoneUntombstonesAndFails) +{ + auto backend = startVictim(); + seed("p/roots/victim/loose_file"); + bool fired = false; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (std::exchange(fired, true)) + return; + fenceSlot(); + }); + + const auto report = run("admin"); + + ASSERT_TRUE(fired); + EXPECT_FALSE(report.member_retired); + EXPECT_EQ(report.warnings.size(), 1u) << fmt::format("{}", fmt::join(report.warnings, "; ")); + expectOwner(3, false, false); /// claim 1, tombstone 2, rollback 3 + EXPECT_TRUE(exists("p/roots/victim/loose_file")) << "nothing was drained"; +} + +/// A renewal between the claim and the confirm changes the ETag and the attempt id, and the confirm must not +/// care; a fence, a farewell, another epoch or another uuid is a loss. +TEST_F(CASDecommission, ConfirmAcceptsARenewalBetweenClaimAndConfirm) { auto backend = std::make_shared(); - { auto victim = openVictim(backend); } /// identity only -- no namespace, so the subtree stays empty + OperationForTest op(*backend); + MountLease body{.server_uuid = UInt128(7), .writer_epoch = 5, .hostname = "h", .pid = 1, + .started_at_ms = 1, .seq = 1, .expires_at_ms = 2, .min_active_build_sequence = 0, + .gc_fenced = false, .write_attempt_id = UInt128(11)}; + ASSERT_TRUE(std::holds_alternative((*op).create(layout.mountKey("v"), encodeMountLease(body), Retry::once()))); + EXPECT_TRUE(adminClaimStillHeld(*op, layout, "v", UInt128(7), 5)); + + MountLease renewed = body; + renewed.seq = 2; + renewed.write_attempt_id = UInt128(12); + ASSERT_TRUE(std::holds_alternative((*op).replace(layout.mountKey("v"), encodeMountLease(renewed), + (*op).read(layout.mountKey("v"), Retry::once())->etag, Retry::once()))); + EXPECT_TRUE(adminClaimStillHeld(*op, layout, "v", UInt128(7), 5)); + + EXPECT_FALSE(adminClaimStillHeld(*op, layout, "v", UInt128(7), 6)) << "another epoch"; + EXPECT_FALSE(adminClaimStillHeld(*op, layout, "v", UInt128(8), 5)) << "another uuid"; + ASSERT_TRUE(std::holds_alternative((*op).replace(layout.mountKey("v"), encodeMountLease(renewed.fencedCopy()), + (*op).read(layout.mountKey("v"), Retry::once())->etag, Retry::once()))); + EXPECT_FALSE(adminClaimStillHeld(*op, layout, "v", UInt128(7), 5)) << "fenced"; + MountLease farewell = renewed; + farewell.min_active_build_sequence = std::numeric_limits::max(); + ASSERT_TRUE(std::holds_alternative((*op).replace(layout.mountKey("v"), encodeMountLease(farewell), + (*op).read(layout.mountKey("v"), Retry::once())->etag, Retry::once()))); + EXPECT_FALSE(adminClaimStillHeld(*op, layout, "v", UInt128(7), 5)) << "farewelled"; + EXPECT_FALSE(adminClaimStillHeld(*op, layout, "absent", UInt128(7), 5)); +} - const Layout layout("p"); - /// Claim the slot once, exactly as `decommissionPoolMember`'s own first step would -- this (re)writes - /// fresh epoch/owner/mount control objects. Closing gracefully (scope exit) stamps the mount lease's - /// farewell, matching what a real slot retirement's `admin.reset()` does right before its delete loop. +/// Another server took the server_root_id between the admin's claim and its tombstone (a rejoin). The +/// tombstone stops before it writes anything. +TEST_F(CASDecommission, StopsWhenTheOwnerUuidChanged) +{ + auto backend = startVictim(std::make_shared()); + backend->arm(); + seed("p/roots/victim/loose_file"); + + expectThrowsCode(ErrorCodes::ABORTED, [&] { run("admin"); }); + + ASSERT_TRUE(backend->fired()); + EXPECT_EQ(owner().server_uuid, UInt128(0xBEEF)); + EXPECT_FALSE(owner().retired_at_ms.has_value()) << "the new owner was not tombstoned"; + EXPECT_TRUE(exists("p/roots/victim/loose_file")) << "nothing was drained"; +} + +/// An absent counter is refused, whether or not an owner is left, and never re-created; nothing is written. +TEST_F(CASDecommission, AbsentCounterIsRefused) +{ + for (const bool owner_left : {true, false}) { - auto admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "chk"}, "victim"); + SCOPED_TRACE(owner_left); + startVictim(); + OperationForTest raw_op(*victim_backend); + ASSERT_EQ((*raw_op).removeCurrent(layout.epochKey("victim"), Retry::once()), Removal::Removed); + if (!owner_left) + ASSERT_EQ((*raw_op).removeCurrent(layout.ownerKey("victim"), Retry::once()), Removal::Removed); + const auto before = controlObjects(); + + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "no epoch counter", [&] { run("admin"); }); + + EXPECT_EQ(controlObjects(), before); } +} - /// Manually strike epoch + owner, leaving the mount -- the legacy partial hand-cleanup shape. +/// The owner was deleted by hand while the counter survived: the run recovers the uuid from the lease, +/// re-creates the owner over the empty subtree and retires it. +TEST_F(CASDecommission, OwnerDeletedByHandIsRecreatedFromTheLease) +{ + auto backend = startVictim(); OperationForTest raw_op(*backend); - for (const String & key : {layout.epochKey("victim"), layout.ownerKey("victim")}) + ASSERT_EQ((*raw_op).removeCurrent(layout.ownerKey("victim"), Retry::once()), Removal::Removed); + + const auto report = run("a2"); + + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.member_retired); + expectOwner(3, true, true); + EXPECT_TRUE(exists(layout.epochKey("victim"))); +} + +TEST_F(CASDecommission, UnconfirmedFarewellIsNotRetired) +{ + auto backend = startVictim(std::make_shared()); + backend->arm(); + DB::Cas::tests::ScopedLoggerCapture capture("CasDecommission", "warning"); + + const auto report = run("admin"); + + ASSERT_TRUE(backend->fired()); + EXPECT_FALSE(report.member_retired); + ASSERT_FALSE(report.warnings.empty()); + for (const String & warning : report.warnings) + EXPECT_NE(capture.captured().find(warning), String::npos) << "not logged: " << warning; + expectOwner(2, true, false); + EXPECT_TRUE(slot().gc_fenced); +} + +/// The mark is written on the control plane after the admin pool is gone, and does not wait for GC to delete +/// the victim's catalog rows. +TEST_F(CASDecommission, VerifiedMarkDoesNotWaitForCatalogGc) +{ + auto backend = startVictim(std::make_shared(), oneTable); + const auto report = run("admin"); + EXPECT_EQ(report.namespaces_removed, 1u); + EXPECT_EQ(report.namespaces_pending_gc, 1u); + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.member_retired); + expectOwner(3, true, true); +} + +/// A rerun over an earlier retirement, verified or not, loses its claim after its tombstone: nothing is rolled +/// back and the owner stays retired. An unverified tombstone landed while its run held the claim, so no +/// member can be running under it. +TEST_F(CASDecommission, RollbackKeepsAPreviousRetirement) +{ + for (const bool verified : {true, false}) { - const auto head = (*raw_op).head(key, Retry::once()); - ASSERT_TRUE(head.has_value()); - ASSERT_EQ((*raw_op).remove(key, head->etag, Retry::once()), Removal::Removed); + SCOPED_TRACE(verified); + auto backend = startVictim(); + if (verified) + ASSERT_TRUE(run("a1").member_retired); + else + tombstoneOwner(); + const uint64_t life_before = owner().owner_life; + + bool fired = false; + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (std::exchange(fired, true)) + return; + fenceSlot(); /// the rerun's tombstone landed; its claim is lost now + }); + const auto rerun = run("a2"); + + ASSERT_TRUE(fired); + EXPECT_FALSE(rerun.member_retired); + EXPECT_EQ(rerun.warnings.size(), 1u) << fmt::format("{}", fmt::join(rerun.warnings, "; ")); + expectOwner(life_before + 1, true, verified); /// the rerun's tombstone, kept } - ASSERT_FALSE((*raw_op).head(layout.epochKey("victim"), Retry::once()).has_value()); - ASSERT_FALSE((*raw_op).head(layout.ownerKey("victim"), Retry::once()).has_value()); - ASSERT_TRUE((*raw_op).head(layout.mountKey("victim"), Retry::once()).has_value()) - << "the mount lease must survive -- it is the resume anchor the fallback reads"; +} - const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a2"}, "victim"); +/// Three runs: one verifies, one refreshes the tombstone and dies, one reruns and is fenced before its +/// confirm. The mark survives all three. +TEST_F(CASDecommission, RefreshKeepsTheVerifiedMark) +{ + auto backend = startVictim(); + ASSERT_TRUE(run("a1").member_retired); + ASSERT_TRUE(owner().retirement_verified); - EXPECT_TRUE(report.warnings.empty()); - EXPECT_EQ(report.namespaces_removed, 0u); - EXPECT_TRUE(report.slot_removed); - EXPECT_FALSE((*raw_op).head(layout.epochKey("victim"), Retry::once()).has_value()); - const auto owner = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); - ASSERT_TRUE(owner.has_value()); - EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); - EXPECT_FALSE((*raw_op).head(layout.mountKey("victim"), Retry::once()).has_value()); + tombstoneOwner(); /// a second run's tombstone, then death: life 4, mark kept + ASSERT_TRUE(owner().retirement_verified); + + bool fired = false; + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (std::exchange(fired, true)) + return; + fenceSlot(); + }); + const auto third = run("a3"); + ASSERT_TRUE(fired); + EXPECT_FALSE(third.member_retired); + expectOwner(5, true, true); +} + +namespace +{ + +/// What an operator leaves after clearing a verified tombstone by hand to restart the member: the mark of the +/// retired life stays. +void clearTombstoneByHand(Backend & backend, const Layout & layout, const String & srid) +{ + OperationForTest op(backend); + const auto current = (*op).read(layout.ownerKey(srid), Retry::once()); + ASSERT_TRUE(current.has_value()); + OwnerObject owner = decodeOwner(current->bytes); + ASSERT_TRUE(owner.retirement_verified); + owner.retired_at_ms.reset(); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.ownerKey(srid), encodeOwner(owner), current->etag, Retry::once()))); +} + +} + +/// The member restarted over a hand-cleared tombstone. The next decommission loses its claim after its +/// tombstone: the mark of the earlier life must not stop the rollback. +TEST_F(CASDecommission, HandClearedTombstoneStillRollsBackALostClaim) +{ + auto backend = startVictim(); + ASSERT_TRUE(run("a1").member_retired); + clearTombstoneByHand(*backend, layout, "victim"); + { auto victim = openVictim(backend); } - expectThrowsCode(ErrorCodes::CORRUPTED_DATA, [&] + bool fired = false; + backend->onWriteCommitted(layout.ownerKey("victim"), [&] { - decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a3"}, "victim"); + if (std::exchange(fired, true)) + return; + fenceSlot(); }); + const auto second = run("a2"); + + ASSERT_TRUE(fired); + EXPECT_FALSE(second.member_retired); + EXPECT_EQ(second.warnings.size(), 1u) << fmt::format("{}", fmt::join(second.warnings, "; ")); + expectOwner(5, false, false); /// first run 3, second tombstone 4, rollback 5 +} + +/// The member restarted over a hand-cleared tombstone. The next decommission's farewell is fenced: without a +/// mark of its own life the run reports rerun. +TEST_F(CASDecommission, HandClearedTombstoneIsNotARetirementOfTheNextRun) +{ + auto backend = startVictim(std::make_shared()); + ASSERT_TRUE(run("a1").member_retired); + clearTombstoneByHand(*backend, layout, "victim"); + { auto victim = openVictim(backend); } + backend->arm(); + + const auto second = run("a2"); + + ASSERT_TRUE(backend->fired()); + EXPECT_FALSE(second.member_retired); + expectOwner(4, true, false); +} + +/// A run dies after each step; a rerun completes over the run's own tombstone. +TEST_F(CASDecommission, ResumesOverItsOwnTombstone) +{ + enum class DieAt { AfterTombstone, AfterFarewell, AfterMark }; + for (const DieAt die_at : {DieAt::AfterTombstone, DieAt::AfterFarewell, DieAt::AfterMark}) + { + SCOPED_TRACE(static_cast(die_at)); + auto backend = startVictim(); + seed("p/roots/victim/loose_file"); + /// `BAD_ARGUMENTS` is a deterministic local failure to the request engine: the attempt is not + /// reissued and not resolved by a read, so the landed write surfaces as a refusal -- the shape of a + /// process that died right after its write landed. + const auto died_with = [](const char * what) + { + return std::make_exception_ptr(DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "{}", what)); + }; + bool died = false; + switch (die_at) + { + case DieAt::AfterTombstone: + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (!std::exchange(died, true)) + std::rethrow_exception(died_with("died after the tombstone")); + }); + break; + case DieAt::AfterFarewell: + /// The farewell is written by `~Pool`, which swallows a throw; die on the verify read instead. + backend->onWriteCommitted(layout.mountKey("victim"), [&] + { + if (slot().isFarewell() && !std::exchange(died, true)) + backend->failNextReadWith(layout.ownerKey("victim"), died_with("died after the farewell")); + }); + break; + case DieAt::AfterMark: + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (owner().retirement_verified && !std::exchange(died, true)) + std::rethrow_exception(died_with("died after the mark")); + }); + break; + } + + DecommissionReport first; + bool threw = false; + try + { + first = run("a1"); + } + catch (const DB::Exception &) + { + threw = true; + } + ASSERT_TRUE(died); + switch (die_at) + { + case DieAt::AfterTombstone: + EXPECT_TRUE(threw) << "the tombstone write is refused after it landed; nothing else ran"; + break; + case DieAt::AfterFarewell: + EXPECT_FALSE(threw) << "a failed verify read is reported, not thrown"; + EXPECT_FALSE(first.member_retired); + break; + case DieAt::AfterMark: + EXPECT_FALSE(threw) << "an unclear mark write is settled by the owner re-read"; + EXPECT_TRUE(first.member_retired) << "the re-read finds the verified mark"; + break; + } + expectOwner(die_at == DieAt::AfterMark ? 3 : 2, true, die_at == DieAt::AfterMark); + + const auto rerun = run("a2"); + EXPECT_TRUE(rerun.warnings.empty()) << fmt::format("{}", fmt::join(rerun.warnings, "; ")); + EXPECT_TRUE(rerun.member_retired); + expectOwner(die_at == DieAt::AfterMark ? 5 : 4, true, true); + EXPECT_FALSE(exists("p/roots/victim/loose_file")); + } +} + +namespace +{ + +/// The first roots delete under the victim lets the admin's local lease deadline pass (the fake clock jumps a +/// TTL); every delete admitted on the mount plane after it is refused. Optionally also GC-fences the body. +class ExpireLeaseOnFirstRootsDeleteBackend : public InMemoryBackend +{ +public: + std::shared_ptr clock; + uint64_t jump_ms = 0; + bool fence_body = false; + bool fired() const { return done; } + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + if (!done && key.starts_with("p/roots/victim/")) + { + done = true; + clock->now += jump_ms; + if (fence_body) + { + const auto current = InMemoryBackend::read(mount_key, access); + const MountLease body = decodeMountLease(current->bytes).fencedCopy(); + if (!InMemoryBackend::write(mount_key, encodeMountLease(body), current->value, access)) + throw std::runtime_error("expire fixture: the fence conflicted"); + } + } + return InMemoryBackend::remove(key, expected_value, access); + } + +private: + inline static const String mount_key = "p/gc/server-roots/victim/mount"; + bool done = false; +}; + +/// On the staging LIST (the request before the roots LIST) lets the admin's local lease deadline pass and +/// GC-fences its body in the store: the fence loss then surfaces on the roots LIST page, not on a delete. +class ExpireLeaseBeforeRootsListBackend : public InMemoryBackend +{ +public: + using Backend::list; + std::shared_ptr clock; + uint64_t jump_ms = 0; + bool fired() const { return done; } + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); + if (!done && prefix == "p/staging/victim/") + { + done = true; + clock->now += jump_ms; + const auto current = InMemoryBackend::read(mount_key, access); + const MountLease body = decodeMountLease(current->bytes).fencedCopy(); + if (!InMemoryBackend::write(mount_key, encodeMountLease(body), current->value, access)) + throw std::runtime_error("expire-before-list fixture: the fence conflicted"); + } + return page; + } + +private: + inline static const String mount_key = "p/gc/server-roots/victim/mount"; + bool done = false; +}; + +/// A store whose `roots/` objects carry content-derived ETags (as S3 does). While `capture` is set, every +/// delete under the victim's prefixes applies and also leaves a copy of the attempt in flight, which the test +/// delivers later with `replay`. +class LateDeleteBackend : public InMemoryBackend +{ +public: + struct Late { String key; Etag etag; }; + std::vector late; + bool capture = false; + + std::optional read(const String & key, TransportAccess & access) override + { + auto raw = InMemoryBackend::read(key, access); + if (raw && isRoots(key)) + raw->value = contentValue(raw->bytes); + return raw; + } + + std::optional head(const String & key, TransportAccess & access) override + { + auto meta = InMemoryBackend::head(key, access); + if (meta && isRoots(key)) + if (const auto raw = InMemoryBackend::read(key, access)) + meta->value = contentValue(raw->bytes); + return meta; + } + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); + for (RawListedKey & listed : page.keys) + if (listed.value && isRoots(listed.key)) + if (const auto raw = InMemoryBackend::read(listed.key, access)) + listed.value = contentValue(raw->bytes); + return page; + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + if (!isRoots(key)) + return InMemoryBackend::write(key, bytes, expected_value, access); + std::optional minted_expected; + if (expected_value) + { + const auto current = InMemoryBackend::read(key, access); + if (!current || contentValue(current->bytes) != *expected_value) + return std::unexpected(RawConflict{}); + minted_expected = current->value; + } + const auto written = InMemoryBackend::write(key, bytes, minted_expected, access); + if (!written) + return written; + return contentValue(bytes); + } + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + if (capture && isVictim(key)) + { + OperationForTest op(*this); + if (const auto meta = (*op).head(key, Retry::once())) + late.push_back({key, meta->etag}); + } + if (!isRoots(key)) + return InMemoryBackend::remove(key, expected_value, access); + const auto current = InMemoryBackend::read(key, access); + if (!current) + return RawRemoval::Gone; + if (contentValue(current->bytes) != expected_value) + return RawRemoval::Mismatch; + return InMemoryBackend::remove(key, current->value, access); + } + + /// Deliver every captured attempt now, through the ordinary exact-incarnation delete. + std::vector> replay() + { + std::vector> outcomes; + OperationForTest op(*this); + for (const Late & attempt : late) + outcomes.emplace_back(attempt.key, (*op).remove(attempt.key, attempt.etag, Retry::once())); + return outcomes; + } + +private: + static bool isRoots(const String & key) { return key.starts_with("p/roots/victim/"); } + static bool isVictim(const String & key) + { + return isRoots(key) || key.starts_with("p/staging/victim/") || key.starts_with("p/cas/manifests/victim/") + || key.starts_with("p/cas/ns/"); + } + static String contentValue(const String & bytes) { return "content-" + DB::Cas::tests::hexOf(bytes); } +}; + +/// Each delete under the victim's roots prefix takes `delay`, as a slow store would. `on_first_delete` runs +/// once, before the first of them; `after_delay` runs after each delay. +class SlowRootsDeleteBackend : public InMemoryBackend +{ +public: + std::chrono::milliseconds delay{0}; + std::function on_first_delete; + std::function after_delay; + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + if (key.starts_with("p/roots/victim/")) + { + if (on_first_delete) + std::exchange(on_first_delete, {})(); + std::this_thread::sleep_for(delay); + if (after_delay) + after_delay(); + } + return InMemoryBackend::remove(key, expected_value, access); + } +}; + +/// The tombstone write lands, GC fences the admin's body, and the write's answer is lost; the owner then +/// cannot be read for `unreadable_ms` of the fake clock, longer than the write's own retry window, so the +/// write gives up after sending. +class OwnerUnreadableAfterTombstoneBackend : public InMemoryBackend +{ +public: + std::shared_ptr clock; + std::atomic unreadable_ms = 0; + bool landed() const { return landed_at.load() != 0; } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + if (!landed() && key == owner_key && decodeOwner(bytes).retired_at_ms) + { + landed_at = clock->now.load(); + const auto current = InMemoryBackend::read(mount_key, access); + const MountLease body = decodeMountLease(current->bytes).fencedCopy(); + if (!InMemoryBackend::write(mount_key, encodeMountLease(body), current->value, access)) + throw std::runtime_error("unreadable-owner fixture: the fence conflicted"); + injectAmbiguousLandedWrite(key); + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + std::optional read(const String & key, TransportAccess & access) override + { + if (landed() && key == owner_key && clock->now.load() < landed_at.load() + unreadable_ms.load()) + throw Poco::TimeoutException("unreadable-owner fixture: the owner does not answer"); + return InMemoryBackend::read(key, access); + } + +private: + inline static const String mount_key = "p/gc/server-roots/victim/mount"; + inline static const String owner_key = "p/gc/server-roots/victim/owner"; + std::atomic landed_at = 0; +}; + +} + +TEST_F(CASDecommission, DrainStopsOnTheAdminLeaseFence) +{ + auto backend = startVictim(std::make_shared()); + seed("p/roots/victim/a_first"); + seed("p/roots/victim/b_second"); + auto clock = farClock(); + backend->clock = clock; + backend->jump_ms = 31'000; /// past the default 30 s TTL + + const auto report = runOnClock("a1", clock); + + ASSERT_TRUE(backend->fired()); + EXPECT_FALSE(report.member_retired); + ASSERT_FALSE(report.warnings.empty()); + EXPECT_EQ(report.mountpoint_objects_removed, 1u) << "the first delete was already sent"; + EXPECT_TRUE(exists("p/roots/victim/b_second")) + << "a delete admitted after the lease deadline is refused locally"; + expectOwner(2, true, false); +} + +/// After the first delete refused by the fence the walk stops: one warning, the later objects untouched. +TEST_F(CASDecommission, DrainStopsWalkingAfterTheFirstRefusedDelete) +{ + auto backend = startVictim(std::make_shared()); + for (const char * name : {"a_first", "b_second", "c_third"}) + seed(String("p/roots/victim/") + name); + auto clock = farClock(); + backend->clock = clock; + backend->jump_ms = 31'000; + + const auto report = runOnClock("a1", clock); + + ASSERT_TRUE(backend->fired()); + const auto refused = std::count_if(report.warnings.begin(), report.warnings.end(), + [](const String & warning) { return warning.starts_with("decommission drain: "); }); + EXPECT_EQ(refused, 1) << fmt::format("{}", fmt::join(report.warnings, "; ")); + EXPECT_TRUE(exists("p/roots/victim/c_third")); +} + +/// The admin is fenced in the store AND past its local deadline: the drain stops, and the rollback goes +/// through the control plane, which the mount fence does not gate. +TEST_F(CASDecommission, RollbackRunsOnTheControlPlaneAfterAFence) +{ + auto backend = startVictim(std::make_shared()); + seed("p/roots/victim/a_first"); + seed("p/roots/victim/b_second"); + auto clock = farClock(); + backend->clock = clock; + backend->jump_ms = 31'000; + backend->fence_body = true; + + const auto report = runOnClock("a1", clock); + + ASSERT_TRUE(backend->fired()); + EXPECT_FALSE(report.member_retired); + expectOwner(3, false, false); /// rolled back + EXPECT_TRUE(slot().gc_fenced); +} + +TEST_F(CASDecommission, FenceLossOnAListPageStillRunsTheConfirm) +{ + auto backend = startVictim(std::make_shared()); + auto clock = farClock(); + backend->clock = clock; + backend->jump_ms = 31'000; + + EXPECT_THROW(runOnClock("a1", clock), DB::Exception); + + ASSERT_TRUE(backend->fired()); + expectOwner(3, false, false); /// the exception exit confirmed and rolled back +} + +/// Real time: the background renewer keeps the admin's lease alive across a drain longer than one TTL (1 s +/// TTL, renewal every 200 ms, three roots deletes of 400 ms each), and the run retires the member. +TEST_F(CASDecommission, LongDrainOutlivesOneTtl) +{ + auto backend = startVictim(std::make_shared()); + backend->delay = std::chrono::milliseconds(400); + for (const char * name : {"a", "b", "c"}) + seed(String("p/roots/victim/") + name); + /// A stop that races a renewal skips the farewell; that race is a known open defect. The last delete + /// waits for a renewal to land and then half a renewal period, so the stop lands between two renewals. + struct MountWrites + { + std::mutex mutex; + std::condition_variable landed; + size_t count = 0; + }; + auto mount_writes = std::make_shared(); + backend->onWriteCommitted(layout.mountKey("victim"), [mount_writes] + { + std::lock_guard lock(mount_writes->mutex); + ++mount_writes->count; + mount_writes->landed.notify_all(); + }); + auto deletes = std::make_shared(0); + backend->after_delay = [mount_writes, deletes] + { + if (++*deletes != 3) + return; + std::unique_lock lock(mount_writes->mutex); + const size_t seen = mount_writes->count; + ASSERT_TRUE(mount_writes->landed.wait_for(lock, std::chrono::seconds(5), [&] { return mount_writes->count > seen; })); + lock.unlock(); + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + }; + + const auto started = std::chrono::steady_clock::now(); + const auto report = run("a1", realTimeConfig()); + + EXPECT_GE(std::chrono::steady_clock::now() - started, std::chrono::milliseconds(1200)); + EXPECT_EQ(report.mountpoint_objects_removed, 3u); + EXPECT_TRUE(report.warnings.empty()) << fmt::format("{}", fmt::join(report.warnings, "; ")); + EXPECT_TRUE(report.member_retired); +} + +/// Once the owner is tombstoned, the admin's own self-remount fails on the owner check and never reclaims +/// under a fresh epoch. +TEST_F(CASDecommission, FencedAdminFailsInsteadOfRemounting) +{ + auto backend = startVictim(); + auto admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim"); + const uint64_t admin_epoch = admin->writerEpoch(); + tombstoneOwner(); + fenceSlot(); + admin->tripMountLost(); /// what the renewer does when it finds its body fenced + EXPECT_FALSE(admin->tryRemountOnce()); + const MountLease body = slot(); + EXPECT_EQ(body.writer_epoch, admin_epoch); + EXPECT_TRUE(body.gc_fenced); + EXPECT_FALSE(admin->mayMutate()); +} + +/// The admin is GC-fenced during its drain; its renewer trips and its self-remount fails +/// (owner retired). The run's exit confirms, finds the claim lost and rolls the tombstone back. Real time: +/// TTL 1 s, renewal every 200 ms, a 400 ms delete per roots object. +TEST_F(CASDecommission, FailedRemountStillRunsTheConfirm) +{ + auto backend = startVictim(std::make_shared()); + backend->delay = std::chrono::milliseconds(400); + for (const char * name : {"a", "b", "c", "d"}) + seed(String("p/roots/victim/") + name); + bool fenced = false; + backend->on_first_delete = [&] + { + fenced = true; + fenceSlot(); + }; + const uint64_t remount_failures_before = ProfileEvents::global_counters[ProfileEvents::CASRemountFailed].load(); + + const auto report = run("a1", realTimeConfig()); + + ASSERT_TRUE(fenced); + EXPECT_FALSE(report.member_retired); + EXPECT_GT(ProfileEvents::global_counters[ProfileEvents::CASRemountFailed].load(), remount_failures_before) + << "the admin's self-remount ran and failed"; + expectOwner(3, false, false); /// rolled back: the claim was lost + EXPECT_TRUE(slot().gc_fenced); +} + +/// A paused run resumes after a rerun refreshed the tombstone: its rollback names an older incarnation and +/// cannot remove the rerun's tombstone. +TEST_F(CASDecommission, PausedRunCannotUntombstoneARerunsTombstone) +{ + auto backend = startVictim(); + std::optional rerun; + bool armed = true; + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (!std::exchange(armed, false)) + return; + /// Run 1 is paused right after its tombstone. GC fences it; the rerun reclaims, refreshes the + /// tombstone, drains, farewells and verifies. + fenceSlot(); + rerun = run("a2"); + }); + + const auto first = run("a1"); + + ASSERT_TRUE(rerun.has_value()); + EXPECT_TRUE(rerun->member_retired); + EXPECT_FALSE(first.member_retired); + EXPECT_EQ(first.warnings.size(), 1u) << fmt::format("{}", fmt::join(first.warnings, "; ")); + expectOwner(4, true, true); /// run 1 tombstone 2, rerun tombstone 3, rerun mark 4 +} + +/// A delete attempt sent by a run and delivered after a rejoin can remove only an object whose key and bytes +/// repeat: the roots objects. Namespace, manifest and staging keys of the new member never match. +TEST_F(CASDecommission, InFlightDeleteAfterRejoinHitsOnlyIdenticalRootsBytes) +{ + auto backend = startVictim(std::make_shared(), oneTable); /// no debris: it would block the mark + const RootNamespace ns("victim/db/t1"); + seed("p/staging/victim/3/old.tmp"); + { + OperationForTest seed_op(*backend); + (*seed_op).create("p/roots/victim/clickhouse_access_check", "probe", Retry::once()); + } + backend->capture = true; + const auto report = run("a1"); + backend->capture = false; + ASSERT_TRUE(report.member_retired); + ASSERT_FALSE(backend->late.empty()); + + /// The rejoin and the new member's first writes, by hand: the next owner life, a new staging object, a + /// new manifest, and the access probe with the same bytes under the same key. + { + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + const OwnerObject rejoined = nextOwnerLife(decodeOwner(current->bytes), std::nullopt, false); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.ownerKey("victim"), encodeOwner(rejoined), current->etag, Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*op).create("p/staging/victim/9/new.tmp", "y", Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*op).create("p/roots/victim/clickhouse_access_check", "probe", Retry::once()))); + } + const ManifestRef new_ref{.writer_epoch = 9, .build_sequence = 1, .manifest_ordinal = 1}; + const ManifestId new_manifest = writeManifestRaw(*backend, layout, ns, new_ref, {}); + + backend->replay(); + EXPECT_FALSE(exists("p/roots/victim/clickhouse_access_check")) + << "identical bytes repeat the ETag: the late delete lands on the new member's probe (the documented residual)"; + EXPECT_TRUE(exists("p/staging/victim/9/new.tmp")); + EXPECT_TRUE(exists(layout.manifestKey(new_manifest))); +} + +/// The tombstone lands and the engine throws before reporting it, while GC fences the admin: the exit reads +/// the owner back, finds this run's tombstone, sees the lost claim and rolls the tombstone back. +TEST_F(CASDecommission, TombstoneLandedBehindAnErrorIsConfirmed) +{ + auto backend = startVictim(); + bool landed = false; + backend->onWriteCommitted(layout.ownerKey("victim"), [&] + { + if (std::exchange(landed, true)) + return; + fenceSlot(); + throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "the tombstone landed and its answer was lost"); + }); + + EXPECT_THROW(run("a1"), DB::Exception); + + ASSERT_TRUE(landed); + expectOwner(3, false, false); /// claim 1, tombstone 2, rollback 3 +} + +/// The tombstone write gives up after sending; the owner answers again later and holds this run's tombstone. +/// GC fenced the admin, so the exit rolls the tombstone back. +TEST_F(CASDecommission, TombstoneWriteThatGaveUpAfterSendingIsConfirmed) +{ + auto backend = startVictim(std::make_shared()); + auto clock = farClock(); + backend->clock = clock; + backend->unreadable_ms = 100'000; /// longer than the tombstone write retries + + EXPECT_THROW(runOnClock("a1", clock), DB::Exception); + + ASSERT_TRUE(backend->landed()); + backend->unreadable_ms = 0; + expectOwner(3, false, false); /// claim 1, tombstone 2, rollback 3 +} + +/// A later run tombstones the owner between this run's farewell and its verify. The owner is past this run's +/// tombstone but unverified: this run reports rerun. +TEST_F(CASDecommission, LaterUnverifiedTombstoneIsNotARetirement) +{ + auto backend = startVictim(); + bool refreshed = false; + backend->onWriteCommitted(layout.mountKey("victim"), [&] + { + if (slot().isFarewell() && !std::exchange(refreshed, true)) + tombstoneOwner(); + }); + + const auto report = run("a1"); + + ASSERT_TRUE(refreshed); + EXPECT_FALSE(report.member_retired); + EXPECT_EQ(report.warnings.size(), 2u) << fmt::format("{}", fmt::join(report.warnings, "; ")); + expectOwner(3, true, false); +} + +/// The owner moves on to a later, unretired life between this run's farewell and its verify (a rejoin, or +/// another run's rollback). Only a verified mark is a retirement: the run reports 0 and says the owner moved. +TEST_F(CASDecommission, OwnerThatMovedOnIsNotReportedRetired) +{ + auto backend = startVictim(); + bool moved = false; + backend->onWriteCommitted(layout.mountKey("victim"), [&] + { + if (!slot().isFarewell() || std::exchange(moved, true)) + return; + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + ASSERT_TRUE(std::holds_alternative((*op).replace(layout.ownerKey("victim"), + encodeOwner(nextOwnerLife(decodeOwner(current->bytes), std::nullopt, false)), current->etag, Retry::once()))); + }); + + const auto report = run("a1"); + + ASSERT_TRUE(moved); + EXPECT_FALSE(report.member_retired); + ASSERT_EQ(report.warnings.size(), 2u) << fmt::format("{}", fmt::join(report.warnings, "; ")); + EXPECT_NE(report.warnings.back().find("moved on to life 3"), String::npos) << report.warnings.back(); + EXPECT_FALSE(owner().retired_at_ms.has_value()); +} + +TEST_F(CASDecommission, RetirementLogsTheUuidAndLife) +{ + auto backend = startVictim(); + DB::Cas::tests::ScopedLoggerCapture capture("CasDecommission", "information"); + + const auto report = run("a1"); + + ASSERT_TRUE(report.member_retired); + const String expected = "retired server_uuid=" + u128ToHex(owner().server_uuid) + " at owner life 2"; + EXPECT_NE(capture.captured().find(expected), String::npos) << capture.captured(); +} + +/// The owner moves under the mark write: the run reports why the mark is missing, not only that it is. +TEST_F(CASDecommission, MarkWriteConflictIsReported) +{ + auto backend = startVictim(); + bool farewell_landed = false; + backend->onWriteCommitted(layout.mountKey("victim"), [&] + { + if (slot().isFarewell()) + farewell_landed = true; + }); + bool moved = false; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (!farewell_landed || std::exchange(moved, true)) + return; + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.ownerKey("victim"), current->bytes, current->etag, Retry::once()))); + }); + + const auto report = run("a1"); + + ASSERT_TRUE(moved); + EXPECT_FALSE(report.member_retired); + const auto mark_warning = std::find_if(report.warnings.begin(), report.warnings.end(), + [](const String & warning) { return warning.starts_with("the verified mark was not written"); }); + EXPECT_NE(mark_warning, report.warnings.end()) << fmt::format("{}", fmt::join(report.warnings, "; ")); + EXPECT_FALSE(owner().retirement_verified); +} + +namespace +{ + +/// Once armed, the next catalog read fails, and so does the slot read after it: a drain that fails on a store +/// outage, followed by a confirm that fails for the same reason. +class DrainThenConfirmFailBackend : public InMemoryBackend +{ +public: + void arm() { armed = true; } + bool confirmFailed() const { return slot_failed; } + + std::optional read(const String & key, TransportAccess & access) override + { + if (armed && !catalog_failed && key == catalog_key) + { + catalog_failed = true; + throw std::runtime_error("injected: the catalog does not answer"); + } + if (catalog_failed && !slot_failed && key == mount_key) + { + slot_failed = true; + throw std::runtime_error("injected: the slot does not answer"); + } + return InMemoryBackend::read(key, access); + } + +private: + inline static const String catalog_key = Layout("p").refCatalogKey(); + inline static const String mount_key = "p/gc/server-roots/victim/mount"; + bool armed = false; + bool catalog_failed = false; + bool slot_failed = false; +}; + +} + +/// The drain fails and the confirm's own read fails too: the drain's error is the one that propagates, and +/// the failed confirm is logged. +TEST_F(CASDecommission, FailedConfirmKeepsTheDrainError) +{ + auto backend = startVictim(std::make_shared()); + backend->onWriteCommitted(layout.ownerKey("victim"), [&] { backend->arm(); }); + DB::Cas::tests::ScopedLoggerCapture capture("CasDecommission", "warning"); + + try + { + run("a1"); + ADD_FAILURE() << "the drain must fail"; + } + catch (const std::exception & e) + { + EXPECT_NE(String(e.what()).find("the catalog does not answer"), String::npos) << e.what(); + } + + ASSERT_TRUE(backend->confirmFailed()); + EXPECT_NE(capture.captured().find("could not confirm the admin claim"), String::npos) << capture.captured(); +} + +TEST_F(CASDecommission, OpenLogsTheSkippedProbe) +{ + auto backend = startVictim(); + DB::Cas::tests::ScopedLoggerCapture capture("CasPool", "information"); + auto admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim"); + EXPECT_NE(capture.captured().find("capability probe was skipped"), String::npos) << capture.captured(); +} + +/// The admin impersonates the owner it retires, so whose life is "ours" is ambiguous: it never takes another +/// uuid's terminal body and refuses before writing anything. +TEST_F(CASDecommission, AdminDoesNotReclaimAForeignTerminalBody) +{ + startVictim(); + { + OperationForTest op(*victim_backend); + const auto got = (*op).read(layout.mountKey("victim"), Retry::once()); + MountLease body = decodeMountLease(got->bytes); + body.server_uuid = UInt128(5); + body.seq += 1; + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.mountKey("victim"), encodeMountLease(body), got->etag, Retry::once()))); + } + const OwnerObject owner_before = owner(); + const MountLease slot_before = slot(); + + expectThrowsCodeWithMessage(ErrorCodes::ABORTED, "alive or contended", [&] { run("admin"); }); + + expectOwner(owner_before.owner_life, /*retired=*/false, /*verified=*/false); + EXPECT_EQ(slot().seq, slot_before.seq); +} + +namespace +{ +/// A rejoin needs a real server uuid: a start without one (uuid 0) never rejoins. +const UInt128 victim_uuid = UInt128(7); + +PoolConfig victimConfig(UInt128 uuid = victim_uuid) +{ + return PoolConfig{.pool_prefix = "p", .server_id = uuid, .server_root_id = "victim"}; +} + +/// One completed decommission of the victim with nothing left for GC: the state a rejoin starts from. +void retireVictim(const std::shared_ptr & backend, const String & admin_srid) +{ + const auto report = decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = admin_srid}, "victim"); + if (!report.member_retired || report.namespaces_pending_gc != 0) + throw std::runtime_error("fixture: the decommission did not retire the victim cleanly"); +} + +bool victimSubtreeEmpty(Backend & backend) +{ + const Layout layout("p"); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + return serverRootSubtreeEmpty(op, layout, "victim", CasRefCatalog::read(op, layout).catalog); +} + +/// GC removes the dropped namespaces' catalog rows and manifests over a few rounds; a rejoin waits for all of them. +void gcUntilVictimSubtreeEmpty(const std::shared_ptr & backend) +{ + auto store = Pool::open(backend, PoolConfig{ + .pool_prefix = "p", .server_root_id = "gc", .gc_fold_threshold = 1, .gc_fold_max_defer_rounds = 0}); + Gc gc(store, UInt128{991}); + for (int round = 0; round < 8 && !victimSubtreeEmpty(*backend); ++round) + runRegularRoundReclaiming(gc); + if (!victimSubtreeEmpty(*backend)) + throw std::runtime_error("fixture: GC left data of the victim's previous life"); +} + +/// The rejoined member writes a part under the old life's namespace name; GC rounds run from another member +/// keep it, and the member's slot stays its own. +void expectWriteSurvivesGc(const std::shared_ptr & backend, const PoolPtr & member, UInt128 member_uuid) +{ + const RootNamespace ns{"victim/db/t1"}; + const String payload = "written after the rejoin"; + PartWriteInfo info; + info.intended_ref = ns.string() + "/r"; + auto build = member->beginPartWrite(info); + ManifestEntry entry; + entry.path = "data.bin"; + entry.placement = EntryPlacement::Blob; + entry.ref = idOf(payload); + entry.blob_size = payload.size(); + const ManifestId manifest = build->stageManifest({entry}); + build->precommitAdd(ns, "r", manifest); + build->putBlob(idOf(payload), BlobSource::fromString(payload)); + build->promote(ns, "r", build->buildId(), manifest); + + { + auto gc_store = Pool::open(backend, PoolConfig{ + .pool_prefix = "p", .server_root_id = "gc", .gc_fold_threshold = 1, .gc_fold_max_defer_rounds = 0}); + Gc gc(gc_store, UInt128{992}); + for (int round = 0; round < 3; ++round) + { + runRegularRoundReclaiming(gc); + member->renewWatermarkOnce(); + } + } + + const Layout layout("p"); + EXPECT_EQ(member->listRefs(ns).size(), 1u); + OperationForTest op(*backend); + EXPECT_TRUE((*op).head(layout.manifestKey(manifest), Retry::once()).has_value()); + EXPECT_TRUE((*op).head(layout.blobKey(idOf(payload)), Retry::once()).has_value()); + const MountLease slot = readSlot(*backend, layout, "victim"); + EXPECT_TRUE(slot.isHeldBy(member_uuid, member->writerEpoch())); + EXPECT_FALSE(slot.gc_fenced); +} + +/// The slot body rewritten by hand; `edit` changes the decoded body before it is written back. +void rewriteSlot(Backend & backend, const Layout & layout, const std::function & edit) +{ + OperationForTest op(backend); + const auto got = (*op).read(layout.mountKey("victim"), Retry::once()); + ASSERT_TRUE(got.has_value()); + MountLease body = decodeMountLease(got->bytes); + edit(body); + body.seq += 1; + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.mountKey("victim"), encodeMountLease(body), got->etag, Retry::once()))); +} +} + +TEST(CASRejoin, SameUuidStartsEmptyAndContinuesTheCounter) +{ + auto backend = std::make_shared(); + { + auto victim = Pool::open(backend, victimConfig()); + makeTableWithRefs(*victim, "victim/db/t1", 2, 0); + } + ASSERT_TRUE(decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim").member_retired); + gcUntilVictimSubtreeEmpty(backend); + const Layout layout("p"); + const OwnerObject retired = readOwner(*backend, layout, "victim"); + ASSERT_TRUE(retired.retirement_verified); + const uint64_t admin_epoch = readSlot(*backend, layout, "victim").writer_epoch; + + auto rejoined = Pool::open(backend, victimConfig()); + + EXPECT_GT(rejoined->writerEpoch(), admin_epoch) << "the kept counter continues past every earlier life"; + const OwnerObject owner = readOwner(*backend, layout, "victim"); + EXPECT_EQ(owner.owner_life, retired.owner_life + 1); + EXPECT_FALSE(owner.retired_at_ms.has_value()); + EXPECT_FALSE(owner.retirement_verified); + EXPECT_TRUE(victimSubtreeEmpty(*backend)) << "the member comes back empty"; + EXPECT_EQ(readSlot(*backend, layout, "victim").writer_epoch, rejoined->writerEpoch()); + expectWriteSurvivesGc(backend, rejoined, victim_uuid); +} + +TEST(CASRejoin, NewUuidReclaimsTheOldLifesFarewell) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + const MountLease farewell = readSlot(*backend, layout, "victim"); + + auto rejoined = Pool::open(backend, victimConfig(UInt128(77))); + + const OwnerObject owner = readOwner(*backend, layout, "victim"); + EXPECT_EQ(owner.server_uuid, UInt128(77)); + EXPECT_FALSE(owner.retired_at_ms.has_value()); + const MountLease slot = readSlot(*backend, layout, "victim"); + EXPECT_EQ(slot.server_uuid, UInt128(77)); + EXPECT_GT(slot.writer_epoch, farewell.writer_epoch); + EXPECT_EQ(slot.writer_epoch, rejoined->writerEpoch()); + expectWriteSurvivesGc(backend, rejoined, UInt128(77)); +} + +/// The old victim reclaimed the admin's farewell under a fresh epoch, saw the tombstone and fenced itself. +TEST(CASRejoin, NewUuidReclaimsASelfFencedOldVictimBody) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + uint64_t victim_epoch = 0; + { + OperationForTest op(*backend); + victim_epoch = allocateWriterEpoch(*op, layout, "victim", [] { return RefCatalog{}; }); + } + rewriteSlot(*backend, layout, [&](MountLease & body) + { + body.writer_epoch = victim_epoch; + body.min_active_build_sequence = 0; + body.gc_fenced = true; + }); + + auto rejoined = Pool::open(backend, victimConfig(UInt128(77))); + + const MountLease slot = readSlot(*backend, layout, "victim"); + EXPECT_EQ(slot.server_uuid, UInt128(77)); + EXPECT_EQ(slot.writer_epoch, rejoined->writerEpoch()); + EXPECT_FALSE(slot.gc_fenced); +} + +TEST(CASRejoin, RefusesWhileTheDecommissionRuns) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + auto rerun_admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); + const OwnerObject before = readOwner(*backend, layout, "victim"); + + for (const UInt128 uuid : {victim_uuid, UInt128(77)}) + { + SCOPED_TRACE(u128ToHex(uuid)); + expectThrowsCodeWithMessage(ErrorCodes::ABORTED, "still running", [&] { Pool::open(backend, victimConfig(uuid)); }); + } + + const OwnerObject after = readOwner(*backend, layout, "victim"); + EXPECT_EQ(after.owner_life, before.owner_life); + EXPECT_TRUE(after.retirement_verified); + EXPECT_EQ(readSlot(*backend, layout, "victim").writer_epoch, rerun_admin->writerEpoch()) << "the admin's live body is untouched"; +} + +/// Each part of the old life that can remain refuses the rejoin, names the part and writes nothing; once the +/// part is gone the start rejoins. The verified mark does not wait for GC; the rejoin does. +TEST(CASRejoin, RefusesWhileDataOfTheOldLifeRemains) +{ + const Layout layout("p"); + const auto control_objects = [&](Backend & backend) + { + OperationForTest op(backend); + std::vector etags; + for (const String & key : {layout.ownerKey("victim"), layout.epochKey("victim"), layout.mountKey("victim")}) + etags.push_back((*op).read(key, Retry::once())->etag); + return etags; + }; + for (const ServerRootLeftover leftover : + {ServerRootLeftover::CatalogRows, ServerRootLeftover::Manifests, ServerRootLeftover::RootsObjects}) + { + SCOPED_TRACE(static_cast(leftover)); + auto backend = std::make_shared(); + OperationForTest op(*backend); + /// The server uuid as `IDisk::checkAccess` formats it for the probe key and the server passes as its id. + const String old_uuid = toString(parseFromString("01234567-89ab-cdef-fedc-ba9876543210")); + String old_uuid_hex = old_uuid; + std::erase(old_uuid_hex, '-'); + const String probe = "p/roots/victim/clickhouse_access_check_" + old_uuid; + { + auto victim = Pool::open(backend, victimConfig(hexToU128(old_uuid_hex))); + if (leftover == ServerRootLeftover::CatalogRows) + makeTableWithRefs(*victim, "victim/db/t1", 1, 0); + } + const auto report = decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + ASSERT_TRUE(report.member_retired); + ASSERT_EQ(report.namespaces_pending_gc, leftover == ServerRootLeftover::CatalogRows ? 1u : 0u); + + String part; + std::function clear; + switch (leftover) + { + case ServerRootLeftover::CatalogRows: + part = "catalog rows of the old life"; + clear = [&] { gcUntilVictimSubtreeEmpty(backend); }; + break; + case ServerRootLeftover::Manifests: + ASSERT_TRUE(std::holds_alternative((*op).create("p/cas/manifests/victim/m", "m", Retry::once()))); + part = "manifests under cas/manifests/victim/"; + clear = [&] { ASSERT_EQ((*op).removeCurrent("p/cas/manifests/victim/m", Retry::once()), Removal::Removed); }; + break; + case ServerRootLeftover::RootsObjects: + ASSERT_TRUE(std::holds_alternative((*op).create(probe, "test", Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*op).create("p/roots/victim/backup.zip", "zip", Retry::once()))); + part = "an object under roots/victim/"; + clear = [&] { ASSERT_EQ((*op).removeCurrent("p/roots/victim/backup.zip", Retry::once()), Removal::Removed); }; + break; + case ServerRootLeftover::None: + FAIL(); + } + const std::vector before = control_objects(*backend); + + try + { + Pool::open(backend, victimConfig(UInt128(77))); + ADD_FAILURE() << "the rejoin was not refused"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ABORTED) << e.message(); + EXPECT_NE(e.message().find(part), String::npos) << e.message(); + if (leftover != ServerRootLeftover::RootsObjects) + EXPECT_NE(e.message().find("on the GC leader"), String::npos) << e.message(); + } + + EXPECT_EQ(control_objects(*backend), before); + if (leftover == ServerRootLeftover::RootsObjects) + EXPECT_FALSE((*op).head(probe, Retry::once()).has_value()) << "the old life's access probe is deleted first"; + clear(); + EXPECT_NO_THROW(Pool::open(backend, victimConfig(UInt128(77)))); + } +} + +/// A rerun over a verified retirement died holding the slot. The rejoin waits until GC fences the lapsed lease. +TEST(CASRejoin, RejoinsOnceGcFencesAKilledRerunsLease) +{ + for (const UInt128 uuid : {victim_uuid, UInt128(77)}) + { + SCOPED_TRACE(u128ToHex(uuid)); + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + OperationForTest op(*backend); + const uint64_t rerun_epoch = allocateWriterEpoch(*op, layout, "victim", [] { return RefCatalog{}; }); + rewriteSlot(*backend, layout, [&](MountLease & body) + { + body.writer_epoch = rerun_epoch; + body.min_active_build_sequence = 0; + }); + tombstoneOwnerForTest(*backend, layout.ownerKey("victim")); + ASSERT_TRUE(readOwner(*backend, layout, "victim").retirement_verified); + + expectThrowsCodeWithMessage(ErrorCodes::ABORTED, "still running", [&] { Pool::open(backend, victimConfig(uuid)); }); + + MountObservationMap observations; + std::set warned; + for (const uint64_t mono_ms : {uint64_t{0}, uint64_t{1'000'000'000}}) + computeHeartbeatFloor(*op, layout, /*now_ms=*/0, [mono_ms] { return mono_ms; }, /*observer_lease_ms=*/30'000, + /*cadence_ms=*/1'000, observations, warned); + ASSERT_TRUE(readSlot(*backend, layout, "victim").gc_fenced); + + auto rejoined = Pool::open(backend, victimConfig(uuid)); + EXPECT_EQ(readOwner(*backend, layout, "victim").server_uuid, uuid); + EXPECT_TRUE(readSlot(*backend, layout, "victim").isHeldBy(uuid, rejoined->writerEpoch())); + } +} + +TEST(CASRejoin, RefusesSlotRetiredBy2664) +{ + const Layout layout("p"); + { + SCOPED_TRACE("26.6.4 end state: tombstone without life or mark, no epoch, no mount"); + auto backend = std::make_shared(); + { auto other = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "other"}); } + OperationForTest op(*backend); + const String tombstone = encodeOwner(OwnerObject{.server_uuid = UInt128{}, .retired_at_ms = 1}); + ASSERT_TRUE(std::holds_alternative((*op).create(layout.ownerKey("victim"), tombstone, Retry::once()))); + try + { + Pool::open(backend, victimConfig()); + ADD_FAILURE() << "the start was not refused"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::CORRUPTED_DATA) << e.message(); + EXPECT_NE(e.message().find("retired by an older version"), String::npos) << e.message(); + EXPECT_EQ(e.message().find("not finished"), String::npos) << "a rerun refuses the absent counter: " << e.message(); + } + EXPECT_EQ((*op).read(layout.ownerKey("victim"), Retry::once())->bytes, tombstone); + EXPECT_FALSE((*op).head(layout.epochKey("victim"), Retry::once()).has_value()); + EXPECT_FALSE((*op).head(layout.mountKey("victim"), Retry::once()).has_value()); + } + { + SCOPED_TRACE("verified retirement whose epoch counter was deleted by hand"); + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + OperationForTest op(*backend); + ASSERT_EQ((*op).removeCurrent(layout.epochKey("victim"), Retry::once()), Removal::Removed); + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "cannot rejoin", [&] { Pool::open(backend, victimConfig()); }); + EXPECT_FALSE((*op).head(layout.epochKey("victim"), Retry::once()).has_value()); + EXPECT_TRUE(readOwner(*backend, layout, "victim").retired_at_ms.has_value()); + } + { + SCOPED_TRACE("verified retirement whose mount slot was deleted by hand"); + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + OperationForTest op(*backend); + ASSERT_EQ((*op).removeCurrent(layout.mountKey("victim"), Retry::once()), Removal::Removed); + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "cannot rejoin", [&] { Pool::open(backend, victimConfig()); }); + EXPECT_FALSE((*op).head(layout.mountKey("victim"), Retry::once()).has_value()); + EXPECT_TRUE(readOwner(*backend, layout, "victim").retired_at_ms.has_value()); + } +} + +/// Starter 88 rewrote the slot and stalled before its owner write; starter 77 then writes its owner and +/// takes the loser's terminal body. A later start of 88 is refused and writes nothing. +TEST(CASRejoin, ConcurrentStartersWithDifferentUuidsOneWins) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + bool loser_wrote_mount = false; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (std::exchange(loser_wrote_mount, true)) + return; + rewriteSlot(*backend, layout, [](MountLease & body) { body.server_uuid = UInt128(88); }); + }); + + auto winner = Pool::open(backend, victimConfig(UInt128(77))); + + ASSERT_TRUE(loser_wrote_mount); + EXPECT_EQ(readOwner(*backend, layout, "victim").server_uuid, UInt128(77)); + EXPECT_EQ(readSlot(*backend, layout, "victim").server_uuid, UInt128(77)); + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "different server", [&] { Pool::open(backend, victimConfig(UInt128(88))); }); + EXPECT_EQ(readSlot(*backend, layout, "victim").writer_epoch, winner->writerEpoch()); +} + +/// Starter 88's owner write loses its `If-Match` to starter 77's owner write: it decides again from a fresh +/// owner read, sees the foreign owner and refuses. It has already rewritten the slot, as every new uuid does +/// before its owner write, and leaves the owner alone. +TEST(CASRejoin, LostOwnerWriteDecidesAgainAndRefuses) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + const OwnerObject retired = readOwner(*backend, layout, "victim"); + const MountLease slot_before = readSlot(*backend, layout, "victim"); + const uint64_t slot_seq = slot_before.seq; + bool winner_wrote = false; + String winner_body; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (std::exchange(winner_wrote, true)) + return; + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + winner_body = encodeOwner(OwnerObject{.server_uuid = UInt128(77), .retired_at_ms = std::nullopt, .owner_life = retired.owner_life + 1}); + ASSERT_TRUE(std::holds_alternative((*op).replace(layout.ownerKey("victim"), winner_body, current->etag, Retry::once()))); + }); + + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "different server", [&] { Pool::open(backend, victimConfig(UInt128(88))); }); + + ASSERT_TRUE(winner_wrote); + OperationForTest op(*backend); + EXPECT_EQ((*op).read(layout.ownerKey("victim"), Retry::once())->bytes, winner_body) << "the loser left the owner alone"; + const MountLease slot = readSlot(*backend, layout, "victim"); + EXPECT_EQ(slot.seq, slot_seq + 1); + EXPECT_EQ(slot.server_uuid, UInt128(88)); + EXPECT_NE(slot.write_attempt_id, slot_before.write_attempt_id); + EXPECT_EQ(slot.pid, static_cast(::getpid())); +} + +/// The old uuid rejoins after a new uuid replaced the mount body, and the new uuid writes the owner first. The old +/// uuid writes no body of its own: under the new owner it would be a farewell a 26.6.4 starter of the old uuid, +/// prechecked before the decommission, could still reclaim. +TEST(CASRejoin, SameUuidLeavesTheMountBodyAlone) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + const OwnerObject retired = readOwner(*backend, layout, "victim"); + rewriteSlot(*backend, layout, [](MountLease & body) { body.server_uuid = UInt128(77); }); + const MountLease replaced = readSlot(*backend, layout, "victim"); + std::atomic fired = false; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (fired.exchange(true)) + return; + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + const String body = encodeOwner( + OwnerObject{.server_uuid = UInt128(77), .retired_at_ms = std::nullopt, .owner_life = retired.owner_life + 1}); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.ownerKey("victim"), body, current->etag, Retry::once()))); + }); + + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "different server", [&] { Pool::open(backend, victimConfig()); }); + + backend->onBeforeWrite(layout.ownerKey("victim"), {}); + ASSERT_TRUE(fired); + const MountLease slot = readSlot(*backend, layout, "victim"); + EXPECT_EQ(slot.server_uuid, UInt128(77)) << "the old uuid wrote no body under the new owner"; + EXPECT_EQ(slot.seq, replaced.seq); +} + +/// Every rejoin write loses its `If-Match`: the start gives up after a bounded number of decisions. +TEST(CASRejoin, OwnerChangingOnEveryDecisionIsBounded) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + bool in_hook = false; + size_t fires = 0; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (std::exchange(in_hook, true)) + return; + ++fires; + OperationForTest op(*backend); + const auto current = (*op).read(layout.ownerKey("victim"), Retry::once()); + ASSERT_TRUE(std::holds_alternative((*op).replace(layout.ownerKey("victim"), current->bytes, current->etag, Retry::once()))); + in_hook = false; + }); + + expectThrowsCodeWithMessage(ErrorCodes::ABORTED, "kept changing", [&] { Pool::open(backend, victimConfig(UInt128(77))); }); + + EXPECT_EQ(fires, 4u); + EXPECT_TRUE(readOwner(*backend, layout, "victim").retired_at_ms.has_value()); +} + +/// A start of the old life allocated its epoch before the decommission and claims only after a rejoin. +TEST(CASRejoin, DelayedLowerEpochClaimAfterRejoinIsRefused) +{ + for (const UInt128 rejoiner : {victim_uuid, UInt128(77)}) + { + SCOPED_TRACE(rejoiner == victim_uuid ? "same uuid" : "new uuid: a foreign farewell"); + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + const Layout layout("p"); + uint64_t delayed_epoch = 0; + { + OperationForTest op(*backend); + delayed_epoch = allocateWriterEpoch(*op, layout, "victim", [] { return RefCatalog{}; }); + } + retireVictim(backend, "admin"); + { auto rejoined = Pool::open(backend, victimConfig(rejoiner)); } + OperationForTest op(*backend); + const auto before = (*op).read(layout.mountKey("victim"), Retry::once()); + const OwnerExpectation old_life{.server_uuid = victim_uuid, .owner_life = 1, .retired = RetiredOwnerPolicy::Refuse}; + if (rejoiner == victim_uuid) + expectThrowsCode(ErrorCodes::ABORTED, [&] { claimMount(*op, layout, "victim", victim_uuid, delayed_epoch, 1, 100, {}, {}, {}, old_life); }); + else + EXPECT_EQ(claimMount(*op, layout, "victim", victim_uuid, delayed_epoch, 1, 100, {}, {}, {}, old_life).kind, MountClaimResult::ForeignOwner); + EXPECT_EQ((*op).read(layout.mountKey("victim"), Retry::once())->etag, before->etag); + } +} + +/// The victim restarts while the admin holds the slot; it reclaims the farewell, sees the tombstone and +/// fences itself. A completed rerun later, the same victim rejoins on its next start. +TEST(CASRejoin, VictimRefusedAfterFarewellRejoinsOnItsNextStart) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + const Layout layout("p"); + auto admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + auto fake_boot = std::make_shared>(0); + bool decommissioned = false; + PoolConfig waiting = victimConfig(); + waiting.boot_ms_fn = [fake_boot] { return fake_boot->load(); }; + waiting.wait_sleep_fn = [&, fake_boot](uint64_t ms) + { + *fake_boot += ms; + if (std::exchange(decommissioned, true)) + return; + tombstoneOwnerForTest(*backend, layout.ownerKey("victim")); + admin.reset(); + }; + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "decommissioned", [&] { Pool::open(backend, waiting); }); + ASSERT_TRUE(readSlot(*backend, layout, "victim").gc_fenced); + + retireVictim(backend, "admin2"); + auto rejoined = Pool::open(backend, victimConfig()); + EXPECT_FALSE(readOwner(*backend, layout, "victim").retired_at_ms.has_value()); + EXPECT_EQ(readSlot(*backend, layout, "victim").writer_epoch, rejoined->writerEpoch()); +} + +TEST(CASRejoin, RefusesWhileAnUnfinishedDecommissionLeftData) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).create("p/roots/victim/kept", "x", Retry::once()))); + } + backend->failWithTokenMismatch("p/roots/victim/kept"); + ASSERT_FALSE(decommissionPoolMember(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim").member_retired); + const Layout layout("p"); + const OwnerObject tombstone = readOwner(*backend, layout, "victim"); + ASSERT_TRUE(tombstone.retired_at_ms.has_value()); + ASSERT_FALSE(tombstone.retirement_verified); + + for (const UInt128 uuid : {victim_uuid, UInt128(77)}) + { + SCOPED_TRACE(u128ToHex(uuid)); + expectThrowsCodeWithMessage(ErrorCodes::CORRUPTED_DATA, "decommission is not finished", [&] { Pool::open(backend, victimConfig(uuid)); }); + } + EXPECT_EQ(readOwner(*backend, layout, "victim").owner_life, tombstone.owner_life); + + backend->disarm(); + retireVictim(backend, "admin2"); + EXPECT_NO_THROW(Pool::open(backend, victimConfig())); +} + +/// A losing racing rejoiner of uuid 88 left its terminal body; the old uuid's own restart still rejoins. +TEST(CASRejoin, SameUuidRejoinsOverAForeignTerminalBody) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + rewriteSlot(*backend, layout, [](MountLease & body) { body.server_uuid = UInt128(88); }); + + auto rejoined = Pool::open(backend, victimConfig()); + + const MountLease slot = readSlot(*backend, layout, "victim"); + EXPECT_EQ(slot.server_uuid, victim_uuid); + EXPECT_EQ(slot.writer_epoch, rejoined->writerEpoch()); +} + +/// Only a start rejoins. A process that was fenced and finds its id retired keeps refusing to remount. +TEST(CASRejoin, RemountDoesNotRejoin) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, victimConfig()); + const Layout layout("p"); + tombstoneOwnerForTest(*backend, layout.ownerKey("victim"), /*verified=*/true); + fenceOutMountForTest(*backend, layout.mountKey("victim")); + expectThrowsCode(ErrorCodes::ABORTED, [&] { store->renewWatermarkOnce(); }); + const OwnerObject before = readOwner(*backend, layout, "victim"); + + EXPECT_FALSE(store->tryRemountOnce()); + + const OwnerObject after = readOwner(*backend, layout, "victim"); + EXPECT_EQ(after.owner_life, before.owner_life); + EXPECT_TRUE(after.retired_at_ms.has_value()); +} + + +/// A 26.6.4 starter of the old uuid read the farewell before the rejoin and reclaims it by `If-Match` with +/// no owner recheck; it fires just before the rejoin's owner write. +TEST(CASRejoin, NewUuidReplacesTheMountBeforeTheOwner) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + OperationForTest raw_op(*backend); + const auto old_farewell = (*raw_op).read(layout.mountKey("victim"), Retry::once()); + ASSERT_TRUE(old_farewell.has_value()); + std::optional old_claim; + backend->onBeforeWrite(layout.ownerKey("victim"), [&] + { + if (old_claim) + return; + MountLease body = decodeMountLease(old_farewell->bytes); + body.writer_epoch += 1; + body.min_active_build_sequence = 0; + body.seq += 1; + OperationForTest op(*backend); + old_claim = (*op).replace(layout.mountKey("victim"), encodeMountLease(body), old_farewell->etag, Retry::once()); + }); + + auto rejoined = Pool::open(backend, victimConfig(UInt128(77))); + + ASSERT_TRUE(old_claim.has_value()); + EXPECT_TRUE(std::holds_alternative(*old_claim)) << "the rejoin rewrote the mount body first"; + EXPECT_EQ(readSlot(*backend, layout, "victim").server_uuid, UInt128(77)); +} + +/// A decommission rerun claims the slot between the rejoin's reads and its first write: the rejoin's +/// `If-Match` fails, the next decision sees the admin's live body and refuses; the owner stays retired. +TEST(CASRejoin, DecommissionRerunDuringARejoinWins) +{ + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig()); } + retireVictim(backend, "admin"); + const Layout layout("p"); + const OwnerObject before = readOwner(*backend, layout, "victim"); + std::atomic fired = false; + PoolPtr rerun_admin; + const auto admin_claims = [&] + { + if (fired.exchange(true)) + return; + rerun_admin = Pool::openForDecommission(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); + }; + backend->onBeforeWrite(layout.mountKey("victim"), admin_claims); + backend->onBeforeWrite(layout.ownerKey("victim"), admin_claims); + + expectThrowsCodeWithMessage(ErrorCodes::ABORTED, "still running", [&] { Pool::open(backend, victimConfig(UInt128(77))); }); + + backend->onBeforeWrite(layout.mountKey("victim"), {}); + backend->onBeforeWrite(layout.ownerKey("victim"), {}); + ASSERT_TRUE(rerun_admin != nullptr); + const OwnerObject after = readOwner(*backend, layout, "victim"); + EXPECT_EQ(after.server_uuid, before.server_uuid); + EXPECT_EQ(after.owner_life, before.owner_life); + EXPECT_TRUE(after.retirement_verified); + EXPECT_EQ(readSlot(*backend, layout, "victim").writer_epoch, rerun_admin->writerEpoch()); +} + +/// A writable open without a server uuid (uuid 0) stays refused over a retired owner, whoever owned it. +TEST(CASRejoin, ZeroUuidDoesNotRejoin) +{ + for (const UInt128 old_uuid : {UInt128(77), UInt128{}}) + { + SCOPED_TRACE(u128ToHex(old_uuid)); + auto backend = std::make_shared(); + { auto victim = Pool::open(backend, victimConfig(old_uuid)); } + retireVictim(backend, "admin"); + const Layout layout("p"); + const OwnerObject before = readOwner(*backend, layout, "victim"); + const MountLease slot_before = readSlot(*backend, layout, "victim"); + + expectThrowsCodeWithMessage( + ErrorCodes::CORRUPTED_DATA, "no server uuid", + [&] { Pool::open(backend, victimConfig(UInt128{})); }); + + const OwnerObject after = readOwner(*backend, layout, "victim"); + EXPECT_EQ(after.owner_life, before.owner_life); + EXPECT_TRUE(after.retired_at_ms.has_value()); + EXPECT_EQ(readSlot(*backend, layout, "victim").seq, slot_before.seq); + } } diff --git a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp index 34cb8b7fad077..09400c1ed89fc 100644 --- a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp +++ b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp @@ -54,6 +54,15 @@ bool slotObjectExists(Backend & backend, const String & leaf) return (*op).head("p/gc/server-roots/victim/" + leaf, Retry::standard()).has_value(); } +OwnerObject readOwnerDuty(Backend & backend) +{ + DB::Cas::tests::OperationForTest op(backend); + const auto current = (*op).read("p/gc/server-roots/victim/owner", Retry::standard()); + if (!current) + throw Exception(ErrorCodes::CORRUPTED_DATA, "fixture owner is absent"); + return decodeOwner(current->bytes); +} + class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend { public: @@ -87,60 +96,7 @@ class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend bool added = false; }; -/// Admits the late catalog entry between the retirement tail's two exact catalog reads -/// (`retirement_catalog_cut`, then `fresh_retirement_catalog`), never before. The mountpoint drain's -/// `list("p/roots/victim/", ...)` is the last LIST call in `decommissionPoolMember` before either -/// read, so it orders the two `read("p/cas/ref_catalog")` calls that follow it: the first is -/// `retirement_catalog_cut`, the second is `fresh_retirement_catalog`. Mutating on the second call -/// makes that read observe a catalog the first read did not. -class MutateCatalogBetweenRetirementReadsBackend final : public InMemoryBackend -{ -public: - /// Unhide the legacy `list` overloads the primitive override below would otherwise hide. - using Backend::list; - void arm() { armed = true; } - bool fired() const { return added; } - - /// Both hooks sit on the PRIMITIVES, which every legacy forwarder reaches too, so the ordering - /// they observe is the physical request order whichever surface issued each request. - RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override - { - RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); - if (armed && !past_mountpoint_drain && prefix == "p/roots/victim/" && cursor.empty()) - past_mountpoint_drain = true; - return page; - } - - std::optional read(const String & key, TransportAccess & access) override - { - if (armed && past_mountpoint_drain && !added && key == "p/cas/ref_catalog") - { - if (!seen_retirement_catalog_cut) - seen_retirement_catalog_cut = true; - else - { - added = true; - CasRequests requests = DB::Cas::tests::openRequestsForTest(*this); - CasOperation op = requests.admit(); - CasRefCatalog::casAdmitEntry( - op, Layout("p"), 1, - CatalogEntry{ - .ns = RootNamespace("victim/db/late"), - .state = NsState::Live, - .incarnation = UInt128{707}}); - } - } - return InMemoryBackend::read(key, access); - } - -private: - bool armed = false; - bool past_mountpoint_drain = false; - bool seen_retirement_catalog_cut = false; - bool added = false; -}; - -TEST(CASDecommissionCatalogDuties, RemovingWithoutCheckpointIsCorruptionAndKeepsSlot) +TEST(CASDecommissionCatalogDuties, RemovingWithoutCheckpointIsCorruptionAndLeavesAnUnverifiedTombstone) { auto backend = std::make_shared(); CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); @@ -166,11 +122,13 @@ TEST(CASDecommissionCatalogDuties, RemovingWithoutCheckpointIsCorruptionAndKeeps EXPECT_TRUE(slotObjectExists(*backend, "owner")); EXPECT_TRUE(slotObjectExists(*backend, "epoch")); EXPECT_TRUE(slotObjectExists(*backend, "mount")); + EXPECT_TRUE(readOwnerDuty(*backend).retired_at_ms.has_value()); + EXPECT_FALSE(readOwnerDuty(*backend).retirement_verified); EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), RootNamespace("victim/db/missing_ckpt")).state, NsState::Removing); } -TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndKeepsSlotForGc) +TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndIsVerifiedWithTheRowPendingGc) { auto backend = std::make_shared(); CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); @@ -193,8 +151,9 @@ TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndKeeps EXPECT_EQ(report.namespaces_already_removed, 1u); EXPECT_EQ(wake_requests.load(), 1u); - EXPECT_FALSE(report.slot_removed); - EXPECT_FALSE(report.warnings.empty()); + EXPECT_TRUE(report.member_retired); + EXPECT_TRUE(report.warnings.empty()); + EXPECT_EQ(report.namespaces_pending_gc, 1u); EXPECT_TRUE(slotObjectExists(*backend, "owner")); const ListPage stream = catalog_op.list(Layout("p").namespaceStreamPrefix(*life), "", 100, Retry::standard()); @@ -246,12 +205,14 @@ TEST(CASDecommissionCatalogDuties, PartialRemovalProgressStillWakesGcWhenLaterNa EXPECT_EQ(wake_requests.load(), 1u) << "progress already made for an earlier life must wake GC even when a later life fails closed"; EXPECT_TRUE(slotObjectExists(*backend, "owner")); + EXPECT_TRUE(readOwnerDuty(*backend).retired_at_ms.has_value()); + EXPECT_FALSE(readOwnerDuty(*backend).retirement_verified); const ListPage progressed_stream = catalog_op.list(Layout("p").namespaceStreamPrefix(*progressed_life), "", 100, Retry::standard()); ASSERT_EQ(progressed_stream.keys.size(), 1u); } -TEST(CASDecommissionCatalogDuties, VictimEntryAppearingBeforeTheOwnershipCutKeepsSlot) +TEST(CASDecommissionCatalogDuties, VictimEntryAppearingAfterTheCutBlocksTheMark) { auto backend = std::make_shared(); CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); @@ -263,31 +224,11 @@ TEST(CASDecommissionCatalogDuties, VictimEntryAppearingBeforeTheOwnershipCutKeep backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); EXPECT_TRUE(backend->fired()); - EXPECT_FALSE(report.slot_removed); - ASSERT_FALSE(report.warnings.empty()); - EXPECT_NE(report.warnings.front().find("pool member decommission underway: 1 namespace(s)"), String::npos) - << report.warnings.front(); - EXPECT_TRUE(slotObjectExists(*backend, "owner")); - EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), RootNamespace("victim/db/late")).state, NsState::Live); -} - -TEST(CASDecommissionCatalogDuties, CatalogTokenMovedBetweenOwnershipCutAndRetirementKeepsSlot) -{ - auto backend = std::make_shared(); - CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); - CasOperation catalog_op = catalog_requests.admit(); - { auto victim = openVictim(backend); } - backend->arm(); - - const DecommissionReport report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); - - EXPECT_TRUE(backend->fired()); - EXPECT_FALSE(report.slot_removed); + EXPECT_FALSE(report.member_retired); ASSERT_FALSE(report.warnings.empty()); - EXPECT_NE(report.warnings.front().find("catalog changed after the victim ownership check"), String::npos) - << report.warnings.front(); - EXPECT_TRUE(slotObjectExists(*backend, "owner")); + EXPECT_NE(report.warnings.front().find("appeared after"), String::npos) << report.warnings.front(); + EXPECT_TRUE(readOwnerDuty(*backend).retired_at_ms.has_value()); + EXPECT_FALSE(readOwnerDuty(*backend).retirement_verified); EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), RootNamespace("victim/db/late")).state, NsState::Live); } @@ -325,7 +266,9 @@ TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAn EXPECT_EQ(wake_requests.load(), 1u); EXPECT_EQ(report.namespaces_already_removed, 1u); - EXPECT_FALSE(report.slot_removed); + EXPECT_TRUE(report.member_retired); + EXPECT_TRUE(report.warnings.empty()); + EXPECT_EQ(report.namespaces_pending_gc, 1u); EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), ns).state, NsState::Removing); std::vector stream_after; for (const ListedKey & key : catalog_op.list(Layout("p").namespaceStreamPrefix(*life), "", 100, Retry::standard()).keys) @@ -350,7 +293,7 @@ TEST(CASDecommissionCatalogDuties, OpaqueLifeDebrisWithoutCatalogOwnershipDoesNo backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); EXPECT_TRUE(report.warnings.empty()); - EXPECT_TRUE(report.slot_removed); + EXPECT_TRUE(report.member_retired); EXPECT_TRUE(catalog_op.head(debris_key, Retry::standard()).has_value()); } diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index 2bee87edb300e..68f65658a6d57 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -1360,6 +1360,40 @@ TEST(CASHeartbeat, CancellationBeforeSendIsNotAttemptedAndAllowsRelease) EXPECT_EQ(renewer.state(), MountLeaseRenewerState::Released); } +/// The stop lands after the renewal began and before its admission. Nothing was sent, so the renewer stays +/// releasable and the teardown still writes the farewell. +TEST(CASHeartbeat, CancellationRacingAdmissionIsNotAttemptedAndAllowsRelease) +{ + auto backend = std::make_shared(); + Layout layout("pool"); + uint64_t wall_ms = 1000; + uint64_t boot_ms = 100; + bool cancelled = false; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); + MountLeaseRenewer renewer( + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), + [&] { return boot_ms; }); + renewer.start(); + backend->attempts.clear(); + backend->read_calls = 0; + const auto stop_then_refuse = [&] + { + cancelled = true; + return false; + }; + const MountRenewResult result = renewer.renew(renewalEnvironment( + boot_ms, /*live=*/stop_then_refuse, /*cancelled=*/[&] { return cancelled; })); + EXPECT_EQ(result.outcome, MountRenewOutcome::NotAttempted); + EXPECT_EQ(result.failure, nullptr); + ASSERT_EQ(renewer.state(), MountLeaseRenewerState::Active); + EXPECT_TRUE(backend->attempts.empty()); + EXPECT_NO_THROW(renewer.release()); + EXPECT_EQ(decodeMountLease(ops.op.read(layout.mountKey("test"), Retry::standard())->bytes).min_active_build_sequence, + std::numeric_limits::max()); +} + TEST(CASHeartbeat, CancellationAfterSendIsTerminalAndForbidsRelease) { auto backend = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_layout.cpp b/src/Disks/tests/gtest_cas_layout.cpp index 1006e87188a2a..48f42a6ff6cf4 100644 --- a/src/Disks/tests/gtest_cas_layout.cpp +++ b/src/Disks/tests/gtest_cas_layout.cpp @@ -3,6 +3,9 @@ #include #include #include "cas_test_helpers.h" +#include +#include +#include using namespace DB::Cas; @@ -380,3 +383,28 @@ TEST(CASLayout, BlobKeyRoundTripsThroughParse) EXPECT_EQ(*parsed_meta, ref); EXPECT_FALSE(layout.parseBlobKey("pool0/blobs/unknown-algo/aa/aa00").has_value()); /// foreign => nullopt } + +TEST(CASLayout, StagingKeyCarriesTheWriterEpoch) +{ + const Layout layout("p"); + EXPECT_EQ(layout.stagingPrefix("a/b"), "p/staging/a/b/"); + const String key = layout.stagingKey("a/b", 7, "x.tmp"); + EXPECT_EQ(key, "p/staging/a/b/7/x.tmp"); + EXPECT_EQ(Layout::stagedEpochOf(std::string_view(key).substr(layout.stagingPrefix("a/b").size())), 7u); + + const std::vector>> cases{ + {"x.tmp", 0}, /// flat key written before keys carried an epoch + {"0/x.tmp", 0}, + {"007/x.tmp", 7}, + {"18446744073709551615/x.tmp", std::numeric_limits::max()}, + {"18446744073709551616/x.tmp", std::nullopt}, /// overflow + {"", std::nullopt}, + {"7/", std::nullopt}, /// a folder marker + {"/x.tmp", std::nullopt}, + {"a/x.tmp", std::nullopt}, /// a nested member's key + {"+7/x.tmp", std::nullopt}, + {"7/a/x.tmp", std::nullopt}, /// deeper than one epoch directory + }; + for (const auto & [rest, expected] : cases) + EXPECT_EQ(Layout::stagedEpochOf(rest), expected) << rest; +} diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 41a6baf861c62..8c05367b269d2 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -9,12 +9,15 @@ #include #include +#include #include #include +#include #include #include #include #include +#include #include #include #include @@ -172,6 +175,51 @@ class OwnerRaceBackend : public InMemoryBackend UInt128 winner; }; +/// Owner ETags derived from the bytes, as S3 derives them: equal bytes bring back an equal ETag. +class ContentEtagOwnerBackend : public InMemoryBackend +{ +public: + std::optional read(const String & key, TransportAccess & access) override + { + auto raw = InMemoryBackend::read(key, access); + if (raw && isOwner(key)) + raw->value = contentValue(raw->bytes); + return raw; + } + + std::optional head(const String & key, TransportAccess & access) override + { + auto meta = InMemoryBackend::head(key, access); + if (meta && isOwner(key)) + if (const auto raw = InMemoryBackend::read(key, access)) + meta->value = contentValue(raw->bytes); + return meta; + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + if (!isOwner(key)) + return InMemoryBackend::write(key, bytes, expected_value, access); + std::optional minted_expected; + if (expected_value) + { + const auto current = InMemoryBackend::read(key, access); + if (!current || contentValue(current->bytes) != *expected_value) + return std::unexpected(RawConflict{}); + minted_expected = current->value; + } + const auto written = InMemoryBackend::write(key, bytes, minted_expected, access); + if (!written) + return written; + return contentValue(bytes); + } + +private: + static bool isOwner(const String & key) { return key.ends_with("/owner"); } + static String contentValue(const String & bytes) { return "content-" + DB::Cas::tests::hexOf(bytes); } +}; + /// Refuses the FIRST write of the epoch key after installing a competing allocator's own epoch, so the /// absent-epoch decision has to be made a second time. `reveal_owned_work` decides whether owned work /// becomes visible at that same instant -- the fact the second decision must re-establish. @@ -265,27 +313,25 @@ class NewerStartBackend : public InMemoryBackend } }; -/// A competitor's conditional bump of the epoch counter lands with the very bytes ours sends, and ours -/// is lost unapplied. Our resolve read then finds our bytes under a moved precondition and reports the -/// allocation committed, so two allocators hold one writer_epoch. -class EpochCollisionBackend : public InMemoryBackend +/// Records every write of `key`; while `lose_first` is set, the first one times out without applying. +class UnappliedFirstWriteBackend : public InMemoryBackend { public: - String epoch_key; - bool armed = false; + String key; + bool lose_first = false; + std::vector>> writes; std::expected write( - const String & key, const String & bytes, + const String & k, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { - if (armed && expected_value && key == epoch_key) + if (k == key) { - armed = false; - if (!InMemoryBackend::write(key, bytes, expected_value, access)) - throw std::runtime_error("EpochCollisionBackend: the competitor's bump conflicted"); - throw Poco::TimeoutException("EpochCollisionBackend: response lost, our attempt did not apply"); + writes.emplace_back(bytes, expected_value); + if (std::exchange(lose_first, false)) + throw Poco::TimeoutException("UnappliedFirstWriteBackend: response lost, the attempt did not apply"); } - return InMemoryBackend::write(key, bytes, expected_value, access); + return InMemoryBackend::write(k, bytes, expected_value, access); } }; @@ -310,6 +356,27 @@ class LandedReplaceAckLostOnceBackend : public InMemoryBackend } }; +/// On the first write of `key` after `competitor` is set, runs `competitor` (another allocator of the +/// same uuid) to completion, then times out our write without applying it. +class TwinAllocatorBackend : public InMemoryBackend +{ +public: + String key; + std::function competitor; + + std::expected write( + const String & k, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + if (competitor && k == key) + { + std::exchange(competitor, {})(); + throw Poco::TimeoutException("TwinAllocatorBackend: response lost, our attempt did not apply"); + } + return InMemoryBackend::write(k, bytes, expected_value, access); + } +}; + /// Counts every request that reaches the store, per primitive, so a test can pin how many a protocol /// step costs rather than only what it produced. class RequestCountingBackend final : public InMemoryBackend @@ -359,36 +426,6 @@ class RenewalLogBackend final : public InMemoryBackend } }; -class ScopedRenewalLogCapture -{ -public: - explicit ScopedRenewalLogCapture(const String & level) - : logger(getLogger("CasMountLeaseRenewer")) - , channel(new Poco::StreamChannel(stream)) - , old_channel(logger->getChannel(), /*shared=*/true) - , old_level(logger->getLevel()) - { - logger->setChannel(channel.get()); - logger->setLevel(level); - } - - ~ScopedRenewalLogCapture() - { - logger->setChannel(old_channel); - logger->setLevel(old_level); - } - - String captured() const { return stream.str(); } - -private: - LoggerPtr logger; - std::ostringstream stream; // STYLE_CHECK_ALLOW_STD_STRING_STREAM - Poco::AutoPtr channel; - /// A real reference (shared=true), so the parked previous channel cannot die while ours is installed. - Poco::AutoPtr old_channel; - int old_level; -}; - size_t countRenewalLogText(const String & haystack, std::string_view needle) { size_t count = 0; @@ -443,7 +480,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) auto backend = std::make_shared(); auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-silent"); - ScopedRenewalLogCapture capture("information"); + DB::Cas::tests::ScopedLoggerCapture capture("CasMountLeaseRenewer", "information"); EXPECT_NO_THROW(store->renewWatermarkOnce()); EXPECT_EQ(countRenewalLogText(capture.captured(), "CAS mount renewal"), 0u); } @@ -452,7 +489,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) auto backend = std::make_shared(); auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-recovered"); - ScopedRenewalLogCapture capture("information"); + DB::Cas::tests::ScopedLoggerCapture capture("CasMountLeaseRenewer", "information"); backend->throw_before_next_overwrite = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); const String output = capture.captured(); @@ -465,7 +502,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) auto backend = std::make_shared(); auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-debug"); - ScopedRenewalLogCapture capture("debug"); + DB::Cas::tests::ScopedLoggerCapture capture("CasMountLeaseRenewer", "debug"); backend->throw_before_next_overwrite = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); EXPECT_EQ(countRenewalLogText(capture.captured(), "physical retry attempt 2"), 1u); @@ -475,7 +512,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) auto backend = std::make_shared(); auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-fenced"); - ScopedRenewalLogCapture capture("information"); + DB::Cas::tests::ScopedLoggerCapture capture("CasMountLeaseRenewer", "information"); /// The lease was claimed at boot 100 with the 1000 ms TTL above, so it expires at 1100. The /// fence admits only while the remaining time strictly clears the safety margin plus whatever /// the attempt reserves, so exactly `margin` remaining (with the reservation on top) refuses. @@ -614,10 +651,113 @@ TEST(CASServerRootClaim, TombstonedSameOwnerFailsClosed) { EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); EXPECT_NE(e.message().find("decommissioned"), String::npos) << e.message(); - EXPECT_EQ(e.message().find("owned by a different server"), String::npos) << e.message(); } } +TEST(CASServerRootClaim, FirstClaimWritesLifeOne) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + EXPECT_EQ(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()), 1u); + const OwnerObject stored = decodeOwner(ops.op.read(l.ownerKey("r"), Retry::standard())->bytes); + EXPECT_EQ(stored.owner_life, 1u); + EXPECT_FALSE(stored.retirement_verified); + /// A restart under the same owner writes nothing and reads the same life. + const Etag before = currentEtag(ops.op, l.ownerKey("r")); + EXPECT_EQ(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()), 1u); + EXPECT_EQ(currentEtag(ops.op, l.ownerKey("r")), before); +} + +TEST(CASServerRootClaim, ClaimOverA2664OwnerReadsLifeZero) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + mustCommit(ops.op.create(l.ownerKey("r"), encodeOwner(OwnerObject{.server_uuid = UInt128(1), .retired_at_ms = std::nullopt}), Retry::standard()), + "owner without a life"); + EXPECT_EQ(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()), 0u); +} + +TEST(CASServerRootClaim, DecommissionPolicyAcceptsARetiredOwnerOfTheSameUuid) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + mustCommit(ops.op.create(l.ownerKey("r"), encodeOwner(OwnerObject{ + .server_uuid = UInt128(1), .retired_at_ms = 1752537600000ULL, .owner_life = 2}), Retry::standard()), "tombstone"); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] + { + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation(), RetiredOwnerPolicy::Refuse); + }); + EXPECT_EQ(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation(), + RetiredOwnerPolicy::AcceptForDecommission), 2u); + /// A retired owner of ANOTHER uuid is still foreign under either policy; the refusal also names a duplicate id. + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "same ", [&] + { + claimOwnerOrThrow(ops.op, l, "r", UInt128(2), emptyCatalogObservation(), RetiredOwnerPolicy::AcceptForDecommission); + }); +} + +/// The owner appears between the claim's read and its create: the conflict re-read applies the same retired +/// policy as the first read. +TEST(CASServerRootClaim, ConflictRereadAppliesTheRetiredPolicy) +{ + for (const RetiredOwnerPolicy policy : + {RetiredOwnerPolicy::Refuse, RetiredOwnerPolicy::AcceptForDecommission, RetiredOwnerPolicy::RejoinIfVerified}) + { + SCOPED_TRACE(static_cast(policy)); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + bool raced = false; + b->onBeforeWrite(l.ownerKey("r"), [&] + { + if (std::exchange(raced, true)) + return; + Ops racer(b); + mustCommit(racer.op.create(l.ownerKey("r"), encodeOwner(OwnerObject{ + .server_uuid = UInt128(1), .retired_at_ms = 1752537600000ULL, .owner_life = 2}), Retry::standard()), + "racing tombstone"); + }); + if (policy != RetiredOwnerPolicy::AcceptForDecommission) + { + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "decommission is not finished", [&] + { + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation(), policy); + }); + } + else + { + EXPECT_EQ(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation(), policy), 2u); + } + EXPECT_TRUE(raced); + } +} + +/// A start over an unverified tombstone is refused and writes neither the counter nor the slot. +TEST(CASMount, PoolOpenRefusesAnUnverifiedRetirement) +{ + auto b = std::make_shared(); + Layout l("p"); + { auto first = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r"}); } + Ops ops(b); + const auto owner = ops.op.read(l.ownerKey("r"), Retry::standard()); + OwnerObject tombstone = decodeOwner(owner->bytes); + tombstone.retired_at_ms = 1; + tombstone.owner_life += 1; + tombstone.retirement_verified = false; + mustCommit(ops.op.replace(l.ownerKey("r"), encodeOwner(tombstone), owner->etag, Retry::standard()), "tombstone"); + const Etag epoch_before = currentEtag(ops.op, l.epochKey("r")); + const Etag mount_before = currentEtag(ops.op, l.mountKey("r")); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "decommission is not finished", [&] + { + Pool::open(b, PoolConfig{.pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r"}); + }); + EXPECT_EQ(currentEtag(ops.op, l.epochKey("r")), epoch_before); + EXPECT_EQ(currentEtag(ops.op, l.mountKey("r")), mount_before); +} + /// Each identity refusal names the supported recovery: the decommission verb for a dead foreign owner, /// a new id for a retired one. Neither advises a hand edit of the owner object as the first option. TEST(CASServerRootClaim, RefusalsNameTheSupportedRecovery) @@ -643,8 +783,8 @@ TEST(CASServerRootClaim, RefusalsNameTheSupportedRecovery) catch (const DB::Exception & e) { EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); - EXPECT_NE(e.message().find("explicitly decommissioned"), String::npos) << e.message(); - EXPECT_NE(e.message().find("new "), String::npos) << e.message(); + EXPECT_NE(e.message().find("was decommissioned"), String::npos) << e.message(); + EXPECT_NE(e.message().find("decommission is not finished"), String::npos) << e.message(); EXPECT_EQ(e.message().find("clear the owner object"), String::npos) << e.message(); } } @@ -655,8 +795,8 @@ TEST(CASServerRootEpoch, AllocatorIsMonotoneAndSurvivesMountConcept) Layout l("r"); Ops ops(b); claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); - const uint64_t e1 = allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); - const uint64_t e2 = allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); + const uint64_t e1 = allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()); + const uint64_t e2 = allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()); EXPECT_GE(e1, 1u); // 0 is a reserved sentinel EXPECT_GT(e2, e1); // strictly increasing @@ -664,7 +804,102 @@ TEST(CASServerRootEpoch, AllocatorIsMonotoneAndSurvivesMountConcept) /// so the removal is a no-op that touches nothing. ASSERT_FALSE(ops.op.head(l.mountKey("r"), Retry::standard()).has_value()); EXPECT_EQ(ops.op.removeCurrent(l.mountKey("r"), Retry::standard()), Removal::Gone); - EXPECT_GT(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), e2); + EXPECT_GT(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), e2); +} + +/// Our counter write is lost while a twin of the same uuid allocates the same next epoch. Its counter +/// body must not read back as our landed write, on both the create and the replace path. +TEST(CASServerRootEpoch, TwinAllocatorsNeverShareAnEpochAfterALostWrite) +{ + for (const bool counter_exists : {false, true}) + { + SCOPED_TRACE(counter_exists ? "replace" : "create"); + auto b = std::make_shared(); + Layout l("p"); + Ops ours(b); + Ops twin(b); + b->key = l.epochKey("r"); + if (counter_exists) + ASSERT_EQ(allocateWriterEpoch(ours.op, l, "r", emptyCatalogObservation()), 1u); + + uint64_t twin_epoch = 0; + b->competitor = [&] + { + twin_epoch = allocateWriterEpoch(twin.op, l, "r", emptyCatalogObservation()); + }; + const uint64_t our_epoch + = allocateWriterEpoch(ours.op, l, "r", emptyCatalogObservation()); + ASSERT_FALSE(b->competitor) << "the twin never ran"; + ASSERT_NE(twin_epoch, 0u); + EXPECT_NE(our_epoch, twin_epoch); + EXPECT_EQ(decodeServerEpoch(ours.op.read(l.epochKey("r"), Retry::standard())->bytes).next_writer_epoch, + std::max(our_epoch, twin_epoch) + 1); + } +} + +/// A resend of one allocation decision carries that decision's nonce, so our own landed write with a +/// lost response is recognized as ours instead of costing a second epoch. +TEST(CASServerRootEpoch, LandedCounterWriteWithLostResponseIsOurs) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + b->key = l.epochKey("r"); + ASSERT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 1u); + + b->armed = true; + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 2u); + ASSERT_FALSE(b->armed) << "the lost response was never injected"; + EXPECT_EQ(decodeServerEpoch(ops.op.read(l.epochKey("r"), Retry::standard())->bytes).next_writer_epoch, 3u); +} + +/// An unapplied write with an unclear outcome is reissued inside the same decision: every physical +/// attempt must carry the same body, nonce included, under the same precondition. +TEST(CASServerRootEpoch, ReissuedCounterWriteKeepsItsNonce) +{ + for (const bool counter_exists : {false, true}) + { + SCOPED_TRACE(counter_exists ? "replace" : "create"); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + b->key = l.epochKey("r"); + if (counter_exists) + ASSERT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 1u); + + b->writes.clear(); + b->lose_first = true; + const uint64_t expected_epoch = counter_exists ? 2 : 1; + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), + expected_epoch); + ASSERT_FALSE(b->lose_first) << "the lost write was never injected"; + ASSERT_GE(b->writes.size(), 2u) << "the unclear write was never reissued"; + for (const auto & [bytes, precondition] : b->writes) + { + EXPECT_EQ(bytes, b->writes.front().first); + EXPECT_EQ(precondition, b->writes.front().second); + } + const ServerEpoch stored = decodeServerEpoch(ops.op.read(l.epochKey("r"), Retry::standard())->bytes); + EXPECT_EQ(stored.next_writer_epoch, expected_epoch + 1); + EXPECT_TRUE(stored.allocation_nonce.has_value()); + } +} + +TEST(CASServerRootEpoch, EveryAllocationWritesAFreshNonce) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + std::vector nonces; + for (int i = 0; i < 2; ++i) + { + (void)allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()); + const std::optional nonce + = decodeServerEpoch(ops.op.read(l.epochKey("r"), Retry::standard())->bytes).allocation_nonce; + ASSERT_TRUE(nonce.has_value()); + nonces.push_back(*nonce); + } + EXPECT_NE(nonces[0], nonces[1]); } /// No writer emits a zero counter; an allocation over one would hand out writer_epoch 0. @@ -680,7 +915,7 @@ TEST(CASServerRootEpoch, ZeroCounterFailsAllocationWithoutWriting) DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - (void)allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); + (void)allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()); }); EXPECT_EQ(ops.op.read(l.epochKey("r"), Retry::standard())->bytes, before->bytes); } @@ -697,7 +932,7 @@ TEST(CASMount, EpochRemintOverExistingMountRefuses) ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/1, /*now_ms=*/1000, /*ttl_ms=*/30000).kind, MountClaimResult::Claimed); /// The epoch object is ABSENT (never created in this sequence) while the mount exists: - EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); /// CORRUPTED_DATA + EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), DB::Exception); /// CORRUPTED_DATA } TEST(CASMount, EpochRemintAuthoritativeAbsenceMints) @@ -706,8 +941,8 @@ TEST(CASMount, EpochRemintAuthoritativeAbsenceMints) Layout l("p"); Ops ops(b); claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); - EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); /// fresh root: both control objects absent - EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u); /// epoch present now: normal conditional bump, no probe + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 1u); /// fresh root: both control objects absent + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 2u); /// epoch present now: normal conditional bump, no probe } /// The probe outcome gates the mint: anything short of authoritative KeyAbsent fails closed. @@ -725,35 +960,7 @@ TEST(CASMount, EpochRemintIndeterminateProbeFailsClosed) Layout l("p"); Ops ops(b); claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); - EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); -} - -/// Decommission over a TERMINAL (expired/fenced) mount with a lost epoch object proceeds and mints -/// an epoch DISTINCT from the surviving mount's — the same-pair state is unrepresentable. -TEST(CASMount, DecommissionRemintOverTerminalMountMintsDistinctEpoch) -{ - auto b = std::make_shared(); - Layout l("p"); - Ops ops(b); - claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); - ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/3, /*now_ms=*/1000, /*ttl_ms=*/100).kind, - MountClaimResult::Claimed); - /// now_ms=5000: the ttl_ms=100 lease above is long expired -> terminal. - EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::DecommissionRecovery, /*now_ms=*/5000, emptyCatalogObservation()), 4u); -} - -/// Decommission over a LIVE mount with a lost epoch refuses — the blind bypass would recreate the -/// forbidden pair (codex round-3 finding 1) and defeat CASDecommission.RefusesLiveMember. -TEST(CASMount, DecommissionRemintOverLiveMountRefuses) -{ - auto b = std::make_shared(); - Layout l("p"); - Ops ops(b); - claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); - ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/1, /*now_ms=*/1000, /*ttl_ms=*/30000).kind, - MountClaimResult::Claimed); - EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::DecommissionRecovery, /*now_ms=*/2000, emptyCatalogObservation()), - DB::Exception); /// ABORTED: live member + EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), DB::Exception); } /// The steady-state path (epoch object PRESENT) must never pay the probe — pins the zero @@ -774,9 +981,9 @@ TEST(CASMount, EpochBumpWithPresentEpochIssuesNoProbe) Layout l("p"); Ops ops(b); claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); - EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); /// bootstrap: ONE probe (absent-epoch branch) + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 1u); /// bootstrap: ONE probe (absent-epoch branch) const int probes_after_bootstrap = b->probes; - EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u); /// epoch present: normal conditional bump... + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", emptyCatalogObservation()), 2u); /// epoch present: normal conditional bump... EXPECT_EQ(b->probes, probes_after_bootstrap) << "...must not probe the mount key"; } @@ -787,7 +994,17 @@ TEST(CASServerRootClaim, MissingOwnerOverNonEmptyRootIsCorrupted) Ops ops(b); /// Simulate existing data without an owner (identity lost): plant a key under roots//. mustCommit(ops.op.create(l.serverRootDataPrefix("r") + "some-data", "x", Retry::standard()), "root debris"); - EXPECT_THROW(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()), DB::Exception); + try + { + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + FAIL() << "the claim over an ownerless non-empty subtree must be refused"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA) << e.message(); + EXPECT_NE(e.message().find("refusing to re-claim"), String::npos) << e.message(); + EXPECT_NE(e.message().find("Configure a new for this disk"), String::npos) << e.message(); + } } TEST(CASServerRootSafety, EveryCatalogLifecycleStateBlocksOwnerAndEpochRecreation) @@ -808,7 +1025,7 @@ TEST(CASServerRootSafety, EveryCatalogLifecycleStateBlocksOwnerAndEpochRecreatio Ops epoch_ops(std::make_shared()); EXPECT_THROW(allocateWriterEpoch( - epoch_ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, observe), DB::Exception); + epoch_ops.op, layout, "root/x", observe), DB::Exception); EXPECT_FALSE(epoch_ops.op.head(layout.epochKey("root/x"), Retry::standard()).has_value()); } } @@ -834,7 +1051,7 @@ TEST(CASServerRootSafety, OpaqueStreamAndStateDebrisAloneDoesNotBlockRecreation) EXPECT_NO_THROW(claimOwnerOrThrow(ops.op, layout, "rootx", UInt128{1}, emptyCatalogObservation())); EXPECT_EQ(allocateWriterEpoch( - ops.op, layout, "rootx", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); + ops.op, layout, "rootx", emptyCatalogObservation()), 1u); } TEST(CASServerRootSafety, ManifestAndLooseRootDebrisStillBlockRecreation) @@ -852,7 +1069,7 @@ TEST(CASServerRootSafety, ManifestAndLooseRootDebrisStillBlockRecreation) claimOwnerOrThrow(ops.op, layout, "root/x", UInt128{1}, emptyCatalogObservation()); }); EXPECT_THROW(allocateWriterEpoch( - ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + ops.op, layout, "root/x", emptyCatalogObservation()), DB::Exception); } } @@ -870,7 +1087,7 @@ TEST(CASServerRootSafety, UnreadableCatalogNeverFallsBackToPhysicalGuesses) claimOwnerOrThrow(ops.op, layout, "root/x", UInt128{1}, unreadable); }); EXPECT_THROW(allocateWriterEpoch( - ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, unreadable), DB::Exception); + ops.op, layout, "root/x", unreadable), DB::Exception); EXPECT_FALSE(ops.op.head(layout.ownerKey("root/x"), Retry::standard()).has_value()); EXPECT_FALSE(ops.op.head(layout.epochKey("root/x"), Retry::standard()).has_value()); } @@ -897,7 +1114,7 @@ TEST(CASServerRootSafety, EpochConflictRecomputesTheWholeEmptinessBundle) Ops ops(backend); const Layout layout("p"); EXPECT_THROW(allocateWriterEpoch( - ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + ops.op, layout, "root/x", emptyCatalogObservation()), DB::Exception); EXPECT_TRUE(backend->fired); ASSERT_TRUE(backend->winner_installed); const auto epoch = ops.op.read(layout.epochKey("root/x"), Retry::standard()); @@ -1185,62 +1402,32 @@ TEST(CASMountAwaitExpiry, LiveRenewingTwinTimesOutAsDoubleStart) namespace { -/// fix-round F5 harness: makes the mount key vanish to EVERY read, unconditionally, while the real -/// underlying object stays put -- forcing `claimMount`'s own read to take the absent-slot race -/// branch every call (its create then conflicts against the real, still-present object, returning -/// `LiveDoubleStart` with no incarnation -- that branch deliberately leaves `.etag` unset, -/// since no re-read was done). That in turn forces `claimMountAwaitingExpiry`'s fallback re-read, -/// which ALSO sees the slot as vanished -- deterministically reproducing "the slot vanished between -/// claimMount's own read and ours" on EVERY loop iteration, not just a lucky one-shot race. -class AlwaysVanishesBackend final : public DB::Cas::Backend +/// Hides the watched key from every read while the object stays underneath, so `claimMount`'s create +/// conflicts with nothing observed and the observation loop's re-read finds the slot absent every time. +class AlwaysVanishesBackend final : public InMemoryBackend { public: - explicit AlwaysVanishesBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} String watched_key; - bool supportsListTokens() const override { return inner->supportsListTokens(); } - - /// The fault is on the read primitive, which is the only way anything now reaches the store. std::optional read(const String & key, TransportAccess & access) override { if (key == watched_key) return std::nullopt; - return inner->read(key, access); - } - std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } - RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } - RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } - void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } - size_t bulkDeleteKeyLimit() const override { return inner->bulkDeleteKeyLimit(); } - std::expected write(const String & key, const String & bytes, - const std::optional & expected_value, TransportAccess & access) override - { - return inner->write(key, bytes, expected_value, access); + return InMemoryBackend::read(key, access); } - std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } - void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } - Dialect dialect() const override { return inner->dialect(); } - -private: - std::shared_ptr inner; }; } -/// fix-round F5 (author-review: `!got -> continue` in the observation loop, with no sleep and outside -/// the restart limit, spins `get`/`claimMount`/`put` at backend RTT under persistent slot churn). A -/// backend that makes the mount slot look vanished to every GET must still terminate (bounded restarts, -/// not an infinite loop) AND must pace itself (the injected `sleep_fn` must actually fire) rather than -/// busy-spin. +/// A slot deleted under the observation loop (a 26.6.4 decommission still deletes it) is re-claimed on the +/// same paced, bounded restart budget as a renewing holder, not spun at backend RTT. TEST(CASMountAwaitExpiry, PersistentSlotVanishPacesAndBoundsRestartsInsteadOfSpinning) { - auto inner = std::make_shared(); + auto vanishing = std::make_shared(); Layout l("p"); - Ops inner_ops(inner); - /// A real slot exists underneath (uuid 1, epoch 7) so `claimMount`'s absent-slot create - /// genuinely conflicts every time (never accidentally re-mints). - ASSERT_EQ(claimMount(inner_ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); - - auto vanishing = std::make_shared(inner); + { + Ops seed(vanishing); + ASSERT_EQ(claimMount(seed.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); + } vanishing->watched_key = l.mountKey("r"); Ops ops(vanishing); @@ -1253,11 +1440,10 @@ TEST(CASMountAwaitExpiry, PersistentSlotVanishPacesAndBoundsRestartsInsteadOfSpi const auto r = claimMountAwaitingExpiry( ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); - EXPECT_EQ(r.kind, MountClaimResult::LiveDoubleStart) << "must terminate (bounded), not loop forever"; - EXPECT_GT(sleeps, 0) << "a persistently vanishing slot must still pace via sleep_fn, not busy-spin"; - /// The real epoch-7 lease is untouched -- every create attempt against it genuinely conflicts - /// (the object is still there), so it is never accidentally re-minted over. - EXPECT_EQ(decodeMountLease(inner_ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 7u); + EXPECT_EQ(r.kind, MountClaimResult::LiveDoubleStart); + EXPECT_GT(sleeps, 0); + vanishing->watched_key.clear(); + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 7u); } TEST(CASMountAwaitExpiry, ForeignUuidFailsClosedImmediately) @@ -1905,37 +2091,51 @@ TEST(CASHeartbeatFloor, RenewalBetweenRoundsRestartsObservation) EXPECT_EQ(obs.at("s1").first_seen_mono_ms, kStableThresholdMs); } -/// fix-round F7 (author-review: `Gc::mount_obs` not pruned for srids gone from LIST -> slow unbounded -/// growth on a long-lived leader, worsened by pool-member decommission). A srid whose `/mount` key is -/// removed ENTIRELY (not merely fenced/terminated -- those already `obs.erase` themselves mid-loop) is -/// never visited by a later LIST pass again, so its observation entry must be pruned at end-of-round, -/// not linger in `obs` forever. -TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) +/// A slot that turns terminal leaves the observation map in the walk that sees it. +TEST(CASHeartbeatFloor, TerminalSlotLeavesTheObservationMap) { auto b = std::make_shared(); Layout l("p"); Ops ops(b); seedMount(ops.op, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); seedMount(ops.op, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); - MountObservationMap obs; std::set warned; computeHeartbeatFloor(ops.op, l, kNowMs, frozenClock(0), kObserverLeaseMs, kCadenceMs, obs, warned); ASSERT_TRUE(obs.contains("s1")); ASSERT_TRUE(obs.contains("s2")); - /// s2's `/mount` key is removed entirely -- e.g. `SYSTEM CAS DROP POOL MEMBER` -- so - /// no future LIST pass will ever visit it again. s1 renews (a live renewer would), so its OWN - /// observation restarts and it stays `live` -- isolating this test to the pruning behavior alone, - /// not confounding it with s1 also becoming fence-eligible (which would erase its `obs` entry too, - /// for an unrelated reason). renewMount(ops.op, l, "s1"); - ASSERT_EQ(ops.op.removeCurrent(l.mountKey("s2"), Retry::standard()), Removal::Removed); + const auto got = ops.op.read(l.mountKey("s2"), Retry::standard()); + MountLease farewell = decodeMountLease(got->bytes); + farewell.min_active_build_sequence = std::numeric_limits::max(); + mustCommit(ops.op.replace(l.mountKey("s2"), encodeMountLease(farewell), got->etag, Retry::standard()), "s2 farewell"); + const HeartbeatFloor floor = computeHeartbeatFloor(ops.op, l, kNowMs, frozenClock(kStableThresholdMs), kObserverLeaseMs, kCadenceMs, obs, warned); + EXPECT_EQ(floor.terminated, 1u); + EXPECT_TRUE(obs.contains("s1")); + EXPECT_FALSE(obs.contains("s2")); +} + +/// A slot deleted by a 26.6.4 decommission or by hand is never listed again; its observation is dropped. +TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + seedMount(ops.op, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(ops.op, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + MountObservationMap obs; + std::set warned; + computeHeartbeatFloor(ops.op, l, kNowMs, frozenClock(0), kObserverLeaseMs, kCadenceMs, obs, warned); + ASSERT_TRUE(obs.contains("s2")); + + /// s1 renews so it stays observed for its own reason; only s2's absence from the listing is under test. + renewMount(ops.op, l, "s1"); + ASSERT_EQ(ops.op.removeCurrent(l.mountKey("s2"), Retry::standard()), Removal::Removed); computeHeartbeatFloor(ops.op, l, kNowMs, frozenClock(kStableThresholdMs), kObserverLeaseMs, kCadenceMs, obs, warned); EXPECT_TRUE(obs.contains("s1")); - EXPECT_FALSE(obs.contains("s2")) - << "a srid removed from the LIST entirely must be pruned from obs, not linger forever"; + EXPECT_FALSE(obs.contains("s2")); } TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) @@ -2643,21 +2843,13 @@ TEST(CASClaimMountMonotonicity, LowerEpochOverANewerBodyFailsWithoutWriting) } } -TEST(CASClaimMountMonotonicity, TwoAllocatorsSharingAnEpochDoNotShareTheSlot) +/// Two processes holding one epoch, as 26.6.4 allocators without the counter nonce could produce. +TEST(CASClaimMountMonotonicity, TwoClaimersOfOneEpochDoNotShareTheSlot) { - auto b = std::make_shared(); + auto b = std::make_shared(); Layout l("p"); Ops ops(b); - b->epoch_key = l.epochKey("r"); - ASSERT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); - - b->armed = true; - const uint64_t shared_epoch - = allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); - ASSERT_FALSE(b->armed) << "the collision was never injected"; - ASSERT_EQ(shared_epoch, 2u); - /// One bump past the shared epoch: the competitor's, which our resolve read took for ours. - ASSERT_EQ(decodeServerEpoch(ops.op.read(l.epochKey("r"), Retry::standard())->bytes).next_writer_epoch, 3u); + const uint64_t shared_epoch = 2; /// The competitor claims first. ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), shared_epoch, 1000, 30000).kind, MountClaimResult::Claimed); @@ -2972,7 +3164,7 @@ TEST(CASServerRoot, AllocateWriterEpochKeepsThePostConflictCorruptionCheck) auto backend = std::make_shared(/*reveal_owned_work=*/true); Ops ops(backend); EXPECT_THROW(allocateWriterEpoch( - ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + ops.op, layout, "root/x", emptyCatalogObservation()), DB::Exception); EXPECT_TRUE(backend->fired); ASSERT_TRUE(backend->winner_installed); } @@ -2980,7 +3172,7 @@ TEST(CASServerRoot, AllocateWriterEpochKeepsThePostConflictCorruptionCheck) auto backend = std::make_shared(/*reveal_owned_work=*/false); Ops ops(backend); EXPECT_EQ(allocateWriterEpoch( - ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u) + ops.op, layout, "root/x", emptyCatalogObservation()), 2u) << "the second decision must allocate from the conflict winner's epoch, not refuse outright"; EXPECT_TRUE(backend->fired); ASSERT_TRUE(backend->winner_installed); @@ -3132,6 +3324,35 @@ TEST(CASServerRootClaim, OwnerLostToARacerIsDecidedFromTheConflictObservation) } } +/// A tombstone sent against the owner's first ETag, delivered after a tombstone and a rejoin. Without a +/// life the rejoined owner's bytes equal the first owner's, so a content-derived ETag would admit it. +TEST(CASOwner, DelayedTombstoneAfterRejoinFailsOnLife) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + const String key = l.ownerKey("r"); + const OwnerObject first{.server_uuid = UInt128(1), .retired_at_ms = std::nullopt, .owner_life = 1}; + mustCommit(ops.op.create(key, encodeOwner(first), Retry::standard()), "first owner"); + const Etag stale = currentEtag(ops.op, key); + + OwnerObject tombstone = first; + tombstone.retired_at_ms = 1000; + tombstone.owner_life = 2; + mustCommit(ops.op.replace(key, encodeOwner(tombstone), currentEtag(ops.op, key), Retry::standard()), "tombstone"); + OwnerObject rejoined = first; + rejoined.owner_life = 3; + mustCommit(ops.op.replace(key, encodeOwner(rejoined), currentEtag(ops.op, key), Retry::standard()), "rejoin"); + ASSERT_NE(currentEtag(ops.op, key), stale) << "the life makes the rejoined bytes differ from the first"; + + OwnerObject late = first; + late.retired_at_ms = 2000; + late.owner_life = 2; + const WriteResult landed = ops.op.replace(key, encodeOwner(late), stale, Retry::standard()); + EXPECT_TRUE(std::holds_alternative(landed)); + EXPECT_EQ(decodeOwner(ops.op.read(key, Retry::standard())->bytes).owner_life, 3u); +} + /// A remount re-anchors its lease BEFORE it arms the fence for the new incarnation, so the fence is /// still latched lost at that moment. The steady-state renewal is refused there — the sibling test /// above pins that — and the remount's own renewal has to be admitted off the fence, or the pool could @@ -3164,3 +3385,446 @@ TEST(CASMountLease, RemountRenewalIsAdmittedOffTheMountFence) CasOperation reader = open_requests.admit(); EXPECT_EQ(decodeMountLease(reader.read(l.mountKey("r"), Retry::standard())->bytes).seq, 2u); } + +namespace +{ +/// A rejoin by hand: the next life under `uuid`, not retired. +void rewriteOwnerAt(Backend & backend, const Layout & l, const String & srid, UInt128 uuid) +{ + DB::Cas::tests::OperationForTest op(backend); + const auto current = (*op).read(l.ownerKey(srid), Retry::once()); + ASSERT_TRUE(current.has_value()); + const OwnerObject next{ + .server_uuid = uuid, .retired_at_ms = std::nullopt, .owner_life = decodeOwner(current->bytes).owner_life + 1, + .retirement_verified = false}; + ASSERT_TRUE(std::holds_alternative( + (*op).replace(l.ownerKey(srid), encodeOwner(next), current->etag, Retry::once()))); +} + +MountLease slotBody(Backend & backend, const Layout & l, const String & srid) +{ + DB::Cas::tests::OperationForTest op(backend); + const auto current = (*op).read(l.mountKey(srid), Retry::once()); + if (!current) + throw std::runtime_error("mount absent: " + l.mountKey(srid)); + return decodeMountLease(current->bytes); +} + +PoolConfig rootConfig() +{ + return PoolConfig{.pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r"}; +} +} + +/// The tombstone lands between the owner claim and the mount write; the only write of the mount key in +/// this open is the reclaim of the first open's farewell. +TEST(CASMount, NormalMountRefusesARetiredOwnerAfterClaim) +{ + auto b = std::make_shared(); + Layout l("p"); + { auto first = Pool::open(b, rootConfig()); } + bool fired = false; + b->onBeforeWrite(l.mountKey("r"), [&] + { + if (std::exchange(fired, true)) + return; + DB::Cas::tests::tombstoneOwnerForTest(*b, l.ownerKey("r")); + }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "decommissioned", [&] { Pool::open(b, rootConfig()); }); + ASSERT_TRUE(fired); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.server_uuid, UInt128(1)); + EXPECT_EQ(slot.writer_epoch, 2u) << "the refused claim's own body, not the first open's farewell"; + EXPECT_TRUE(slot.gc_fenced) << "the refused claimer fences itself so nobody waits its lease out"; +} + +/// A victim prechecked before a decommission wakes after a new-uuid rejoin and reclaims its own farewell. +TEST(CASMount, ClaimRecheckRefusesAForeignOwner) +{ + auto b = std::make_shared(); + Layout l("p"); + { auto first = Pool::open(b, rootConfig()); } + bool fired = false; + b->onBeforeWrite(l.mountKey("r"), [&] + { + if (std::exchange(fired, true)) + return; + rewriteOwnerAt(*b, l, "r", UInt128(2)); + }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "different server", [&] { Pool::open(b, rootConfig()); }); + ASSERT_TRUE(fired); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.server_uuid, UInt128(1)); + EXPECT_EQ(slot.writer_epoch, 2u); + EXPECT_TRUE(slot.gc_fenced); +} + +TEST(CASMount, ClaimRecheckRefusesAnAbsentOwner) +{ + auto b = std::make_shared(); + Layout l("p"); + { auto first = Pool::open(b, rootConfig()); } + bool fired = false; + b->onBeforeWrite(l.mountKey("r"), [&] + { + if (std::exchange(fired, true)) + return; + DB::Cas::tests::OperationForTest op(*b); + ASSERT_EQ((*op).removeCurrent(l.ownerKey("r"), Retry::once()), Removal::Removed); + }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "vanished", [&] { Pool::open(b, rootConfig()); }); + ASSERT_TRUE(fired); + EXPECT_TRUE(slotBody(*b, l, "r").gc_fenced); +} + +/// The owner read refuses, and the self-fence write cannot reach the store: the refusal is still the +/// owner's reason and the slot keeps the claimer's live body for GC to fence. +TEST(CASMount, RecheckRefusalSurvivesAFailedSelfFence) +{ + auto b = std::make_shared(); + Layout l("p"); + { auto first = Pool::open(b, rootConfig()); } + bool fired = false; + b->onBeforeWrite(l.mountKey("r"), [&] + { + if (std::exchange(fired, true)) + return; + DB::Cas::tests::tombstoneOwnerForTest(*b, l.ownerKey("r")); + /// The claim write has already passed its armed-failure check, so this arming fails the next + /// mount write: the self-fence. + b->failNextWriteWith(l.mountKey("r"), std::make_exception_ptr( + DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "store refused the fence write"))); + }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "decommissioned", [&] { Pool::open(b, rootConfig()); }); + ASSERT_TRUE(fired); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.writer_epoch, 2u) << "the claim itself landed"; + EXPECT_FALSE(slot.gc_fenced) << "the fence write was refused; the body is untouched"; +} + +/// The GC fenced our lease; the remount reclaims it, and the owner was tombstoned in between. +TEST(CASMount, FenceRecoveryReclaimRechecksTheOwner) +{ + auto b = std::make_shared(); + Layout l("p"); + auto store = Pool::open(b, rootConfig()); + DB::Cas::tests::fenceOutMountForTest(*b, l.mountKey("r")); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::ABORTED, [&] { store->renewWatermarkOnce(); }); + bool fired = false; + b->onBeforeWrite(l.mountKey("r"), [&] + { + if (std::exchange(fired, true)) + return; + DB::Cas::tests::tombstoneOwnerForTest(*b, l.ownerKey("r")); + }); + EXPECT_FALSE(store->tryRemountOnce()); + ASSERT_TRUE(fired); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.writer_epoch, 2u) << "the remount's fresh epoch reached the slot"; + EXPECT_TRUE(slot.gc_fenced) << "and was fenced by the refused recheck"; + EXPECT_FALSE(store->mayMutate()); +} + +/// Another process of the same uuid rejoined (the life moved) while this one was fenced. The remount sees the +/// moved life before it allocates an epoch and stops for good instead of retrying. +TEST(CASMount, RemountRefusesAfterASameUuidRejoin) +{ + auto b = std::make_shared(); + Layout l("p"); + auto store = Pool::open(b, rootConfig()); + rewriteOwnerAt(*b, l, "r", UInt128(1)); /// life 2, same uuid, not retired + DB::Cas::tests::fenceOutMountForTest(*b, l.mountKey("r")); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::ABORTED, [&] { store->renewWatermarkOnce(); }); + Ops ops(b); + const String counter_before = ops.op.read(l.epochKey("r"), Retry::standard())->bytes; + + EXPECT_FALSE(store->tryRemountOnce()); + EXPECT_EQ(store->lifecycle(), PoolLifecycle::IdentityLost); + EXPECT_FALSE(store->tryRemountOnce()); + + EXPECT_EQ(ops.op.read(l.epochKey("r"), Retry::standard())->bytes, counter_before) << "no epoch was allocated"; + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.writer_epoch, 1u); + EXPECT_TRUE(slot.gc_fenced); +} + +namespace +{ +void seedOwner(Ops & ops, const Layout & l, const OwnerObject & owner) +{ + mustCommit(ops.op.create(l.ownerKey("r"), encodeOwner(owner), Retry::standard()), "owner"); +} + +MountLease liveBody(UInt128 uuid, uint64_t epoch) +{ + return MountLease{ + .server_uuid = uuid, .writer_epoch = epoch, .hostname = "h", .pid = 1, .started_at_ms = 1, .seq = 1, + .expires_at_ms = 100, .min_active_build_sequence = 0, .gc_fenced = false, .write_attempt_id = UInt128(9)}; +} +} + +/// A refusing recheck rewrites only its own live `(uuid, epoch)` body; every other slot body keeps its bytes. +TEST(CASMount, RecheckSelfFenceLeavesAnyOtherBodyAlone) +{ + MountLease other_epoch = liveBody(UInt128(1), 6); + MountLease foreign = liveBody(UInt128(2), 7); + MountLease farewell = liveBody(UInt128(1), 7); + farewell.min_active_build_sequence = std::numeric_limits::max(); + MountLease fenced = liveBody(UInt128(1), 7); + fenced.gc_fenced = true; + const std::vector> cases{ + {"same uuid, another epoch", other_epoch}, {"foreign uuid", foreign}, {"clean farewell", farewell}, + {"already fenced", fenced}}; + for (const auto & [name, body] : cases) + { + SCOPED_TRACE(name); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + seedOwner(ops, l, OwnerObject{.server_uuid = UInt128(1), .retired_at_ms = std::nullopt, .owner_life = 2, + .retirement_verified = false}); + mustCommit(ops.op.create(l.mountKey("r"), encodeMountLease(body), Retry::standard()), "slot"); + const auto before = ops.op.read(l.mountKey("r"), Retry::standard()); + ASSERT_TRUE(before.has_value()); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, "owner moved from life 1 to life 2", [&] + { + recheckOwnerAfterClaim(ops.op, l, "r", + OwnerExpectation{.server_uuid = UInt128(1), .owner_life = 1, .retired = RetiredOwnerPolicy::Refuse}, 7); + }); + const auto after = ops.op.read(l.mountKey("r"), Retry::standard()); + ASSERT_TRUE(after.has_value()); + EXPECT_EQ(after->bytes, before->bytes); + EXPECT_EQ(after->etag, before->etag); + } +} + +/// The decommission admin accepts a retired owner, but not a foreign one or one at another life. +TEST(CASMount, RecheckForDecommissionStillRefusesForeignOwnerAndMovedLife) +{ + const OwnerExpectation admin{.server_uuid = UInt128(1), .owner_life = 2, .retired = RetiredOwnerPolicy::AcceptForDecommission}; + struct Case + { + String name; + OwnerObject owner; + String refusal; + }; + const std::vector cases{ + {"retired at the expected life", {.server_uuid = UInt128(1), .retired_at_ms = 5, .owner_life = 2, .retirement_verified = false}, ""}, + {"foreign uuid", {.server_uuid = UInt128(2), .retired_at_ms = 5, .owner_life = 2, .retirement_verified = false}, "different server"}, + {"moved life", {.server_uuid = UInt128(1), .retired_at_ms = 5, .owner_life = 3, .retirement_verified = false}, "owner moved from life 2 to life 3"}}; + for (const Case & c : cases) + { + SCOPED_TRACE(c.name); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + seedOwner(ops, l, c.owner); + mustCommit(ops.op.create(l.mountKey("r"), encodeMountLease(liveBody(UInt128(1), 7)), Retry::standard()), "slot"); + const auto run = [&] { recheckOwnerAfterClaim(ops.op, l, "r", admin, 7); }; + if (c.refusal.empty()) + { + EXPECT_NO_THROW(run()); + EXPECT_FALSE(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).gc_fenced); + } + else + { + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::CORRUPTED_DATA, c.refusal, run); + EXPECT_TRUE(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).gc_fenced); + } + } +} + +namespace +{ +MountLease terminalBody(UInt128 uuid, uint64_t epoch, bool fenced) +{ + MountLease body = liveBody(uuid, epoch); + if (fenced) + body.gc_fenced = true; + else + body.min_active_build_sequence = std::numeric_limits::max(); + return body; +} + +const OwnerExpectation uuid_one_at_life_two{.server_uuid = UInt128(1), .owner_life = 2, .retired = RetiredOwnerPolicy::Refuse}; + +OwnerObject liveOwner(UInt128 uuid, uint64_t life) +{ + return OwnerObject{.server_uuid = uuid, .retired_at_ms = std::nullopt, .owner_life = life, .retirement_verified = false}; +} + +/// The slot's body with another `server_uuid`, everything else kept: what a rejoin under a new uuid leaves. +void rewriteSlotUuid(Backend & backend, const Layout & l, const String & srid, UInt128 uuid) +{ + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(l.mountKey(srid), Retry::standard()); + ASSERT_TRUE(got.has_value()); + MountLease m = decodeMountLease(got->bytes); + m.server_uuid = uuid; + m.seq += 1; + ASSERT_TRUE(std::holds_alternative( + (*op).replace(l.mountKey(srid), encodeMountLease(m), got->etag, Retry::standard()))); +} +} + +/// After a rejoin under a new uuid the slot can hold another uuid's farewell or self-fence; the server +/// whose owner life it is takes it over. +TEST(CASMount, ForeignTerminalBodyIsReclaimable) +{ + for (const bool fenced : {false, true}) + { + SCOPED_TRACE(fenced ? "gc_fenced" : "clean farewell"); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + seedOwner(ops, l, liveOwner(UInt128(1), 2)); + mustCommit(ops.op.create(l.mountKey("r"), encodeMountLease(terminalBody(UInt128(2), 7, fenced)), Retry::standard()), "slot"); + const MountClaimResult claim = claimMount(ops.op, l, "r", UInt128(1), 8, 1000, 100, {}, {}, {}, uuid_one_at_life_two); + ASSERT_EQ(claim.kind, MountClaimResult::Claimed); + EXPECT_EQ(claim.prior, fenced ? MountPriorState::Fenced : MountPriorState::Clean); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.server_uuid, UInt128(1)); + EXPECT_EQ(slot.writer_epoch, 8u); + } +} + +/// The owner is read before the reclaim: a reclaim that a later recheck refuses would leave an old-uuid +/// body that a 26.6.4 starter (no recheck) could take. +TEST(CASMount, ForeignReclaimReadsTheOwnerFirst) +{ + struct Case + { + String name; + std::optional owner; + std::optional expected; + }; + OwnerObject retired = liveOwner(UInt128(1), 2); + retired.retired_at_ms = 5; + const std::vector cases{ + {"owner of another uuid", liveOwner(UInt128(3), 2), uuid_one_at_life_two}, + {"owner retired", retired, uuid_one_at_life_two}, + {"owner at another life", liveOwner(UInt128(1), 3), uuid_one_at_life_two}, + {"owner absent", std::nullopt, uuid_one_at_life_two}, + {"no expectation passed", liveOwner(UInt128(1), 2), std::nullopt}}; + for (const Case & c : cases) + { + SCOPED_TRACE(c.name); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + if (c.owner) + seedOwner(ops, l, *c.owner); + mustCommit(ops.op.create(l.mountKey("r"), encodeMountLease(terminalBody(UInt128(2), 7, false)), Retry::standard()), "slot"); + const auto before = ops.op.read(l.mountKey("r"), Retry::standard()); + EXPECT_EQ(claimMount(ops.op, l, "r", UInt128(1), 8, 1000, 100, {}, {}, {}, c.expected).kind, MountClaimResult::ForeignOwner); + EXPECT_EQ(ops.op.read(l.mountKey("r"), Retry::standard())->etag, before->etag); + } +} + +/// A foreign body at or above our epoch is never replaced: a delayed claimer with a lower epoch stays out. +TEST(CASMount, ForeignTerminalBodyAtOrAboveOurEpochStaysForeign) +{ + for (const uint64_t body_epoch : {8, 9}) + { + SCOPED_TRACE(body_epoch); + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + seedOwner(ops, l, liveOwner(UInt128(1), 2)); + mustCommit(ops.op.create(l.mountKey("r"), encodeMountLease(terminalBody(UInt128(2), body_epoch, false)), Retry::standard()), "slot"); + const auto before = ops.op.read(l.mountKey("r"), Retry::standard()); + EXPECT_EQ(claimMount(ops.op, l, "r", UInt128(1), 8, 1000, 100, {}, {}, {}, uuid_one_at_life_two).kind, MountClaimResult::ForeignOwner); + EXPECT_EQ(ops.op.read(l.mountKey("r"), Retry::standard())->etag, before->etag); + } +} + +/// A live body of another uuid is never taken, whatever the owner says. +TEST(CASMount, ForeignLiveBodyStaysForeignOwner) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + seedOwner(ops, l, liveOwner(UInt128(1), 2)); + mustCommit(ops.op.create(l.mountKey("r"), encodeMountLease(liveBody(UInt128(2), 7)), Retry::standard()), "slot"); + const auto before = ops.op.read(l.mountKey("r"), Retry::standard()); + EXPECT_EQ(claimMount(ops.op, l, "r", UInt128(1), 8, 1000, 100, {}, {}, {}, uuid_one_at_life_two).kind, MountClaimResult::ForeignOwner); + EXPECT_EQ(ops.op.read(l.mountKey("r"), Retry::standard())->etag, before->etag); +} + +/// A start and a self-remount pass their owner expectation, so both take over a terminal body of another uuid. +TEST(CASMount, ClaimPathsReclaimAForeignTerminalBodyOfTheirOwnOwner) +{ + { + SCOPED_TRACE("start"); + auto b = std::make_shared(); + Layout l("p"); + { auto first = Pool::open(b, rootConfig()); } + rewriteSlotUuid(*b, l, "r", UInt128(2)); + auto store = Pool::open(b, rootConfig()); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.server_uuid, UInt128(1)); + EXPECT_EQ(slot.writer_epoch, store->writerEpoch()); + } + { + SCOPED_TRACE("self-remount"); + auto b = std::make_shared(); + Layout l("p"); + auto store = Pool::open(b, rootConfig()); + DB::Cas::tests::fenceOutMountForTest(*b, l.mountKey("r")); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::ABORTED, [&] { store->renewWatermarkOnce(); }); + rewriteSlotUuid(*b, l, "r", UInt128(2)); + EXPECT_TRUE(store->tryRemountOnce()); + const MountLease slot = slotBody(*b, l, "r"); + EXPECT_EQ(slot.server_uuid, UInt128(1)); + EXPECT_FALSE(slot.gc_fenced); + } +} + +/// A terminal row reads its owner: a tombstone makes it `retired`, and the row carries whether the retirement is +/// verified. A live row never reads its owner (its owner here is undecodable). +TEST(CASListMounts, TerminalSlotOfATombstonedOwnerIsRetired) +{ + const auto state_of = [](Ops & ops, const Layout & l, const String & srid) -> String + { + for (const MountInfo & m : listMounts(ops.op, l, /*now_ms=*/0, /*skew_margin_ms=*/0)) + if (m.srid == srid) + return m.state + (m.retirement_verified ? "+verified" : ""); + return "absent"; + }; + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + { auto first = Pool::open(b, rootConfig()); } + EXPECT_EQ(state_of(ops, l, "r"), "terminated"); + DB::Cas::tests::tombstoneOwnerForTest(*b, l.ownerKey("r")); + EXPECT_EQ(state_of(ops, l, "r"), "retired") << "an unverified tombstone"; + DB::Cas::tests::tombstoneOwnerForTest(*b, l.ownerKey("r"), /*verified=*/true); + EXPECT_EQ(state_of(ops, l, "r"), "retired+verified"); + DB::Cas::tests::fenceOutMountForTest(*b, l.mountKey("r")); + EXPECT_EQ(state_of(ops, l, "r"), "retired+verified") << "a fenced slot of a retired owner"; + rewriteOwnerAt(*b, l, "r", UInt128(2)); + EXPECT_EQ(state_of(ops, l, "r"), "fenced") << "a rejoined owner"; + + mustCommit(ops.op.create(l.ownerKey("x"), "garbage-not-a-proto", Retry::standard()), "undecodable owner of x"); + mustCommit(ops.op.create(l.mountKey("x"), encodeMountLease(liveBody(UInt128(1), 4)), Retry::standard()), "live slot of x"); + EXPECT_EQ(state_of(ops, l, "x"), "live") << "a non-terminal row keeps its lease state and never reads its owner"; +} + +TEST(CASListMounts, TerminalSlotWithAnUndecodableOwnerIsCorrupt) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + { auto first = Pool::open(b, rootConfig()); } + DB::Cas::tests::OperationForTest op(*b); + const auto current = (*op).read(l.ownerKey("r"), Retry::once()); + ASSERT_TRUE(current.has_value()); + ASSERT_TRUE(std::holds_alternative((*op).replace(l.ownerKey("r"), "garbage-not-a-proto", current->etag, Retry::once()))); + mustCommit(ops.op.create(l.mountKey("other"), encodeMountLease(liveBody(UInt128(1), 4)), Retry::standard()), "live slot of other"); + std::map by_srid; + for (const MountInfo & m : listMounts(ops.op, l, /*now_ms=*/0, /*skew_margin_ms=*/0)) + by_srid[m.srid] = m.state; + EXPECT_EQ(by_srid.size(), 2u); + EXPECT_EQ(by_srid["r"], "corrupt"); + EXPECT_EQ(by_srid["other"], "live"); +} diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index 2c68bf8492fd0..3d87eab377a7a 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -11,10 +11,12 @@ #include "cas_sweep_test_support.h" #include "cas_test_helpers.h" #include +#include #include #include #include #include +#include using namespace DB::Cas; using namespace DB::Cas::tests; @@ -178,6 +180,72 @@ TEST(CASOrphanManifestSweep, EligibleAndUnownedIsDeleted) EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } +/// The decommission drain runs this on its admin pool: once that pool lost its mount, nothing is deleted. +TEST(CASOrphanManifestSweep, LostMountDeletesNothing) +{ + auto backend = std::make_shared(); + auto store = openPoolForTest(backend); + const RootNamespace ns{"00/aa@cas@"}; + registerNamespaceRaw(*backend, store->layout(), ns); + const ManifestRef r = ref(5, 0xAB); + writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active_build_sequence*/6); + seedConsumedSealCursor(*backend, store->layout(), ns); + seedEmptyRecoveryAuthority(*backend, store->layout(), ns); + store->tripMountLost(); + + std::vector warnings; + EXPECT_THROW(sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}, &warnings), + DB::Exception); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); +} + +namespace +{ +/// Runs `on_first_manifest_remove` once, inside the first delete of a manifest, after it applied. +class HookOnFirstManifestRemoveBackend : public InMemoryBackend +{ +public: + std::function on_first_manifest_remove; + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + const RawRemoval removed = InMemoryBackend::remove(key, expected_value, access); + if (on_first_manifest_remove && key.starts_with("p/cas/manifests/")) + std::exchange(on_first_manifest_remove, {})(); + return removed; + } +}; +} + +/// The mount is lost after the first delete: the walk stops at the first refused key instead of adding one +/// warning per remaining key. +TEST(CASOrphanManifestSweep, LostMountStopsTheWalkAtTheFirstRefusal) +{ + auto backend = std::make_shared(); + auto store = openPoolForTest(backend); + const RootNamespace ns{"00/aa@cas@"}; + registerNamespaceRaw(*backend, store->layout(), ns); + std::vector keys; + for (const uint64_t ordinal : {0xABu, 0xACu, 0xADu, 0xAEu}) + { + const ManifestRef r = ref(5, ordinal); + writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); + keys.push_back(store->layout().manifestKey(ManifestId{ns, r})); + } + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active_build_sequence*/6); + seedConsumedSealCursor(*backend, store->layout(), ns); + seedEmptyRecoveryAuthority(*backend, store->layout(), ns); + backend->on_first_manifest_remove = [&] { store->tripMountLost(); }; + + std::vector warnings; + sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}, &warnings); + + EXPECT_EQ(warnings.size(), 1u) << fmt::format("{}", fmt::join(warnings, "; ")); + const auto kept = std::count_if(keys.begin(), keys.end(), [&](const String & key) { return headExists(*backend, key); }); + EXPECT_GE(kept, 2); +} + /// The orphan sweep must not turn a forged same-id snapshot at an OLDER `EpochSeal` into an empty owner /// view. The base differs from `last_epoch_seal`, so metadata equality cannot catch it; the candidate /// remains retained until the checkpoint is repaired. diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 46639ec4d8fc0..c546b31b78857 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -1654,6 +1654,19 @@ TEST(CASPoolRemount, FenceOutThenSelfRemountRestoresWrites) EXPECT_NO_THROW(store->renewWatermarkOnce()); } +TEST(CASPool, RestartKeepsTheOwnerLife) +{ + auto backend = std::make_shared(); + String owner_key; + { auto store = DB::Cas::tests::openPoolForTest(backend); owner_key = store->layout().ownerKey(store->poolConfig().server_root_id); } + DB::Cas::tests::OperationForTest op(*backend); + const auto before = (*op).read(owner_key, DB::Cas::Retry::once()); + ASSERT_TRUE(before.has_value()); + EXPECT_EQ(DB::Cas::decodeOwner(before->bytes).owner_life, 1u); + auto again = DB::Cas::tests::openPoolForTest(backend); + EXPECT_EQ((*op).read(owner_key, DB::Cas::Retry::once())->etag, before->etag) << "a plain restart writes no owner"; +} + TEST(CASPoolRemount, OldEpochBuildFailsClosedAfterRemount) { auto backend = std::make_shared(); @@ -3862,7 +3875,7 @@ TEST(CASPoolRemount, NaturalTerminalTransitionMakesBothPersistentWorkersSelfExit "test", sink, runtimeRenewBudget(), [&] { if (terminal == PoolLifecycle::IdentityLost) - runtime_ptr->enterIdentityLost(); + runtime_ptr->enterIdentityLost("forced by the test"); else runtime_ptr->enterVanished(PoolLifecycle::VanishedReplaced, "injected natural replacement"); transitioned.arriveAndWait(); @@ -3968,7 +3981,7 @@ TEST(CASPoolRemount, ParkedRenewalCannotMissNaturalTerminalPublication) { parked_predicate_sampled_false.wait(); if (terminal == PoolLifecycle::IdentityLost) - runtime_ptr->enterIdentityLost(); + runtime_ptr->enterIdentityLost("forced by the test"); else runtime_ptr->enterVanished(PoolLifecycle::VanishedReplaced, "injected parked-wait replacement"); /// Before the fix, terminal publication does not wait for `driver_mutex`, so it reaches diff --git a/src/Disks/tests/gtest_cas_probe.cpp b/src/Disks/tests/gtest_cas_probe.cpp index 6b9da190dcade..136f36370547b 100644 --- a/src/Disks/tests/gtest_cas_probe.cpp +++ b/src/Disks/tests/gtest_cas_probe.cpp @@ -48,7 +48,17 @@ TEST(CASProbe, FailsClosedOnDeleteMarkers) b->setSimulateDeleteMarkers(true); // versioning enabled on the prefix auto requests = makeRequests(*b); auto op = requests.admit(); - EXPECT_THROW(runCapabilityProbe(op, "p/.cas_probe"), DB::Exception); + String message; + try + { + runCapabilityProbe(op, "p/.cas_probe"); + } + catch (const DB::Exception & e) + { + message = e.message(); + } + EXPECT_NE(message.find("does not make a versioned bucket usable"), String::npos) << message; + EXPECT_EQ(message.find("has no override"), String::npos) << message; } TEST(CASProbe, PassesOnEmulatedLocal) diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index 7919711c17531..64441e5e80e65 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -404,7 +404,7 @@ void burnEpochsUpTo(const BackendPtr & backend, const Layout & layout, uint64_t CasRequests requests(backend, Fence::open()); CasOperation op = requests.admit(); for (uint64_t e = 1; e < target_live_epoch; ++e) - allocateWriterEpoch(op, layout, "test", EpochMintPolicy::NormalMount, 0, [] { return RefCatalog{}; }); + allocateWriterEpoch(op, layout, "test", [] { return RefCatalog{}; }); } /// One ordinary transaction at `id`, publishing `ref` (prepending the birth op when `birth`). diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index bb34f6b198448..5b127709194ac 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -257,8 +257,7 @@ uint64_t allocateWriterEpochForTest(const BackendPtr & backend, const Layout & l { CasRequests requests(backend, Fence::open()); CasOperation op = requests.admit(); - return allocateWriterEpoch(op, layout, server_root_id, EpochMintPolicy::NormalMount, 0, - [] { return RefCatalog{}; }); + return allocateWriterEpoch(op, layout, server_root_id, [] { return RefCatalog{}; }); } /// A fixture enumeration on an open fence: the primitive `list` override in this file's test backend diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 42b320ef68ce7..caf65b6222507 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -1026,13 +1026,116 @@ TEST(CASStagingSweeper, RemovesOnlyObjectsUnderGivenMountPrefix) put(root + "/p/staging/mountA/two.tmp", "a2"); put(root + "/p/staging/mountB/three.tmp", "b1"); /// a DIFFERENT mount's staging — must survive - DB::Cas::sweepOwnMountStaging(*storage, root + "/p/staging/mountA/"); + DB::Cas::sweepOwnMountStaging(*storage, root + "/p/staging/mountA/", /*live_writer_epoch=*/1); EXPECT_FALSE(storage->exists(DB::StoredObject(root + "/p/staging/mountA/one.tmp"))); EXPECT_FALSE(storage->exists(DB::StoredObject(root + "/p/staging/mountA/two.tmp"))); EXPECT_TRUE(storage->exists(DB::StoredObject(root + "/p/staging/mountB/three.tmp"))); } +TEST(CASStagingSweeper, OldEpochSweepKeepsSuccessorStaging) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + const std::string root = storage->getCommonKeyPrefix(); + auto put = [&](const std::string & key) + { + auto buf = storage->writeObject(DB::StoredObject(key), DB::WriteMode::Rewrite); + buf->write("x", 1); + buf->finalize(); + }; + put(root + "/p/staging/m/1/a.tmp"); + put(root + "/p/staging/m/2/b.tmp"); + put(root + "/p/staging/m/3/c.tmp"); + + DB::Cas::sweepOwnMountStaging(*storage, root + "/p/staging/m/", /*live_writer_epoch=*/3); + + EXPECT_FALSE(storage->exists(DB::StoredObject(root + "/p/staging/m/1/a.tmp"))); + EXPECT_FALSE(storage->exists(DB::StoredObject(root + "/p/staging/m/2/b.tmp"))); + EXPECT_TRUE(storage->exists(DB::StoredObject(root + "/p/staging/m/3/c.tmp"))) << "the live epoch's own staging"; +} + +TEST(CASStagingSweeper, FlatLegacyKeysAreSwept) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + const std::string root = storage->getCommonKeyPrefix(); + auto buf = storage->writeObject(DB::StoredObject(root + "/p/staging/m/legacy.tmp"), DB::WriteMode::Rewrite); + buf->write("x", 1); + buf->finalize(); + DB::Cas::sweepOwnMountStaging(*storage, root + "/p/staging/m/", /*live_writer_epoch=*/1); + EXPECT_FALSE(storage->exists(DB::StoredObject(root + "/p/staging/m/legacy.tmp"))) << "a flat key counts as epoch 0"; +} + +TEST(CASStagingSweeper, NestedOtherMemberKeysAreKept) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + const std::string root = storage->getCommonKeyPrefix(); + auto put = [&](const std::string & key) + { + auto buf = storage->writeObject(DB::StoredObject(key), DB::WriteMode::Rewrite); + buf->write("x", 1); + buf->finalize(); + }; + put(root + "/p/staging/a/b/1/x.tmp"); /// member `a/b`, nested under `a` + put(root + "/p/staging/a/1/own.tmp"); + DB::Cas::sweepOwnMountStaging(*storage, root + "/p/staging/a/", /*live_writer_epoch=*/5); + EXPECT_TRUE(storage->exists(DB::StoredObject(root + "/p/staging/a/b/1/x.tmp"))) << "`b` is not an epoch"; + EXPECT_FALSE(storage->exists(DB::StoredObject(root + "/p/staging/a/1/own.tmp"))); +} + +TEST(CASStagingSweeper, DeeperPathsUnderAnEpochDirectoryAreKept) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + const std::string root = storage->getCommonKeyPrefix(); + auto buf = storage->writeObject(DB::StoredObject(root + "/p/staging/a/7/x/y.tmp"), DB::WriteMode::Rewrite); + buf->write("x", 1); + buf->finalize(); + DB::Cas::sweepOwnMountStaging(*storage, root + "/p/staging/a/", /*live_writer_epoch=*/9); + EXPECT_TRUE(storage->exists(DB::StoredObject(root + "/p/staging/a/7/x/y.tmp"))) << "member `a/7`'s key, not `a`'s"; +} + +/// The staging key's epoch and the admission generation the finalize rechecks come from one capture: a +/// self-remount between the two reads refuses the write instead of staging new-generation bytes under the +/// old epoch. Without the hook, the key carries the live epoch. +TEST(ContentAddressedTransaction, StagingEpochMatchesAdmissionGeneration) +{ + auto object_storage = makeFakeNativeCopyStorage(/*native_only_copy_supported=*/true); + auto metadata_storage = makeS3StagingMetadataStorageForTest(object_storage, "mountE"); + metadata_storage->startup(); + const DB::Cas::PoolPtr pool = metadata_storage->poolForTest(); + const uint64_t epoch_before = pool->liveWriterEpoch(); + const std::string path = "a11/a11a11a1-1111-4111-8111-111111111111/all_1_1_0/data.bin"; + + { + auto tx = metadata_storage->createTransaction(); + auto & ca_tx = dynamic_cast(*tx); + writeThroughS3Transaction(ca_tx, path, std::string(300, 'x')); + DB::RelativePathsWithMetadata staged; + object_storage->listObjects(metadata_storage->stagingKeyPrefix(), staged, /*max_keys=*/0); + ASSERT_EQ(staged.size(), 1u); + EXPECT_NE(staged.front()->relative_path.find("/" + std::to_string(epoch_before) + "/"), std::string::npos) + << staged.front()->relative_path; + } + + const String mount_key = pool->layout().mountKey("mountE"); + bool remounted = false; + pool->setIncarnationCaptureHookForTest([&] + { + if (std::exchange(remounted, true)) + return; + DB::Cas::tests::fenceOutMountForTest(*pool->poolBackendPtr(), mount_key); + try { pool->renewWatermarkOnce(); } catch (const DB::Exception &) {} + ASSERT_TRUE(pool->tryRemountOnce()); + }); + auto tx = metadata_storage->createTransaction(); + auto & ca_tx = dynamic_cast(*tx); + EXPECT_THROW(ca_tx.writeFile(path, 65536, DB::WriteMode::Rewrite, {}), DB::Exception); + ASSERT_TRUE(remounted); + EXPECT_EQ(pool->liveWriterEpoch(), epoch_before + 1); + DB::RelativePathsWithMetadata staged; + object_storage->listObjects(metadata_storage->stagingKeyPrefix() + std::to_string(epoch_before) + "/", staged, 0); + EXPECT_EQ(staged.size(), 1u) << "only the first write's key; the refused write staged nothing under the old epoch"; +} + /// (d) GC's blob-discovery LISTs ONLY `Layout::blobsPrefix()` (`/blobs/`) — a top-level prefix /// strictly disjoint from the S3-staging area (`/staging//`), so a staging object can /// never be listed, HEAD'd, or condemned as an orphan blob by GC's fold (`CasGc.cpp`, `CasFsck.cpp`). @@ -1058,7 +1161,6 @@ TEST(CASS3Staging, GcBlobDiscoveryPrefixExcludesStagingObjects) std::string staging_prefix = full_staging_prefix.substr(physical_root.size()); if (!staging_prefix.empty() && staging_prefix.front() == '/') staging_prefix.erase(0, 1); - staging_prefix += "/"; const std::string staging_key = staging_prefix + "aaa.tmp"; const DB::Cas::Layout layout(metadata_storage->poolForTest()->poolConfig().pool_prefix); @@ -1074,7 +1176,7 @@ TEST(CASS3Staging, GcBlobDiscoveryPrefixExcludesStagingObjects) /// Guard: the drain lists `/staging//` as a LOGICAL key and the backend applies the /// local root itself, while the writer mints its keys through `physicalKey`. Local prefix: /// `EmulatedSingleProcess` over a `LocalObjectStorage`, whose physical keys carry its root. -TEST(CASDecommission, StagingDrainRemovesWhatTheWriterStagedUnderTheLocalPrefix) +TEST(CASDecommissionStaging, StagingDrainRemovesWhatTheWriterStagedUnderTheLocalPrefix) { auto object_storage = makeFakeNativeCopyStorage(/*native_only_copy_supported=*/true); auto metadata_storage = makeS3StagingMetadataStorageForTest(object_storage, "victimLocal"); @@ -1452,7 +1554,7 @@ TEST(CASS3Staging, GenerationBackendMayUseNativeOnlyCopy) /// Native prefix: keys pass through unchanged on both sides, so a `physicalKey` that started adding a /// root for Native, or a drain that started adding one, would miss the writer's objects. -TEST(CASDecommission, StagingDrainRemovesWhatTheWriterStagedUnderTheNativePrefix) +TEST(CASDecommissionStaging, StagingDrainRemovesWhatTheWriterStagedUnderTheNativePrefix) { auto object_storage = makeFakeGenerationObjectStorageForTest(); auto metadata_storage = makeS3StagingMetadataStorageForTest(object_storage, "victimNative"); diff --git a/src/Disks/tests/gtest_cas_server_root_format.cpp b/src/Disks/tests/gtest_cas_server_root_format.cpp index 42fcf33e484f0..691174ad20ad0 100644 --- a/src/Disks/tests/gtest_cas_server_root_format.cpp +++ b/src/Disks/tests/gtest_cas_server_root_format.cpp @@ -1,8 +1,11 @@ #include "cas_format_test_battery.h" #include #include +#include #include +#include #include +#include using namespace DB::Cas; @@ -46,10 +49,48 @@ TEST(CASFormatBattery, ServerEpoch) { ServerEpoch e; e.next_writer_epoch = 7; + e.allocation_nonce = hexToU128("0123456789abcdeffedcba9876543210"); + const String golden = currentFormatHeader("cas_epoch") + + "{\"next_writer_epoch\":\"7\",\"allocation_nonce\":\"0123456789abcdeffedcba9876543210\"}\n"; + const ServerEpoch back = decodeServerEpoch(golden); + EXPECT_EQ(back.next_writer_epoch, 7u); + EXPECT_EQ(back.allocation_nonce, e.allocation_nonce); runFormatBattery({FormatId::ServerEpoch, [&] { return sealObject(FormatId::ServerEpoch, encodeServerEpoch(e)); }, [](std::string_view s) { decodeServerEpoch(std::string(openObject(FormatId::ServerEpoch, s))); }, - currentFormatHeader("cas_epoch") + "{\"next_writer_epoch\":\"7\"}\n"}); + golden}); +} + +/// The counter body written by 26.6.4, before the allocation nonce existed. +TEST(CASServerEpochFormat, BodyWithoutNonceDecodes) +{ + const ServerEpoch e = decodeServerEpoch(currentFormatHeader("cas_epoch") + "{\"next_writer_epoch\":\"7\"}\n"); + EXPECT_EQ(e.next_writer_epoch, 7u); + EXPECT_FALSE(e.allocation_nonce.has_value()); +} + +/// The 26.6.4 decoder's loop: it knows only `next_writer_epoch` and hands every other key to `skipUnknown`. +TEST(CASServerEpochFormat, NonceBodyDecodesWithThe2664KeySet) +{ + const String body = encodeServerEpoch(ServerEpoch{ + .next_writer_epoch = 7, .allocation_nonce = hexToU128("0123456789abcdeffedcba9876543210")}); + DB::ReadBufferFromMemory in(body.data(), body.size()); + expectHeaderLine(in, FormatId::ServerEpoch); + const String line = readLine(in, traitsFor(FormatId::ServerEpoch).line_cap, "server-epoch"); + DB::ReadBufferFromMemory line_in(line.data(), line.size()); + JsonObjectReader r(line_in, "server-epoch"); + std::optional next; + String key; + while (r.nextKey(key)) + { + if (key == "next_writer_epoch") + next = r.readU64String(); + else + r.skipUnknown(key); + } + EXPECT_EQ(next, 7u); + EXPECT_TRUE(line_in.eof()); + EXPECT_TRUE(in.eof()); } TEST(CASServerEpochFormat, ZeroNextWriterEpochIsRejected) @@ -181,3 +222,61 @@ TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) expectMessage(header + "{" + uuid + attempt + rest + "\n", "CAS mount-lease: missing writer_epoch"); expectMessage(header + "{" + uuid + epoch + rest + "\n", "CAS mount-lease: missing or zero write_attempt_id"); } + +TEST(CASOwnerFormat, LifeAndVerifiedRoundTrip) +{ + OwnerObject o; + o.server_uuid = hexToU128("0123456789abcdeffedcba9876543210"); + o.retired_at_ms = 1752537600000ULL; + o.owner_life = 7; + o.retirement_verified = true; + EXPECT_EQ(encodeOwner(o), currentFormatHeader("cas_owner") + + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"retired_at_ms\":1752537600000," + "\"owner_life\":\"7\",\"retirement_verified\":true}\n"); + const OwnerObject back = decodeOwner(encodeOwner(o)); + EXPECT_EQ(back.owner_life, 7u); + EXPECT_TRUE(back.retirement_verified); + EXPECT_EQ(back.retired_at_ms, o.retired_at_ms); +} + +/// The owner body written by 26.6.4 and by a never-retired owner: no life, no mark. +TEST(CASOwnerFormat, BodyWithoutLifeReadsAsZeroAndUnverified) +{ + const OwnerObject o = decodeOwner(currentFormatHeader("cas_owner") + + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\"}\n"); + EXPECT_EQ(o.owner_life, 0u); + EXPECT_FALSE(o.retirement_verified); + OwnerObject zero; + zero.server_uuid = o.server_uuid; + EXPECT_EQ(encodeOwner(zero), currentFormatHeader("cas_owner") + + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\"}\n") + << "a zero life and a false mark are not written, so the 26.6.4 bytes stay reachable"; +} + +/// The 26.6.4 decoder's loop knows `server_uuid` and `retired_at_ms` and hands every other key to `skipUnknown`. +TEST(CASOwnerFormat, NewFieldsDecodeWithThe2664KeySet) +{ + const String body = encodeOwner(OwnerObject{ + .server_uuid = hexToU128("0123456789abcdeffedcba9876543210"), + .retired_at_ms = 5, .owner_life = 3, .retirement_verified = true}); + DB::ReadBufferFromMemory in(body.data(), body.size()); + expectHeaderLine(in, FormatId::Owner); + const String line = readLine(in, traitsFor(FormatId::Owner).line_cap, "owner"); + DB::ReadBufferFromMemory line_in(line.data(), line.size()); + JsonObjectReader r(line_in, "owner"); + String key; + std::optional uuid; + std::optional retired; + while (r.nextKey(key)) + { + if (key == "server_uuid") + uuid = r.readHex128(); + else if (key == "retired_at_ms") + retired = r.readU64Number(); + else + r.skipUnknown(key); + } + EXPECT_EQ(uuid, hexToU128("0123456789abcdeffedcba9876543210")); + EXPECT_EQ(retired, 5u); + EXPECT_TRUE(line_in.eof()); +} diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index dc4295ff49938..e92d0ef416bd6 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -1111,8 +1111,9 @@ BlockIO InterpreterSystemQuery::execute() {"manifest_debris_removed", std::make_shared()}, {"staging_objects_removed", std::make_shared()}, {"mountpoint_objects_removed", std::make_shared()}, - {"slot_removed", std::make_shared()}, + {"member_retired", std::make_shared()}, {"warnings", std::make_shared()}, + {"namespaces_pending_gc", std::make_shared()}, }}; Block sample_block; for (const auto & column : columns) @@ -1128,8 +1129,9 @@ BlockIO InterpreterSystemQuery::execute() res_columns[i++]->insert(report.manifest_debris_removed); res_columns[i++]->insert(report.staging_objects_removed); res_columns[i++]->insert(report.mountpoint_objects_removed); - res_columns[i++]->insert(static_cast(report.slot_removed)); + res_columns[i++]->insert(static_cast(report.member_retired)); res_columns[i++]->insert(fmt::format("{}", fmt::join(report.warnings, "; "))); + res_columns[i++]->insert(report.namespaces_pending_gc); size_t num_rows = res_columns[0]->size(); auto source = std::make_shared(std::make_shared(std::move(sample_block)), Chunk(std::move(res_columns), num_rows)); diff --git a/src/Storages/System/StorageSystemContentAddressedMounts.cpp b/src/Storages/System/StorageSystemContentAddressedMounts.cpp index 6c07aa43b9c53..b2ffa229a2901 100644 --- a/src/Storages/System/StorageSystemContentAddressedMounts.cpp +++ b/src/Storages/System/StorageSystemContentAddressedMounts.cpp @@ -48,7 +48,7 @@ StorageSystemContentAddressedMounts::StorageSystemContentAddressedMounts(const S {"expires_at", std::make_shared(3), "Time when the lease expires."}, {"min_active_build_sequence", std::make_shared(), "Oldest in-flight build sequence (UINT64_MAX means the mount said farewell)."}, {"gc_fenced", std::make_shared(), "1 if GC fenced this slot out (terminal)."}, - {"state", std::make_shared(), "Mount slot state: live, expired, terminated, fenced or corrupt."}, + {"state", std::make_shared(), "Mount slot state: live, expired, terminated, fenced, retired or corrupt."}, {"is_leader", std::make_shared(std::make_shared()), "1 if this server's GC scheduler holds this disk's leadership lease. NULL on rows describing other servers' mounts."}, {"pending_reclaim", std::make_shared(std::make_shared()), "Cumulative condemned-minus-deleted backlog observed by this process's GC on this disk. NULL on rows describing other servers' mounts."}, {"last_success_age_seconds", std::make_shared(std::make_shared()), "Seconds since this disk's GC last led a round (0 if it never led). NULL on rows describing other servers' mounts."}, @@ -60,6 +60,7 @@ StorageSystemContentAddressedMounts::StorageSystemContentAddressedMounts(const S {"renewal_in_progress", std::make_shared(std::make_shared()), "1 while this server renews its mount lease. NULL on rows describing other servers' mounts."}, {"renewal_attempts", std::make_shared(std::make_shared()), "Conditional PUTs sent by the current lease renewal, or by the last one once it ended. NULL on rows describing other servers' mounts."}, {"last_renewal_error", std::make_shared(std::make_shared()), "The last failed request of that renewal: ' : '. NULL when none failed, and on rows describing other servers' mounts."}, + {"retirement_verified", std::make_shared(), "1 if a decommission verified the retirement of a retired member, 0 otherwise."}, })); storage_metadata.setVirtuals(createVirtuals()); setInMemoryMetadata(storage_metadata); @@ -107,6 +108,7 @@ Pipe StorageSystemContentAddressedMounts::read( MutableColumnPtr col_renewal_in_progress = ColumnNullable::create(ColumnUInt8::create(), ColumnUInt8::create()); MutableColumnPtr col_renewal_attempts = ColumnNullable::create(ColumnUInt64::create(), ColumnUInt8::create()); MutableColumnPtr col_last_renewal_error = ColumnNullable::create(ColumnString::create(), ColumnUInt8::create()); + MutableColumnPtr col_retirement_verified = ColumnUInt8::create(); const uint64_t now_ms = static_cast( std::chrono::duration_cast( @@ -210,6 +212,7 @@ Pipe StorageSystemContentAddressedMounts::read( col_min_active->insert(m.lease.min_active_build_sequence); col_fenced->insert(static_cast(m.lease.gc_fenced)); col_state->insert(m.state); + col_retirement_verified->insert(static_cast(m.retirement_verified)); /// GC health is a process-local fact about THIS server's scheduler. Stamping it onto /// peer rows misreads as "peer B is GC leader" during incidents — NULL there instead. @@ -255,6 +258,7 @@ Pipe StorageSystemContentAddressedMounts::read( col_min_active->insertDefault(); col_fenced->insertDefault(); col_state->insertDefault(); + col_retirement_verified->insertDefault(); col_is_leader->insertDefault(); col_pending->insertDefault(); col_last_success->insertDefault(); @@ -288,6 +292,7 @@ Pipe StorageSystemContentAddressedMounts::read( res_columns.emplace_back(std::move(col_renewal_in_progress)); res_columns.emplace_back(std::move(col_renewal_attempts)); res_columns.emplace_back(std::move(col_last_renewal_error)); + res_columns.emplace_back(std::move(col_retirement_verified)); UInt64 num_rows = res_columns.at(0)->size(); Chunk chunk(std::move(res_columns), num_rows); diff --git a/tests/integration/test_cas_drop_pool_member/test.py b/tests/integration/test_cas_drop_pool_member/test.py index 3db394e13c4dd..13bad950529dc 100644 --- a/tests/integration/test_cas_drop_pool_member/test.py +++ b/tests/integration/test_cas_drop_pool_member/test.py @@ -72,6 +72,110 @@ def _disks(node, query): ) +# A stop that lands after the admin's renewal write was sent ends its renewer terminally: the farewell is +# skipped (fail-safe), the run reports rerun with exactly these warnings, and the admin's live body holds the +# slot until GC fences it: about one lease TTL plus a GC round. Tolerated until the stop waits for an in-flight renewal +# before the farewell, a tracked follow-up. +FAREWELL_SKIPPED_BY_A_RACING_STOP = ( + "the slot does not hold this run's farewell; " + "the retirement is not verified; rerun SYSTEM CAS DROP POOL MEMBER" +) + +# The result row of SYSTEM CAS DROP POOL MEMBER, in column order. +REPORT_COLUMNS = [ + "server_root_id", + "namespaces_removed", + "namespaces_already_removed", + "committed_refs_removed", + "precommits_removed", + "manifest_debris_removed", + "staging_objects_removed", + "mountpoint_objects_removed", + "member_retired", + "warnings", + "namespaces_pending_gc", +] + + +def _drop_member(node): + # Retried through the "wait for its lease to lapse" refusal, bounded: the slot may hold the live body + # FAREWELL_SKIPPED_BY_A_RACING_STOP describes. + for _ in range(90): + try: + report_tsv = node.query( + "SYSTEM CAS DROP POOL MEMBER '{}' FROM DISK '{}'".format(SRID2, CA_DISK) + ).rstrip("\n") + except Exception as e: + if "alive or contended" not in str(e): + raise + time.sleep(1.0) + continue + fields = report_tsv.split("\t") + assert len(fields) == len(REPORT_COLUMNS), report_tsv + return report_tsv, dict(zip(REPORT_COLUMNS, fields)) + pytest.fail("decommission kept refusing: lease never lapsed within the bound") + + +def _assert_retired_or_racing_stop(report, report_tsv): + if int(report["member_retired"]) == 1: + assert report["warnings"] == "", report_tsv + else: + # TSV escapes the apostrophe. + assert report["warnings"].replace("\\'", "'") == FAREWELL_SKIPPED_BY_A_RACING_STOP, report_tsv + + +MANIFESTS_PREFIX = POOL + "/cas/manifests/" + SRID2 + "/" +ROOTS_PREFIX = POOL + "/roots/" + SRID2 + "/" + + +def _srid2_mount(node1): + return tuple( + node1.query( + "SELECT state, server_uuid FROM system.cas_mounts " + "WHERE server_root_id = '{}' AND disk = '{}'".format(SRID2, CA_DISK) + ) + .strip() + .split("\t") + ) + + +def _srid2_retirement(node1): + return tuple( + node1.query( + "SELECT state, retirement_verified FROM system.cas_mounts " + "WHERE server_root_id = '{}' AND disk = '{}'".format(SRID2, CA_DISK) + ) + .strip() + .split("\t") + ) + + +def _retire_srid2(node1): + # A rejoin needs a verified retirement, no catalog row left for GC and no object left under the id. + # The test polls namespaces_pending_gc through the report of a rerun, which is safe only because node2 is + # stopped; an operator runs SYSTEM CAS GC RUN after a verified run and does not rerun the decommission. + report_tsv = "" + for _ in range(60): + report_tsv, report = _drop_member(node1) + if ( + int(report["member_retired"]) == 1 + and int(report["namespaces_pending_gc"]) == 0 + and _count(MANIFESTS_PREFIX) == 0 + and _count(ROOTS_PREFIX) == 0 + ): + return + node1.query("SYSTEM CAS GC RUN '{}'".format(CA_DISK)) + time.sleep(1.0) + pytest.fail("node2 never reached a drained, verified retirement: {}".format(report_tsv)) + + +def _gc_rounds(nodes, rounds=3): + # GC RUN acts only on the GC leader and does nothing elsewhere; running it on every node reaches the leader. + for _ in range(rounds): + for node in nodes: + node.query("SYSTEM CAS GC RUN '{}'".format(CA_DISK)) + + def test_drop_dead_pool_member_heals_the_pool(): node1 = cluster.instances["node1"] node2 = cluster.instances["node2"] @@ -162,43 +266,27 @@ def test_drop_dead_pool_member_heals_the_pool(): # (6) Decommission the dead member from node1 -- PHASE 1 of a two-phase heal. SYSTEM queries do # not accept a FORMAT clause (ParserSystemQuery is not part of ParserQueryWithOutput), so parse - # the default TSV row. Column order matches the interpreter's ColumnsDescription: - # server_root_id, namespaces_removed, namespaces_already_removed, committed_refs_removed, - # precommits_removed, manifest_debris_removed, staging_objects_removed, - # mountpoint_objects_removed, slot_removed, warnings. + # the default TSV row (REPORT_COLUMNS). # # t2 is still `Live` at this point, so THIS call's normal drop path is what appends its # removal terminal and moves its catalog row to `Removing` -- catalog deletion stays GC's job # (`224aacd8eb9`), never the decommission command's, so a row this same call just legitimately - # transitioned still counts as "owned" and the retirement fence correctly refuses the slot. - # This is a success with GC completion pending, not a failure. + # transitioned is reported under `namespaces_pending_gc`. + # This is a verified retirement with GC completion pending, not a failure. # The mounts-table poll above and the decommission command judge liveness by DIFFERENT # predicates on purpose: the table renders TTL arithmetic over the last observed mount row, # while the command re-reads the mountpoint object and refuses while the lease could still # be live under its conservative safety margin. The destructive side being stricter is the # fail-close direction, so the table saying "not live" does not guarantee the command is # ready yet -- under sanitizer slowdowns the gap is wide enough to hit. Retry the command - # itself through its `alive or contended` refusal until the slot turns `fenced`, bounded. - report_tsv = None - for _ in range(90): - try: - report_tsv = node1.query( - "SYSTEM CAS DROP POOL MEMBER '{}' FROM DISK '{}'".format(SRID2, CA_DISK) - ).rstrip("\n") - break - except Exception as e: - if "alive or contended" not in str(e): - raise - time.sleep(1.0) - assert report_tsv is not None, "decommission kept refusing as alive or contended: the slot never turned fenced within the bound" - fields = report_tsv.split("\t") - assert len(fields) == 10, report_tsv - assert fields[0] == SRID2, report_tsv - assert int(fields[1]) >= 1, report_tsv # namespaces_removed: the drop half did its work - assert int(fields[8]) == 0, report_tsv # slot_removed: not yet -- GC owns the row now - assert "pool member decommission underway" in fields[9], report_tsv - - # (6b) PHASE 2: drive GC and re-run the decommission until the slot retires. Folding a fresh + # itself through the documented "wait for its lease to lapse" refusal, bounded. + report_tsv, report = _drop_member(node1) + assert report["server_root_id"] == SRID2, report_tsv + assert int(report["namespaces_removed"]) >= 1, report_tsv # the drop half did its work + assert int(report["namespaces_pending_gc"]) >= 1, report_tsv # GC owns the row now + _assert_retired_or_racing_stop(report, report_tsv) + + # (6b) PHASE 2: drive GC and re-run the decommission until no row is pending. Folding a fresh # terminal and pruning its catalog row are two separate GC rounds by design (a fold-then-prune # handoff -- see `CasDecommissionCatalogDuties.FoldedTerminalRemainsGcOwnedAndOnlyRequestsAnotherRound`), # so poll rather than assume one round suffices. The explicit `GC RUN` is the same idiom @@ -207,30 +295,26 @@ def test_drop_dead_pool_member_heals_the_pool(): # Stage A's destructive-reclaim suppression, so this heals under the current Stage-A posture. for _ in range(30): node1.query("SYSTEM CAS GC RUN '{}'".format(CA_DISK)) - report_tsv = node1.query( - "SYSTEM CAS DROP POOL MEMBER '{}' FROM DISK '{}'".format(SRID2, CA_DISK) - ).rstrip("\n") - fields = report_tsv.split("\t") - assert len(fields) == 10, report_tsv - if int(fields[8]) == 1: + report_tsv, report = _drop_member(node1) + _assert_retired_or_racing_stop(report, report_tsv) # on every rerun + if int(report["namespaces_pending_gc"]) == 0 and int(report["member_retired"]) == 1: break - assert "pool member decommission underway" in fields[9], report_tsv else: pytest.fail("pool never healed after driving GC: {}".format(report_tsv)) - assert fields[9] == "", report_tsv # warnings: the pool healed cleanly - # (7) node1's own data survives the whole flow untouched. assert int(node1.query("SELECT count() FROM t1")) == n1_count assert int(node1.query("SELECT sum(id) FROM t1")) == n1_sum - # (8) node2's server_root_id is gone from the mounts table (only true once the slot above actually - # retired -- checked after PHASE 2, not right after the first, still-pending call). + # (8) node2's mount slot stays and reads as `retired`: the decommission keeps + # the control objects and tombstones the owner. assert ( node1.query( - "SELECT count() FROM system.cas_mounts WHERE server_root_id = '{}'".format(SRID2) + "SELECT state FROM system.cas_mounts WHERE server_root_id = '{}' AND disk = '{}'".format( + SRID2, CA_DISK + ) ).strip() - == "0" + == "retired" ) # (9) Drive GC (node1's background GC is already running against the shared pool) to reclaim @@ -280,7 +364,7 @@ def test_drop_dead_pool_member_heals_the_pool(): assert deleted > 0, "the pool drained but GC's own bookkeeping reports no deletion" # node2's decommissioned-and-healed namespace (t2) left canonical dead-life residue behind: its - # catalog row is gone (that is what let the slot retire above), but its `_ckpt`/`_files`/`_log` + # catalog row is gone (pruned by GC after the drop), but its `_ckpt`/`_files`/`_log` # objects are the perpetual namespace janitor's job, not decommission's or GC's own destructive # round. t1's row is pruned the same way once dropped. That residue must DRAIN to zero -- the # janitor deletes one bounded page per round, so this is polled rather than read once -- and it @@ -301,14 +385,10 @@ def test_drop_dead_pool_member_heals_the_pool(): "the dead-life residue from the healed decommission and the t1 drop never drained: {}".format(fsck) ) - # (10) Re-run the same command: decommission tombstones the owner anchor in place rather than - # deleting it, so the slot is not "unknown" -- the tombstone is found and the re-run is refused - # with CORRUPTED_DATA and a message telling the operator this server-root was explicitly - # decommissioned and will not silently resume. - err = node1.query_and_get_error( - "SYSTEM CAS DROP POOL MEMBER '{}' FROM DISK '{}'".format(SRID2, CA_DISK) - ) - assert "explicitly decommissioned" in err, err + # (10) Re-run the same command: the owner carries a verified retirement, so the rerun is accepted + # and reports the member retired again. + report_tsv, report = _drop_member(node1) + _assert_retired_or_racing_stop(report, report_tsv) def test_drop_pool_member_rejected_on_readonly_disk(): @@ -331,3 +411,56 @@ def test_drop_pool_member_refuses_id_with_slash(): ) assert "BAD_ARGUMENTS" in err, err assert "contains '/'" in err, err + + +def test_retired_member_rejoins_empty(): + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + + # Start from a retired node2 whatever ran before: a stop of a stopped server only warns. + node2.stop_clickhouse(kill=True) + _retire_srid2(node1) + assert _srid2_retirement(node1) == ("retired", "1") + + # Same uuid: the restart rejoins, then the member writes a table of its own. + node2.start_clickhouse() + state, old_uuid = _srid2_mount(node1) + assert state == "live" + node2.query("DROP TABLE IF EXISTS t3 SYNC") + node2.query( + "CREATE TABLE t3 (id Int64, s String) ENGINE = MergeTree() ORDER BY id " + "SETTINGS storage_policy = '{}'".format(STORAGE_POLICY) + ) + node2.query("INSERT INTO t3 SELECT number, toString(number) FROM numbers(1000)") + _gc_rounds([node1, node2]) + assert int(node2.query("SELECT count() FROM t3")) == 1000 + assert _srid2_mount(node1) == ("live", old_uuid) + + node2.stop_clickhouse() + _retire_srid2(node1) + assert _srid2_retirement(node1) == ("retired", "1") + + # The same uuid comes back empty: the local table definition attaches with no rows and takes writes. + node2.start_clickhouse() + assert _srid2_mount(node1) == ("live", old_uuid) + assert int(node2.query("SELECT count() FROM t3")) == 0 + node2.query("INSERT INTO t3 SELECT number, toString(number) FROM numbers(10)") + _gc_rounds([node1, node2]) + assert int(node2.query("SELECT count() FROM t3")) == 10 + assert _srid2_mount(node1) == ("live", old_uuid) + + # A new uuid (the local uuid file regenerated) rejoins the same id, also empty. + node2.stop_clickhouse() + _retire_srid2(node1) + node2.exec_in_container(["rm", "-f", "/var/lib/clickhouse/uuid"], user="root") + node2.start_clickhouse() + state, new_uuid = _srid2_mount(node1) + assert state == "live" + assert new_uuid != old_uuid + assert int(node2.query("SELECT count() FROM t3")) == 0 + node2.query("INSERT INTO t3 SELECT number, toString(number) FROM numbers(10)") + _gc_rounds([node1, node2]) + assert int(node2.query("SELECT count() FROM t3")) == 10 + assert _srid2_mount(node1) == ("live", new_uuid) + + node2.query("DROP TABLE t3 SYNC")