Skip to content
Open
3 changes: 2 additions & 1 deletion docs/en/antalya/cas/architecture/backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,8 @@ battery, skipping the battery used to skip it too, which is exactly why the thir
writable generation-token mount; every other backend still skips only its permitted access-check
I/O. This is what makes the exact-token delete check below unskippable on an *ordinary* writable
mount. Decommissioning a pool member is the deliberate exception — it opens writable with the
battery skipped, because the fail-closed tradeoff inverts there: refusing an ordinary mount costs
battery skipped and logs one line naming what went unverified (versioning, conditional writes,
list-after-write), because the fail-closed tradeoff inverts there: refusing an ordinary mount costs
availability and protects data, whereas refusing a decommission strands a pool with a dead replica
in it and leaves the operator no way forward.
- `checkConditionalWriteSingleAttemptSupport` — inside the battery. Refuses to mount writable unless the underlying
Expand Down
55 changes: 42 additions & 13 deletions docs/en/antalya/cas/architecture/mounts-and-leases.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,22 @@ and never reassigned — decommission only tombstones it in place.
| Observed at `gc/server-roots/<server_root_id>/owner` | Action |
|---|---|
| present, same `server_uuid`, not tombstoned | proceed |
| present, `retired_at_ms` set | `CORRUPTED_DATA` — explicitly decommissioned; the refusal says to configure a new `cas_server_root_id` |
| present, `retired_at_ms` set | `CORRUPTED_DATA` for a server start or remount; the decommission admin accepts its own victim's tombstone |
| present, different `server_uuid` | `CORRUPTED_DATA` — names the regenerated-uuid-file cause and, for a dead owner, `SYSTEM CAS DROP POOL MEMBER` |
| absent, subtree provably empty | `putIfAbsent` the owner (claim) |
| absent, subtree non-empty | `CORRUPTED_DATA` — identity lost over existing data |
| absent, subtree provably empty | `putIfAbsent` the owner (claim) |
| lost the `putIfAbsent` race | re-read; equal uuid proceeds, else `CORRUPTED_DATA` |

The owner carries `owner_life`, a counter every owner write sets to the read value + 1 by `If-Match`, so
two owner writes never repeat their bytes and a delayed `If-Match` against an older incarnation cannot
land. A process records the life it started under; after every mount claim (start, self-remount,
decommission) it reads the owner once more and requires its own `server_uuid` and that life
(`recheckOwnerAfterClaim`); otherwise it fences its own mount body and fails, so a successor does not wait
the refused lease out. A self-remount also compares the life before it allocates an epoch: an owner at
another life, not retired, moves the pool to `IdentityLost` instead of retrying. `retirement_verified`
marks a decommission whose claim, tombstone and farewell all held.

"Provably empty" requires both an authoritative decoded catalog naming no life owned by `server_root_id` and
a 1-key `LIST` probe finding nothing under `cas/manifests/<server_root_id>/` or `roots/<server_root_id>/`.

Expand Down Expand Up @@ -145,8 +155,7 @@ lease the slot's body declares (`expires_at_ms − started_at_ms`). A body witho
non-positive interval (a farewell), declares nothing. A re-mounting server waits out a predecessor the
same way with the step `max(1, floor(period/2))`. `period` is always the observer's own. The two stamps
are compared only with each other, never with the observer's clock. `expires_at_ms` alone is a
writer-stamped diagnostic used by `system.cas_mounts` and the non-authoritative decommission
epoch-recovery precheck. Local fencing is derived from the confirmed request's pre-I/O `BOOTTIME`
writer-stamped diagnostic used by `system.cas_mounts`. Local fencing is derived from the confirmed request's pre-I/O `BOOTTIME`
anchor plus the TTL, and wall-clock `now` stays audit-only.

`cas_mount_lease_ttl_ms` is local to each server. It is how long this server may write after its last
Expand All @@ -165,9 +174,15 @@ the first renewal attempt of a period can begin, where `envelope = attempt_timeo

| Counter | Storage | Scope | Protects against |
|---|---|---|---|
| `writer_epoch` | durable, `gc/server-roots/<server_root_id>/epoch` (`ServerEpoch::next_writer_epoch`, CAS-bumped by `allocateWriterEpoch`) | across crashes and restarts | a same-`(uuid, epoch)` twin: a present mount under a normal claim attempt is `CORRUPTED_DATA` |
| `writer_epoch` | durable, `gc/server-roots/<server_root_id>/epoch` (`ServerEpoch::next_writer_epoch`, CAS-bumped by `allocateWriterEpoch`); never deleted: a retired `server_root_id` keeps its counter | across crashes and restarts | a same-`(uuid, epoch)` twin: a present mount under a normal claim attempt is `CORRUPTED_DATA` |
| `build_seq` | in-memory only, `CasMountRuntime::next_build_seq`, reset to 1 on every process start | one process incarnation | orders builds *within* an epoch; combined with `writer_epoch` it gives GC a total order |

Each bump of the `epoch` counter also writes a fresh random 128-bit `allocation_nonce`. When a
counter write's outcome is unclear and the counter has since moved, the allocator counts the write
as landed only if the stored bytes match its own exactly. Two starts of the same `server_uuid` running this allocator that bump
to the same value write different nonces, so they never both adopt one `writer_epoch`. Allocators
of 26.6.4 write no nonce, so two of them can still collide. 26.6.4 readers skip the field.

The absent-epoch branch of `allocateWriterEpoch` is deliberately paranoid: absent with a
non-empty subtree is `CORRUPTED_DATA` (reset hazard); absent with an empty subtree decides by an
authoritative probe, never by plain-`get` absence, because a transport fault must not be flattened
Expand Down Expand Up @@ -221,7 +236,7 @@ stateDiagram-v2
Live --> Live: same-uuid claim, proven-dead token via UncleanObserved
Live --> Live: same-uuid claim under cas_unsafe_remount_no_delay, no observation
Fenced --> Fenced: same uuid and epoch claim, FencedSelf, no write
Live --> Absent: decommission tail, mount then epoch then owner tombstone
Terminated --> Terminated: decommission farewell, owner tombstoned and verified
Terminated --> [*]
```

Expand All @@ -234,7 +249,7 @@ stateDiagram-v2
Live --> TransientNotLive: renewal failure, tripMountLost, lost=true
TransientNotLive --> Live: self-remount succeeds with a fresh epoch
TransientNotLive --> TransientNotLive: probe inconclusive, retry with backoff
TransientNotLive --> IdentityLost: pool meta and owner both authoritatively absent
TransientNotLive --> IdentityLost: pool meta and owner both absent, or the owner at another life
TransientNotLive --> VanishedReplaced: foreign pool_id observed
Live --> VanishedForgotten: SYSTEM CAS FORGET
IdentityLost --> [*]
Expand Down Expand Up @@ -280,13 +295,27 @@ while a stale conditional request from the predecessor is still in flight.
and waiting out the token-stability observation, or the GC leader fencing the slot first, after
which any reclaim is instant.

**Permanent removal** of a dead replica (`Cas::decommissionPoolMember`, driven by
`SYSTEM CAS DROP POOL MEMBER '<server_root_id>' FROM DISK '<disk>'`) claims the victim's mount slot
as an administrative writer with a no-wait policy (refuses unless the slot is terminal:
`terminated` or `fenced`; an `expired` slot is refused until the disk's GC leader fences it),
drops every ref-bearing namespace, sweeps manifest debris before the slot (deleting the mount
removes the watermark authority), drains staging and roots, then — only with zero warnings —
retires in order: mount, epoch, a final liveness re-check, owner tombstone.
**Permanent removal** of a dead replica (`Cas::decommissionPoolMember`, `SYSTEM CAS DROP POOL MEMBER '<server_root_id>' FROM DISK '<disk>'`)
keeps `owner`, `epoch` and `mount` and removes only data, in six steps:

1. Claim the victim's slot as an administrative writer with a no-wait policy. The claim needs a terminal
slot (`terminated` or `fenced`); an `expired` slot is refused until the disk's GC leader fences it. A
member without an epoch counter is refused.
2. Tombstone the owner at the next `owner_life`, on the admin's fenced mount plane: no attempt is sent
after the admin's lease deadline, so a late retry cannot land on a victim that restarted after it.
3. Confirm the slot still holds the admin's claim; otherwise roll the tombstone back, unless the owner was
already retired before this run. Every exit between (2) and (5) runs this confirm.
4. Drop every namespace, sweep manifest debris, drain `staging/` and `roots/`, all on the admin's fenced
mount plane.
5. Write the admin's clean farewell.
6. Read the farewell and the tombstone back and set `retirement_verified`.

`member_retired = 1` only when the owner carries a verified retirement. An owner that moved on to a later,
unretired life after the tombstone (a rejoin, or another run's rollback) reports `member_retired = 0` with
the warning `the owner moved on to life N`. Catalog rows GC still has to delete are reported as
`namespaces_pending_gc`, not as a failure. The operator reruns after `member_retired = 0`; a rerun refused
as alive or contended waits until GC fences the previous run's slot: about one mount lease TTL plus one GC
round, with a GC leader running.

## `system.cas_mounts` {#mounts-table}

Expand Down
2 changes: 1 addition & 1 deletion docs/en/antalya/cas/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ where `envelope = cas_attempt_timeout_ms + 2 × cap` (7000 ms with defaults) and
retrying until `confirmed deadline − cas_lease_safety_margin_ms`.

The `expires_at_ms` stamped into the mount object is a writer-stamped diagnostic used by
`system.cas_mounts` and by the non-authoritative decommission epoch-recovery precheck; it never
`system.cas_mounts`; it never
authorizes a reclaim or a GC fence-out. Local fencing is derived instead from the confirmed
request's pre-I/O `CLOCK_BOOTTIME` anchor plus the TTL.

Expand Down
3 changes: 1 addition & 2 deletions docs/en/antalya/cas/operations/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,7 @@ row followed by a `recovered` one — and the older `unresolved_reason`, `deadli
closed admission while the operation was active), `deterministic_failure` (a local
failure, or, for a bounded renewal, a store answer proving the write never applied;
the background renewal sends a refused write again), and `vanished` (an exact resolve read proved the mount
slot absent — the pool directory was removed or renamed out of band, or a decommission raced the
renewal). `terminal_unclassified` means the renewal terminated through a path that assigned no
slot absent — the pool directory was removed or renamed out of band, or a 26.6.4 decommission deleted the slot; a decommission of this version keeps it). `terminal_unclassified` means the renewal terminated through a path that assigned no
classification; that is a defect to report together with the surrounding rows, not an operator
condition. Do not collapse these into a generic timeout — the action
differs by classification, and only `external_lease_deadline` and `request_deadline` are about a
Expand Down
42 changes: 29 additions & 13 deletions docs/en/antalya/cas/operations/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,11 @@ irreversible operation from an ordinary restart or a temporary outage: it fences
`server_root_id` and reclaims the storage attributable only to it.

`SYSTEM CAS DROP POOL MEMBER` claims the victim's mount slot as an administrative writer (refusing
immediately if the member is still alive), drops every table namespace the member owned, sweeps
manifest debris, drains its staging and mountpoint objects, and — only once every drain is
confirmed — retires the mount slot itself. It emits ordinary ref-edge deltas rather than a GC
immediately if the member is still alive), tombstones its owner, drops every table namespace the member
owned, drains its debris and marks the retirement verified; the steps are in
[mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle). The member's control
objects (`owner`, `epoch`, `mount`) are kept: the epoch counter keeps epochs from being reused if the
`server_root_id` is ever reused. It emits ordinary ref-edge deltas rather than a GC
transition: it does not synchronously reclaim shared blob content, it only makes the now-unreferenced
blobs eligible for an ordinary GC round to reclaim later.

Expand All @@ -192,13 +194,14 @@ clickhouse-disks -C config.xml --disk cas cas-drop-member 'replica-2'
```

The command returns one row (or, offline, one line per field) with `namespaces_removed`,
`namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`,
`manifest_debris_removed`, `staging_objects_removed`, `mountpoint_objects_removed`, and
`slot_removed`. It is resumable: a rerun skips namespaces already marked removed and reports them
under `namespaces_already_removed` rather than redoing the work. A per-object drain failure is
recorded as a `warning` rather than raised as an exception, leaving the slot terminated but not
fully drained so a later invocation can resume; a non-empty `warnings` means exactly that, and the
mount slot stays in place as a resume anchor rather than being fully retired.
`namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`, `manifest_debris_removed`,
`staging_objects_removed`, `mountpoint_objects_removed`, `member_retired`, `warnings` and
`namespaces_pending_gc`; the SQL row ends with `namespaces_pending_gc`. It is resumable: a rerun skips namespaces already marked removed and reports them
under `namespaces_already_removed` rather than redoing the work. A per-object drain failure is recorded as
a `warning`; the owner keeps an unverified tombstone and the next run resumes over it. `member_retired = 1`
means the owner carries a verified retirement; rows still listed under `namespaces_pending_gc` are deleted
by GC and need no rerun. See [troubleshooting](/antalya/cas/operations/troubleshooting) for
`member_retired = 0`.

**Preconditions.** Confirm the member is actually and permanently dead before running this: the
operation fences that `server_root_id` out even if the server comes back online, and it deletes
Expand All @@ -210,12 +213,25 @@ row is refused with `alive or contended`; a hard-killed member stays `expired` u
(`is_leader = 1` in `system.cas_mounts`) can trigger the fence, but the run that fences must come at
least that long after the first run that saw the lease; two back-to-back runs do not fence.

**Verification.** After the command reports `slot_removed = true` with no warnings, the member's
`server_root_id` no longer appears as a row in `system.cas_mounts` on any peer, and a subsequent
`SYSTEM CAS GC RUN` on the pool will no longer wait on or fence its heartbeat. See
**Verification.** After `member_retired = 1`, `system.cas_mounts` shows the member's
row as `terminated`; a `SYSTEM CAS GC RUN` no longer waits on its heartbeat. See
[mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle) for how the
claim, drain, and retirement steps fit into the mount-slot lifecycle.

### Upgrading and downgrading {#decommission-compatibility}

| Case | Behaviour |
|---|---|
| A `server_root_id` retired by 26.6.4 (its `epoch` and `mount` deleted) | not repaired: a decommission rerun refuses the absent counter; use a new `server_root_id` |
| A 26.6.4 decommission that stopped midway | the same refusal; use a new `server_root_id` |
| Flat 26.6.4 staging keys `staging/<srid>/<name>.tmp` | swept as epoch 0 at the next mount |
| Downgrade to 26.6.4 with a retired `server_root_id` kept by this version | 26.6.4 refuses to start it and its GC treats the farewell as terminal |
| Downgrade: `owner_life`, `retirement_verified`, epoch staging keys | skipped by 26.6.4 readers; staging swept recursively |
| Mixed versions: a 26.6.4 decommission tool rewrites an owner | the life drops to 0; servers of this version fail their owner check and stop (fail-closed) |
| Mixed versions: a 26.6.4 member restarts during a decommission | its owner check refuses after the tombstone; before it there is no recheck, as in 26.6.4 |

Rolling upgrades keep the old races on the old binaries; decommission from an upgraded server.

## Recreating a member whose id contains `/` {#recreate-id-with-slash}

A `cas_server_root_id` that contains `/` is refused: the disk fails to open with `BAD_ARGUMENTS` and a
Expand Down
Loading
Loading