filimonov · Oct 17, 2026 · Oct 17, 2026 · Oct 17, 2026 · Oct 17, 2026 · Oct 17, 2026
diff --git a/docs/en/antalya/cas/architecture/backend.md b/docs/en/antalya/cas/architecture/backend.md
@@ -125,7 +125,8 @@ battery, skipping the battery used to skip it too, which is exactly why the thir
   writable generation-token mount; every other backend still skips only its permitted access-check
   I/O. This is what makes the exact-token delete check below unskippable on an *ordinary* writable
   mount. Decommissioning a pool member is the deliberate exception — it opens writable with the
-  battery skipped, because the fail-closed tradeoff inverts there: refusing an ordinary mount costs
+  battery skipped and logs one line naming what went unverified (versioning, conditional writes,
+  list-after-write), because the fail-closed tradeoff inverts there: refusing an ordinary mount costs
   availability and protects data, whereas refusing a decommission strands a pool with a dead replica
   in it and leaves the operator no way forward.
 - `checkConditionalWriteSingleAttemptSupport` — inside the battery. Refuses to mount writable unless the underlying

diff --git a/docs/en/antalya/cas/architecture/garbage-collection.md b/docs/en/antalya/cas/architecture/garbage-collection.md
@@ -194,7 +194,8 @@ token of each `mount` object.
   deletes
 - **Safety:** fences only after this leader's *own* monotonic clock has watched the mount's
   write-token hold unchanged for `L + 5% + cas_mount_renew_period_ms`, where `L` is the longer of this server's
-  `cas_mount_lease_ttl_ms` and the lease the slot's body declares (defaults 30 s and 10 s); the write is
+  `cas_mount_lease_ttl_ms` and the lease the slot's body declares (defaults 30 s and 10 s), and only at the
+  start of a round, so a killed member becomes `fenced` up to one GC round later than that; the write is
   guarded by that exact token. A completed fence-out stays valid even if this GC later loses
   leadership. `expires_at_ms` (another host's wall clock) is never trusted.
 - **Fails the round if:** nothing — a per-mount `PUT` conflict is re-observed and re-classified
@@ -1009,12 +1010,12 @@ checksum disagrees with its seal).
 
 | Command | Effect |
 |---|---|
-| `SYSTEM CAS GC RUN [<disk>]` | One synchronous round execution on the contacted node; only the lease holder makes progress. The disk is optional: without it every content-addressed disk on the node runs one round. It runs even while the scheduler is stopped |
+| `SYSTEM CAS GC RUN [<disk>]` | One synchronous round execution on the contacted node; only the lease holder (the GC leader, `is_leader = 1` in `system.cas_mounts`) makes progress, and on any other node the command does nothing. The disk is optional: without it every content-addressed disk on the node runs one round. It runs even while the scheduler is stopped |
 | `SYSTEM CAS GC STOP <disk>` / `SYSTEM CAS GC START <disk>` | Stop or resume future background rounds on that disk's scheduler, preserving its identity. The disk is required |
 | `SYSTEM CAS GC REBUILD [FORCE] <disk>` (`clickhouse-disks cas-gc-rebuild`) | Fail-closed disaster-recovery path for a lost or corrupt `GC` baseline — the `CORRUPTED_DATA` errors that name it in their message (missing adopted seal, snapshot without a surviving log, cursor/apply mismatch). An `ABORTED` commit conflict or a `LOGICAL_ERROR` delete marker is not a reason to rebuild. It deliberately over-protects — it prefers bounded leaks over risking an under-count — and cannot delete live data directly: deletions it produces still flow through the normal round's condemn, graduate, exact-token path. The disk is required |
 | `clickhouse-disks cas-gc-dryrun` | Opens the disk read-only, constructs a non-leader `GC`, and prints what would be deleted with a reason per entry. Write-free, resolves runs through the seal's references. Documented caveat: it does not fold new owner events, so away from quiescence it can **over-report** — the subset guarantee holds only at quiescence, and its output must never feed a real delete |
 
-`SYSTEM CAS DROP POOL MEMBER '<server_root_id>' FROM DISK '<disk>'` — permanent removal of a dead
+`SYSTEM CAS DROP POOL MEMBER '<server_root_id>' FROM DISK '<disk>'` — removal of a dead
 replica, distinct from ordinary `GC` — is covered on the
 [mounts-and-leases page](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle).
 `SYSTEM CAS FSCK <disk>` (`clickhouse-disks cas-fsck`) and its `dangling`/`unreachable` vocabulary

diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md
diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md
@@ -117,7 +117,8 @@ token holding stable on the observer's own `CLOCK_BOOTTIME` for `L + floor(L / 2
 longer of the observer's `cas_mount_lease_ttl_ms` and the lease the holder's body declares
 (`expires_at_ms − started_at_ms`; a body without stamps or with a non-positive interval declares
 nothing). Startup's step is `max(1, floor(cas_mount_renew_period_ms / 2))` and GC's is
-`cas_mount_renew_period_ms`, both the observer's own. Mixed values are therefore safe in any order;
+`cas_mount_renew_period_ms`, both the observer's own. GC checks only at the start of a round, so a
+fence-out can land up to one GC round after that. Mixed values are therefore safe in any order;
 a longer lease on one server delays recovery after that server's crash. A server built without this
 rule waits only its own lease, so keep the values equal until every server of the pool runs a build
 with it.
@@ -132,7 +133,7 @@ where `envelope = cas_attempt_timeout_ms + 2 × cap` (7000 ms with defaults) and
 retrying until `confirmed deadline − cas_lease_safety_margin_ms`.
 
 The `expires_at_ms` stamped into the mount object is a writer-stamped diagnostic used by
-`system.cas_mounts` and by the non-authoritative decommission epoch-recovery precheck; it never
+`system.cas_mounts`; it never
 authorizes a reclaim or a GC fence-out. Local fencing is derived instead from the confirmed
 request's pre-I/O `CLOCK_BOOTTIME` anchor plus the TTL.
 

diff --git a/docs/en/antalya/cas/operations/debugging.md b/docs/en/antalya/cas/operations/debugging.md
@@ -142,8 +142,7 @@ row followed by a `recovered` one — and the older `unresolved_reason`, `deadli
   shutdown), `fence_or_lifecycle_lost` (another local fence loss or a terminal lifecycle transition
   closed admission while the operation was active), `deterministic_failure` (the store's own
   answer proved the write never applied), and `vanished` (an exact resolve read proved the mount
-  slot absent — the pool directory was removed or renamed out of band, or a decommission raced the
-  renewal). `terminal_unclassified` means the renewal terminated through a path that assigned no
+  slot absent — the pool directory was removed or renamed out of band, or a 26.6.4 decommission deleted the slot; a decommission of this version keeps it). `terminal_unclassified` means the renewal terminated through a path that assigned no
   classification; that is a defect to report together with the surrounding rows, not an operator
   condition. Do not collapse these into a generic timeout — the action
   differs by classification, and only `external_lease_deadline` and `request_deadline` are about a
@@ -191,6 +190,9 @@ available:
 SYSTEM CAS GC RUN cas;
 ```
 
+Run it on the GC leader (`is_leader = 1` in `system.cas_mounts`). On any other node it does nothing and
+returns no error.
+
 One row per disk it ran on: `disk`, `acquired_lease`, `deferred`, `round`, `candidates_marked`,
 `objects_deleted`, `objects_absent`, `objects_replaced`, `objects_spared`, `manifests_deleted`,
 `entries_condemned`, `entries_graduated`, `entries_redeleted`, `entries_redelete_failed`, `fence_outs`,
@@ -225,8 +227,8 @@ SYSTEM CAS FORGET cas;
 
 It is an assertion, not a proof of erasure: the disk stays registered and answers further store-class
 access with a typed error, and a server restart re-registers the name. This is different from
-[`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission), which permanently
-retires one pool *member*'s identity across the whole shared pool — `FORGET` only affects this node's
+[`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission), which decommissions
+one pool *member* across the whole shared pool — `FORGET` only affects this node's
 own local view of one disk.
 
 ## Offline tools {#offline-tools}

diff --git a/docs/en/antalya/cas/operations/migration.md b/docs/en/antalya/cas/operations/migration.md
@@ -1,5 +1,5 @@
 ---
-description: 'Adding a content-addressed disk to an existing deployment, moving a partition onto it with ALTER TABLE MOVE PARTITION, rolling back, and permanently decommissioning a pool member.'
+description: 'Adding a content-addressed disk to an existing deployment, moving a partition onto it with ALTER TABLE MOVE PARTITION, rolling back, and decommissioning a pool member.'
 sidebar_label: 'Migration'
 sidebar_position: 1
 slug: /antalya/cas/operations/migration
@@ -164,17 +164,20 @@ directly (the raw disk, not the cache) was refused with `All parts of partition
 already on disk 'cas_cache'. (UNKNOWN_DISK)` — a real error message from the run, kept here because
 it is exactly what an operator sees after guessing the wrong disk name.
 
-## Permanently removing a pool member {#decommission}
+## Removing a pool member {#decommission}
 
 A `CAS` pool can be shared by several servers (see [`server_root_id`](/antalya/cas/architecture/mounts-and-leases#server-root-id)).
-Scaling down — permanently removing a server that will never rejoin the pool — is a distinct,
-irreversible operation from an ordinary restart or a temporary outage: it fences the member's
-`server_root_id` and reclaims the storage attributable only to it.
+Scaling down — removing a server and its data from the pool — is a distinct operation from an ordinary
+restart or a temporary outage: it fences the member's current life and reclaims the storage attributable
+only to it. The data is gone for good; the `server_root_id` can rejoin later as an empty member
+([rejoin](#rejoin)).
 
 `SYSTEM CAS DROP POOL MEMBER` claims the victim's mount slot as an administrative writer (refusing
-immediately if the member is still alive), drops every table namespace the member owned, sweeps
-manifest debris, drains its staging and mountpoint objects, and — only once every drain is
-confirmed — retires the mount slot itself. It emits ordinary ref-edge deltas rather than a GC
+immediately if the member is still alive), tombstones its owner, drops every table namespace the member
+owned, drains its debris and marks the retirement verified; the steps are in
+[mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle). The member's control
+objects (`owner`, `epoch`, `mount`) are kept: the epoch counter keeps epochs from being reused when the
+`server_root_id` rejoins. It emits ordinary ref-edge deltas rather than a GC
 transition: it does not synchronously reclaim shared blob content, it only makes the now-unreferenced
 blobs eligible for an ordinary GC round to reclaim later.
 
@@ -192,22 +195,85 @@ clickhouse-disks -C config.xml --disk cas cas-drop-member 'replica-2'
 ```
 
 The command returns one row (or, offline, one line per field) with `namespaces_removed`,
-`namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`,
-`manifest_debris_removed`, `staging_objects_removed`, `mountpoint_objects_removed`, and
-`slot_removed`. It is resumable: a rerun skips namespaces already marked removed and reports them
-under `namespaces_already_removed` rather than redoing the work. A per-object drain failure is
-recorded as a `warning` rather than raised as an exception, leaving the slot terminated but not
-fully drained so a later invocation can resume; a non-empty `warnings` means exactly that, and the
-mount slot stays in place as a resume anchor rather than being fully retired.
-
-**Preconditions.** Confirm the member is actually and permanently dead before running this: the
-operation fences that `server_root_id` out even if the server comes back online, and it deletes
-namespace and drain state that cannot be recovered. Check `system.cas_mounts` for the member's
+`namespaces_already_removed`, `committed_refs_removed`, `precommits_removed`, `manifest_debris_removed`,
+`staging_objects_removed`, `mountpoint_objects_removed`, `member_retired`, `warnings` and
+`namespaces_pending_gc`; the SQL row ends with `namespaces_pending_gc`. It is resumable: a rerun skips namespaces already marked removed and reports them
+under `namespaces_already_removed` rather than redoing the work. A per-object drain failure is recorded as
+a `warning`; the owner keeps an unverified tombstone and the next run resumes over it. `member_retired = 1`
+means the owner carries a verified retirement; rows still listed under `namespaces_pending_gc` are deleted
+by GC and need no rerun. See [troubleshooting](/antalya/cas/operations/troubleshooting) for
+`member_retired = 0`.
+
+**Preconditions.** Confirm the member is actually dead before running this: the
+operation deletes namespace and drain state that cannot be recovered, and a server that comes back under
+that id after the run starts empty. Check `system.cas_mounts` for the member's
 `state` and `last_success_age_seconds` first — a `live` row, or one with a recent lease renewal,
 means the member is not a decommission candidate yet.
 
-**Verification.** After the command reports `slot_removed = true` with no warnings, the member's
-`server_root_id` no longer appears as a row in `system.cas_mounts` on any peer, and a subsequent
-`SYSTEM CAS GC RUN` on the pool will no longer wait on or fence its heartbeat. See
+**Verification.** After `member_retired = 1`, `system.cas_mounts` shows the member's
+row as `retired` with `retirement_verified = 1` (an unfinished run leaves `retirement_verified = 0`); a `SYSTEM CAS GC RUN` no longer waits on its heartbeat. Run it on the GC leader (`is_leader = 1` in `system.cas_mounts`); on another node it does nothing. See
 [mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle) for how the
 claim, drain, and retirement steps fit into the mount-slot lifecycle.
+
+### Bringing a member back {#rejoin}
+
+Restart the server. A server whose `server_root_id` was decommissioned rejoins the pool on its next start,
+with the same or a new `server_uuid`. It comes back empty: its local table definitions attach with no data.
+
+The start rejoins only after the decommission finished:
+
+- the last run reported `member_retired = 1`, and `system.cas_mounts` shows the member as `retired` with
+  `retirement_verified = 1`;
+- GC removed the catalog rows and manifests of the old life: run `SYSTEM CAS GC RUN '<disk>'` on the GC leader
+  (`is_leader = 1` in `system.cas_mounts`) after the verified run; it can take several rounds.
+
+| Start fails with | Action |
+|---|---|
+| `cannot rejoin` | `epoch` or `mount` is missing: the id was retired by 26.6.4 or the object was deleted by hand. Configure a new `cas_server_root_id` |
+| `decommission is not finished` | finish the decommission under the [rerun rule](#rejoin-rerun-rule) until `member_retired = 1`, then restart |
+| `previous life remains` | the message names the part left. Catalog rows or manifests: run `SYSTEM CAS GC RUN '<disk>'` on the GC leader, `is_leader = 1` (it can take several rounds) and restart. An object under `roots/<server_root_id>/`: delete it by hand and restart |
+| `still running` | a decommission run, a killed run or an interrupted restart holds the slot: wait for the run to finish, or for the lease to lapse and GC to fence it (up to one GC round after the lease threshold), then restart |
+| `kept changing` | other starts or a decommission run kept rewriting the owner or the slot during this start: restart |
+| `no server uuid` | the process (for example `clickhouse disks` before a uuid file exists) has no server uuid and never rejoins: start the server that should own the id |
+
+Keep `/var/lib/clickhouse/uuid` (the server uuid file) on a persistent volume. A lost uuid file makes the
+server start as a different server: a `server_root_id` that is not decommissioned refuses it as owned by a
+different server, and a decommissioned one is rejoined under the new `server_uuid`.
+
+#### Rerun rule {#rejoin-rerun-rule}
+
+Rerun `SYSTEM CAS DROP POOL MEMBER` to finish a decommission only while both hold:
+
+- `system.cas_mounts` shows the id as `retired` with `retirement_verified = 0`;
+- no decommission of that id is running.
+
+A run killed after its tombstone leaves the row `live`, then `expired`, then `retired` with
+`retirement_verified = 0` once GC fences its lease, up to one GC round after the lease threshold. A run killed
+before its tombstone ends `fenced` with the owner not retired. Do not rerun on any other state. `retired` with
+`retirement_verified = 1` is finished: restart the member to rejoin. A `live`, `expired`, `terminated` or
+`fenced` row belongs to whoever owns the id now, a rejoined member included, and a rerun would delete its data.
+A run that warns `the tombstone was rolled back` lost its claim and left the id unretired: running the command
+again is a new decommission of the id.
+
+A tombstone can land late: the store may apply it after the run reported an error. If a server of the id ran
+after the tombstone time printed in its start error, do not rerun: the rerun deletes what that server wrote.
+Instead keep the owner object's header line, replace its body line with
+`{"server_uuid":"<uuid>","owner_life":"<life + 1>"}` (the same `server_uuid`, the life from the error plus one)
+and restart that member.
+
+### Upgrading and downgrading {#decommission-compatibility}
+
+| Case | Behaviour |
+|---|---|
+| A `server_root_id` retired by 26.6.4 (its `epoch` and `mount` deleted) | not repaired: a decommission rerun refuses the absent counter and it cannot rejoin; use a new `server_root_id` |
+| A 26.6.4 decommission that stopped midway | the same refusal; use a new `server_root_id` |
+| Flat 26.6.4 staging keys `staging/<srid>/<name>.tmp` | swept as epoch 0 at the next mount |
+| Downgrade to 26.6.4 with a retired `server_root_id` kept by this version | 26.6.4 refuses to start it and its GC treats the farewell as terminal |
+| Downgrade: `owner_life`, `retirement_verified`, epoch staging keys | skipped by 26.6.4 readers; staging swept recursively |
+| Mixed versions: a 26.6.4 decommission tool rewrites an owner | the life drops to 0; servers of this version fail their owner check and stop (fail-closed) |
+| Downgrade: a `server_root_id` rejoined under a new `server_uuid` | start it once on this version before downgrading; 26.6.4 then starts it normally |
+| Downgrade: a `server_root_id` rejoined under the same `server_uuid` | 26.6.4 starts it and continues the counter |
+| Mixed versions: a 26.6.4 server starts with a retired `server_root_id` | refused, as in 26.6.4: it cannot rejoin |
+| Mixed versions: a 26.6.4 member restarts during a decommission | its owner check refuses after the tombstone; before it there is no recheck, as in 26.6.4 |
+
+Rolling upgrades keep the old races on the old binaries; decommission from an upgraded server.