CAS: refuse new nested server_root_id, delete S3 staging on abort, actionable member refusals

docs/en/antalya/cas/architecture/mounts-and-leases.md

@@ -20,8 +20,8 @@ otherwise regenerate the same `ServerUUID` from a wiped local state directory, w
 silently steal an existing identity.
 
 Validation (`validateServerRootId`, `Pool/CasServerRoot.h`) is fail-closed `BAD_ARGUMENTS`, no
-sanitizing fallback: non-empty, at most 255 bytes, no empty/`.`/`..` path segment, no `_files` or
-`_manifests` segment.
+sanitizing fallback: non-empty, at most 255 bytes, no `/`, not `.` or `..`, not `_files` or
+`_manifests`.
 
 It roots four subtrees and owns catalog names at or below `<server_root_id>`:
 
@@ -38,6 +38,11 @@ dedup work. Ref/namespace keys are also deliberately opaque and do not embed the
 Each replica sharing a backend endpoint must use a distinct `cas_server_root_id`; omitting the setting
 is a startup error.
 
+The id may not contain `/`, and this holds for an id that already owns its owner object too: decommission
+and the staging sweep select a member's objects by key prefix, so a member `a/b` would be drained by a
+decommission of `a`. A server whose id contains `/` refuses to open the disk; see
+[recreating a member whose id contains `/`](/antalya/cas/operations/migration#recreate-id-with-slash).
+
 ## The owner claim {#owner-claim}
 
 `claimOwnerOrThrow` binds `server_root_id` ↔ `server_uuid` **permanently**. The owner object is never deleted
@@ -46,10 +51,10 @@ and never reassigned — decommission only tombstones it in place.
 | Observed at `gc/server-roots/<server_root_id>/owner` | Action |
 |---|---|
 | present, same `server_uuid`, not tombstoned | proceed |
-| present, `retired_at_ms` set | `CORRUPTED_DATA` — explicitly decommissioned, refuses to resume |
-| present, different `server_uuid` | `CORRUPTED_DATA` — names the regenerated-uuid-file cause |
-| absent, subtree provably empty | `putIfAbsent` the owner (claim) |
+| present, `retired_at_ms` set | `CORRUPTED_DATA` — explicitly decommissioned; the refusal says to configure a new `cas_server_root_id` |
+| present, different `server_uuid` | `CORRUPTED_DATA` — names the regenerated-uuid-file cause and, for a dead owner, `SYSTEM CAS DROP POOL MEMBER` |
 | absent, subtree non-empty | `CORRUPTED_DATA` — identity lost over existing data |
+| absent, subtree provably empty | `putIfAbsent` the owner (claim) |
 | lost the `putIfAbsent` race | re-read; equal uuid proceeds, else `CORRUPTED_DATA` |
 
 "Provably empty" requires both an authoritative decoded catalog naming no life owned by `server_root_id` and
@@ -261,7 +266,8 @@ which any reclaim is instant.
 
 **Permanent removal** of a dead replica (`Cas::decommissionPoolMember`, driven by
 `SYSTEM CAS DROP POOL MEMBER '<server_root_id>' FROM DISK '<disk>'`) claims the victim's mount slot
-as an administrative writer with a no-wait policy (refuses immediately if the member is alive),
+as an administrative writer with a no-wait policy (refuses unless the slot is terminal:
+`terminated` or `fenced`; an `expired` slot is refused until the disk's GC leader fences it),
 drops every ref-bearing namespace, sweeps manifest debris before the slot (deleting the mount
 removes the watermark authority), drains staging and roots, then — only with zero warnings —
 retires in order: mount, epoch, a final liveness re-check, owner tombstone.

docs/en/antalya/cas/architecture/part-lifecycle.md

@@ -109,7 +109,7 @@ something recoverable; no row loses data or leaves a dangling reference.
 
 | # | Crash window | Left behind | Who cleans it |
 |---|---|---|---|
-| C1 | During staging | Local temp files, or S3 staging objects | Local: unconditional cleanup plus buffer destructor. S3: the mount's own staging sweep at next mount — never deleted on abort |
+| C1 | During staging | Local temp files, or S3 staging objects | Local: unconditional cleanup plus buffer destructor. S3: the transaction deletes it on commit or abort (best-effort); after a crash, the next mount of the same `server_root_id` sweeps it, or `SYSTEM CAS DROP POOL MEMBER` drains it |
 | C2 | After `stageManifest`, before `precommitAdd` | An unreferenced manifest body | Writer's best-effort exact-token delete; durable backstop is the orphan-manifest sweep |
 | C3 | `precommitAdd` returned Unresolved | A possibly-live precommit binding | Intent recorded pre-append; `abandon` appends the exact removal, tolerating absence. The body is never writer-deleted |
 | C4 | Between `precommitAdd` and `promote` | A live precommit plus uploaded blobs | No resume path exists. Removed by `abandon`, else by a fenced successor's stale-precommit sweep |

docs/en/antalya/cas/bucket-requirements.md

@@ -45,6 +45,12 @@ Because that check is part of the mount battery, `skip_access_check = true` is r
 generation-token disk. Mount the disk read-only if you need to start before the access check can
 pass.
 
+**Incomplete multipart uploads.** With `cas_staging_backend = s3`, a process killed during a
+multipart upload leaves an incomplete upload under `staging/<server_root_id>/`. Object `LIST` does
+not show it, so neither the transaction's own delete nor the mount-start sweep can remove it, and its
+parts stay billed. Add an `AbortIncompleteMultipartUpload` lifecycle rule on the bucket (one day is
+enough: a staging upload completes within its own transaction or never).
+
 ## Soft delete is an operator precondition {#soft-delete}
 
 Object **soft delete must be disabled** on a `CAS` bucket, and unlike versioning this one is *not*

docs/en/antalya/cas/configuration.md

@@ -89,7 +89,7 @@ entirely before release. Treat this table as a snapshot of the current build, no
 
 | Setting | Default | Description |
 |---|---|---|
-| `cas_server_root_id` | — (required) | Explicit layout subtree identity; macros expand as in the `s3` `endpoint`. Anchored in the pool by a write-once owner claim — a colliding identity is refused at mount |
+| `cas_server_root_id` | — (required) | Explicit layout subtree identity; macros expand as in the `s3` `endpoint`. Anchored in the pool by a write-once owner claim — a colliding identity is refused at mount. May not contain `/`, also after macro expansion: `SYSTEM CAS DROP POOL MEMBER` selects objects by key prefix, so a member `a/b` would be drained by a decommission of `a`. A disk configured with such an id refuses to open, including one whose id already owns its owner object in the pool; see [recreating a member whose id contains `/`](/antalya/cas/operations/migration#recreate-id-with-slash) |
 | `cas_scratch_path` | `<clickhouse-path>/disks/<disk_name>/cas_scratch/` | Server-local scratch dir for the write-buffer spill; a relative value is anchored to the server data path |
 | `cas_gc_enabled` | `true` | Run the background GC scheduler on this disk. `false` is a debugging aid, not an operating mode: garbage then accumulates indefinitely and silently — watch `system.cas_gc_log` for round activity if you ever toggle it |
 | `cas_gc_interval_sec` | `60` | Seconds between background GC rounds (≥ 1) |
@@ -110,7 +110,7 @@ entirely before release. Treat this table as a snapshot of the current build, no
 | `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove, conditional write), at least 1. Together with the connect cap it forms the attempt envelope (`cas_attempt_timeout_ms + 2 × cap`; the cap is `cas_attempt_timeout_ms` itself when the disk's `connect_timeout_ms` is `0`, else `min(connect_timeout_ms, cas_attempt_timeout_ms)`) that the lease arithmetic reserves: one TCP connect and one TLS handshake under the cap each, send/receive bounded per socket operation by `cas_attempt_timeout_ms`. With background renewal the cadence check requires `cas_mount_renew_period_ms + 2 × envelope + cas_lease_safety_margin_ms < cas_mount_lease_ttl_ms`, which puts an effective ceiling on the frozen connect cap: under the defaults (TTL 30000, period 10000, margin 2000) the envelope must stay under 9000, so a disk `connect_timeout_ms` of 2000 ms or more refuses to open writable — lower the connect timeout or raise the TTL if you hit this. Blob body publication does not use this budget: its object-creating request is sent once under the disk's `request_timeout_ms`, with the connect timeout under the same cap |
 | `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: the attempt envelope + `cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, and `cas_mount_renew_period_ms` + 2 × envelope + `cas_lease_safety_margin_ms` too, or the disk refuses to open writable |
 | `cas_unsafe_remount_no_delay` | `0` | Reclaim a mount slot that carries this server's own uuid at once after a hard restart, without observing the slot's token for the lease TTL. Unsafe whenever two processes can hold the same `server_uuid` (a copied uuid file, a stalled predecessor). After such a reclaim the predecessor can still start conditional writes until its own cutoff (`confirmed deadline − cas_lease_safety_margin_ms − 2 × envelope`) or until its next renewal meets the token guard, and a request it already sent may still materialize later. That is not a data hazard: ref-log keys carry `(writer_epoch, sequence)` and creates are conditional, so two writers can never commit different bodies to one key, and recovery's epoch seal settles any straggler (recovery fails closed after 64 successive seal-create attempts displaced by newly materializing old-epoch transactions). The exposure is availability, not data. Intended for test stands and deployments that guarantee one process per uuid |
-| `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount |
+| `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount. A transaction deletes its staging objects on commit and on abort; mount start removes leftovers of the same `server_root_id`. Neither can see a multipart upload that was never completed, so add an `AbortIncompleteMultipartUpload` lifecycle rule on the bucket to reclaim those |
 
 Both settings are local. Startup reclaim and GC's fence-out judge liveness by the mount slot's write
 token holding stable on the observer's own `CLOCK_BOOTTIME` for `L + floor(L / 20) + step`. `L` is the

docs/en/antalya/cas/operations/migration.md

@@ -202,12 +202,30 @@ mount slot stays in place as a resume anchor rather than being fully retired.
 
 **Preconditions.** Confirm the member is actually and permanently dead before running this: the
 operation fences that `server_root_id` out even if the server comes back online, and it deletes
-namespace and drain state that cannot be recovered. Check `system.cas_mounts` for the member's
-`state` and `last_success_age_seconds` first — a `live` row, or one with a recent lease renewal,
-means the member is not a decommission candidate yet.
+namespace and drain state that cannot be recovered. The command claims the slot without waiting
+and trusts no wall clock, so it accepts only a terminal slot in `system.cas_mounts`:
+`state = 'terminated'` (clean stop) or `state = 'fenced'` (GC fenced the lease). A `live` or `expired`
+row is refused with `alive or contended`; a hard-killed member stays `expired` until the disk's GC leader fences it at the start of a GC round, once it has seen the lease unchanged for about one
+`cas_mount_lease_ttl_ms` plus one `cas_mount_renew_period_ms`; the wait can be one GC round longer. `SYSTEM CAS GC RUN` on the leader
+(`is_leader = 1` in `system.cas_mounts`) can trigger the fence, but the run that fences must come at
+least that long after the first run that saw the lease; two back-to-back runs do not fence.
 
 **Verification.** After the command reports `slot_removed = true` with no warnings, the member's
 `server_root_id` no longer appears as a row in `system.cas_mounts` on any peer, and a subsequent
 `SYSTEM CAS GC RUN` on the pool will no longer wait on or fence its heartbeat. See
 [mount, unmount, crash](/antalya/cas/architecture/mounts-and-leases#mount-lifecycle) for how the
 claim, drain, and retirement steps fit into the mount-slot lifecycle.
+
+## Recreating a member whose id contains `/` {#recreate-id-with-slash}
+
+A `cas_server_root_id` that contains `/` is refused: the disk fails to open with `BAD_ARGUMENTS` and a
+message that contains `contains '/'`. Pools created by earlier builds can hold such an id. To bring the
+server back:
+
+1. Set `<cas_server_root_id>` to a new value without `/`, for example `shard-01-replica-a` instead of
+   `shard-01/replica-a`. If the id comes from macros, remove the `/` from the template.
+2. Restart the server. The disk claims the new id over an empty subtree.
+
+Data stored under the old id is not visible under the new id. The old id's objects stay in the pool
+and keep their blobs referenced. `SYSTEM CAS DROP POOL MEMBER` and `clickhouse-disks cas-drop-member`
+refuse an id with `/`, so they cannot reclaim them.

docs/en/antalya/cas/operations/troubleshooting.md

@@ -22,6 +22,10 @@ tools.
 | A dangling-access exception or `CORRUPTED_DATA` on read | Run `clickhouse-disks cas-fsck --detail` and check `dangling` specifically — it is the one class that means data loss, distinct from `unreachable`/`awaiting-gc`, which are just waiting for graduation | A nonzero `dangling` count is a real incident: collect the `--detail` output (see [what to collect before filing a bug](/antalya/cas/operations/debugging#filing-a-bug)) before taking any destructive action |
 | `SYSTEM CAS FSCK` or `clickhouse-disks cas-fsck` times out on a large pool | The scan is bounded by `--timeout` (default 600s / the `SYSTEM` form has no override); a large `roots/` prefix can make the scan slow | Retry with `--partial` to see the counts accumulated so far instead of aborting empty-handed, or `--namespace <prefix>` to scope the scan to a subset of namespaces |
 | `SYSTEM CAS DROP POOL MEMBER` returns a non-empty `warnings` column | A per-object drain step could not confirm emptiness; the mount slot is left terminated but not fully drained, as a resume anchor | Rerun the same command — it is resumable and skips namespaces already marked removed, reporting them under `namespaces_already_removed` |
+| `SYSTEM CAS DROP POOL MEMBER` refuses with `alive or contended` although the member is down | The claim trusts no wall clock: `expired` in `system.cas_mounts` is not a certificate of death; only a terminal slot (`terminated` or `fenced`) is reclaimable | Wait until the member's row shows `state = 'terminated'` (clean stop) or `state = 'fenced'` (the disk's GC leader fences the stale lease at the start of a GC round, once it has seen it unchanged for about one `cas_mount_lease_ttl_ms` plus one `cas_mount_renew_period_ms`, so the wait can be one GC round longer; `SYSTEM CAS GC RUN` on the leader, `is_leader = 1` in `system.cas_mounts`, can trigger the fence, but the run that fences must come at least that long after the first run that saw the lease), then rerun |
+| A server refuses to start with `owned by a different server` | The owner object binds the `server_root_id` to another `server_uuid`: the local uuid file was regenerated, or two servers share one `cas_server_root_id` | Restore the uuid file (keep it on a persistent volume); or start this server under a new `cas_server_root_id`. To reclaim the old id's data once its owning server is permanently gone, decommission it with `SYSTEM CAS DROP POOL MEMBER` from a surviving member, once its slot is `terminated` or `fenced`; the two steps are independent, and in a single-member pool starting under the new id comes first, because a hard-killed old id is fenced only by a running GC leader |
+| A `CAS` disk refuses to open with `contains '/'` | Its `cas_server_root_id` contains `/`, directly or through a macro; such ids are refused, including ids that already own their owner object in the pool | Configure a new `cas_server_root_id` without `/` and restart; see [recreating a member whose id contains `/`](/antalya/cas/operations/migration#recreate-id-with-slash) |
+| A server refuses to start with `explicitly decommissioned` | Its `server_root_id` was retired by `SYSTEM CAS DROP POOL MEMBER`; the tombstone is permanent so writer epochs never repeat | Configure a new `cas_server_root_id` for the disk; do not clear the tombstone by hand |
 | Writes or `ALTER`s on a `CAS` disk fail with a `READONLY`-class error | The disk's metadata storage rejects every mutating entry point; this is deliberate for a disk opened with `<readonly>true</readonly>`, used by every offline `clickhouse-disks` tool | Confirm whether the disk was intentionally configured read-only (offline inspection, `cas-fsck`, `cas-gc-dryrun`, `cas-gc-rebuild`, `cas-drop-member` all require it); a production disk serving writes must not carry `<readonly>true</readonly>` |
 | A table stays unavailable after a transient network error during startup | `AsyncLoader` has no retry/requeue path for a failed table load job: a transient S3 `NETWORK_ERROR` during `CAS` ref-table startup recovery can leave the job permanently `FAILED` | Restart the server, or issue a fresh load for the table; this is a one-shot job design, not a `CAS`-specific bug |
 | A mounted pool directory was removed or renamed out of band | Renewal observes an absent, foreign, successor, or otherwise conflicting mount body and terminates the keeper with a typed fail-closed exception; the runtime closes the local write fence and requests remount rather than adopting the body | Never remove or rename a live pool's storage path. To retire a member permanently use [`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission) instead of raw filesystem operations; collect the `watermark_renew` classification (`vanished` for an absent body, `conflict` for a foreign, successor, or otherwise conflicting one) and the subsequent `mount_remount` step |

docs/en/sql-reference/statements/system.md

@@ -594,6 +594,13 @@ are confirmed. This is a **destructive, irreversible** operation: only run it on
 confirmed permanently dead, since it fences the member out even if it later comes back online, and
 it deletes namespace and drain state that cannot be recovered.
 
+The claim accepts only a terminal mount slot (`state` `terminated` or `fenced` in
+`system.cas_mounts`). A hard-killed member shows `expired` until the disk's GC leader fences it at the start of a GC round, once it has
+seen the lease unchanged for about one lease TTL plus one renew period; the wait can be one GC round longer. Until then the command
+refuses with `alive or contended`. `SYSTEM CAS GC RUN` on the leader (`is_leader = 1` in
+`system.cas_mounts`) can trigger the fence, but the run that fences must come at least that long after
+the first run that saw the lease; two back-to-back runs do not fence.
+
 It is a writer operation, not GC: it emits ordinary ref-edge deltas rather than inventing GC
 transitions, and it does not synchronously reclaim shared blob content — removing the ref edges only
 makes the now-unreferenced blobs eligible for an ordinary GC round to reclaim later.
@@ -602,8 +609,9 @@ makes the now-unreferenced blobs eligible for an ordinary GC round to reclaim la
 SYSTEM CAS DROP POOL MEMBER 'server_root_id' FROM DISK 'disk_name' [ON CLUSTER cluster_name]
 ```
 
-Both `server_root_id` and `disk_name` are required string literals (a `server_root_id` is an opaque server-root path
-that may contain `/`, not a plain identifier, so it cannot be written unquoted).
+Both `server_root_id` and `disk_name` are required string literals (a `server_root_id` is an opaque string, not a
+plain identifier, so it cannot be written unquoted). A `server_root_id` that contains `/` is refused with
+`BAD_ARGUMENTS`.
 
 The operation is resumable: a rerun skips namespaces already marked `Removed` and reports them
 separately from namespaces newly removed by this invocation. Per-object drain failures are recorded