Rejoin a decommissioned CAS server root on server start

[filimonov]

A CAS server whose server_root_id was decommissioned with SYSTEM CAS DROP POOL MEMBER now rejoins the pool as an empty member on its next start, with the same or a new server uuid. Before, the id was refused for good and needed a hand-edited owner object or a new server_root_id.

Built on PR B; until it merges, this diff also shows PR B's commits. Base: antalya-26.6-20261008.

Related: #2567

Changes

• Rejoin on a server start. Pool::open over a tombstoned owner takes the id over when epoch and mount exist, the retirement is verified, the slot is terminal and the old life's data is gone. The owner moves to a new life by If-Match, the epoch counter continues, and the member starts empty. A conflict re-reads the owner and decides again. The rejoin logs a warning with the old and new uuid and life.
• New server uuid. The mount body is rewritten for the new uuid before the owner is claimed. A same-uuid rejoin leaves the mount body alone.
• Refused: a self-remount (never rejoins), a server uuid of 0, a missing epoch or mount (an id retired by 26.6.4, or deleted by hand; checked first, so such an id is never sent to a rerun), an unverified tombstone, a held slot, data of the old life, and an owner or slot that changes on every decision.
• Foreign terminal slot. A starter may reclaim a foreign terminal slot body only after a fresh owner read shows its own uuid, the recorded life and no tombstone. A live foreign body stays ForeignOwner, and the decommission tool never takes a foreign body.
• Old access-probe key. Before the emptiness check, the rejoin deletes, by ETag, the previous owner's late roots/<srid>/clickhouse_access_check_<uuid> key. Any other object there still blocks the rejoin.
• Refusal texts state the cause and point at the docs. previous life remains names the part left and, for catalog rows and manifests, SYSTEM CAS GC RUN '<disk>' on the GC leader. The foreign-owner refusal also names a duplicate server_root_id between two running servers.
• system.cas_mounts shows a decommissioned member as retired (one more GET per terminated or fenced row) and adds the last column retirement_verified: 1 once the retirement is verified, else 0. A rerun of SYSTEM CAS DROP POOL MEMBER is for retired with retirement_verified = 0 only.
• Docs: rejoin, its refusals and the rerun rule (operations, troubleshooting, mounts-and-leases, system.cas_mounts, SYSTEM CAS DROP POOL MEMBER). Decommission is no longer called permanent: the data is gone for good, the id is not.

Risks / notes

• Late tombstone. The admin's fenced plane bounds when the tombstone is sent, not when the store applies it. PR B withholds the admin's farewell after an unresolved tombstone, so the victim waits for a GC fence; a tombstone applied later than that is a timing assumption, like an in-flight drain request. The docs keep the repair: if a server of the id ran after the tombstone time in its start error, do not rerun; rewrite the owner's body line to the next life and restart that member.
• State and column change. There is no retiring state: a tombstoned owner's terminal row is retired, and retirement_verified (new last column) tells verified from not. Queries that matched retiring from the earlier revision of this PR must read the column.
• Fail-closed becomes rejoin. A restarted decommissioned server used to be refused; it now comes back empty under the same server_root_id. Its local table definitions attach with no rows.
• Late-object corner. A rejoin that died between its two writes, followed by a late roots/ object, leaves a slot the next start refuses, and a decommission rerun cannot clear it (the admin never takes a foreign body). The refusal says what to delete by hand.
• clickhouse disks without a server uuid file does not rejoin: it keeps the refusal.
• Mixed versions: a 26.6.4 server still refuses a retired id. An id retired by 26.6.4 cannot rejoin; configure a new cas_server_root_id. Run SYSTEM CAS DROP POOL MEMBER only from upgraded servers: a 26.6.4 admin concurrent with a same-uuid rejoin can delete epoch and mount under it.
• Residuals, not coded (CAS-385):
  • A rerun over a verified retirement racing a same-uuid rejoin can land its tombstone on the rejoined owner and downgrade it to unverified. The rerun rule forbids a rerun outside retired with retirement_verified = 0; PR B's fenced tombstone narrows the window.
  • A new-uuid mount rewrite racing another starter's full rejoin, claim and farewell between our owner read and our mount write leaves our clean body under their owner until their next start reclaims it; a DROP POOL MEMBER of them in that window is refused.
• Rerunning a decommission bound to the original life is not done here.

Testing

• gtests: a rejoined member's part (same uuid and new uuid) survives three GC rounds run from another member; a killed rerun's lease blocks the rejoin until GC fences it; an id without epoch and mount is refused with "configure a new <cas_server_root_id>" and no rerun advice; retired with retirement_verified 0 and 1 in listMounts; one test over every ServerRootLeftover (control objects unchanged, probe key built from toString(UUID)).
• test_cas_drop_pool_member: after each post-rejoin insert, SYSTEM CAS GC RUN on every node, then the rows are rechecked; a verified retirement reads retired with retirement_verified = 1. Module: 3/3.
• CAS* unit tests: release 2747, ASan 2751, all passed. Every commit builds and passes release CAS*.
• TLA+ (CaEpochAllocCore round 14): with the tombstone allowed to land until GC fences the admin's slot, a clean farewell after an unresolved tombstone puts it on a running member (NoTombOverLive); with the farewell withheld the late-landing, paused-run and 26.6.4 rows are green under the rerun rule. The default battery (187 rows) meets every expectation.
• Soak S47 (decommission under load through the SQL verb, the executor killed after each of the six steps, rerun, rejoin with the same and a new uuid), 60 min, seed 47, all eight cycles: 139 of 140 checks pass. Every kill after the tombstone shows retired with retirement_verified = 0 and the rerun converges; a kill after the verified mark shows retirement_verified = 1 and needs no rerun; every rejoin comes back empty and its new rows survive three GC rounds; dangling = 0 after every cycle; soak oracle PHASE3 OK. The one red is the NoWait refusal text, which does not name the state to wait for; that text belongs to the base series, not this PR.

Commits (on top of PR B)

• ba336d2c60a9 Let a CAS start reclaim another uuid's terminal body of its own owner
• b4ede038732f Show retired CAS members in system.cas_mounts
• 85df7b155084 Rejoin a decommissioned CAS server root at server start
• e20348282ab0 Document CAS member rejoin and the retired state

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

A CAS server whose server_root_id was decommissioned with SYSTEM CAS DROP POOL MEMBER now comes back on its next start as an empty member, with the same or a new server uuid, instead of being refused. system.cas_mounts shows a decommissioned member as retired, with a new retirement_verified column.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [e203482]