CAS: decommission keeps a member's control objects; owner life and epoch nonce

[filimonov]

SYSTEM CAS DROP POOL MEMBER no longer deletes a member's owner, epoch and mount objects; it deletes only the member's data. Deleting those objects let a restarting victim be issued a writer epoch it already used and let a late tombstone land on a live successor. Keeping them makes epochs strictly increasing per server_root_id across lives and removes the tail's race handling.

Changes

• Writer-epoch counter nonce. Every counter write carries a fresh allocation_nonce. Two processes with the same server_uuid that race on an ambiguous counter write no longer resolve to the same epoch: their bodies differ, so the byte-equality resolve cannot adopt the other's write.

• Owner life. The owner object gains owner_life, a counter every owner write bumps by If-Match, and retirement_verified. A mount records the life it started under. After every slot claim (start and self-remount) it re-reads the owner and requires its own uuid, that life, and no tombstone; otherwise it fences its own slot body and fails. A self-remount compares the life before it allocates an epoch: an owner at another life ends the remount loop in IdentityLost, and a restart picks up the current life.

• Decommission run:

  1. claim the victim's slot;
  2. tombstone the owner (new life) on the admin's fenced mount plane, so no attempt is sent after the admin's lease deadline;
  3. confirm the claim still holds, otherwise roll the tombstone back on the unfenced control plane, unless the owner was already retired;
  4. drain the data on the admin's fenced mount plane, so a fenced admin stops;
  5. write the farewell;
  6. verify the slot and the owner and mark the retirement verified.

  Every exit between steps 2 and 5 runs the confirm. When the tombstone's outcome is unknown and the owner does not read back as the tombstone, the run fails without its clean farewell (Pool::withholdFarewell): the slot keeps the admin's live body, so the victim restarts only after GC fences it (about 1.05 × the lease TTL plus a GC round) instead of reclaiming at once. member_retired = 1 only for a verified mark. An owner that moved on to a later unretired life (a rejoin, or another run's rollback) reports member_retired = 0 with its own warning. A run that cannot tell whether its tombstone or verified mark landed re-reads the owner before advising a rerun.

• S3 staging keys carry the writer epoch (staging/<srid>/<epoch>/<random>.tmp), built and parsed in Layout. The startup sweep deletes only earlier epochs, so a paused older process can no longer delete its successor's staging. The epoch and the admission generation come from one capture.

• Removed: the epoch re-mint policy for decommission (EpochMintPolicy). A member whose counter is missing (left by an older decommission) is refused, not repaired.

• A mount that skips the capability probe (skip_access_check, always the case for decommission) logs it.

• Lease renewer fix, all CAS mounts. The stop flag was read before the renewal's admission. A stop landing in between made the renewal terminal with nothing sent, so a clean shutdown skipped its farewell and the next start paid the unclean-reclaim wait.

Risks / notes

• Result columns change. slot_removed is renamed member_retired at the same position, and namespaces_pending_gc is appended after warnings. Rows still Removing after the drain are reported in namespaces_pending_gc; GC deletes them and no rerun is needed.
• A retired server_root_id refuses to mount, as before. The refusal names the owner object, its retirement and its life; the troubleshooting table gives the owner rewrite a restart accepts. The alternative is a new server_root_id. Rejoining a retired id without a hand edit is a separate change.
• Format: allocation_nonce, owner_life and retirement_verified are optional plain keys; 26.6.4 readers skip them. An owner without owner_life reads as life 0, and a body without the new fields keeps its exact bytes.
• Mixed versions: a 26.6.4 decommission tool still deletes the control objects, and a 26.6.4 server has no owner recheck; both keep their old races. Use this version's tool for decommission. Downgrade: 26.6.4 refuses a retired member as before and treats the kept farewell as terminal.
• Known shutdown race, not fixed here. A stop that lands after a renewal write was sent still ends the renewer and skips the farewell. This is fail-safe: a decommission reports member_retired = 0 with "the slot does not hold this run's farewell" and asks for a rerun, which waits until GC fences the slot (about one mount lease TTL plus a GC round). test_cas_drop_pool_member accepts exactly this outcome; CASDecommission.LongDrainOutlivesOneTtl places the stop between two renewals and requires the farewell. Follow-up: shutdown waits for an in-flight renewal.
• Late tombstone. The admin's fenced plane bounds when the tombstone is sent, not when the store applies it. Withholding the farewell makes the victim wait for a GC fence; a tombstone applied later than that is a timing assumption, like an in-flight drain request.
• A remount over a retired owner still retries (reads only, no epoch, no slot write) until the server restarts and refuses.

Testing

• New gtests fail on the base for the stated reason, among them:
  • twin allocators sharing an epoch, and a resend keeping its nonce;
  • a late tombstone after a same-uuid rejoin, failing on the owner life;
  • the claim recheck: absent, foreign, retired or moved owner; self-fence only of its own body; recheck on remount; a remount over a moved life allocates no epoch and stops;
  • decommission: keeps epoch and farewell; a tombstone retry after the lease deadline is not sent; the owner rewritten before the tombstone aborts; a rollback keeps an earlier retirement, verified or not; an owner that moved on is not reported retired; a member written by 26.6.4 rolls back and restarts (pins existing behaviour); victim claiming after the tombstone; fence before the tombstone; rollback on the control plane; a paused run cannot clear a rerun's tombstone; resume after a crash at each step; drain stops on a fence;
  • staging: an older-epoch sweep keeps a successor's staging; capture vs remount; staging key shapes;
  • a slot deleted under the observation loop is re-claimed on the paced, bounded budget; the observation map drops an unlisted slot;
  • the renewer stop race;
  • CASDecommission.UnresolvedTombstoneLeavesNoFarewell: an unresolved tombstone leaves the admin's live body, and the victim reclaims only after a fence.
• Model: CaEpochAllocCore round 12 (docs/superpowers/models/CaEpochAllocCore_RESULTS.md on the CAS design branch): the fenced tombstone keeps NoTombOverLive; without it, and with the wider rollback rule, the sabotages are red.
• CAS* unit tests: release 2720, ASan 2724, all passed. test_cas_drop_pool_member: 2/2.
• Every commit builds and passes release CAS*.

Commits

• 8c0f14fd7fc4 Add an allocation nonce to the CAS writer-epoch counter
• 86b20da89e7f Record the CAS owner life and recheck the owner after a mount claim
• d1f194489a2c Keep the CAS control objects on decommission and fence the tombstone
• 133ee5786ae3 Put the writer epoch into CAS S3 staging keys
• e25a8282614e Remove the CAS epoch re-mint policy for decommission
• 93adba90c85d Keep a CAS renewal refused by a mid-renewal stop releasable
• 3a2641dc7032 Document the CAS member lifecycle and decommission

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

SYSTEM CAS DROP POOL MEMBER keeps the member's owner, epoch and mount objects, tombstones the owner before draining and verifies the retirement. Its result column slot_removed is renamed member_retired, and namespaces_pending_gc is appended. Every CAS mount re-reads its owner after claiming its slot. Writer-epoch allocation is unique between same-uuid processes, and S3 staging keys carry the writer epoch. A clean CAS shutdown no longer skips its farewell when the stop races the start of a lease renewal. A decommission whose tombstone outcome is unknown no longer writes its clean farewell, so the member restarts only after GC fences its slot.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [3a2641d]

[filimonov]

Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) and Stateless tests (amd_binary, cas s3 storage, parallel): Server died with Logical error: GroupingAggregatedTransform pushed bucket A twice. The stack is in aggregation (MergingAggregatedMemoryEfficientTransform), no CAS frames; the same failure hits other PRs on this base. Not related to this PR. Tracked upstream: ClickHouse#115986