Renew the CAS mount lease until a definitive answer; an expired lease is a pause

docs/en/antalya/cas/architecture/mounts-and-leases.md

@@ -81,14 +81,17 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve
   after a flat pause and settled by the reissue's own answer (a 2xx) or by the exact `GET` that
   follows its `412`. The renewer adopts the result only when the complete body, including `write_attempt_id`,
   equals its immutable request. If the predecessor token is still current, another identical `PUT`
-  may follow bounded backoff. A same-pair twin, GC-fenced body, successor, foreign holder, or absent
+  follows after a pause. A same-pair twin, GC-fenced body, successor, foreign holder, or absent
   body is never treated as this renewal.
-- **Absolute deadline.** Renewal uses `CLOCK_BOOTTIME`, not `CLOCK_MONOTONIC`, so a VM resumed from
-  suspend correctly observes itself expired. Its absolute deadline is the minimum of the existing
-  request-operation budget and the last confirmed lease deadline minus the safety margin. The
-  controller checks that one attempt envelope still fits before each backend `PUT` or resolving
-  `GET`, after each interruptible backoff, and before accepting success. A retry, `GET`, response
-  timestamp, or wall-clock step never extends authority.
+- **Deadline from the start.** Renewal uses `CLOCK_BOOTTIME`, not `CLOCK_MONOTONIC`, so a VM resumed
+  from suspend correctly observes itself expired. A confirmed renewal sets the deadline to its own
+  start plus the lease; a retry, `GET`, response timestamp, or wall-clock step never moves it. The
+  background renewal has no deadline of its own: it retries one body every 0.8–1.2 s (measured from
+  the start of the previous attempt) until the store answers about the slot, or until a stop, a
+  remount park request, a lost fence or a pool leaving `Live` ends it. A
+  refusal (`AccessDenied`, `MalformedXML`, …) sends the same body again one renewal period later.
+  The bounded renewals — the startup re-anchor, the remount re-anchor, and the background renewal of a
+  pool opened for decommission — stay bounded by the last confirmed lease minus the safety margin.
 - **Cadence.** The runtime normally starts a logical renewal every `cas_mount_renew_period_ms` (default
   10 s), with TTL `cas_mount_lease_ttl_ms` (default 30 s, TTL/3 renewal ratio). The next beat is anchored
   at the committed body's pre-I/O BOOTTIME start. A slow recovery therefore causes an immediate
@@ -103,19 +106,32 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve
 
 **Losing the lease is neither read-only mode nor a process abort.** `MountLeaseRenewer` is a
 synchronous durable-slot state machine. A committed result advances its token, sequence, confirmed
-BOOTTIME deadline, and cadence anchor. Any admitted deterministic failure, confirmed conflict, or
-ambiguity left at the deadline/attempt limit moves it to `RenewalTerminal`; it cannot mint another
-body or publish a clean farewell. Owner cancellation before any request is the only
-`NotAttempted` result and leaves clean release possible. Cancellation after a request was sent is
-terminal because that request may still land.
+BOOTTIME deadline, and cadence anchor. A deterministic failure, a confirmed conflict, or, for a
+bounded renewal, ambiguity left at its deadline moves it to `RenewalTerminal`; it cannot mint another
+body or publish a clean farewell. Owner cancellation before any request, or between two calls of the
+background renewal after a refusal or after a read that found the slot unchanged, is `NotAttempted` and
+leaves clean release possible. Cancellation after a request of the current call was sent, or while the
+reads after a refused precondition repeat, is terminal: that request may still land, or the slot may
+have moved.
+
+**An expired lease is a pause.** While the store does not answer, writes stop being admitted once the
+remaining lease cannot cover one write: `2 × envelope + safety_margin` before the deadline. From that
+moment `system.cas_mounts` shows `lifecycle = 'not_live'` and `lifecycle_reason = 'lease_expired'`,
+and the server logs one `WARNING`, at the next renewal attempt, that ends with the last renewal error:
+`CAS mount lease of '<server_root_id>' under writer_epoch <E> has too little time left for a write`.
+Nothing is remounted. A confirmed renewal resumes writes under the same `writer_epoch` only if its start
+plus the lease leaves more than `2 × envelope + safety_margin`; a renewal confirmed later than that
+leaves writes paused, and the next one starts at once. A clean shutdown during a pause skips the
+farewell, which has no lease room left, and leaves the slot for GC to observe. Such a late renewal stores an `expires_at_ms` already in the past, and GC still does
+not fence the slot for it: GC fences only a token it watched unchanged for its whole threshold.
 
 After the renewer call returns, `CasMountRuntime` consumes the result. A terminal result trips the
 local fence (latches `lost`, bumps the fence generation, moves the in-process runtime to
 `TransientNotLive`) and latches one self-remount generation. A confirmed foreign/successor or
 same-pair conflict remains a typed fail-closed error; it is never adopted. A real fence still costs
 only an epoch: recovery reclaims with a fresh one, bounded at three whole-chain attempts. This is the
-general CAS posture: doubt about the source fails closed, while transport ambiguity may retry only
-inside authority already proved by the last confirmed lease.
+general CAS posture: doubt about the source fails closed, while transport ambiguity is retried until a
+definitive answer and writes are admitted only inside the last confirmed lease.
 
 GC's own view of a dead server is symmetric and clock-skew-immune: a slot becomes fence-eligible
 only after the leader observes the *same* renewal token hold stable, on its own monotonic clock,

docs/en/antalya/cas/operations/debugging.md

@@ -129,19 +129,21 @@ row followed by a `recovered` one — and the older `unresolved_reason`, `deadli
 `stop_cause` keys are gone; everything they used to distinguish is now named directly by
 `classification`. Interpret the sequence as follows:
 
-- `outcome = 'recovered'` means an in-budget renewal landed, in the same epoch. `classification`
+- `outcome = 'recovered'` means a renewal landed after a retry or a settling read, in the same epoch.
+  Writes are re-admitted only if it left more than `2 × envelope + safety_margin` of lease. `classification`
   says how: `committed_by_read` means an exact `GET` proved a landed request; `committed_after_retry`
   means a later identical physical `PUT` completed and the response itself proved it.
-- `outcome = 'failed'` carries the decisive `classification`: `external_lease_deadline` (the
-  confirmed lease's own safety margin, not the request policy, ran out first — check object-store
-  latency or `BOOTTIME` advancement before anything else), `request_deadline` (the ninety-second
-  request policy exhausted first), `unresolved` (every attempt was ambiguous and never settled by
+- `outcome = 'failed'` carries the decisive `classification`: `external_lease_deadline` (a
+  bounded renewal only: the confirmed lease's own safety margin, not the request policy, ran out first — check object-store
+  latency or `BOOTTIME` advancement before anything else), `request_deadline` (a bounded
+  renewal only: the ninety-second request policy exhausted first), `unresolved` (every attempt was ambiguous and never settled by
   the time the operation gave up), `conflict` (an exact resolve read found another body — a
   same-pair twin, a GC-fenced body, a successor epoch, or a foreign holder), `cancelled` (a
   renewal in flight was cancelled by shutdown or a remount park request; expected during graceful
   shutdown), `fence_or_lifecycle_lost` (another local fence loss or a terminal lifecycle transition
-  closed admission while the operation was active), `deterministic_failure` (the store's own
-  answer proved the write never applied), and `vanished` (an exact resolve read proved the mount
+  closed admission while the operation was active), `deterministic_failure` (a local
+  failure, or, for a bounded renewal, a store answer proving the write never applied;
+  the background renewal sends a refused write again), and `vanished` (an exact resolve read proved the mount
   slot absent — the pool directory was removed or renamed out of band, or a decommission raced the
   renewal). `terminal_unclassified` means the renewal terminated through a path that assigned no
   classification; that is a defect to report together with the surrounding rows, not an operator

docs/en/antalya/cas/operations/monitoring.md

@@ -58,7 +58,7 @@ window and correlate them with the `server_root_id` in `system.cas_log`.
 | `CASMountRenewalRetries` | One per physical renewal `PUT` after the first in the same logical renewal | Positive growth shows in-period retry, not a later cadence beat |
 | `CASMountRenewalResolved` | One per logical renewal proved committed by an exact resolving `GET` | A response was ambiguous, but exact bytes and `write_attempt_id` proved the write |
 | `CASMountRenewalRecovered` | One per logical renewal committed after a retry or exact resolving `GET` | Recovered object-store blips that retained the existing mount incarnation |
-| `CASMountRenewalDeadlineExceeded` | One per logical renewal stopped by the external lease-safety deadline | The last confirmed lease no longer left enough safe time; this is narrower than request-budget exhaustion |
+| `CASMountRenewalDeadlineExceeded` | One per bounded logical renewal (startup or remount re-anchor, or a decommission's background renewal) stopped by the external lease-safety deadline | Growth means a bounded renewal found too little confirmed lease left; a mount's own background renewal never counts here |
 | `CASRemountAttempts` | One per invocation of the existing whole-chain remount attempt | Includes both successful and failed attempts |
 | `CASRemountSucceeded` | One per whole-chain attempt that restored `Live` under a fresh writer epoch | Must be a subset of `CASRemountAttempts` |
 | `CASRemountFailed` | One per whole-chain attempt that returned without restoring `Live` | Includes a named step exception or a step that returned transiently |

docs/en/antalya/cas/operations/troubleshooting.md

@@ -38,16 +38,21 @@ Start with the `watermark_renew` timeline described in
    `CASMountRenewalRecovered` rises while `CASMountLeaseLost` and all remount counters stay flat. No
    intervention is needed unless the rate is sustained; investigate backend throttling/latency before
    the blips consume the lease budget.
-2. **External lease-safety exhaustion.** The failed row has
-   `classification = 'external_lease_deadline'`; `CASMountRenewalDeadlineExceeded` and
-   `CASMountLeaseLost` rise. The runtime correctly refused to manufacture authority beyond the last
-   confirmed lease. Check object-store latency and BOOTTIME/suspend history, then follow the ensuing
-   remount. `classification = 'request_deadline'` is the sibling case: the ninety-second request
-   policy exhausted first rather than the lease's own safety margin.
-3. **Cancellation.** `classification = 'cancelled'` after a sent request is terminal and suppresses a
-   clean farewell because the request may still land. Cancellation before any request remains
-   `Active` and emits no failed aggregate row; during graceful shutdown that is the expected
-   clean-release path.
+2. **External lease-safety exhaustion.** Only a bounded renewal ends this way: the startup re-anchor,
+   the remount re-anchor, or the background renewal of a pool opened for decommission. A mount's own
+   background renewal keeps retrying; a long outage shows as
+   `lifecycle_reason = 'lease_expired'` and resumes under the same `writer_epoch`. The failed row has
+   `classification = 'external_lease_deadline'` and `CASMountRenewalDeadlineExceeded` rises. The
+   runtime refused to claim authority beyond the last confirmed lease. Check object-store latency and
+   BOOTTIME/suspend history. At startup the disk does not start; during a remount the remount loop
+   retries. `classification = 'request_deadline'` is the sibling case for the same bounded renewals:
+   the ninety-second request policy exhausted first rather than the lease's own safety margin.
+3. **Cancellation.** `classification = 'cancelled'` after a request of the current engine call was sent
+   is terminal and suppresses a clean farewell because the request may still land; so is cancellation
+   while the reads after a refused precondition repeat. Cancellation before any request, or between two
+   calls of the background renewal after a refusal, remains `Active` and emits no failed aggregate row;
+   during graceful shutdown that is the expected clean-release path while the lease has room for the
+   farewell. During a pause it has none: the slot is left for observation.
 4. **Confirmed conflict.** `classification = 'conflict'` means exact resolution found another body;
    inspect `server_root_id`, `writer_epoch`, `seq`, and `write_attempt_id`. Same-pair twins, GC-fenced
    bodies, successor epochs, and foreign holders all remain fail closed. Do not delete or rewrite the

docs/en/antalya/metrics_and_events.md

@@ -285,10 +285,10 @@ window.
 | Event | Type | Description |
 | --- | --- | --- |
 | `CASMountExclusivityViolation` | `Number` | Counts CAS mount releases where a runtime that still BELIEVED IT OWNED the mount (no deposition ever observed) found a FOREIGN occupant in the slot. This is the single-writer guarantee being broken, not a failover: the release refuses, the slot is left untouched, and the write fence is latched so the runtime stops trusting itself. This must always be zero. |
-| `CASMountLeaseLost` | `Number` | Counts exactly once per operational CAS mount-lease Live-to-TransientNotLive loss/recovery generation. The initiating external loss or the first ordinary terminal renewal consumer owns the increment, including external lease-safety deadline exhaustion; parked/classification/shutdown paths do not duplicate it. |
+| `CASMountLeaseLost` | `Number` | Counts exactly once per operational CAS mount-lease Live-to-TransientNotLive loss/recovery generation. The initiating external loss or the first ordinary terminal renewal consumer owns the increment; parked/classification/shutdown paths do not duplicate it. |
 | `CASMountReleaseSkippedForeignOccupant` | `Number` | Counts conclusive CAS mount-renewal observations that found a FOREIGN successor in the slot. This is the EXPECTED end state of a failover: renewal fences the deposed runtime, terminal teardown skips the farewell without backend I/O, and the successor's slot is left byte-for-byte untouched. Steady non-zero values on a cluster that is not failing over are worth investigating; a value that tracks failovers is normal. |
 | `CASMountRenewalAttempts` | `Number` | Number of physical conditional renewal PUTs sent for CAS mount leases. This counts transport attempts, not logical renewals. |
-| `CASMountRenewalDeadlineExceeded` | `Number` | Number of logical CAS mount-lease renewals stopped by the external lease-safety deadline. Growth means the last confirmed lease no longer had enough safe time for another physical attempt. |
+| `CASMountRenewalDeadlineExceeded` | `Number` | Number of bounded CAS mount-lease renewals (the startup and remount re-anchors, and the background renewal of a pool opened for decommission) stopped by the external lease-safety deadline. Growth means the last confirmed lease no longer had enough safe time for another physical attempt. |
 | `CASMountRenewalRecovered` | `Number` | Number of logical CAS mount-lease renewals that committed after a physical retry or exact resolving GET. |
 | `CASMountRenewalResolved` | `Number` | Number of CAS mount-lease renewals whose committed outcome was proved by an exact resolving GET. |
 | `CASMountRenewalRetries` | `Number` | Number of physical conditional renewal PUTs sent after the first attempt of one logical CAS mount-lease renewal. |

docs/en/operations/system-tables/cas_mounts.md

@@ -38,9 +38,9 @@ rows.
 - `last_success_age_seconds` ([Nullable(UInt64)](/sql-reference/data-types/nullable)) — Seconds since this disk's GC last led a round (`0` if it has never led or GC is not running here).
 - `wedged_namespace_count` ([Nullable(UInt64)](/sql-reference/data-types/nullable)) — Ref-append lanes currently wedged on this disk (an uncertain `PUT` exhausted its retry budget).
 - `lifecycle` ([String](/sql-reference/data-types/string)) — This server's content-addressed pool lifecycle for the disk (a non-gated snapshot, always populated so a not-live disk stays visible): one of `live`, `not_live`, `identity_lost`, `vanished`, `constructing` (never started), or `shutdown` (torn down).
-- `lifecycle_reason` ([String](/sql-reference/data-types/string)) — The enum-clean sub-state word for a `vanished` disk: `replaced` or `forgotten`. Empty for every other lifecycle, so `lifecycle || '(' || lifecycle_reason || ')'` reads e.g. `vanished(forgotten)`.
-- `lifecycle_detail` ([String](/sql-reference/data-types/string)) — The full typed reason text naming the actual cause when not live: the vanish diagnosis (a data root replaced by a foreign pool, or decommissioned by `SYSTEM CAS FORGET` at a given time) or the identity-loss message. Empty when live.
-- `lifecycle_since` ([Nullable(DateTime)](/sql-reference/data-types/nullable)) — When this server entered the current non-live lifecycle state. `NULL` when live, or when the state has no backing pool to date from.
+- `lifecycle_reason` ([String](/sql-reference/data-types/string)) — The enum-clean sub-state word: `replaced` or `forgotten` for a `vanished` disk, `lease_expired` for a `not_live` disk whose mount lease has too little time left to admit a write while its renewal keeps retrying. Empty for every other lifecycle, so `lifecycle || '(' || lifecycle_reason || ')'` reads e.g. `vanished(forgotten)`.
+- `lifecycle_detail` ([String](/sql-reference/data-types/string)) — The full typed reason text naming the actual cause when not live: the vanish diagnosis (a data root replaced by a foreign pool, or decommissioned by `SYSTEM CAS FORGET` at a given time) or the identity-loss message. Empty when live, and for `not_live` with `lifecycle_reason = 'lease_expired'`.
+- `lifecycle_since` ([Nullable(DateTime)](/sql-reference/data-types/nullable)) — When this server entered the current non-live lifecycle state. For `lease_expired`, when writes paused. `NULL` when live, or when the state has no backing pool to date from.
 - `renewal_in_progress` ([Nullable(UInt8)](/sql-reference/data-types/nullable)) — `1` while this server renews its mount lease.
 - `renewal_attempts` ([Nullable(UInt64)](/sql-reference/data-types/nullable)) — Conditional `PUT`s sent by the current renewal, or by the last one once it ended. Updated per attempt, so a renewal stuck in retries is visible before it ends.
 - `last_renewal_error` ([Nullable(String)](/sql-reference/data-types/nullable)) — The last failed request of that renewal, as `<VERB> <key>: <class> <code> <message>`. `class` is `refused` when the store definitely refused the request (access denied, malformed request, entity too large), `transport` otherwise; `code` is the store's error name, such as `SlowDown`. `NULL` when no request failed.