Say whether a failed CAS ref-log commit can still take effect

docs/en/antalya/cas/architecture/part-lifecycle.md

@@ -1,5 +1,5 @@
 ---
-description: 'The part-add protocol from local build through blob upload to promote, its nine crash points and their cleaners, and how each MergeTree operation maps onto it.'
+description: 'The part-add protocol from local build through blob upload to promote, its crash points and their cleaners, and how each MergeTree operation maps onto it.'
 sidebar_label: 'Part lifecycle'
 sidebar_position: 6
 slug: /antalya/cas/architecture/part-lifecycle
@@ -104,8 +104,9 @@ manifest never loses an owner, so it is net zero.
 
 ## Crash points and their cleaners {#crash-points}
 
-This table is the single best summary of the design's crash-safety story: every row leaks
-something recoverable; no row loses data or leaves a dangling reference.
+This table is the single best summary of the design's crash-safety story: no row loses committed
+data or leaves a dangling reference. In C6 and C10 the part itself may take effect; C3 and C11
+leave only a precommit binding.
 
 | # | Crash window | Left behind | Who cleans it |
 |---|---|---|---|
@@ -114,10 +115,15 @@ something recoverable; no row loses data or leaves a dangling reference.
 | C3 | `precommitAdd` returned Unresolved | A possibly-live precommit binding | Intent recorded pre-append; `abandon` appends the exact removal, tolerating absence. The body is never writer-deleted |
 | C4 | Between `precommitAdd` and `promote` | A live precommit plus uploaded blobs | No resume path exists. Removed by `abandon`, else by a fenced successor's stale-precommit sweep |
 | C5 | Mid blob fan-out | Already-uploaded blobs | Nothing merged; blobs become `GC`-reclaimable debris; the part is not published |
-| C6 | `promote` append Unresolved | The ref may or may not be committed | Commit state Uncertain — the relink layer maps this to "retry the whole fetch", never to a byte fetch |
+| C6 | `promote` append Unresolved | The ref may or may not be committed | Commit state Uncertain — the relink layer maps this to "retry the whole fetch", never to a byte fetch. The client sees `outcome is unknown` |
 | C7 | A later part throws after earlier parts published | A partial multi-part commit | Precise rollback: drop only the refs this call created, matching the exact manifest — never clobbers a concurrent writer's repoint |
 | C8 | Transaction destroyed uncommitted | Open builds | Destructor abandons every build |
 | C9 | Namespace dropped mid-build | — | One atomic flag; every further op fails closed at the alive check |
+| C10 | `promote` ref-log `PUT` committed, then the `_ckpt` frontier publish failed | A durable promotion this server has not installed; the ref lane needs recovery | The next ref-log recovery installs it. The in-memory part was already rolled back, and part cleanup later drops the ref; a restart before that drop brings the part back. The client sees `is durable but was not acknowledged` |
+| C11 | `precommitAdd` ref-log `PUT` committed, then the `_ckpt` frontier publish failed | A durable precommit binding this server has not installed | The next ref-log recovery installs the binding; the part is never published. `abandon`, else the stale-precommit sweep, removes it. The client sees the same message as in C10, although the part never appears |
+
+What each client-visible message means, and when a retry is safe, is in
+[CAS commit errors](/antalya/cas/operations/troubleshooting#cas-commit-errors).
 
 ## The repoint {#repoint}
 

docs/en/antalya/cas/operations/troubleshooting.md

@@ -18,6 +18,7 @@ tools.
 |---|---|---|
 | A server keeps losing its mount lease and self-remounting | Check `system.cas_mounts` for the server's own `state`/`expires_at`, then correlate `watermark_renew` and `mount_remount` in `system.cas_log`; losing the lease trips a local fence and latches a remount generation | Read the failed renewal's `classification` before changing anything — it alone now says why (see [the decision flow](#mount-renewal-remount-flow)). Look for object-store latency consuming the confirmed lease or BOOTTIME advancement; see [the mount lease](/antalya/cas/architecture/mounts-and-leases#mount-lease) |
 | Writes slow down or stall under load, with no exception reaching the client | S3 `SlowDown`/`ServiceUnavailable`/`RequestTimeout`/`InternalError` (5xx) responses are not on the request engine's `isDefinitelyRefusedWrite` definite-failure list (only malformed-request, entity-too-large, and access-denied that no credential refresh can fix are), so they classify as ambiguous and are retried automatically. Confirm with `sum(ProfileEvents['CASConditionalWriteUnresolved'])` rising alongside `sum(ProfileEvents['CASConditionalWriteAttempts'])` over `system.query_log` for the affected window (or `ProfileEvent_CASConditionalWriteUnresolved` in `system.metric_log` for a cumulative view across queries), and check `system.blob_storage_log` for `disk_name = '<cas>'` rows with a nonzero `error_code` around the same window | Nothing to configure per-request: the request engine retries the same `(key, bytes)` with capped-exponential backoff (200ms initial, capped at 5s, full jitter) until the 90-second operation deadline — for these conditional writes there is no separate attempts ceiling, only the deadline. A blob body upload is different: it is never resent as is after an ambiguous outcome, and each retry publishes a fresh envelope, up to 8 attempts, and no new attempt starts after the 90-second deadline. An attempt already started can run longer: each creating request waits up to the disk's `request_timeout_ms`, and multipart part uploads keep their own retries. `CASBlobPublishAttemptFailed` counts failed attempts; `CASBlobPublishGaveUp` counts uploads that failed at least once and then ended with an exception, at the attempt cap or the deadline. Meanwhile the mount-lease renewer keeps extending the fence across the disruption — this is the "blips, throttling, partial outages" case the write path is built to survive. Confirm the mount lease itself is still renewing (`system.cas_mounts.expires_at` moving forward, `last_success_age_seconds` not climbing) — if it is, this is expected and self-resolving. If `SlowDown` responses are sustained rather than transient, check the bucket's request-rate limits against the pool's actual PUT/GET rate (see [bucket requirements](/antalya/cas/bucket-requirements)) and consider lowering `cas_blob_upload_pool_size` to reduce concurrent upload traffic; a write only surfaces a client-visible `NETWORK_ERROR` if the 90-second deadline passes before the store recovers (an attempt started before the deadline can finish after it), and that error is retried by the ordinary merge/insert backoff, not silently dropped |
+| A write fails with a message containing `CAS ref-log txn`, `is durable but was not acknowledged`, `outcome is unknown` or `nothing is durable` | The phrase says whether the write can still take effect. A write blocked behind an earlier unresolved transaction (a wedged or needs-recovery lane) carries that transaction's text instead; its own write was never attempted | Read [CAS commit errors](#cas-commit-errors) before retrying: two of the three outcomes can duplicate on retry |
 | `GC` never seems to reclaim space after tables are dropped | `SELECT * FROM system.cas_gc_log WHERE event_type='Finish' ORDER BY event_time DESC LIMIT 5` — check `outcome`; also `SELECT is_leader FROM system.cas_mounts` on this node | If `outcome != 'Success'`/`'Deferred'`, see [reading GC health](/antalya/cas/operations/monitoring#gc-health); if this node is not the leader (`is_leader = 0`), it never reclaims for this disk — check the peer holding leadership. Reclamation also needs at least two full rounds past condemnation by design (the grace period is rounds, not acks) — a single manual `SYSTEM CAS GC RUN` will not finish it |
 | A dangling-access exception or `CORRUPTED_DATA` on read | Run `clickhouse-disks cas-fsck --detail` and check `dangling` specifically — it is the one class that means data loss, distinct from `unreachable`/`awaiting-gc`, which are just waiting for graduation | A nonzero `dangling` count is a real incident: collect the `--detail` output (see [what to collect before filing a bug](/antalya/cas/operations/debugging#filing-a-bug)) before taking any destructive action |
 | `SYSTEM CAS FSCK` or `clickhouse-disks cas-fsck` times out on a large pool | The scan is bounded by `--timeout` (default 600s / the `SYSTEM` form has no override); a large `roots/` prefix can make the scan slow | Retry with `--partial` to see the counts accumulated so far instead of aborting empty-handed, or `--namespace <prefix>` to scope the scan to a subset of namespaces |
@@ -28,6 +29,36 @@ tools.
 | Stale-looking part metadata after an out-of-band change to the pool | The part-folder view cache may be serving a retained (not re-validated) view | Set the disk-level `cas_part_folder_cache_bytes = 0` to disable view retention and `cas_manifest_decode_cache_bytes = 0` to make every manifest read fetch the body, then run `fsck`; both are diagnostic kill switches, not steady-state settings |
 | A wide merge (many thousands of columns) fails with a port-exhaustion error from the network layer | Each column in a wide part can cost a separate object-store operation in one merge, and a very wide part can issue on the order of the column count in requests, exhausting local ephemeral TCP ports under load | Reduce concurrent merge parallelism on that table, or increase the host's ephemeral port range; this is a general high-fan-out-merge limit, not specific to content addressing |
 
+## CAS commit errors and retries {#cas-commit-errors}
+
+A failed ref-log commit of this write names one of three outcomes in its message. The error code, usually
+`NETWORK_ERROR` (210), does not tell them apart; the phrase does. The original error is kept inside
+the message, and the server log may also show the inner `CAS write could not be committed` line.
+Each phrase describes the ref-log transaction that carried this part's change; one transaction can
+batch several callers. A write blocked behind an earlier unresolved transaction (a wedged or
+needs-recovery lane) carries that transaction's text instead, and its own write was never attempted.
+
+| Message contains | What happened to this transaction | Retrying this part's write |
+|---|---|---|
+| `nothing is durable, so a retry cannot duplicate this transaction` | The store refused the write, or it was refused before any request was sent | Cannot duplicate this part. See the deduplication note below |
+| `outcome is unknown` | A request may have been sent and nothing settled its result. The append lane stays wedged until the same key resolves | May duplicate: the write may still take effect |
+| `is durable but was not acknowledged` | It landed (named as `txn <epoch>-<seq>`), then a later step failed: the `_ckpt` frontier publish, the mount fence, or the install admission. The next ref-log recovery installs it | May duplicate: the write may still take effect |
+
+Scope: an `INSERT` that spans several partitions commits one part at a time. Earlier parts stay
+committed; a failed statement does not roll them back. None of these phrases proves that re-running
+the whole query is duplicate-free.
+
+For an `INSERT` in the last two rows, the part is rolled back in memory, so a `SELECT` right after
+the error does not show its rows. If the transaction landed and the server restarts before part
+cleanup drops the ref, the rows appear. A failure at the precommit step never publishes the part,
+but the message does not say which step failed.
+
+`non_replicated_deduplication_window` does not make these retries safe. `MergeTree` writes the
+block id to the deduplication log before the CAS commit and does not remove it when the commit
+fails. While the matching block ID remains in the deduplication log, a retry of the same block is
+reported as success and skipped as a duplicate, so its rows appear only if the original write
+takes effect.
+
 ## Mount renewal and remount decision flow {#mount-renewal-remount-flow}
 
 Start with the `watermark_renew` timeline described in

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h

@@ -121,7 +121,9 @@ inline std::optional<Etag> orThrow(WriteResult && result, std::string_view what)
                 case GaveUp::Why::Deadline:
                     throwCasWriteRetryLater(fmt::format("{}: gave up at the {} deadline after {} attempt(s)", what, g.deadline_source == GaveUp::Source::Lease ? "lease" : "policy", g.sent_any ? "one or more" : "zero"));
                 case GaveUp::Why::Unresolved:
-                    throwCasWriteRetryLater(fmt::format("{}: the write is unresolved (sent, resolve read found {})", what, renderObservation(g.last_seen)));
+                    throwCasWriteRetryLater(g.sent_any
+                        ? fmt::format("{}: the write is unresolved (sent, resolve read found {})", what, renderObservation(g.last_seen))
+                        : fmt::format("{}: no write was sent; a read of the current object did not complete ({})", what, renderObservation(g.last_seen)));
             }
             UNREACHABLE();
         }}, result);

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp

@@ -267,19 +267,13 @@ CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const N
         /// get there, so the write is resolved rather than skipped.
         return decisions > 1 ? CkptPublishOutcome::Published : CkptPublishOutcome::IdenticalSkip;
     }
-    if (const auto * gave_up = std::get_if<GaveUp>(&result))
-    {
-        /// A lost fence is an expected, transient control signal, so it is a value rather than a
-        /// throw. It says nothing about the object: the engine reports it both before an attempt is
-        /// sent and after one is proven durable, so the caller must re-read rather than assume.
-        /// Every other give-up leaves the contribution unpublished, and the caller must be told that
-        /// rather than left to assume it landed.
-        if (gave_up->why == GaveUp::Why::FenceLost)
-            return CkptPublishOutcome::FencedOut;
-        throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string()
-            + "': persistent CAS contention, the checkpoint contribution was not published");
-    }
-    /// Only `Conflict` and `Refused` are left, and both are an exception.
+    /// A lost fence is an expected, transient control signal, so it is a value rather than a throw.
+    /// It says nothing about the object: the engine reports it both before an attempt is sent and
+    /// after one is proven durable, so the caller must re-read rather than assume.
+    if (const auto * gave_up = std::get_if<GaveUp>(&result); gave_up && gave_up->why == GaveUp::Why::FenceLost)
+        return CkptPublishOutcome::FencedOut;
+    /// Every other give-up, conflict and refusal leaves the contribution unpublished, and `orThrow`
+    /// tells the caller so with the real reason.
     const String what = "CAS _ckpt for namespace '" + life.ns.string() + "'";
     orThrow(std::move(result), what);
     UNREACHABLE();