diff --git a/docs/en/antalya/cas/architecture/part-lifecycle.md b/docs/en/antalya/cas/architecture/part-lifecycle.md index 8d2e7175e5518..02eb552f3411a 100644 --- a/docs/en/antalya/cas/architecture/part-lifecycle.md +++ b/docs/en/antalya/cas/architecture/part-lifecycle.md @@ -1,5 +1,5 @@ --- -description: 'The part-add protocol from local build through blob upload to promote, its nine crash points and their cleaners, and how each MergeTree operation maps onto it.' +description: 'The part-add protocol from local build through blob upload to promote, its crash points and their cleaners, and how each MergeTree operation maps onto it.' sidebar_label: 'Part lifecycle' sidebar_position: 6 slug: /antalya/cas/architecture/part-lifecycle @@ -104,8 +104,9 @@ manifest never loses an owner, so it is net zero. ## Crash points and their cleaners {#crash-points} -This table is the single best summary of the design's crash-safety story: every row leaks -something recoverable; no row loses data or leaves a dangling reference. +This table is the single best summary of the design's crash-safety story: no row loses committed +data or leaves a dangling reference. In C6 and C10 the part itself may take effect; C3 and C11 +leave only a precommit binding. | # | Crash window | Left behind | Who cleans it | |---|---|---|---| @@ -114,10 +115,15 @@ something recoverable; no row loses data or leaves a dangling reference. | C3 | `precommitAdd` returned Unresolved | A possibly-live precommit binding | Intent recorded pre-append; `abandon` appends the exact removal, tolerating absence. The body is never writer-deleted | | C4 | Between `precommitAdd` and `promote` | A live precommit plus uploaded blobs | No resume path exists. Removed by `abandon`, else by a fenced successor's stale-precommit sweep | | C5 | Mid blob fan-out | Already-uploaded blobs | Nothing merged; blobs become `GC`-reclaimable debris; the part is not published | -| C6 | `promote` append Unresolved | The ref may or may not be committed | Commit state Uncertain — the relink layer maps this to "retry the whole fetch", never to a byte fetch | +| C6 | `promote` append Unresolved | The ref may or may not be committed | Commit state Uncertain — the relink layer maps this to "retry the whole fetch", never to a byte fetch. The client sees `outcome is unknown` | | C7 | A later part throws after earlier parts published | A partial multi-part commit | Precise rollback: drop only the refs this call created, matching the exact manifest — never clobbers a concurrent writer's repoint | | C8 | Transaction destroyed uncommitted | Open builds | Destructor abandons every build | | C9 | Namespace dropped mid-build | — | One atomic flag; every further op fails closed at the alive check | +| C10 | `promote` ref-log `PUT` committed, then the `_ckpt` frontier publish failed | A durable promotion this server has not installed; the ref lane needs recovery | The next ref-log recovery installs it. The in-memory part was already rolled back, and part cleanup later drops the ref; a restart before that drop brings the part back. The client sees `is durable but was not acknowledged` | +| C11 | `precommitAdd` ref-log `PUT` committed, then the `_ckpt` frontier publish failed | A durable precommit binding this server has not installed | The next ref-log recovery installs the binding; the part is never published. `abandon`, else the stale-precommit sweep, removes it. The client sees the same message as in C10, although the part never appears | + +What each client-visible message means, and when a retry is safe, is in +[CAS commit errors](/antalya/cas/operations/troubleshooting#cas-commit-errors). ## The repoint {#repoint} diff --git a/docs/en/antalya/cas/operations/troubleshooting.md b/docs/en/antalya/cas/operations/troubleshooting.md index ef63acf17a337..19f2c2d150d24 100644 --- a/docs/en/antalya/cas/operations/troubleshooting.md +++ b/docs/en/antalya/cas/operations/troubleshooting.md @@ -18,6 +18,7 @@ tools. |---|---|---| | A server keeps losing its mount lease and self-remounting | Check `system.cas_mounts` for the server's own `state`/`expires_at`, then correlate `watermark_renew` and `mount_remount` in `system.cas_log`; losing the lease trips a local fence and latches a remount generation | Read the failed renewal's `classification` before changing anything — it alone now says why (see [the decision flow](#mount-renewal-remount-flow)). Look for object-store latency consuming the confirmed lease or BOOTTIME advancement; see [the mount lease](/antalya/cas/architecture/mounts-and-leases#mount-lease) | | Writes slow down or stall under load, with no exception reaching the client | S3 `SlowDown`/`ServiceUnavailable`/`RequestTimeout`/`InternalError` (5xx) responses are not on the request engine's `isDefinitelyRefusedWrite` definite-failure list (only malformed-request, entity-too-large, and access-denied that no credential refresh can fix are), so they classify as ambiguous and are retried automatically. Confirm with `sum(ProfileEvents['CASConditionalWriteUnresolved'])` rising alongside `sum(ProfileEvents['CASConditionalWriteAttempts'])` over `system.query_log` for the affected window (or `ProfileEvent_CASConditionalWriteUnresolved` in `system.metric_log` for a cumulative view across queries), and check `system.blob_storage_log` for `disk_name = ''` rows with a nonzero `error_code` around the same window | Nothing to configure per-request: the request engine retries the same `(key, bytes)` with capped-exponential backoff (200ms initial, capped at 5s, full jitter) until the 90-second operation deadline — for these conditional writes there is no separate attempts ceiling, only the deadline. A blob body upload is different: it is never resent as is after an ambiguous outcome, and each retry publishes a fresh envelope, up to 8 attempts, and no new attempt starts after the 90-second deadline. An attempt already started can run longer: each creating request waits up to the disk's `request_timeout_ms`, and multipart part uploads keep their own retries. `CASBlobPublishAttemptFailed` counts failed attempts; `CASBlobPublishGaveUp` counts uploads that failed at least once and then ended with an exception, at the attempt cap or the deadline. Meanwhile the mount-lease renewer keeps extending the fence across the disruption — this is the "blips, throttling, partial outages" case the write path is built to survive. Confirm the mount lease itself is still renewing (`system.cas_mounts.expires_at` moving forward, `last_success_age_seconds` not climbing) — if it is, this is expected and self-resolving. If `SlowDown` responses are sustained rather than transient, check the bucket's request-rate limits against the pool's actual PUT/GET rate (see [bucket requirements](/antalya/cas/bucket-requirements)) and consider lowering `cas_blob_upload_pool_size` to reduce concurrent upload traffic; a write only surfaces a client-visible `NETWORK_ERROR` if the 90-second deadline passes before the store recovers (an attempt started before the deadline can finish after it), and that error is retried by the ordinary merge/insert backoff, not silently dropped | +| A write fails with a message containing `CAS ref-log txn`, `is durable but was not acknowledged`, `outcome is unknown` or `nothing is durable` | The phrase says whether the write can still take effect. A write blocked behind an earlier unresolved transaction (a wedged or needs-recovery lane) carries that transaction's text instead; its own write was never attempted | Read [CAS commit errors](#cas-commit-errors) before retrying: two of the three outcomes can duplicate on retry | | `GC` never seems to reclaim space after tables are dropped | `SELECT * FROM system.cas_gc_log WHERE event_type='Finish' ORDER BY event_time DESC LIMIT 5` — check `outcome`; also `SELECT is_leader FROM system.cas_mounts` on this node | If `outcome != 'Success'`/`'Deferred'`, see [reading GC health](/antalya/cas/operations/monitoring#gc-health); if this node is not the leader (`is_leader = 0`), it never reclaims for this disk — check the peer holding leadership. Reclamation also needs at least two full rounds past condemnation by design (the grace period is rounds, not acks) — a single manual `SYSTEM CAS GC RUN` will not finish it | | A dangling-access exception or `CORRUPTED_DATA` on read | Run `clickhouse-disks cas-fsck --detail` and check `dangling` specifically — it is the one class that means data loss, distinct from `unreachable`/`awaiting-gc`, which are just waiting for graduation | A nonzero `dangling` count is a real incident: collect the `--detail` output (see [what to collect before filing a bug](/antalya/cas/operations/debugging#filing-a-bug)) before taking any destructive action | | `SYSTEM CAS FSCK` or `clickhouse-disks cas-fsck` times out on a large pool | The scan is bounded by `--timeout` (default 600s / the `SYSTEM` form has no override); a large `roots/` prefix can make the scan slow | Retry with `--partial` to see the counts accumulated so far instead of aborting empty-handed, or `--namespace ` to scope the scan to a subset of namespaces | @@ -28,6 +29,36 @@ tools. | Stale-looking part metadata after an out-of-band change to the pool | The part-folder view cache may be serving a retained (not re-validated) view | Set the disk-level `cas_part_folder_cache_bytes = 0` to disable view retention and `cas_manifest_decode_cache_bytes = 0` to make every manifest read fetch the body, then run `fsck`; both are diagnostic kill switches, not steady-state settings | | A wide merge (many thousands of columns) fails with a port-exhaustion error from the network layer | Each column in a wide part can cost a separate object-store operation in one merge, and a very wide part can issue on the order of the column count in requests, exhausting local ephemeral TCP ports under load | Reduce concurrent merge parallelism on that table, or increase the host's ephemeral port range; this is a general high-fan-out-merge limit, not specific to content addressing | +## CAS commit errors and retries {#cas-commit-errors} + +A failed ref-log commit of this write names one of three outcomes in its message. The error code, usually +`NETWORK_ERROR` (210), does not tell them apart; the phrase does. The original error is kept inside +the message, and the server log may also show the inner `CAS write could not be committed` line. +Each phrase describes the ref-log transaction that carried this part's change; one transaction can +batch several callers. A write blocked behind an earlier unresolved transaction (a wedged or +needs-recovery lane) carries that transaction's text instead, and its own write was never attempted. + +| Message contains | What happened to this transaction | Retrying this part's write | +|---|---|---| +| `nothing is durable, so a retry cannot duplicate this transaction` | The store refused the write, or it was refused before any request was sent | Cannot duplicate this part. See the deduplication note below | +| `outcome is unknown` | A request may have been sent and nothing settled its result. The append lane stays wedged until the same key resolves | May duplicate: the write may still take effect | +| `is durable but was not acknowledged` | It landed (named as `txn -`), then a later step failed: the `_ckpt` frontier publish, the mount fence, or the install admission. The next ref-log recovery installs it | May duplicate: the write may still take effect | + +Scope: an `INSERT` that spans several partitions commits one part at a time. Earlier parts stay +committed; a failed statement does not roll them back. None of these phrases proves that re-running +the whole query is duplicate-free. + +For an `INSERT` in the last two rows, the part is rolled back in memory, so a `SELECT` right after +the error does not show its rows. If the transaction landed and the server restarts before part +cleanup drops the ref, the rows appear. A failure at the precommit step never publishes the part, +but the message does not say which step failed. + +`non_replicated_deduplication_window` does not make these retries safe. `MergeTree` writes the +block id to the deduplication log before the CAS commit and does not remove it when the commit +fails. While the matching block ID remains in the deduplication log, a retry of the same block is +reported as success and skipped as a duplicate, so its rows appear only if the original write +takes effect. + ## Mount renewal and remount decision flow {#mount-renewal-remount-flow} Start with the `watermark_renew` timeline described in diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h index 72679c54b64a0..722c6c2aa7449 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h @@ -121,7 +121,9 @@ inline std::optional orThrow(WriteResult && result, std::string_view what) case GaveUp::Why::Deadline: throwCasWriteRetryLater(fmt::format("{}: gave up at the {} deadline after {} attempt(s)", what, g.deadline_source == GaveUp::Source::Lease ? "lease" : "policy", g.sent_any ? "one or more" : "zero")); case GaveUp::Why::Unresolved: - throwCasWriteRetryLater(fmt::format("{}: the write is unresolved (sent, resolve read found {})", what, renderObservation(g.last_seen))); + throwCasWriteRetryLater(g.sent_any + ? fmt::format("{}: the write is unresolved (sent, resolve read found {})", what, renderObservation(g.last_seen)) + : fmt::format("{}: no write was sent; a read of the current object did not complete ({})", what, renderObservation(g.last_seen))); } UNREACHABLE(); }}, result); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp index d7a30304eb688..0ad8dee9b5d7d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp @@ -267,19 +267,13 @@ CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const N /// get there, so the write is resolved rather than skipped. return decisions > 1 ? CkptPublishOutcome::Published : CkptPublishOutcome::IdenticalSkip; } - if (const auto * gave_up = std::get_if(&result)) - { - /// A lost fence is an expected, transient control signal, so it is a value rather than a - /// throw. It says nothing about the object: the engine reports it both before an attempt is - /// sent and after one is proven durable, so the caller must re-read rather than assume. - /// Every other give-up leaves the contribution unpublished, and the caller must be told that - /// rather than left to assume it landed. - if (gave_up->why == GaveUp::Why::FenceLost) - return CkptPublishOutcome::FencedOut; - throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string() - + "': persistent CAS contention, the checkpoint contribution was not published"); - } - /// Only `Conflict` and `Refused` are left, and both are an exception. + /// A lost fence is an expected, transient control signal, so it is a value rather than a throw. + /// It says nothing about the object: the engine reports it both before an attempt is sent and + /// after one is proven durable, so the caller must re-read rather than assume. + if (const auto * gave_up = std::get_if(&result); gave_up && gave_up->why == GaveUp::Why::FenceLost) + return CkptPublishOutcome::FencedOut; + /// Every other give-up, conflict and refusal leaves the contribution unpublished, and `orThrow` + /// tells the caller so with the real reason. const String what = "CAS _ckpt for namespace '" + life.ns.string() + "'"; orThrow(std::move(result), what); UNREACHABLE(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp index 196984a9aa68c..6740346b7e354 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp @@ -3725,6 +3725,23 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt } const RefAppendAttempt & active_attempt = *rt->append_attempt; + /// Callers must learn whether a retry can duplicate this transaction, so the two outcomes that + /// leave that open say so, and keep the original code and cause. + const auto durable_unconfirmed = [&](int code, const String & cause) + { + return std::make_exception_ptr(Exception(code, + "CAS ref-log txn {}-{} for namespace '{}' is durable but was not acknowledged ({}); the next " + "ref-log recovery installs it, so the write may still take effect and a retry may duplicate it", + id.writer_epoch, id.ref_sequence, ns.string(), cause)); + }; + const auto outcome_unknown = [&](int code, const String & cause) + { + return std::make_exception_ptr(Exception(code, + "CAS ref-log txn {}-{} for namespace '{}' outcome is unknown ({}): it may or may not take " + "effect; a retry may duplicate it", + id.writer_epoch, id.ref_sequence, ns.string(), cause)); + }; + std::optional written; try { @@ -3736,14 +3753,13 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// local failure raised at a point where an attempt may already have been sent. That is /// ambiguous, and it therefore transfers ownership from `Writing` to `Wedged`; the exact /// attempt remains installed. - const std::exception_ptr write_error = std::current_exception(); { std::lock_guard lock(rt->state_mutex); if (rt->lane_state == RefLaneState::Writing && rt->append_attempt && rt->append_attempt->txn_id == id) rt->lane_state = RefLaneState::Wedged; } - complete_error(chunk_survivors, write_error); + complete_error(chunk_survivors, outcome_unknown(getCurrentExceptionCode(), getCurrentExceptionMessage(false, false, false, false))); return false; } @@ -3782,11 +3798,10 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt rt->lane_state = RefLaneState::Wedged; } ProfileEvents::increment(ProfileEvents::CASRefAppendWedged); - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}': the store refused txn {}-{}'s create and the " - "settling read named no occupant (observed {}) — the append lane is wedged until the " - "SAME key resolves durable or a conclusive rejection is observed", - ns.string(), id.writer_epoch, id.ref_sequence, detail::renderObservation(conflict->seen)))); + complete_error(chunk_survivors, outcome_unknown(ErrorCodes::NETWORK_ERROR, fmt::format( + "the store refused the create and the settling read named no occupant (observed {}); " + "the append lane is wedged until the same key resolves", + detail::renderObservation(conflict->seen)))); return false; } @@ -3899,7 +3914,8 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt ProfileEvents::increment(ProfileEvents::CASRefAppendDefiniteFailure); complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( "CAS ref-log append for namespace '{}' definitively failed ({}); " - "cached state is unchanged and txn id {}-{} was never used (a retry re-derives it)", + "cached state is unchanged and txn id {}-{} was never used (a retry re-derives it); " + "nothing is durable, so a retry cannot duplicate this transaction", ns.string(), refused->message, id.writer_epoch, id.ref_sequence))); return false; } @@ -3950,7 +3966,8 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( "CAS ref-log append for namespace '{}' txn {}-{} was refused BEFORE any request was " "sent — the append lane is NOT wedged (nothing can be durable, so there is " - "nothing to resolve) and the txn id is not consumed (a retry re-derives it)", + "nothing to resolve) and the txn id is not consumed (a retry re-derives it); " + "nothing is durable, so a retry cannot duplicate this transaction", ns.string(), id.writer_epoch, id.ref_sequence))); return false; } @@ -3961,11 +3978,9 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt rt->lane_state = RefLaneState::Wedged; } ProfileEvents::increment(ProfileEvents::CASRefAppendWedged); - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}' txn {}-{} is UNCERTAIN (last observed {}) — " - "the append lane is wedged until the SAME key resolves durable or a conclusive rejection " - "is observed; this outcome is unproven, not failure", - ns.string(), id.writer_epoch, id.ref_sequence, detail::renderObservation(gave_up->last_seen)))); + complete_error(chunk_survivors, outcome_unknown(ErrorCodes::NETWORK_ERROR, fmt::format( + "gave up after a request was sent, last observed {}; the append lane is wedged until the " + "same key resolves", detail::renderObservation(gave_up->last_seen)))); return false; } @@ -3985,12 +4000,12 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt } catch (...) { - const std::exception_ptr frontier_error = std::current_exception(); { std::lock_guard lock(rt->state_mutex); requireRecovery(*rt, ns, "committed-frontier publication"); } - complete_error(chunk_survivors, frontier_error); + complete_error(chunk_survivors, durable_unconfirmed(getCurrentExceptionCode(), + "publishing its checkpoint frontier was not confirmed: " + getCurrentExceptionMessage(false, false, false, false))); return false; } if (frontier_outcome == CkptPublishOutcome::FencedOut) @@ -3999,10 +4014,8 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt std::lock_guard lock(rt->state_mutex); requireRecovery(*rt, ns, "committed-frontier publication fence"); } - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}': txn {}-{} is durable, but the mount fence " - "moved before its checkpoint frontier was published; the lane needs recovery", - ns.string(), id.writer_epoch, id.ref_sequence))); + complete_error(chunk_survivors, durable_unconfirmed(ErrorCodes::NETWORK_ERROR, + "the mount fence moved before the frontier was published")); return false; } @@ -4022,8 +4035,10 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt } catch (...) { - install_admission_error = std::current_exception(); requireRecovery(*rt, ns, "post-frontier install admission"); + install_admission_error = durable_unconfirmed(getCurrentExceptionCode(), + "its install was not admitted after the frontier was published: " + + getCurrentExceptionMessage(false, false, false, false)); } /// Only this leader mutates `rt->state`, so the candidate's base snapshot is still the /// current one: there is one append-lane leader per table at a time (the `leader_active` diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index ddbf9f33c26f1..f68c0f219b301 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -771,8 +771,22 @@ TEST(CASRefCheckpoint, AnExhaustedDeadlineUnderPersistentConflictThrowsRetryLate }; const RefCkpt advance{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = std::nullopt}; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishCkpt(op, layout, life, advance); }); + String message; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + { + try + { + publishCkpt(op, layout, life, advance); + } + catch (const DB::Exception & e) + { + message = e.message(); + throw; + } + }); backend->after_read = nullptr; + EXPECT_NE(message.find("policy deadline"), String::npos) << message; + EXPECT_EQ(message.find("contention"), String::npos) << message; EXPECT_FALSE(clock.sleeps.empty()) << "the loop must back off between attempts, not spin"; EXPECT_EQ(readCkptOrFail(setup, layout, life), base) << "no partial state: every attempt either " "committed the complete merged body or wrote nothing"; diff --git a/src/Disks/tests/gtest_cas_ref_install_safety.cpp b/src/Disks/tests/gtest_cas_ref_install_safety.cpp index d3da467cb415e..e1e55f5892c9c 100644 --- a/src/Disks/tests/gtest_cas_ref_install_safety.cpp +++ b/src/Disks/tests/gtest_cas_ref_install_safety.cpp @@ -306,8 +306,10 @@ TEST(CASRefInstallSafety, UnresolvedAlwaysRecordsTheWedge) /// backend's `putIfAbsent`, so the request reached it), the single-attempt budget is then spent, and /// the lane wedges. The message must say so -- and must NOT say "no attempt was sent", which is the /// only shape allowed to skip the wedge. - EXPECT_NE(message.find("is UNCERTAIN"), String::npos) + EXPECT_NE(message.find("outcome is unknown"), String::npos) << "the reason must reach the wedge message rather than defaulting: " << message; + EXPECT_NE(message.find("a retry may duplicate it"), String::npos) << message; + EXPECT_EQ(message.find("nothing is durable"), String::npos) << message; EXPECT_EQ(message.find("BEFORE any request was sent"), String::npos) << "an ambiguous PUT is not a pre-attempt refusal: " << message; } @@ -350,6 +352,7 @@ TEST(CASRefInstallSafety, PreAttemptRefusalDoesNotWedgeTheLane) EXPECT_NE(message.find("refused BEFORE any request was sent"), String::npos) << "the caller must be told WHY, and this is the reason the no-wedge decision rests on: " << message; + EXPECT_NE(message.find("nothing is durable, so a retry cannot duplicate this transaction"), String::npos) << message; EXPECT_FALSE(store->refLaneWedgedForTest(ns)) << "nothing was sent, so nothing can be durable: there is no ambiguity for a wedge to resolve"; EXPECT_TRUE(store->wedgedKeyForTest(ns).empty()); diff --git a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp index 9fcd010643a95..106608834a21e 100644 --- a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp +++ b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp @@ -490,7 +490,7 @@ void wedgeLaneOverADurableObject(VirtualRetryClock & clock, WedgeTestBackend & b backend.landed_ack_lost_latched = true; backend.fail_get_substr = logPrefix(store, ns); backend.fail_get_latched = true; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, ref); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "outcome is unknown", [&] { store->dropRef(ns, ref); }); backend.landed_ack_lost_latched = false; backend.landed_ack_lost_substr.clear(); backend.fail_get_latched = false; @@ -512,7 +512,7 @@ void wedgeLaneOnUnresolvedAppend(VirtualRetryClock & clock, WedgeTestBackend & b backend.ambiguous_substr = logPrefix(store, ns); backend.ambiguous_latched = true; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, drive); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "outcome is unknown", drive); backend.ambiguous_latched = false; backend.ambiguous_substr.clear(); @@ -1401,7 +1401,8 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthName /// settling read proves absence and the conflict comes back naming no occupant at all. backend->refuse_precondition_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), genesis); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishEmptyPart(store, ns, "x"); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "outcome is unknown", + [&] { publishEmptyPart(store, ns, "x"); }); const auto ckpt_after = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); @@ -1467,7 +1468,8 @@ TEST(CASRefWedgeEveryAttempt, AppendSiteWedgesWhenTheSettlingReadNamesNoOccupant const uint64_t deferred_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(); const uint64_t wedged_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "outcome is unknown", + [&] { store->dropRef(ns, "x"); }); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(), deferred_before + 1) << "the deferral is the one quiet arm here -- it must be counted or a starved loud path is invisible"; @@ -1513,7 +1515,8 @@ TEST(CASRefWedgeEveryAttempt, AppendSiteWedgesWhenTheSettlingReadItselfIsRefused backend->refuse_read_after_precondition = true; const uint64_t deferred_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "outcome is unknown", + [&] { store->dropRef(ns, "x"); }); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(), deferred_before + 1); EXPECT_TRUE(store->mayMutate()); @@ -1661,7 +1664,8 @@ TEST(CASRefLane, RefusedReturnsTheAttemptToReadyAndDoesNotWedge) const uint64_t wedged_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(); const uint64_t definite_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendDefiniteFailure].load(); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, + "nothing is durable, so a retry cannot duplicate this transaction", [&] { store->dropRef(ns, "x"); }); EXPECT_FALSE(store->refLaneWedgedForTest(ns)) << "a proven refusal wrote nothing, so there is nothing for a wedge to resolve"; diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index bb34f6b198448..5921c1a632d03 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -1558,7 +1558,8 @@ TEST(CASRefWriterAppendLane, CheckpointConflictAfterLogCommitRequiresRecoveryWit auto clock = VirtualRetryClock::installOn(store); backend->ckpt_conflict_key = ckpt_key; backend->ckpt_conflict_latched = true; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, + "is durable but was not acknowledged", [&] { store->dropRef(ns, "x"); }); backend->disarmFaults(); EXPECT_GT(clock->pauseCount(), 1u) @@ -1593,7 +1594,9 @@ TEST(CASRefWriterAppendLane, FenceMovementAtCheckpointPublicationRequiresRecover backend->ckpt_get_hook_key = ckpt_key; backend->ckpt_get_hook = [&] { store->tripMountLost(); }; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + SCOPE_EXIT({ backend->ckpt_get_hook = nullptr; }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, + "is durable but was not acknowledged", [&] { store->dropRef(ns, "x"); }); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_before) diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index 31383f78c117b..5faedf5b81611 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -223,6 +223,13 @@ TEST(CASWriteResult, OrThrowMapsEveryAlternative) .sent_any = false, .last_seen = NotObserved{}}; for (const GaveUp & gave_up : {deadline, unresolved, fence_lost}) expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{gave_up}, "t"); }); + + GaveUp unsent_unresolved = unresolved; + unsent_unresolved.sent_any = false; + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "no write was sent", + [&] { orThrow(WriteResult{unsent_unresolved}, "t"); }); + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "the write is unresolved (sent", + [&] { orThrow(WriteResult{unresolved}, "t"); }); } TEST(CASFence, OpenFenceAdmitsEverythingAndNeverMoves)