Repository navigation
CAS: packed parts - #2579
CAS: packed parts#2579
Conversation
This reverts commit 133b29d.
…eature/cas-packed-parts
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com> # Conflicts: # programs/local/LocalServer.cpp # programs/server/Server.cpp # src/Common/ErrorCodes.cpp # src/Common/FailPoint.cpp # src/Core/ServerSettings.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/AlterDropPartitionExecutor.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/AlterDropPartitionExecutor.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
…acked-parts Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com> # Conflicts: # .github/workflows/master.yml # .github/workflows/pull_request.yml
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
|
@k-morozov does it supersede #2540 ? or this is a standalone work? maybe I exclude that PR from the scope? |
Yes, this is standalone work and will be merged after #2540. |
PR #2579 CI VerificationVerdictNo failure is attributable to this PR. Every one of the PR's own tests passes. CI does not block approval of #2579, but it does not clear the stack either: six of The bigger problem is not a red check. The ten
How the list was reducedEight jobs are red. 227 individual rows. 1. Setup errors, 225 rows, one cause. Shards 1/8 (91 errors) and 6/8 (134 errors) are That image is the old-version node used by backward-compatibility fixtures, and it is no 2. Three real integration failures, two distinct assertions. After the setup errors, 3. Four regression-suite jobs. Findings
|
| Feature | On 26.8 |
Everything else |
|---|---|---|
/oauth/access control |
64 fails / 64 runs (100 %) | 0 / 473 |
/oauth/groups |
64 fails / 64 runs (100 %) | 0 / 473 |
It fails on the branch itself (REFs/antalya-26.8/ab9c4003d72ad373f88f9695b) and on
unrelated PRs #2545 and
#2588. antalya-26.8 broke it; #2579
inherits it. Someone needs to own it, but not in this PR.
/swarms/…/wrong cluster name and /swarms/…/cpu overload all swarm nodes — regression, owner antalya-26.8
Same shape. Both are 109 fails / 109 runs (100 %) on 26.8, and 0 / 584 everywhere
else, across 10 distinct commits. The error is
Code: 279. DB::Exception: Cannot connect to any replica for query execution.
(ALL_CONNECTION_TRIES_FAILED)
(query: SELECT hostName(), count() FROM s3Cluster('swarm', 'http://minio:9000/warehouse/…'))
Nothing in this PR touches cluster discovery, s3Cluster, or swarm node startup.
test_storage_gcp_auth::test_gcp_auth_ordinary_contract and test_cas_gcs::test_ordinary_goog4_traffic_keeps_upstream_semantics — regression, owner the antalya-26.8 CAS port
These two are the ones worth reading carefully, because the naive rate comparison points
straight at this PR and is wrong.
| Test | Branch runs (pr = 0) |
Branch fails | #2579 runs | #2579 fails |
|---|---|---|---|---|
test_gcp_auth_ordinary_contract |
86 | 4 | 7 | 7 |
test_ordinary_goog4_traffic_keeps_upstream_semantics |
85 | 0 | 7 | 7 |
Never-fails-on-branch against 7-of-7-here is the textbook PR-caused pattern. It does not
hold here, for two reasons.
The baseline is the wrong branch. Every pull_request_number = 0 run of these two tests
is on antalya-26.6 or a v26.6.* tag. There is no antalya-26.8 branch run for either
test, because the CAS code that ships them has not landed on antalya-26.8 — it lives on
the stacked branch this PR sits on. The baseline measures a branch that cannot run the code
under test.
A sibling PR fails identically.
#2647 (CAS: ref-log owner Move for part renames, feature/antalya-26.8/cas-disk-transactions → antalya-26.8) fails both tests on
2026-10-23, and fails the same 26 test_database_backup rows. #2647 shares nothing with
#2579 except the base.
The mechanism has no path from this diff. Both tests fail on the same assertion — an
If-Match header present on an ordinary, non-CAS object-store request:
assert 'if-match' not in headers
E AssertionError: {'headers': {..., 'authorization': 'GOOG4-HMAC-SHA256 …'}, 'key': 'ordinary/ordinary.txt'}
Whatever adds that header lives in S3ObjectStorage or under src/IO/S3/. This PR's own
commits touch neither; its only I/O changes are ReadBufferFromFileView and
PackedFilesReader, and neither sets HTTP headers.
/lightweight delete/immediate removal/…/delete most of the table — pre-existing-flaky
One scenario in each CAS lane: VersionedCollapsingMergeTree in cas_lightweight_delete_3,
MergeTree in cas_s3_cache_lightweight_delete_3. The failing line is a wall-clock budget,
not a correctness check — the data assertion one line above it passed:
r = node.query(f"SELECT count(*) FROM {table_name} FORMAT TabSeparated")
assert r.output == output, error()
assert execution_time < 20, error() # todo rewrite valueThe scenario ran for about 40 s against 100 partitions × 100 000 rows.
| Scenario | Lane | Runs | Fails | Rate |
|---|---|---|---|---|
MergeTree/delete most of the table |
cas_s3_cache_lightweight_delete_3 |
179 | 6 | 3.4 % |
VersionedCollapsingMergeTree/delete most of the table |
cas_s3_cache_lightweight_delete_3 |
179 | 5 | 2.8 % |
VersionedCollapsingMergeTree/delete most of the table |
cas_lightweight_delete_3 |
181 | 3 | 1.7 % |
MergeTree/delete most of the table |
cas_lightweight_delete_3 |
181 | 2 | 1.1 % |
| every engine | suite (non-CAS) |
275 | 1–5 each | 0.4–1.8 % |
It fails on the plain non-CAS lane too. Two lanes each tripping one of seven engines is what
a ~2 % per-scenario rate predicts. A deterministic code defect would hit the same engine in
both lanes; these hit different ones.
/swarms/…/object storage cluster profile events with overloaded node — pre-existing-flaky
6.6 % on 26.8 (7 of 106) against 5.4 % elsewhere (31 of 575), over 13 distinct commits. The
rates are the same on both sides.
What passed, and what was never run
The feature's own tests all pass, on both the debug and the ASan/UBSan stateless lanes:
| Test | Result |
|---|---|
05217_cas_packed_parts |
OK |
05218_cas_packed_fetch_partition |
OK |
05219_cas_packed_backup_restore |
OK |
05220_cas_unique_key_mutation_keeps_full |
OK |
05301_cas_mutation_preserves_source_storage |
OK |
05385_cas_packed_merge_disabled_destination |
OK |
test_cas_s3/test.py::test_cas_s3_packed_parts |
OK |
Unit tests (asan_ubsan) (23 631 rows, incl. the new CAS and file-view gtests) |
success |
The full CAS regression suite ran and was green apart from the two flaky lightweight-delete
scenarios above: cas, cas_s3_cache, cas_aggregate_functions_*, cas_alter_*,
cas_atomic_insert, cas_engines, and the rest.
Two gaps, and they are the reason not to read the green as full coverage.
The CAS stateless lanes were skipped. All ten are SKIPPED on this PR — amd_binary, cas storage, parallel; amd_binary, cas s3 storage, parallel; arm_binary, cas s3 storage, parallel; and the amd_asan_ubsan, amd_tsan, amd_msan batches. The database has zero
rows for any of them on any commit of #2579, while other antalya-26.8 CAS PRs
(#2644,
#2623,
#2614) run them with tens of thousands of
rows each. This PR sets cas_allow_packed_parts = 1 in
tests/config/config.d/cas_storage_policy_for_merge_tree_by_default.xml and
cas_s3_storage_policy_for_merge_tree_by_default.xml — making the whole stateless suite use
packed MergeTree parts on CAS — and that configuration was never executed.
The PR removes the CAS stress lanes. Commit 762c37ae ("Revert port of #2292") deletes
Stress test (amd_debug | amd_asan_ubsan | amd_tsan | amd_msan, cas s3 storage) from
ci/defs/job_configs.py, along with the USE_CAS_S3_STORAGE_FOR_MERGE_TREE plumbing in
ci/jobs/stress_job.py and its unit tests. Those four lanes exist on the base branch and run
on #2644, #2623 and #2614. The PR description does not mention the removal.
The CAS regression lanes do not cover packed parts. cas_allow_packed_parts appears
nowhere in the clickhouse-regression repository, so every green Regression release cas_*
job ran with packed parts off. Those jobs are still meaningful — they exercise the refactored
*InTransactionOrDisk read helpers on Full parts over CAS, which is a live behaviour change
in this PR — but they say nothing about the packed path.
Recommendations
- Un-skip the CAS stateless lanes and re-run, or state explicitly why packed-on-CAS
across the whole stateless suite does not need to run before merge. This is the one item
that would change the verification result. - Explain or revert the deletion of the four CAS stress lanes, which is unrelated to
packed parts and silently reduces coverage on the base branch. - Resolve the merge conflict with
antalya-26.8and add the DCO sign-off. - Rerun integration shards 1/8 and 6/8 once the
clickhouse-server:23.3.19.33.altinitystable
image is back, or pin the compatibility fixtures to an image that still exists. This is a
branch-wide problem, not a CAS: packed parts #2579 problem. - The
/oauth,/swarmsand GCSIf-Matchbreaks need owners onantalya-26.8and its CAS
port. They are 100 % reproducible there and will block every PR on that branch until fixed.
PR #2579 Audit ReviewVerdictNo High or Medium distributed-systems defect in reviewed scope. Four Low findings, all in the resource and configuration/observability classes. The change that One coverage gap worth closing before merge: nothing exercises two servers with two separate CAS What the change doesA CAS disk previously forced every new MergeTree part to 1. Opt-in capability. New setting 2. The gate moved from the policy to the destination disk. 3. Transaction-aware reads, shared between the two part storages. The read-your-writes 4. Reading the archive through a buffer instead of a disk path. 5. Borrowed transactions. A packed projection sub-part that rides the parent's transaction 6. 7. Model
Why replicas cannot diverge, despite choosing the format independently
A part's On the fetch path the two formats are handled separately and deliberately The remaining question is ordering inside a single build: a borrowed packed projection must get FindingsBackpressure & resource limitsLow: a packed part on CAS is built entirely in memory and published by one commit, and mutating
Configuration & version skewLow: the capability gates creation only, so a move imports packed parts onto a disk configured
Observability & diagnosabilityLow: an in-flight file with no blob yet is reported as a file with no remote paths.
Low:
Coverage summary
Deferred: none. Assumptions and limits: static review of the PR's own commits at head, with no build and no Coverage gap to close before mergeNo test exercises two servers with two separate CAS pools exchanging a packed part. A two-node integration test — insert on node 1, let node 2 fetch, |
filimonov
left a comment
There was a problem hiding this comment.
LGTM in general. There are few small details to fix (in the follow-up PRs)
- DataPartStorageOnDiskPacked::resetReader reads the archive index with a 4 KiB buffer instead of the caller's full-size prefetching one.
- allowsPackedParts forwarded by MetadataStorageFromCacheObjectStorage and MetadataStorageWithPathWrapper.
The merge-base changed after approval.
…tions #2579 landed 267672c (fail point in the Packed commit) and 7dd8e6f (`clickhouse-disks` exit codes by the `cas-` name prefix) after this branch had 504b5c88257 and f084f04084e for the same two problems. Both conflicts resolve to this branch's version: the Packed commit keeps the pause fail points next to the failing one, and the exit code follows `ICommand::exits_nonzero_on_error`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added basic support for Packed parts on CAS disks.
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: