Repository navigation
Pin chdb-core v26.11.0-rc.2 - #34
Merged
Merged
Conversation
Move the engine baseline from v26.7.3 to the v26.11.0-rc.1 pre-release (ClickHouse 26.11 baseline) to validate it before the stable v26.11.1. - scripts/engine.properties and <chdb.engine.version>: version, tag, asset base and the four libchdb SHA-256s, matching the GitHub release digests. chdb_version() of the release library returns 26.11.0-rc.1. - chdb-jni/include/chdb.h: copied verbatim from the release tarball (identical in all four). The ABI change is additive only: chdb_insert_arrow_array/_stream and doc updates. All four libraries export the same 57 chdb_* symbols, including every one the shim uses. - licenses/engine-third-party-26.11.0-rc.1.tsv: regenerated from system.licenses of the new engine. 1082 entries (968 before), 138 added, all permissive; 24 removed, among them numactl (LGPL), whose contrib/ is gone from chdb-core. LicenseInventoryIT's known copyleft set and the licence README written by build-native.sh drop numactl. - build.yml: the AlmaLinux 8 job's chdb.it.engine.version. - Comments and docs that state the pinned baseline or the inventory. macOS minos (11.0 arm64, 10.15 x86_64) and the Linux glibc floor (GLIBC_2.4 x86_64, GLIBC_2.17 aarch64) are unchanged, read from the assets themselves. Per work plan section 4.3 a binding on an engine RC is itself an RC and cannot be 1.0.0, so the POMs stay on 1.0.0-SNAPSHOT and the stable engine has to be re-pinned before GA.
The macos-aarch64 and linux-x86_64 runner images now ship Maven 3.10.0
(resolver 2.0.24), which records the serving repository in
_remote.repositories as <id>-<SHA-1 of the repository URL> rather than
the bare id. The provenance check only knew `chdb-verify=`, so it
rejected artifacts that did come from the throwaway repository.
Accept that form too, but only with the hash of this run's own
file://${REPO} URL, so a mirror is still refused. Reproduced and
checked locally with Maven 3.10.0 and 3.9.16; unrelated to the engine.
Read off this PR's CI "Package the platform JAR" step (du -h of each JAR) and the four release libraries: the macOS x86_64 and both Linux packages grew with the 26.11 engine, and the unpacked size quoted was already below what the Linux libraries take.
The JAR figures are du -h, in MiB; the unpacked range was in decimal MB. In du units the staged package is 346 MiB of engine on macOS arm64 and 543 MiB in all on Linux x86_64.
v26.11.0-rc.2 is v26.11.0-rc.1 plus chdb-core #312 on the same ClickHouse baseline (26.11.1.1). It is still a chdb-core pre-release, so work plan 4.3 still applies: the POMs stay on 1.0.0-SNAPSHOT and the next cut from this baseline is 1.0.0-rc.2. - engine.properties and <chdb.engine.version>: version, tag, asset base and the four libchdb SHA-256s. They are the release API digests, and the release's SHA256SUMS agrees with them. All four downloaded tarballs match, and chdb_version() of the linux-x86_64 library returns 26.11.0-rc.2, the CHDB_VERSION in all four headers. - chdb-jni/include/chdb.h: copied unchanged from the release tarball. It differs from rc.1's only in CHDB_VERSION. All four platform libraries export the same 57 chdb_* symbols as before. - licenses/engine-third-party-26.11.0-rc.2.tsv: regenerated from this engine's system.licenses. It is byte-identical to the rc.1 inventory (1082 entries), so the copyleft set is unchanged. - build.yml: the AlmaLinux 8 job's -Dchdb.it.engine.version. - Comments and docs that name the pinned baseline. Findings section 2 gains a column; the tag's in-tree constant still says 26.11.0-dev. Re-read from the new assets rather than carried over: macOS minos is 11.0 (arm64) and 10.15 (x86_64), the engine's glibc floor is GLIBC_2.4 (x86_64) and GLIBC_2.17 (aarch64), and the wheels keep their tags. run-signal-window-test.sh measure 3 reported hostHandlersAtSigDfl=0/0/0 and stress 6 survived 6 of 6 on Linux x86_64. The URL-settings check in docs/memory.md holds on this engine.
Through v26.11.0-rc.1, chdb_shutdown() returned 1 after a MergeTree table had been used, even with every connection closed, and printed "chDB global pool shutdown refused ... MergeTreeBackgroundExecutorThreads" when the driver's hook ran. chdb-core #312, in v26.11.0-rc.2, destroys the MergeTree background executors on shutdown, and chdb_shutdown() now also stops jemalloc's background threads. ProcessLifecycleIT.chdbShutdownSucceedsAfterMergeTree asserts the first fix in a forked JVM: a MergeTree table on disk, every connection closed, then chdb_shutdown() must return 0 with no refusal on stderr. It passes on rc.2 and fails on rc.1 (shutdown=1). Findings section 9 and docs/memory.md no longer describe the refusal as current. Listing the JVM's threads after a successful call on Linux x86_64, rc.1 left four unnamed engine threads running and rc.2 leaves none.
The registry was a function-local static, constructed at the first connect, so its destructor ran inside exit() after the statics the engine constructs during the first query were already destroyed. It then closed every handle still registered. A JVM that leaked a streaming ResultSet and ran with -Dchdb.shutdownHook=false died that way on every run: SIGSEGV in chdb_close_conn() called from ~HandleRegistry (gdb, Linux x86_64, engine v26.11.0-rc.1 and rc.2). A C program that does the same from an exit handler registered before the query crashes the same way, and leaving the handles open exits 0. The registry is now allocated once and never destroyed, so leaked handles are left to the engine's own exit teardown. ProcessLifecycleIT.hookCanBeDisabled used to assert only that the switch is honoured, because the abort was taken to be the engine's. It now asserts exit 0. That fails with the old shim (134) and passes with this one. 20 of 20 such exits are clean on rc.2, against 20 of 20 crashes before. The comments in ProcessLifecycleIT and ShutdownCleanup, findings section 9, docs/unsupported.md and docs/memory.md no longer say that no driver change reaches this abort. The hook stays: closing the connections is what lets chdb_shutdown() join the engine's threads.
I read them off this PR's CI "Package the platform JAR" step (du -h of each JAR, run 42065207169): 112M, 144M, 158M and 177M, the same as on rc.1. The unpacked range also still holds: 346 MiB of engine on macOS arm64 and 543 MiB staged on Linux x86_64.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Moves the engine baseline from chdb-core v26.7.3 to v26.11.0-rc.2, to validate the ClickHouse 26.11 engine before the full release. The branch first pinned v26.11.0-rc.1. v26.11.0-rc.2 is rc.1 plus chdb-core #312 on the same ClickHouse baseline (26.11.1.1), and this PR now pins it. It also fixes a JVM-exit crash in the JNI shim, found while testing #312.
This engine is a release candidate. chdb-core v26.11.0-rc.2 is a GitHub pre-release (published 2026-11-05), it is not on PyPI, and the full release will be v26.11.1. Work plan §4.3 says a binding built on an engine RC is itself an RC and cannot be V1 GA. The POMs stay on
1.0.0-SNAPSHOT, becauserc-release.ymlsets the version from the tag. Anything cut from this baseline has to be a-rc.<n>version: the next is1.0.0-rc.2, and1.0.0-rc.1stays immutable on 26.7.3. v26.11.1 has to be pinned before1.0.0.Changes
scripts/engine.propertiesand<chdb.engine.version>: version, tag, asset base and the four libchdb SHA-256s. They are the release APIdigestvalues; the release'sSHA256SUMSagrees with them, and all four downloaded tarballs match.chdb_version()of the release library returns26.11.0-rc.2, the same asCHDB_VERSIONin every tarball'schdb.h.chdb-jni/include/chdb.h: copied unchanged from the release tarball, as findings §2 requires. Against v26.7.3 the ABI change is additive only (chdb_insert_arrow_array/chdb_insert_arrow_stream, plus doc updates), and rc.2's header differs from rc.1's only inCHDB_VERSION. All four platform libraries export the same 57chdb_*symbols, including every symbol the shim uses. The in-tree header at the tag still says26.11.0-dev; findings §2 has a column per release.licenses/engine-third-party-26.11.0-rc.2.tsv: regenerated from the engine'ssystem.licenses. It is byte-identical to the rc.1 inventory: 1082 entries, against 968 on 26.7.3, with 138 added and 24 removed. Every added entry has a permissive licence. One removed entry isnumactl(LGPL), because chdb-core no longer hascontrib/numactl. That is whyLicenseInventoryIT's known copyleft set and the licence README thatbuild-native.shwrites drop it (twelve → eleven). No component joined the copyleft set.build.yml: the AlmaLinux 8 job's hard-coded-Dchdb.it.engine.version.chdb-jni/src/chdb_jni_handles.h, own commit): the handle registry is never destroyed. It was a function-local static, and its destructor ran insideexit()after the statics the engine constructs during the first query were gone. It then closed every handle still registered. A JVM that leaked a streamingResultSetand ran with-Dchdb.shutdownHook=falsedied that way on every run: SIGSEGV inchdb_close_conn()called from~HandleRegistry(gdb, Linux x86_64, rc.1 and rc.2).ProcessLifecycleIT.hookCanBeDisablednow asserts exit 0. It fails with the old shim (134) and passes with the new one.ShutdownCleanup,ProcessLifecycleIT, findings §9,unsupported.mdandmemory.md.chdb_shutdown()returned 1 afterMergeTreeuse with every connection closed. The driver's hook also printedchDB global pool shutdown refused ... MergeTreeBackgroundExecutorThreadson stderr. rc.2 returns 0, and it now also stops jemalloc's background threads. After a successful call rc.1 left four engine threads running and rc.2 leaves none.ProcessLifecycleIT.chdbShutdownSucceedsAfterMergeTree: it passes on rc.2 and fails on rc.1.memory.mdno longer describe the refusal as current.scripts/verify-consumer.sh(rc.1 round, not engine-related): Maven 3.10.0 runner images write<id>-<SHA-1 of the repository URL>to_remote.repositories. The provenance check now also accepts that key, but only with this run's own repository URL, so a mirror is still refused.v1-progress.md,upstream-findings.md,memory.md,publishing.md,release-readiness.md. Prose that records a measurement made on an older engine still names that engine. The README's native package sizes were re-read from this PR's CI packaging step on rc.2 (112M, 144M, 158M and 177M), and they are unchanged from rc.1.Re-read from the rc.2 assets rather than carried over:
LC_BUILD_VERSION.Verified locally (Linux x86_64 only, Temurin JDKs, Maven 3.9.16, rc.2 release assets)
mvn -pl chdb-jdbc -am test): 178/178 on JDK 11, 17, 21, 25 and 26.manylinux_2_28withbuild-native-in-container.sh. The linkage and glibc gates pass, with a package floor of glibc 2.25.mvn -pl chdb-integration-tests -am verify -Djvm=..., Maven on JDK 11, stdin closed as in CI): 177/177 on JDK 11, 17, 21 and 25. That is the 176 from before plus the new MergeTree shutdown test, and no crash report was written.QuickStartruns from the packaged JAR.scripts/verify-consumer.sh linux-x86_64-gnupasses with Maven 3.9.16 and with 3.10.0.oldest-supported-linuxsteps: the suite on AlmaLinux 8 with its own OpenJDK 17 passed 173/173, andrun-memory-limit-test.shreturned a catchable code 241 in both cases.sanitizerssteps: the ASan+UBSan native harness passed 107 checks with 0 failures, and the JDBC suite under UBSan passed 173/173.run-signal-window-test.sh:measure 3reportedhostHandlersAtSigDfl=0/0/0, andstress 6survived 6 of 6. I ran both twice, before and after the shim fix.latestURL, no vendored JDK headers or binaries).ResultSetleakedMergeTreeconnection open at exitNot verified locally
LicenseInventoryITon each platform's own engine, and the newhookCanBeDisabledexpectation on macOS, where the old abort was a libc++ assertion rather than a SIGSEGV. CI passed on all four platforms at 0f21523 and d142b37: 177/177 integration tests on Java 11, 17, 21 and 25 in each platform job, including both new expectations on both macOS jobs.Worth knowing
terminateon "Cannot close file". The rate is the same on rc.1 and rc.2 and with either shim. Findings §9 records it as not reachable from the driver.MergeTree-open crashes were inside libchdb's own exit handlers, with no shim frames on the stack. I saw none in 400 runs with the new shim, but that is too few to call the problem fixed.build-native.shstill gives the binding version as<engine>.<revision>(now26.11.0-rc.2.1), which is the retired versioning scheme. That predates this PR and is left as is.🤖 Generated with Claude Code
Note
Pin engine baseline to chdb-core v26.11.0-rc.2
chdb.engine.versionin pom.xml), engine.properties with new asset SHA-256 checksums, the JNI header (CHDB_VERSIONin chdb.h), and CIchdb_insert_arrow_array,chdb_insert_arrow_stream) and anchdb_arrow_insert_optionsstruct in chdb.hHandleRegistry.instancein chdb_jni_handles.h a heap-allocated, never-destroyed singleton, so handle cleanup no longer races with engine static teardownChdbNative.shutdown()succeeding after MergeTree use, and tightens the shutdown-hook-disabled test to require a zero exit codechdb_shutdowndocs now require C/C++/Go hosts to call shutdown before runtime teardownMacroscope summarized d142b37.