Skip to content

Pin chdb-core v26.11.0-rc.2 - #34

Merged
wudidapaopao merged 8 commits into
mainfrom
bump-engine-v26.11.0-rc.1
Nov 5, 2026
Merged

wudidapaopao merged 8 commits into
mainfrom
bump-engine-v26.11.0-rc.1

Conversation

@wudidapaopao

@wudidapaopao wudidapaopao commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What

Moves the engine baseline from chdb-core v26.7.3 to v26.11.0-rc.2, to validate the ClickHouse 26.11 engine before the full release. The branch first pinned v26.11.0-rc.1. v26.11.0-rc.2 is rc.1 plus chdb-core #312 on the same ClickHouse baseline (26.11.1.1), and this PR now pins it. It also fixes a JVM-exit crash in the JNI shim, found while testing #312.

This engine is a release candidate. chdb-core v26.11.0-rc.2 is a GitHub pre-release (published 2026-11-05), it is not on PyPI, and the full release will be v26.11.1. Work plan §4.3 says a binding built on an engine RC is itself an RC and cannot be V1 GA. The POMs stay on 1.0.0-SNAPSHOT, because rc-release.yml sets the version from the tag. Anything cut from this baseline has to be a -rc.<n> version: the next is 1.0.0-rc.2, and 1.0.0-rc.1 stays immutable on 26.7.3. v26.11.1 has to be pinned before 1.0.0.

Changes

  • scripts/engine.properties and <chdb.engine.version>: version, tag, asset base and the four libchdb SHA-256s. They are the release API digest values; the release's SHA256SUMS agrees with them, and all four downloaded tarballs match. chdb_version() of the release library returns 26.11.0-rc.2, the same as CHDB_VERSION in every tarball's chdb.h.
  • chdb-jni/include/chdb.h: copied unchanged from the release tarball, as findings §2 requires. Against v26.7.3 the ABI change is additive only (chdb_insert_arrow_array / chdb_insert_arrow_stream, plus doc updates), and rc.2's header differs from rc.1's only in CHDB_VERSION. All four platform libraries export the same 57 chdb_* symbols, including every symbol the shim uses. The in-tree header at the tag still says 26.11.0-dev; findings §2 has a column per release.
  • licenses/engine-third-party-26.11.0-rc.2.tsv: regenerated from the engine's system.licenses. It is byte-identical to the rc.1 inventory: 1082 entries, against 968 on 26.7.3, with 138 added and 24 removed. Every added entry has a permissive licence. One removed entry is numactl (LGPL), because chdb-core no longer has contrib/numactl. That is why LicenseInventoryIT's known copyleft set and the licence README that build-native.sh writes drop it (twelve → eleven). No component joined the copyleft set.
  • build.yml: the AlmaLinux 8 job's hard-coded -Dchdb.it.engine.version.
  • JNI shim fix (chdb-jni/src/chdb_jni_handles.h, own commit): the handle registry is never destroyed. It was a function-local static, and its destructor ran inside exit() after the statics the engine constructs during the first query were gone. It then closed every handle still registered. A JVM that leaked a streaming ResultSet and ran with -Dchdb.shutdownHook=false died that way on every run: SIGSEGV in chdb_close_conn() called from ~HandleRegistry (gdb, Linux x86_64, rc.1 and rc.2).
    • Leaked handles are now left to the engine's own exit teardown.
    • ProcessLifecycleIT.hookCanBeDisabled now asserts exit 0. It fails with the old shim (134) and passes with the new one.
    • The texts that said no driver change could reach this abort are corrected: ShutdownCleanup, ProcessLifecycleIT, findings §9, unsupported.md and memory.md.
  • What rc.2 fixed, own commit: on rc.1 (and 26.7.3), chdb_shutdown() returned 1 after MergeTree use with every connection closed. The driver's hook also printed chDB global pool shutdown refused ... MergeTreeBackgroundExecutorThreads on stderr. rc.2 returns 0, and it now also stops jemalloc's background threads. After a successful call rc.1 left four engine threads running and rc.2 leaves none.
    • New test ProcessLifecycleIT.chdbShutdownSucceedsAfterMergeTree: it passes on rc.2 and fails on rc.1.
    • Findings §9 and memory.md no longer describe the refusal as current.
  • scripts/verify-consumer.sh (rc.1 round, not engine-related): Maven 3.10.0 runner images write <id>-<SHA-1 of the repository URL> to _remote.repositories. The provenance check now also accepts that key, but only with this run's own repository URL, so a mirror is still refused.
  • Comments and docs that state the pinned baseline or the inventory facts: README, v1-progress.md, upstream-findings.md, memory.md, publishing.md, release-readiness.md. Prose that records a measurement made on an older engine still names that engine. The README's native package sizes were re-read from this PR's CI packaging step on rc.2 (112M, 144M, 158M and 177M), and they are unchanged from rc.1.

Re-read from the rc.2 assets rather than carried over:

  • macOS minos is 11.0 (arm64) and 10.15 (x86_64) in LC_BUILD_VERSION.
  • The engine's glibc floor is GLIBC_2.4 (x86_64) and GLIBC_2.17 (aarch64).
  • The Linux wheels are still tagged manylinux_2_17.

Verified locally (Linux x86_64 only, Temurin JDKs, Maven 3.9.16, rc.2 release assets)

  • Unit tests (mvn -pl chdb-jdbc -am test): 178/178 on JDK 11, 17, 21, 25 and 26.
  • Shim built in manylinux_2_28 with build-native-in-container.sh. The linkage and glibc gates pass, with a package floor of glibc 2.25.
  • Integration tests (mvn -pl chdb-integration-tests -am verify -Djvm=..., Maven on JDK 11, stdin closed as in CI): 177/177 on JDK 11, 17, 21 and 25. That is the 176 from before plus the new MergeTree shutdown test, and no crash report was written.
  • Packaging and loading:
    • The platform JAR layout check passes (manifest, sha256sums, licences, SBOM).
    • QuickStart runs from the packaged JAR.
    • scripts/verify-consumer.sh linux-x86_64-gnu passes with Maven 3.9.16 and with 3.10.0.
  • The oldest-supported-linux steps: the suite on AlmaLinux 8 with its own OpenJDK 17 passed 173/173, and run-memory-limit-test.sh returned a catchable code 241 in both cases.
  • The sanitizers steps: the ASan+UBSan native harness passed 107 checks with 0 failures, and the JDBC suite under UBSan passed 173/173.
  • run-signal-window-test.sh: measure 3 reported hostHandlersAtSigDfl=0/0/0, and stress 6 survived 6 of 6. I ran both twice, before and after the shim fix.
  • Hygiene checks (pom/engine.properties agreement, JNI ABI agreement, no latest URL, no vendored JDK headers or binaries).
  • Exit paths in forked JVMs, JDK 21, with this branch's shim. The counts are runs that exited non-zero or wrote a crash report:
Shape Hook on Hook off
Streaming ResultSet leaked 0 / 10 0 / 20 (old shim: 20 / 20)
Five engine restarts, then a leaked stream — 0 / 20
Five engine restarts, then an open connection — 0 / 10
MergeTree connection open at exit 0 / 10 0 / 400 (old shim: 2 / 291)
Everything closed 0 / 10 0 / 10
Queries still running at exit 0 / 60 1 / 10

Not verified locally

  • linux-aarch64, macos-aarch64 and macos-x86_64 (no such hosts here). CI's four-platform matrix covers them, and §4.3 requires that run for an engine change. That includes LicenseInventoryIT on each platform's own engine, and the new hookCanBeDisabled expectation on macOS, where the old abort was a libc++ assertion rather than a SIGSEGV. CI passed on all four platforms at 0f21523 and d142b37: 177/177 integration tests on Java 11, 17, 21 and 25 in each platform job, including both new expectations on both macOS jobs.

Worth knowing

  • Exiting while a thread is still running a query is still racy inside the engine. A few percent of such exits crash on the way out: SIGSEGV in the fetching thread, or a terminate on "Cannot close file". The rate is the same on rc.1 and rc.2 and with either shim. Findings §9 records it as not reachable from the driver.
  • The old shim's 2 of 291 MergeTree-open crashes were inside libchdb's own exit handlers, with no shim frames on the stack. I saw none in 400 runs with the new shim, but that is too few to call the problem fixed.
  • The SBOM written by build-native.sh still gives the binding version as <engine>.<revision> (now 26.11.0-rc.2.1), which is the retired versioning scheme. That predates this PR and is left as is.

🤖 Generated with Claude Code

Note

Pin engine baseline to chdb-core v26.11.0-rc.2

  • Updates the pinned engine version from v26.7.3 to v26.11.0-rc.2 across Maven (chdb.engine.version in pom.xml), engine.properties with new asset SHA-256 checksums, the JNI header (CHDB_VERSION in chdb.h), and CI
  • Adds new Arrow insert C APIs (chdb_insert_arrow_array, chdb_insert_arrow_stream) and an chdb_arrow_insert_options struct in chdb.h
  • Fixes an exit-time crash by making HandleRegistry.instance in chdb_jni_handles.h a heap-allocated, never-destroyed singleton, so handle cleanup no longer races with engine static teardown
  • Updates license inventory: 1082 components, eleven copyleft (numactl removed); refreshed TSV inventories, docs, build notice in build-native.sh, and the copyleft allowlist in LicenseInventoryIT.java
  • Adds integration tests for ChdbNative.shutdown() succeeding after MergeTree use, and tightens the shutdown-hook-disabled test to require a zero exit code
  • Behavioral Change: the pinned engine is a pre-release; leaked handles at exit are now left to engine exit teardown instead of being closed by registry destruction, and chdb_shutdown docs now require C/C++/Go hosts to call shutdown before runtime teardown

Macroscope summarized d142b37.

Move the engine baseline from v26.7.3 to the v26.11.0-rc.1 pre-release
(ClickHouse 26.11 baseline) to validate it before the stable v26.11.1.

- scripts/engine.properties and <chdb.engine.version>: version, tag,
  asset base and the four libchdb SHA-256s, matching the GitHub release
  digests. chdb_version() of the release library returns 26.11.0-rc.1.
- chdb-jni/include/chdb.h: copied verbatim from the release tarball
  (identical in all four). The ABI change is additive only:
  chdb_insert_arrow_array/_stream and doc updates. All four libraries
  export the same 57 chdb_* symbols, including every one the shim uses.
- licenses/engine-third-party-26.11.0-rc.1.tsv: regenerated from
  system.licenses of the new engine. 1082 entries (968 before), 138
  added, all permissive; 24 removed, among them numactl (LGPL), whose
  contrib/ is gone from chdb-core. LicenseInventoryIT's known copyleft
  set and the licence README written by build-native.sh drop numactl.
- build.yml: the AlmaLinux 8 job's chdb.it.engine.version.
- Comments and docs that state the pinned baseline or the inventory.

macOS minos (11.0 arm64, 10.15 x86_64) and the Linux glibc floor
(GLIBC_2.4 x86_64, GLIBC_2.17 aarch64) are unchanged, read from the
assets themselves.

Per work plan section 4.3 a binding on an engine RC is itself an RC and
cannot be 1.0.0, so the POMs stay on 1.0.0-SNAPSHOT and the stable
engine has to be re-pinned before GA.
The macos-aarch64 and linux-x86_64 runner images now ship Maven 3.10.0
(resolver 2.0.24), which records the serving repository in
_remote.repositories as <id>-<SHA-1 of the repository URL> rather than
the bare id. The provenance check only knew `chdb-verify=`, so it
rejected artifacts that did come from the throwaway repository.

Accept that form too, but only with the hash of this run's own
file://${REPO} URL, so a mirror is still refused. Reproduced and
checked locally with Maven 3.10.0 and 3.9.16; unrelated to the engine.
Read off this PR's CI "Package the platform JAR" step (du -h of each
JAR) and the four release libraries: the macOS x86_64 and both Linux
packages grew with the 26.11 engine, and the unpacked size quoted was
already below what the Linux libraries take.
The JAR figures are du -h, in MiB; the unpacked range was in decimal MB. In du units the staged package is 346 MiB of engine on macOS arm64 and 543 MiB in all on Linux x86_64.
v26.11.0-rc.2 is v26.11.0-rc.1 plus chdb-core #312 on the same
ClickHouse baseline (26.11.1.1). It is still a chdb-core pre-release,
so work plan 4.3 still applies: the POMs stay on 1.0.0-SNAPSHOT and the
next cut from this baseline is 1.0.0-rc.2.

- engine.properties and <chdb.engine.version>: version, tag, asset base
  and the four libchdb SHA-256s. They are the release API digests, and
  the release's SHA256SUMS agrees with them. All four downloaded
  tarballs match, and chdb_version() of the linux-x86_64 library
  returns 26.11.0-rc.2, the CHDB_VERSION in all four headers.
- chdb-jni/include/chdb.h: copied unchanged from the release tarball.
  It differs from rc.1's only in CHDB_VERSION. All four platform
  libraries export the same 57 chdb_* symbols as before.
- licenses/engine-third-party-26.11.0-rc.2.tsv: regenerated from this
  engine's system.licenses. It is byte-identical to the rc.1 inventory
  (1082 entries), so the copyleft set is unchanged.
- build.yml: the AlmaLinux 8 job's -Dchdb.it.engine.version.
- Comments and docs that name the pinned baseline. Findings section 2
  gains a column; the tag's in-tree constant still says 26.11.0-dev.

Re-read from the new assets rather than carried over: macOS minos is
11.0 (arm64) and 10.15 (x86_64), the engine's glibc floor is GLIBC_2.4
(x86_64) and GLIBC_2.17 (aarch64), and the wheels keep their tags.
run-signal-window-test.sh measure 3 reported hostHandlersAtSigDfl=0/0/0
and stress 6 survived 6 of 6 on Linux x86_64. The URL-settings check
in docs/memory.md holds on this engine.
Through v26.11.0-rc.1, chdb_shutdown() returned 1 after a MergeTree
table had been used, even with every connection closed, and printed
"chDB global pool shutdown refused ... MergeTreeBackgroundExecutorThreads"
when the driver's hook ran. chdb-core #312, in v26.11.0-rc.2, destroys
the MergeTree background executors on shutdown, and chdb_shutdown() now
also stops jemalloc's background threads.

ProcessLifecycleIT.chdbShutdownSucceedsAfterMergeTree asserts the first
fix in a forked JVM: a MergeTree table on disk, every connection
closed, then chdb_shutdown() must return 0 with no refusal on stderr.
It passes on rc.2 and fails on rc.1 (shutdown=1).

Findings section 9 and docs/memory.md no longer describe the refusal
as current. Listing the JVM's threads after a successful call on Linux
x86_64, rc.1 left four unnamed engine threads running and rc.2 leaves
none.
The registry was a function-local static, constructed at the first
connect, so its destructor ran inside exit() after the statics the
engine constructs during the first query were already destroyed. It
then closed every handle still registered. A JVM that leaked a
streaming ResultSet and ran with -Dchdb.shutdownHook=false died that
way on every run: SIGSEGV in chdb_close_conn() called from
~HandleRegistry (gdb, Linux x86_64, engine v26.11.0-rc.1 and rc.2).
A C program that does the same from an exit handler registered before
the query crashes the same way, and leaving the handles open exits 0.

The registry is now allocated once and never destroyed, so leaked
handles are left to the engine's own exit teardown.
ProcessLifecycleIT.hookCanBeDisabled used to assert only that the
switch is honoured, because the abort was taken to be the engine's.
It now asserts exit 0. That fails with the old shim (134) and passes
with this one. 20 of 20 such exits are clean on rc.2, against 20 of 20
crashes before.

The comments in ProcessLifecycleIT and ShutdownCleanup, findings
section 9, docs/unsupported.md and docs/memory.md no longer say that
no driver change reaches this abort. The hook stays: closing the
connections is what lets chdb_shutdown() join the engine's threads.
@wudidapaopao wudidapaopao changed the title Pin chdb-core v26.11.0-rc.1 Pin chdb-core v26.11.0-rc.2 Nov 5, 2026
I read them off this PR's CI "Package the platform JAR" step (du -h of
each JAR, run 42065207169): 112M, 144M, 158M and 177M, the same as on
rc.1. The unpacked range also still holds: 346 MiB of engine on macOS
arm64 and 543 MiB staged on Linux x86_64.
@wudidapaopao
wudidapaopao merged commit 50fdcdd into main Nov 5, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant