Pin chdb-core v26.11.0-rc.2

.github/workflows/build.yml

@@ -370,7 +370,7 @@ jobs:
               ldd '"$LIBS"'/libchdb_java_jni.so
 
               echo
-              java -Xmx512m                 -Dchdb.library.path='"$LIBS"'                 -Dchdb.it.engine.version=26.7.3                 -cp "'"$PWD"'/chdb-jdbc/target/classes:'"$PWD"'/chdb-integration-tests/target/test-classes:'"$(cat target/it-cp.txt)"'"                 org.junit.platform.console.ConsoleLauncher execute                 --select-package=org.chdb.it                 --include-classname=".*IT"                 --details=summary                 --disable-banner                 --fail-if-no-tests
+              java -Xmx512m                 -Dchdb.library.path='"$LIBS"'                 -Dchdb.it.engine.version=26.11.0-rc.2                 -cp "'"$PWD"'/chdb-jdbc/target/classes:'"$PWD"'/chdb-integration-tests/target/test-classes:'"$(cat target/it-cp.txt)"'"                 org.junit.platform.console.ConsoleLauncher execute                 --select-package=org.chdb.it                 --include-classname=".*IT"                 --details=summary                 --disable-banner                 --fail-if-no-tests
             '
 
       - name: A query over the memory limit returns an exception, not an OOM kill

README.md

@@ -40,8 +40,12 @@ and is loaded from a real packaged JAR, in CI:
   engine does. Each package records the figure it was built against in its
   `manifest.properties`, and CI runs the whole test suite on AlmaLinux 8 to demonstrate the
   floor rather than infer it from symbol versions.
-- **Engine:** chDB Core **26.7.3**, pinned. The C ABI is version-locked, so the driver
+- **Engine:** chDB Core **26.11.0-rc.2**, pinned. The C ABI is version-locked, so the driver
   refuses to run against a different engine build rather than risking a struct-layout mismatch.
+  That release is a chdb-core *pre-release*, pinned to validate the ClickHouse 26.11 baseline;
+  a binding on an engine RC is itself an RC and is not the V1 GA
+  ([work plan §4.3](CHDB_JAVA_V1_WORK_PLAN.md)), so the stable chdb-core release has to be
+  pinned before `1.0.0`.
 - **Not supported:** Windows, musl (Alpine), 32-bit, GraalVM Native Image, Android.
 
 ## Installing
@@ -101,15 +105,16 @@ architecture — declare the driver plus each native package you need:
 The BOM keeps the driver and every native package on one version, which they must be.
 
 **A native package is large**, because it contains the whole engine: 112 MB for macOS arm64,
-128 MB for macOS x86_64, 130 MB for Linux aarch64, 167 MB for Linux x86_64, and around 350 MB
-unpacked. There is no all-platforms package, on purpose: it would be the sum of those four.
+144 MB for macOS x86_64, 158 MB for Linux aarch64 and 177 MB for Linux x86_64 on engine
+26.11.0-rc.2 (unchanged from rc.1), and about 350 to 545 MB unpacked. There is no
+all-platforms package, on purpose: it would be the sum of those four.
 
 ### Versioning
 
 SemVer, on the binding alone: `1.0.0` is the first release and a major bump means a breaking
 change to the Java API. The engine version is not part of it — it is in each package's
 `manifest.properties`, pinned in [`scripts/engine.properties`](scripts/engine.properties) and
-named in the release notes (`26.7.3` today), and the driver refuses to load any other build.
+named in the release notes (`26.11.0-rc.2` today), and the driver refuses to load any other build.
 
 The first test version is the RC `1.0.0-rc.1`; later candidates increment the final number, and
 the first stable Maven release is `1.0.0`. RCs use the permanent `com.clickhouse.chdb` groupId,

chdb-integration-tests/src/test/java/org/chdb/it/LicenseInventoryIT.java

@@ -139,6 +139,10 @@ void inventoryMatchesTheEngine() throws Exception {
      * <p>These are the whole of what a redistribution review has to reach a position on. LGPL
      * carries obligations under static linking that it does not under dynamic; MPL-2.0 is
      * weaker, file-level copyleft, but it is still copyleft and still has no alternative half.
+     *
+     * <p>numactl (LGPL) was in this set through engine v26.7.3 and left it with v26.11.0-rc.1,
+     * whose chdb-core tree no longer has {@code contrib/numactl} at all, so it is a component
+     * gone from the engine rather than one relicensed.
      */
     private static Map<String, Set<String>> knownCopyleftOnly() {
         Map<String, Set<String>> known = new TreeMap<>();
@@ -147,7 +151,7 @@ private static Map<String, Set<String>> knownCopyleftOnly() {
                 new TreeSet<>(
                         java.util.Arrays.asList(
                                 "lemmagen-c", "libgsasl", "libssh", "mariadb-connector-c",
-                                "numactl", "xz")));
+                                "xz")));
         known.put(
                 "MPL-2.0",
                 new TreeSet<>(

chdb-integration-tests/src/test/java/org/chdb/it/ProcessLifecycleIT.java

@@ -136,19 +136,21 @@ private static final class ForkResult {
     /**
      * A JVM that exits with a streaming result set still open.
      *
-     * <p>Without the shutdown hook this aborts inside the engine rather than exiting:
+     * <p>Without the shutdown hook this used to abort rather than exit -- on macOS arm64 as
      *
      * <pre>
      * libc++ Hardening assertion !empty() failed: front() called on an empty vector
      * </pre>
      *
-     * with SIGABRT (exit 134) and a stack naming ClickHouse internals. Narrow but easy to hit:
-     * exiting with a Connection open is fine, and closing the Connection while a stream is open
-     * is fine, because that closes the stream. Only an open stream at exit does it.
+     * with SIGABRT (exit 134) and a stack naming ClickHouse internals, on Linux x86_64 as a
+     * SIGSEGV. Narrow but easy to hit: exiting with a Connection open was fine, and closing the
+     * Connection while a stream is open is fine, because that closes the stream. Only an open
+     * stream at exit did it.
      *
-     * <p>Still true on engine 26.7.2-rc.2, the release that added {@code
-     * chdb_shutdown()}: that call declines to do anything while a connection is open, so it
-     * does not replace the hook's drain. See {@link #chdbShutdownStopsTheEngine()}.
+     * <p>Still true on engine 26.7.2-rc.2, the release that added {@code chdb_shutdown()}:
+     * that call declines to do anything while a connection is open, so it does not replace the
+     * hook's drain. See {@link #chdbShutdownStopsTheEngine()}. What finally removed the abort
+     * was a fix in the shim, not the engine; see {@link #hookCanBeDisabled()}.
      */
     public static final class ExitsWithOpenStream {
         public static void main(String[] args) throws Exception {
@@ -177,30 +179,30 @@ void exitWithOpenStreamIsClean() throws Exception {
     }
 
     @Test
-    @DisplayName("the same JVM with the hook disabled reaches the engine's own teardown path")
+    @DisplayName("the same JVM with the hook disabled reaches the engine's own teardown path, and"
+            + " exits cleanly")
     @Timeout(value = 3, unit = TimeUnit.MINUTES)
     void hookCanBeDisabled() throws Exception {
-        // Still asserting only that the switch is honoured, not that the engine aborts.
-        //
-        // chdb_shutdown() arriving in the 26.7.2-rc.2 baseline was the event this test was
-        // written to survive, and surviving it is exactly what happened: the abort is
-        // unchanged, because chdb_shutdown() refuses to act while a connection is open and
-        // this JVM's connection is open. So there is still no correct exit code to pin here,
-        // and pinning 134 would turn a real upstream fix into a CI failure. What matters is
-        // that a host managing its own teardown can turn the hook off.
-        //
-        // Issue #22 did not change that either. It removed the hook's other hazard -- see
-        // exitWithQueriesInFlightIsPrompt -- but this shape's abort is the engine's own
-        // behaviour on an exit with a stream open, which no driver change reaches. The switch
-        // is therefore still the escape hatch and still only assertable as a switch.
+        // This used to assert only that the switch is honoured, on the grounds that the abort
+        // with the hook off was the engine's own behaviour on an exit with a stream open, which
+        // no driver change could reach -- so pinning 134 would have turned an upstream fix into
+        // a CI failure. The abort was the shim's, as traced on Linux x86_64 with gdb: SIGSEGV
+        // in chdb_close_conn() called from ~HandleRegistry, on every run -- 20 of 20 on engine
+        // v26.11.0-rc.2, and the same on rc.1. The handle registry was a function-local static,
+        // and its destructor closed the leaked stream and connection from inside exit(), after
+        // statics the engine had constructed during the query were already destroyed. The
+        // registry is never destroyed now, so the leaked handles are left to the engine's own
+        // exit teardown, and this JVM exits 0, 20 of 20.
         ForkResult result =
                 fork(ExitsWithOpenStream.class.getName(), List.of("-Dchdb.shutdownHook=false"));
         assertTrue(result.output.contains("leaked an open stream"), result.output);
-        if (result.exitCode != 0) {
-            System.out.println(
-                    "with the hook off, this engine exits " + result.exitCode
-                            + " -- which is what the hook exists to prevent");
-        }
+        assertEquals(
+                0,
+                result.exitCode,
+                "with the hook off the leaked stream is left to the engine's exit teardown, which"
+                        + " exits cleanly; a crash here means handles are being closed from a"
+                        + " static destructor inside exit() again.\n"
+                        + result.output);
     }
 
     /**
@@ -452,6 +454,49 @@ void chdbShutdownStopsTheEngine() throws Exception {
                 ? "reconnecting is refused" : "reconnecting is allowed"));
     }
 
+    /**
+     * The same after a {@code MergeTree} table has been used, on the storage path given as the
+     * argument: every connection closed, then {@code chdb_shutdown()}.
+     */
+    public static final class ShutsTheEngineDownAfterMergeTree {
+        public static void main(String[] args) throws Exception {
+            try (Connection connection = DriverManager.getConnection("jdbc:chdb:" + args[0]);
+                    Statement statement = connection.createStatement()) {
+                statement.execute("CREATE TABLE t (x UInt32) ENGINE = MergeTree ORDER BY x");
+                statement.executeUpdate("INSERT INTO t SELECT number FROM numbers(1000)");
+                try (ResultSet rs = statement.executeQuery("SELECT count() FROM t")) {
+                    rs.next();
+                    System.out.println("rows=" + rs.getLong(1));
+                }
+            }
+            System.out.println("shutdown=" + ChdbNative.shutdown());
+        }
+    }
+
+    @Test
+    @DisplayName("chdb_shutdown() also succeeds after a MergeTree table was used")
+    @Timeout(value = 3, unit = TimeUnit.MINUTES)
+    void chdbShutdownSucceedsAfterMergeTree() throws Exception {
+        // Through engine v26.11.0-rc.1 this returned 1 with every connection closed: the
+        // MergeTree background executors outlived the last connection, their idle workers kept
+        // the global thread pool busy, and rc.1 said so on stderr ("chDB global pool shutdown
+        // refused ... MergeTreeBackgroundExecutorThreads"). chdb-core #312, in v26.11.0-rc.2,
+        // destroys them on shutdown. Findings section 9 has the measurement.
+        Path dir = Files.createTempDirectory("chdb shutdown mergetree");
+        ForkResult result =
+                fork(ShutsTheEngineDownAfterMergeTree.class.getName(), List.of(), dir.toString());
+        assertEquals(0, result.exitCode, result.output);
+        assertTrue(result.output.contains("rows=1000"), result.output);
+        assertTrue(
+                result.output.contains("shutdown=0"),
+                "chdb_shutdown() should stop the engine once every connection is closed, whether"
+                        + " or not a MergeTree table was used:\n"
+                        + result.output);
+        assertFalse(
+                result.output.contains("shutdown refused"),
+                "the engine should not report a refused shutdown:\n" + result.output);
+    }
+
     /** A JVM that closes everything before exiting, which must also be clean. */
     public static final class ExitsCleanly {
         public static void main(String[] args) throws Exception {

chdb-jdbc/src/main/java/org/chdb/internal/ChdbNative.java

@@ -57,7 +57,7 @@ private ChdbNative() {
     /** ABI version the loaded shim was built with. */
     public static native int jniAbiVersion();
 
-    /** {@code chdb_version()} of the loaded libchdb, e.g. {@code "26.7.3"}. */
+    /** {@code chdb_version()} of the loaded libchdb, e.g. {@code "26.11.0-rc.2"}. */
     public static native String engineVersion();
 
     /** Build provenance of the shim: git commit, compiler, engine headers it compiled against. */
@@ -142,10 +142,10 @@ public static native long query(
      * {@code chdb_classify_query_n}. Decides whether a statement has a result set to stream
      * without executing it, using the engine's own parser.
      *
-     * <p>Optional symbol: it landed in chdb-core v26.7.2-rc.2, so the pinned v26.7.3 baseline has it,
-     * so it is resolved by {@code dlsym} rather than linked. It is present on the baseline;
-     * the null path is what an older engine gets, and means the caller must fall back to
-     * {@code StatementShape}'s own analysis.
+     * <p>Optional symbol: it landed in chdb-core v26.7.2-rc.2, so the pinned v26.11.0-rc.2
+     * baseline has it, so it is resolved by {@code dlsym} rather than linked. It is present on
+     * the baseline; the null path is what an older engine gets, and means the caller must fall
+     * back to {@code StatementShape}'s own analysis.
      *
      * @return {@code {queryClass, statementCount, flags}} where {@code queryClass} follows
      *     {@code chdb_query_class} (0 READ_ONLY, 1 MUTATING, 2 MUTATING_GLOBAL, 3 CONTROL,
@@ -204,9 +204,9 @@ public static native long rowBinaryOpen(
      * every result destroyed first, and the engine cannot be used again in this process.
      *
      * <p>Optional symbol, like {@link #classifyQuery(long, byte[])}: it landed in chdb-core
-     * v26.7.2-rc.2, so the pinned v26.7.3 baseline has it, and it is resolved by {@code dlsym} rather
-     * than linked. Not having it is not a failure -- the threads it would join are reaped by
-     * process exit either way.
+     * v26.7.2-rc.2, so the pinned v26.11.0-rc.2 baseline has it, and it is resolved by {@code
+     * dlsym} rather than linked. Not having it is not a failure -- the threads it would join are
+     * reaped by process exit either way.
      *
      * <p>1 is the ordinary answer while any connection is still open: the engine declines to
      * tear itself down under a live connection rather than leaving it dangling. Callers that

chdb-jdbc/src/main/java/org/chdb/jdbc/ChdbStatement.java

@@ -365,7 +365,7 @@ private void checkDeadlineSurvivedTheOpen(String sql) throws SQLException {
     private StatementShape.Route route(String sql) throws SQLException {
         int[] analysis;
         try {
-            // Null when the engine predates chdb_classify_query_n, which the v26.7.3
+            // Null when the engine predates chdb_classify_query_n, which the v26.11.0-rc.2
             // baseline does not -- but a loader pointed at another libchdb still can, in which
             // case StatementShape falls back to its keyword scan.
             analysis = ChdbNative.classifyQuery(connection.handle(), Utf8.encode(sql));

chdb-jdbc/src/main/java/org/chdb/jdbc/ShutdownCleanup.java

@@ -13,9 +13,8 @@
 /**
  * Closes connections the application forgot, at JVM shutdown.
  *
- * <h2>What this prevents</h2>
- * A JVM that exits with a streaming result set still open aborts inside the engine rather than
- * exiting:
+ * <h2>What this prevented</h2>
+ * A JVM that exited with a streaming result set still open used to abort rather than exit:
  *
  * <pre>
  * libc++ Hardening assertion !empty() failed: front() called on an empty vector
@@ -28,6 +27,14 @@
  * dies with SIGABRT and a stack that names ClickHouse internals, which tells the application
  * nothing about the result set it leaked.
  *
+ * <p>The abort was the JNI shim's, as traced on Linux x86_64, where it is a SIGSEGV in
+ * {@code chdb_close_conn()}: the shim's handle registry was a function-local static whose
+ * destructor closed the leaked stream and connection from inside {@code exit()}, after statics
+ * the engine constructed during the query were already gone. The registry is never destroyed
+ * now, so with this hook turned off the leaked handles are left to the engine's own exit
+ * teardown and the JVM exits cleanly. The hook still closes them on the way out, because that
+ * is what lets {@link #stopEngine()} join the engine's threads.
+ *
  * <p>Closing connections in a shutdown hook covers it: {@code ChdbConnection.close()} closes
  * its statements, which close their result sets, which is the state the engine tolerates. Once
  * they are closed, {@link #stopEngine()} asks the engine to join its own threads.

chdb-jdbc/src/main/java/org/chdb/jdbc/StatementShape.java

@@ -29,10 +29,10 @@
  * <ol>
  *   <li>{@code chdb_classify_query_n}, which parses the SQL with the engine's own parser and
  *       settings and executes nothing. Available from engine v26.7.2-rc.2, so present on the
- *       pinned v26.7.3 baseline and asked in practice. Authoritative about <em>whether</em> there is a
- *       result set, but silent on <em>which door</em> delivers it: {@code
- *       CHDB_QUERY_READ_ONLY} covers a {@code SELECT} and a {@code SHOW} alike, so it cannot
- *       tell the two entry points apart. See {@link #route(int[], String)}.
+ *       pinned v26.11.0-rc.2 baseline and asked in practice. Authoritative about
+ *       <em>whether</em> there is a result set, but silent on <em>which door</em> delivers it:
+ *       {@code CHDB_QUERY_READ_ONLY} covers a {@code SELECT} and a {@code SHOW} alike, so it
+ *       cannot tell the two entry points apart. See {@link #route(int[], String)}.
  *   <li>the leading-keyword scan below, which answers the half the classifier does not, and
  *       answers both halves for an engine that does not export it. Nothing older than the
  *       baseline is supported, but the loader can be pointed at another {@code libchdb} of the

chdb-jdbc/src/test/java/org/chdb/jdbc/StatementShapeTest.java

@@ -13,7 +13,7 @@
  * an entry point that refuses it -- which is what issue #12 was -- so all three are covered in
  * both directions.
  *
- * <p>The pinned v26.7.3 baseline does export chdb_classify_query_n, so the scan is no
+ * <p>The pinned v26.11.0-rc.2 baseline does export chdb_classify_query_n, so the scan is no
  * longer the only thing deciding. It is still what decides whether a result set is streamed or
  * materialized, because the classifier does not distinguish those, as well as
  * deciding both halves whenever the classifier is absent or refuses the text -- hence the cases