[jasperblues]

CI took 23 to 32 minutes on every pull request: test (Oracle) ran pytest -k oracle in one job, 23 to 27 minutes, and each of test (3.11) to test (3.14) ran the whole suite again in 10 to 17. This splits each of them into parallel jobs, each with its own containers, and moves the slow tests, and those that rest on a timing or a plan, to a nightly set. It is the pytest version of liberation-data/drivine#679.

How it's split

pytest --shard i/n (tests/shards.py, a plugin of the tests' own conftest) runs part i of n of what -k and -m picked. A module goes whole into one part, with its module-scoped fixtures, in collection order. The parts are balanced by how long each module took in CI, as tests/durations.json records it (oracle and default). Each job uploads --record-durations output as an artifact (3.11 and Oracle), to refresh that file when the parts drift. I picked this over pytest-split because pytest-split splits test by test, which would run each module fixture in every part and break up modules that share state, such as the write-loop timings' TIMES.

Job	What it runs
test (<python>, 1/3) … 3/3	-m "not nightly" --shard i/3: on a pull request on 3.11 and 3.14 only, on a full run on all four. Part 1 also runs ruff, pyright and unasync.py --check.
test (3.12, unit), test (3.13, unit)	on a pull request only: tests/unit, with no containers (-o drivine_test_containers=false), plus the same lint steps
test (<python>, nightly)	-m nightly, on all four, on a full run only
test (Oracle, 1/4) … 4/4	ORACLE_TESTS=true -k oracle -m "not nightly" --shard i/4, on 3.14
test (Oracle, nightly)	-k oracle -m nightly, on a full run only
test (Oracle)	keeps the old check name: passes only when every Oracle part passed
record the tree, cypher-spec	unchanged

A pull request runs the database suite on the oldest and newest Python, 3.11 and 3.14. On 3.12 and 3.13 it runs tests/unit alone (tests/sync and tests/docs need a database). The nightly, a push to main, full-ci and the release gate run the database suite on all four, as before. A pull request's run is 14 jobs, down from 19.

main has no branch protection or rulesets, and nothing in the org reads test (3.x) by name (drivine-release's preflight reads the run's conclusion), so only test (Oracle) gets an aggregator.

What a pull request leaves out (marker nightly)

Each one is in tests/integration, and its sync copy in tests/sync_integration:

• Timings: sql/test_compiled_write_loop_time.py (ten times the rows; on Postgres it also vacuums and analyzes the tables, the big-row and shm case), postgres/test_compiled_write_keyed_read_time.py (keyed reads, its own graph, VACUUM of 20,000 rows), oracle/test_oracle_era_parse.py.
• Oracle's plans: oracle/test_oracle_keyed_label_plan.py, oracle/test_oracle_node_equality.py, oracle/test_oracle_adaptive_plans.py.
• Texts past 32 KB: oracle/test_oracle_long_text_memory.py; in test_cypher_compiled.py, the five Oracle tests whose names end "leaves the session open" (TS's OracleLongParameter and OracleLongProperty); test_object_manager_saving.py::TestLongText (TS's StatelessGraphObjectManager.longText).

Everything else stays on pull requests, the Cypher spec on every store among it. The full set runs on a push to main, in nightly.yml (17:00 UTC), in release.yml's gate (full: true), and on a pull request labelled full-ci.

Four Oracle tests that never ran

test (Oracle) picks tests with -k oracle, so an Oracle-only test without "oracle" in its name was skipped in the matrix and deselected in the Oracle job. That happened to four tests in test_cypher_compiled.py, in async and sync, eight in all. They are renamed with _on_oracle (one is now ..._for_two_rows_leaves_oracles_session_open, to fit the line), and they pass: the first in the PR set, the other three in the nightly set.

Release and trust

• A pull request's run of the PR set records ci-tree-pr-set-<tree>, and scripts/ci_trust.py doesn't trust it. It ignores it straight away instead of waiting for it. There's a new test in tests/unit/test_ci_trust.py.
• A full run still records ci-tree-<tree>. Checked: the full-ci run recorded ci-tree-e0ca6fd… and the PR-set run ci-tree-pr-set-e0ca6fd….
• Prepare release opens its PR with the full-ci label.

Nightly

nightly.yml calls ci.yml with full: true, with top-level permissions: contents: read. It takes over ci.yml's latest-images schedule (04:17 before, now 17:00). The FalkorDB image test is now github.workflow == 'Nightly'. When the nightly fails, its report job (issues: write) opens an issue labelled nightly, or comments on the one already open, with:

• the failed jobs;
• pytest's FAILED and ERROR lines for each;
• the commits since the last green nightly.

A green run closes the issue. On a branch other than main it only writes to the step summary.

#680: the write-loop check (one test-side change)

liberation-data/drivine#680 found an untimed check in the write-loop timing that took 655 s on Postgres. drivine-python has the same check. On main it took only seconds, among the other modules. In the nightly part, run alone, [postgres-CREATE] (which times every write) took 505 s and 498 s (async and sync), and the 3.11 nightly part took 18m28s. The MERGE check MATCH (:CyLoop)-[r:CY_SELF]->(:CyLoop) RETURN count(r) joins the node table twice, and the suite leaves that table's statistics saying it is empty (#430). It now reads MATCH (n:CyLoop)-[r:CY_SELF]->(n) RETURN count(r), which joins the table once and also checks that each relationship is a self-loop. The CREATE case dropped to 4 s, and the nightly part to 1m55s. The compiled read itself is still slow in that state, and #680 tracks it. drivine4j has the same check: liberation-data/drivine4j#673.

Measured

After the change to 3.12 and 3.13: PR run 42456968539 took 7m50s wall clock, and its slowest job test (3.11, 2/3) took 6m23s. The unit jobs took 1m22s and 1m31s. The release dry run's full gate (42456968569) took 13m19s and passed on all four Pythons. The table below is from before that change, when every Python ran the database suite on a pull request.

	Before	After (PR set)
Wall clock	23m28s (PR 42443378332), 31m37s (push 42439433895)	10m37s (42446350484, the only run that didn't wait in a queue); then 16m06s, 22m41s and 31m36s (attempts 2 to 4 of 42448953585)
Slowest job	test (Oracle) 23m25s / 26m37s	6m21s to 6m39s on the balanced parts (8m51s before they were balanced)

The wall clock went over 12 minutes on the reruns because jobs waited for runners, not because they ran longer. The longest wait for a job in each attempt was 1.7, 11.5, 25.1, 10.2 and 16.6 minutes. Runs in liberation-data/drivine4j were using the org's runners at the same time (three to five of them, at each check). A PR-set run is 19 jobs. With free runners it takes about the slowest job plus a minute: about 8 minutes.

The full set (label full-ci, run 42450537394) took 15m19s, and its slowest job took 7m24s. Release dry-run gates with the full set: 42446350553 (before the #680 fix, 36m20s, the 3.11 nightly part at 18m28s) and 42448953588 (after, 23m04s). Both passed.

Test counts

After the change to 3.12 and 3.13, from pytest's summary lines (passed + skipped + xfailed):

	PR set (42456968539)	full set (gate of 42456968539's dry run, 42456968569)
3.11	8,315 (1,739 + 3,359 + 3,217)	8,567 (with nightly 252)
3.12	2,672 (tests/unit)	8,567
3.13	2,672 (tests/unit)	8,567
3.14	8,315	8,567
Oracle	2,526 (519 + 531 + 931 + 545)	2,626 (with nightly 100)

The full set on each Python is main's 8,557 plus the 10 new tests, and Oracle is main's 2,618 plus the 8 renamed tests. tests/unit is 2,662 on main, plus the same 10.

Before that change:

From pytest's summary lines (passed + skipped + xfailed) in the full-ci run, and pytest --collect-only -q for each job's arguments, run in CI on 3.11 and 3.14 (a temporary workflow, now deleted), against main's collection:

	main	PR set	nightly	full
Each Python, without Oracle	8,557	8,315	252	8,567
Oracle job (-k oracle)	2,618	2,526	100	2,626

• Without Oracle, +10: the new tests, 9 in test_shards.py and 1 in test_ci_trust.py.
• Oracle, +8: the four renamed tests, async and sync, which -k oracle now reaches.
• No test is in two parts of the PR set, and none is in both the PR set and the nightly set. Every test on main's lists is in one of them, with the renames mapped.

The nightly, shown working

A workflow can be dispatched only once it is on main. So a temporary branch, now deleted, ran nightly.yml on push, with a unit test made to fail (run 42447210074). Its report job wrote, for example:

### [ci / test (3.12, 3/3)](…/job/128628553435)
FAILED tests/unit/test_nightly_demo_failure.py::test_fails_on_purpose_to_show_the_nightly_report - AssertionError: …

The demo file also failed ruff in part 1, and for that job the report said "No failing test in its log: it failed in a step around the tests". It ended with "No nightly has passed on main yet". Off main it writes only to the summary, so no issue was opened. The labels full-ci and nightly now exist.

Review: 3 passes, the last CLEAN. Pass 1 capped the nightly issue at 60,000 characters (GitHub refuses longer) and added tests for the shard hooks; pass 2 was CLEAN; pass 3 made a scheduled nightly report a cancelled or timed-out job (a hanging test) instead of exiting silently, and stripped the pr-set- prefix from a ci_trust message.

[jasperblues]

Merged after a rerun: the first run's Oracle 4/4 lost its connection (DPY-4011) on the spec query "collect skips nulls"; the rerun passed 517/517. That matches the Oracle session-poisoning family of drivine4j#670, which Python's own compiler still has until it moves to the shared compiler. If the nightly shows it again, the culprit test gets held for the switch.