Repository navigation
fix: run full local CI lanes in parallel and speed up Python codegen - #450
Merged
Merged
Conversation
… the root The flagless scripts/ci-local.sh took about 85 minutes, every lane one after another. Measured on the same 24-core machine it now takes 10m17s, green, with the same tests: every suite reports the same counts, less the duplicate below. - Parallel lanes. The tree-writing steps (install, drift gates, one workspace build) run first; the rest run as independent lanes, at most MO_CI_JOBS (8) at once, each into its own log printed as one block when it ends. The summary lists every step's time, the slowest steps and each lane's wall time. --quick and --only stay serial and select exactly what they did. - Each test runs once. tests/integration ran inside "conformance: python" and again as "integration-tests (python)"; the java and kotlin conformance classes ran inside the reactor and again on their own; the workspace was rebuilt and reinstalled by several lanes; mvn install ran three times. - Python codegen ran ruff as `python -m ruff`, a second interpreter per call, twice per emitted file: ~90% of codegen time. It now runs ruff's binary (find_ruff_bin), same version, same output. Python conformance 16m49s -> 1m20s alone (with xdist). - The shared Java test logback config was never wired, so every module but metadata logged at DEBUG (1.86M lines a run). Surefire now points every module at it. - Runners in parallel: pytest-xdist, bun test --parallel for the TS integration suite, surefire forkCount=3 for the JVM integration modules. TS integration 15m52s -> 2m49s, Java 6m57s -> 1m55s, Kotlin 4m52s -> 2m33s, Python integration 5m51s -> 56s, each measured alone. - The Postgres sidecar is named per checkout; each scenario still gets its own database.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
scripts/ci-local.shfull (flagless) run now runs its lanes in parallel, capped byMO_CI_JOBS(default 8). Tree-writing steps (install, drift gates, workspace build) run first in a serial prelude. Output stays per-lane and ordered, and the exit code reflects every lane.--quick,--onlyandMO_CI_LIST_ONLY=1still run and list the same selections as before.MO_CI_PY_WORKERS, default 8),bun test --parallelfor the TS integration suite (MO_CI_TS_WORKERS, 6), and surefireforkCountfor JVM integration modules (MO_CI_JVM_FORKS, 3). Database containers use per-scenario databases, OS-assigned host ports and per-checkout container names. The Pythontests/integrationsuite now runs once in the full run, not twice.ruffas its own binary (viafind_ruff_bin, cached), falling back topython -m ruff. Output is unchanged.server/java/pom.xmlwires the shared test logback config, so modules stop logging at DEBUG.CHANGELOG.md,CONTRIBUTING.mdanddocs/CONFORMANCE.mdare updated to match.Before / after (flagless
scripts/ci-local.sh, same machine)Total wall time: about 85m → 10m17s (my validation run). The gate's two back-to-back runs on the final tree took 10m38s and 8m55s, both green.
"After" is from the gate's run #1. Some steps got slower in wall-clock terms because they now share the machine with seven other lanes. The total is set by the longest lane, which is mutation.
Test counts: nothing dropped or skipped
tests/integration(378 scenarios) used to run twice: once inside the conformance step and again asintegration-tests (python). It now runs once, in the integration step: 4131 − 378 + 1 new test = 3754. Under--no-integration, or when docker is down, the conformance step still runs it.mainafter the baseline log was taken. The Java and Kotlin rows compare the same tree before and after.--quick,--onlyandMO_CI_LIST_ONLY=1list the same steps as before.no-mistakes runs: 1. It had two fix rounds: review, plus test (the C# port race, fixed by binding Kestrel and the containers to OS-assigned ports with no retries, and the MySQL native-AIO limit).
Validation
Risk Assessment
✅ Low: The fix round's one change in
gate_conf_pythonmatches the proberun_integration_foruses (docker info) and restores the old behavior when docker is down; the rest of the diff was reviewed in round 1 with no new defects found.Testing
Two consecutive full flagless ci-local runs on the final tree both passed, in 10m38s and 8m55s, against roughly 85m before the change. The final tree includes the OS-assigned-port fix and the MySQL native-AIO fix. The runs are the complete regression suite: all ports, conformance, integration, cube and the mutation gate. The C# integration lane passed 357/357 each time with no port-bind errors.
address already in usein either logPipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
scripts/ci-local.sh:737- gate_conf_python now drops tests/integration whenever --no-integration is NOT set. It assumesintegration-tests (python)will run that directory instead, but that only happens when docker is up. Concrete path:scripts/ci-local.sh --only pythonwith docker down and no --strict-toolchains. The conformance step runspytest tests/ --ignore=tests/integration. run_integration_for then records only a SKIP (ci-local.sh:921-923). The docker-free half of tests/integration (the generated-lane and normalization tests) runs nowhere, and the run reports 'PASSED (with 1 skipped)'. Before this change, the same run executed tests/integration in the conformance step. postgres_container.py does not skip without docker, so the old run went red. Fix: only add --ignore=tests/integration when the integration step will actually run. Use the samedocker infocheck that run_integration_for uses, orinteg_readyin the full run. In the full run with docker down, the FAIL 'integration-tests (docker down)' already makes the run red, so only the serial --only path is affected.scripts/ci-local.sh:58- Fixed the stale help text for--only cube. It said the full run adds cube 'after the integration suite'. It now says the full run runs cube as its own parallel lane.scripts/ci-local.sh:783- Fixed the stale gate_cube comment. It said 'the full run after the integration suite'. It now says 'the full run'scubelane'.scripts/ci-local.sh:459- Fixed the gate_showcase_all_ports comment. It said the workspace is built bygate_ts_build_typecheckearlier in the same run. The full run now builds the workspace in its serial prelude (gate_ts_build), so the comment now says that.🔧 Fix applied.
✅ The docker-down Python integration finding is fixed in the pipeline (commit b732c18). The conformance step now drops
tests/integrationonly when the integration step will run it, meaning integration is enabled anddocker infosucceeds. Otherwise conformance runs it as before.3 infos still open:
scripts/ci-local.sh:58- Fixed the stale help text for--only cube. It said the full run adds cube 'after the integration suite'. It now says the full run runs cube as its own parallel lane.scripts/ci-local.sh:783- Fixed the stale gate_cube comment. It said 'the full run after the integration suite'. It now says 'the full run'scubelane'.scripts/ci-local.sh:459- Fixed the gate_showcase_all_ports comment. It said the workspace is built bygate_ts_build_typecheckearlier in the same run. The full run now builds the workspace in its serial prelude (gate_ts_build), so the comment now says that.✅ **Test** - passed after one fix round
Final state: the first full run found an intermittent C# integration failure (
address already in use). The cause was a port race: a free port was picked, released, then bound again. It is fixed at the root by binding Kestrel tohttp://127.0.0.1:0, so the OS assigns the port, andPickFreePortis deleted. The same sweep moved the other servers and containers to OS-assigned ports. There are no retries and nothing was serialised. Parallel MySQL containers also exhausted the kernelfs.aio-max-nrlimit, so all five MySQL helpers now start InnoDB with--innodb-use-native-aio=0. After those fixes, two full flagless runs passed back to back, in 10m38s and 8m55s.scripts/ci-local.sh---integration-onlynow runs through the parallel full-run scheduler, where before it ran as one serial step. This is intended.docker rm -f <name>stops it.address already in usein either logscripts/ci-local.sh(full flagless run) fix(cli): scaffold outDir defaults to src/generated, not ./src/db #1 on the final tree: exit 0, wall 638sscripts/ci-local.sh(full flagless run) FR-019: shared + @provided enums (all five ports) + typecheck-gate repair #2, back to back: exit 0, wall 535sgrep -rn PickFreePort server/: 0 hits✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.