From 0e6bf2f7a370f7e0f4250ee1b5d571725d787cbb Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 23 Oct 2026 00:40:23 -0400 Subject: [PATCH 1/5] ci: run the full local CI as parallel lanes and fix the slow steps at the root The flagless scripts/ci-local.sh took about 85 minutes, every lane one after another. Measured on the same 24-core machine it now takes 10m17s, green, with the same tests: every suite reports the same counts, less the duplicate below. - Parallel lanes. The tree-writing steps (install, drift gates, one workspace build) run first; the rest run as independent lanes, at most MO_CI_JOBS (8) at once, each into its own log printed as one block when it ends. The summary lists every step's time, the slowest steps and each lane's wall time. --quick and --only stay serial and select exactly what they did. - Each test runs once. tests/integration ran inside "conformance: python" and again as "integration-tests (python)"; the java and kotlin conformance classes ran inside the reactor and again on their own; the workspace was rebuilt and reinstalled by several lanes; mvn install ran three times. - Python codegen ran ruff as `python -m ruff`, a second interpreter per call, twice per emitted file: ~90% of codegen time. It now runs ruff's binary (find_ruff_bin), same version, same output. Python conformance 16m49s -> 1m20s alone (with xdist). - The shared Java test logback config was never wired, so every module but metadata logged at DEBUG (1.86M lines a run). Surefire now points every module at it. - Runners in parallel: pytest-xdist, bun test --parallel for the TS integration suite, surefire forkCount=3 for the JVM integration modules. TS integration 15m52s -> 2m49s, Java 6m57s -> 1m55s, Kotlin 4m52s -> 2m33s, Python integration 5m51s -> 56s, each measured alone. - The Postgres sidecar is named per checkout; each scenario still gets its own database. --- CHANGELOG.md | 10 + CONTRIBUTING.md | 8 + scripts/ci-local.sh | 675 +++++++++++++----- scripts/integration-test.sh | 59 +- server/java/pom.xml | 16 +- server/python/pyproject.toml | 5 +- .../python/src/metaobjects/codegen/format.py | 20 +- server/python/tests/codegen/test_format.py | 16 + server/python/uv.lock | 26 + 9 files changed, 655 insertions(+), 180 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a8d3dab7..49302bb73 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,16 @@ here.** ## [Unreleased] +### Changed + +- **Python: `metaobjects gen` and `verify --codegen` run ruff as its own binary.** Every emitted + file is import-sorted and formatted by ruff twice, and each call used to go through + `python -m ruff`, which starts a second interpreter only to exec the binary the ruff package + installed. That start was about 90% of codegen time (59 ms against 6 ms a call, measured). The + formatter now resolves that binary once with ruff's own `find_ruff_bin`, so the ruff version + and the generated output are unchanged; it falls back to `python -m ruff` if the binary cannot + be found. + ## [1.1.0] — 2026-10-22 _npm `1.1.0` (all 14 `@metaobjectsdev/*` packages in lockstep), PyPI `1.1.0`, NuGet `1.1.0` and diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5e8888de5..9b7cd3e79 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,6 +91,14 @@ scripts/ci-local.sh --quick # hygiene.yml in full + the TypeScript half of MO_CI_LIST_ONLY=1 scripts/ci-local.sh --quick # print the steps, run nothing ``` +The full run does its tree-writing steps first (install, the drift gates, one workspace +build), then runs the rest as parallel lanes: at most `MO_CI_JOBS` at once, default 8. Each +lane's output prints as one block when the lane finishes, so lanes never interleave. The +summary gives every step's time, the slowest steps and each lane's wall time. `--quick` and +`--only` run serially. `MO_CI_PY_WORKERS` (default 8), `MO_CI_TS_WORKERS` (default 6) and +`MO_CI_JVM_FORKS` (default 3) set the worker counts for pytest-xdist, the TypeScript +integration suite and the JVM integration modules' surefire forks. + It SKIPS (loudly, not silently) any port whose toolchain (bun/dotnet/uv/mvn) isn't installed. The `.githooks/pre-push` hook additionally runs the TS build+typecheck gate and a Java pom-version drift guard on every push (activate hooks once per diff --git a/scripts/ci-local.sh b/scripts/ci-local.sh index 297a37da2..5a4f007aa 100755 --- a/scripts/ci-local.sh +++ b/scripts/ci-local.sh @@ -75,13 +75,32 @@ # # machine where skips indicate real problems. # scripts/ci-local.sh --help # +# THE FULL RUN IS PARALLEL (see "Lanes" below). The flagless run does the steps that +# write into the tree first, in order: install, the drift gates, one workspace build. +# Then it runs the rest as independent LANES side by side, at most MO_CI_JOBS at once +# (default 8): java (the reactor, which already holds every java/kotlin conformance +# class), python, mutation, csharp, ts, ts-integ, java-integ and kotlin-integ (both +# after java, which installs the jars they read), showcase (after java and csharp), cube +# and gates. Each lane writes to its own log; the log prints as one block when the lane +# ends, so output never interleaves. The summary lists every step with its time, the +# slowest steps and each lane's wall time. Every step still runs and still fails the +# run; only the order changes. --quick and --only run serially as before. +# Worker counts inside the runners: MO_CI_PY_WORKERS (pytest-xdist, default 8), +# MO_CI_TS_WORKERS (bun test --parallel for the integration suite, default 6) and +# MO_CI_JVM_FORKS (surefire forkCount for the JVM integration modules, default 3). +# # POSTGRES SIDECAR (see ensure_pg_sidecar below). Any lane that runs integration # tests shares ONE Postgres container instead of booting a Testcontainers instance # per scenario. It is started (or reused) automatically and costs ~6s cold, ~1s warm. # Measured on this repo: python integration 6m54s -> 11s, kotlin integration 47s with -# zero containers booted. If METAOBJECTS_TEST_PG_URL is already set — as local-ci.yml -# does from its `services:` block — nothing here changes. Opt out with -# MO_CI_NO_PG_SIDECAR=1; rename it with MO_PG_SIDECAR_NAME. +# zero containers booted. Every port's helper creates its own database per scenario, +# so lanes that share the container never share a database. Its name carries a hash of +# this checkout's path, so two checkouts on one machine never share or stop each +# other's. It is left running for the next run (that is the ~1s warm start), so each +# checkout keeps one; `docker rm -f metaobjects-ci-sidecar-` removes it. If +# METAOBJECTS_TEST_PG_URL is already set — as local-ci.yml does from its +# `services:` block — nothing here changes. Opt out with MO_CI_NO_PG_SIDECAR=1; rename +# it with MO_PG_SIDECAR_NAME. # # MO_CI_LEAK_BASE overrides the ref the leak scan diffs against (default origin/main, # else HEAD~1); hygiene.yml points it at the PR's base branch. MO_CI_JACOCO=1 keeps @@ -127,6 +146,10 @@ done [ -n "$ONLY" ] && [ "$QUICK" -eq 1 ] && { echo "--only and --quick are mutually exclusive" >&2; exit 2; } [ "$INTEG_ONLY" -eq 1 ] && [ "$QUICK" -eq 1 ] && { echo "--integration-only and --quick are mutually exclusive" >&2; exit 2; } [ "$INTEG_ONLY" -eq 1 ] && [ "$NO_INTEG" -eq 1 ] && { echo "--integration-only and --no-integration are mutually exclusive" >&2; exit 2; } +MO_CI_JOBS="${MO_CI_JOBS:-8}" +case "$MO_CI_JOBS" in + ''|*[!0-9]*|0) echo "MO_CI_JOBS must be a positive integer (got '${MO_CI_JOBS}')" >&2; exit 2 ;; +esac in_lane() { # in_lane
— true when the section is selected [ -z "$ONLY" ] && return 0 @@ -145,9 +168,21 @@ want_any() { # want_any — true when ANY listed section's should r local s; for s in "$@"; do want "$s" && return 0; done; return 1 } -PASS=(); FAIL=(); SKIP=() have() { command -v "$1" >/dev/null 2>&1; } +# Every verdict is a line in a results file — "\t\t" — rather +# than an entry in a shell array, because the full run's lanes are background subshells and +# an array a subshell appends to dies with it. Each lane writes its own file; the summary +# reads them all back in lane order. +CI_T0=$SECONDS +CI_TMP="$(mktemp -d "${TMPDIR:-/tmp}/mo-ci-local.XXXXXX")" +CI_RESULTS="$CI_TMP/main.tsv"; : > "$CI_RESULTS" +record() { printf '%s\t%s\t%s\n' "$1" "$2" "$3" >> "$CI_RESULTS"; } # record +fmt_dur() { if [ "$1" -ge 60 ]; then printf '%dm%02ds' $(($1 / 60)) $(($1 % 60)); else printf '%ds' "$1"; fi; } +step_passed() { # step_passed "" — true when a step of that name has PASSED in this run + cat "$CI_TMP"/*.tsv 2>/dev/null | awk -F'\t' -v n="$1" '$1 == "PASS" && $3 == n { f = 1 } END { exit !f }' +} + # Every Maven call a gate makes — here and in the scripts it spawns (integration-test.sh, # fatjar-smoke.sh) — resolves and INSTALLS into a CI-owned local repository, never the # developer's ~/.m2. The Java lanes `mvn install` the tree at the CURRENT version, and on the @@ -163,22 +198,32 @@ export MAVEN_ARGS="${MAVEN_ARGS:+$MAVEN_ARGS }-Dmaven.repo.local=${METAOBJECTS_C # ssh2/cpu-features chain needs node-gyp on the machine — `npm i -g node-gyp` # fixes it for good). Retry once as defense-in-depth for transient installer # flakes; a genuinely broken install still fails both attempts. -bun_install() { bun install || bun install; } +# +# The full run installs ONCE, before its lanes fork (MO_CI_DEPS_INSTALLED=1), so the install +# every gate keeps for running on its own cannot rewrite node_modules under a sibling lane. +bun_install() { [ "${MO_CI_DEPS_INSTALLED:-0}" = "1" ] && return 0; bun install || bun install; } step() { # step "" local name="$1"; shift + local t0=$SECONDS dt echo "" echo "── ▶ $name ──────────────────────────────────────────────" - if "$@"; then PASS+=("$name"); else FAIL+=("$name"); echo " ✖ $name FAILED" >&2; fi + if "$@"; then + dt=$((SECONDS - t0)); record PASS "$dt" "$name" + echo "── ✓ $name ($(fmt_dur "$dt"))" + else + dt=$((SECONDS - t0)); record FAIL "$dt" "$name" + echo " ✖ $name FAILED ($(fmt_dur "$dt"))" >&2 + fi } step_if() { # step_if "" — SKIP (or FAIL under --strict-toolchains) when tool absent local tool="$1"; local name="$2"; shift 2 if have "$tool"; then step "$name" "$@"; elif [ "$STRICT" -eq 1 ]; then echo ""; echo " ✖ $name — '$tool' not installed (strict-toolchains)" >&2 - FAIL+=("$name (no $tool, strict)") + record FAIL 0 "$name (no $tool, strict)" else echo ""; echo "── ⊘ $name — SKIPPED ('$tool' not installed) ──" >&2 - SKIP+=("$name (no $tool)") + record SKIP 0 "$name (no $tool)" fi } @@ -516,6 +561,9 @@ gate_ts_build_typecheck() { bun_install && bun run --filter '*' build && bun run # Build only (no typecheck) — the ts-slow lane needs the workspace dist/ to run the # integration tests, but the typecheck already runs in ts-fast, so don't repeat it. gate_ts_build() { bun_install && bun run --filter '*' build; } +# The full run's other half of gate_ts_build_typecheck: it builds before its lanes fork (every +# lane that runs the CLI or the integration suite reads dist/) and typechecks inside one. +gate_ts_typecheck() { bun run --filter '*' typecheck; } # migrate-ts real-Postgres suites — apply / lifecycle / rollback / introspection against a # real engine. ADR-0015 makes migrate-ts the project's ONLY migrate engine, so this is the @@ -595,17 +643,23 @@ gate_ts_unit() { # any warm developer box — which is exactly what happened the first time they ran. # Scoped, not `--filter '*'`: metadata (its `constants` subpath is the fix under test) # plus every published browser package. ~3s cold, so the lane stays cheap. - bun run --filter '@metaobjectsdev/metadata' \ - --filter '@metaobjectsdev/runtime-web' \ - --filter '@metaobjectsdev/react' \ - --filter '@metaobjectsdev/tanstack' \ - --filter '@metaobjectsdev/angular' build || return 1 - # sdk's suite asserts that `agent-context/` — the gitignored copy bundled INTO the - # package for publishing — exists. That copy is produced by sdk's build, so on a clean - # checkout the assertion fails; on a developer box it passes off a stale artifact. Run - # the bundle step alone rather than sdk's full `build`, which also runs tsc and would - # drag in every dependency's dist for no gain here. - bun run --filter '@metaobjectsdev/sdk' bundle-agent-context || return 1 + # + # The full run has already built the WHOLE workspace before its lanes fork + # (MO_CI_WORKSPACE_BUILT=1), which covers both steps below; rebuilding metadata's dist/ + # here would rewrite it under the lanes running the CLI and the integration suite. + if [ "${MO_CI_WORKSPACE_BUILT:-0}" != "1" ]; then + bun run --filter '@metaobjectsdev/metadata' \ + --filter '@metaobjectsdev/runtime-web' \ + --filter '@metaobjectsdev/react' \ + --filter '@metaobjectsdev/tanstack' \ + --filter '@metaobjectsdev/angular' build || return 1 + # sdk's suite asserts that `agent-context/` — the gitignored copy bundled INTO the + # package for publishing — exists. That copy is produced by sdk's build, so on a clean + # checkout the assertion fails; on a developer box it passes off a stale artifact. Run + # the bundle step alone rather than sdk's full `build`, which also runs tsc and would + # drag in every dependency's dist for no gain here. + bun run --filter '@metaobjectsdev/sdk' bundle-agent-context || return 1 + fi # Every server-side TS package whose suite no other lane runs in full. The three # originals (metadata/render/runtime-ts) plus, from 2026-08-09, the seven that were # gated by NOTHING AT ALL: gate_conf_ts covers migrate-ts / codegen-ts / cli in full @@ -669,9 +723,20 @@ gate_conf_python() { # ModuleNotFoundError on 9 modules (it only passed locally where the venv had # been synced with the extra by a prior run). This matches every other # tests/integration invocation in the repo (scripts/integration-test.sh, - # docs/CONFORMANCE.md). The PG-backed scenarios connect to the lane's Postgres - # sidecar via METAOBJECTS_TEST_PG_URL (set in local-ci.yml). - ( cd server/python && uv run --extra integration pytest tests/ -q ) + # docs/CONFORMANCE.md). + # + # EXCEPT tests/integration when this selection also runs `integration-tests (python)`, + # which is that directory: running it here too ran its ~380 Postgres and MySQL scenarios + # twice per full run. Under --no-integration (conformance.yml) it stays in, because half + # of it needs no docker (the generated-lane and normalization tests) and this is then the + # only step that runs it. + # + # -n: pytest-xdist. The suite is ~4,000 independent tests, many of which shell out to a + # formatter or compile generated code, so it is wall-clock bound, not CPU bound; serial it + # was the longest step in the full run. MO_CI_PY_WORKERS overrides the worker count. + local skip_integ="--ignore=tests/integration" + [ "$NO_INTEG" -eq 1 ] && skip_integ="" + ( cd server/python && uv run --extra integration pytest tests/ $skip_integ -q -n "${MO_CI_PY_WORKERS:-8}" ) } gate_conf_kotlin() { ( cd server/java \ @@ -690,6 +755,10 @@ gate_java_reactor() { [ "${MO_CI_JACOCO:-0}" = "1" ] && jacoco="" ( cd server/java && mvn -q clean install $jacoco ) } +# The full run's JVM integration lanes need the reactor installed and must not install it +# themselves (two concurrent `mvn install`s clobber the shared module jars). When the reactor +# step is not selected — `--integration-only` — this installs it without its tests. +gate_java_install() { ( cd server/java && mvn -q -DskipTests -Djacoco.skip=true install ); } gate_completeness() { ( cd server/typescript/packages/metadata && bun run conformance:mutation ); } gate_doc_template_drift() { bash scripts/sync-doc-templates.sh \ @@ -726,9 +795,9 @@ run_cube_lane() { echo " real Cube. Start docker and run: scripts/ci-local.sh --only cube" >&2 echo " ════════════════════════════════════════════════════════════════════════" >&2 if [ "$STRICT" -eq 1 ]; then - FAIL+=("cube-model live check (docker down, strict)") + record FAIL 0 "cube-model live check (docker down, strict)" else - SKIP+=("cube-model live check (docker down)") + record SKIP 0 "cube-model live check (docker down)" fi } @@ -753,11 +822,27 @@ run_cube_lane() { # no readiness gate and that was the dominant flake: the first connect to a # transiently-slow sidecar hung to the test timeout. # +# * the container name is PER CHECKOUT (a checksum of this checkout's path). Several +# checkouts of this repository run on one machine at once, and a shared name made them +# reuse one database: the migrate-ts and runtime-ts suites create fixed table names in +# it, so two checkouts' runs could drop each other's tables mid-test. +# # Opt out with MO_CI_NO_PG_SIDECAR=1 to get the old per-scenario-container behaviour. -MO_PG_SIDECAR_NAME="${MO_PG_SIDECAR_NAME:-metaobjects-ci-sidecar}" +MO_PG_SIDECAR_NAME="${MO_PG_SIDECAR_NAME:-metaobjects-ci-sidecar-$(printf '%s' "$ROOT" | cksum | cut -d' ' -f1)}" +MO_CI_PG_URL="" # set by start_pg_sidecar; exported only by use_pg_sidecar +# ensure_pg_sidecar — start (or reuse) the sidecar and export its URL into THIS shell. ensure_pg_sidecar() { [ -n "${METAOBJECTS_TEST_PG_URL:-}" ] && return 0 # CI, or an explicit override + start_pg_sidecar && use_pg_sidecar +} + +# start_pg_sidecar — start (or reuse) the sidecar and record its URL in MO_CI_PG_URL, exporting +# nothing. The full run calls it once before its lanes fork and lets each lane that runs an +# integration suite export the URL for itself, so a lane with no database (ts unit suites) +# never sees one: runtime-ts's suite runs its real-PG files whenever the URL is set, against +# the same fixed table names the ts-integration lane is using at that moment. +start_pg_sidecar() { [ "${MO_CI_NO_PG_SIDECAR:-0}" = "1" ] && return 0 local name="$MO_PG_SIDECAR_NAME" port @@ -804,19 +889,25 @@ ensure_pg_sidecar() { # exactly as the ts-slow job does, so migrate-ts's real-Postgres suites run locally # instead of self-skipping. # - # The two *_EXPECT sentinels are armed HERE and nowhere else — at the one point where a - # database is proven up and its URL exported. Every fallback above returns before this, + # The two *_EXPECT sentinels are armed by use_pg_sidecar and nowhere else — only once a + # database is proven up here and its URL recorded. Every fallback above returns before this, # so a box with no docker, no published port or a sidecar that never became ready still # self-skips as it always did rather than failing for want of a database it was never # going to have. Armed, they turn "the URL stopped being set" from a silent 26-test skip # reported as ✓ into a loud failure. Hosted CI sets both from its `services:` block; the # local run had neither, which is why a local ✓ and a hosted ✖ could disagree about the # only real-engine gate on migration correctness. - export METAOBJECTS_TEST_PG_URL="postgres://metaobjects:metaobjects@localhost:${port}/metaobjects_test" + MO_CI_PG_URL="postgres://metaobjects:metaobjects@localhost:${port}/metaobjects_test" + echo " METAOBJECTS_TEST_PG_URL -> localhost:${port} (stop it: docker rm -f $name)" +} + +# use_pg_sidecar — export the URL start_pg_sidecar recorded (nothing when it fell back). +use_pg_sidecar() { + [ -n "$MO_CI_PG_URL" ] || return 0 + export METAOBJECTS_TEST_PG_URL="$MO_CI_PG_URL" export MIGRATE_TS_PG_URL="$METAOBJECTS_TEST_PG_URL" export MIGRATE_TS_PG_EXPECT=1 export RUNTIME_TS_PG_EXPECT=1 - echo " METAOBJECTS_TEST_PG_URL -> localhost:${port} (stop it: docker rm -f $name)" } run_integration_for() { # run_integration_for "