Shard the CI test batches across parallel jobs (#730)

.github/workflows/build_uw3_and_test.yaml

@@ -7,6 +7,10 @@ name: test_uw3
 # 60+ minutes on conda-forge solves and timing the runner out before tests
 # could start. The pixi.lock matches what local development uses, so CI now
 # runs against the same dependency state developers see.
+#
+# The serial batches run SHARDED across parallel jobs (#730). Wall-clock no
+# longer tracks total test time: it tracks the slowest shard. See
+# scripts/test.sh for the shard assignment and the measurements behind it.
 
 on:
   push:
@@ -43,19 +47,50 @@ env:
   WORKERS: 4
 
 jobs:
-  test:
+  # How many shards to run is NOT stated here. scripts/test.sh assigns each
+  # batch to a shard by hand, from measurements, and --print-shards reads the
+  # count back out of those assignments. The matrix is built from that, so the
+  # number has exactly one definition.
+  #
+  # A second copy would be a silent-coverage bug, not a nuisance: a matrix of
+  # [1,2,3] against an assignment using 4 runs green while shard 4's batches —
+  # the largest group, 194 files — never run. That is #721, files going dark
+  # with nothing to say so.
+  plan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    outputs:
+      shards: ${{ steps.shards.outputs.list }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Read the shard count out of the batch assignment
+        id: shards
+        run: |
+          list=$(./scripts/test.sh --print-shards)
+          echo "shards: $list"
+          echo "list=$list" >> "$GITHUB_OUTPUT"
+
+  shards:
+    needs: plan
+    strategy:
+      # Every shard reports. Without this, the first shard to fail cancels the
+      # others and the log shows one failure where there may be four — which is
+      # the opposite of what a sharded run is for.
+      fail-fast: false
+      matrix:
+        shard: ${{ fromJSON(needs.plan.outputs.shards) }}
+
+    # strategy.job-total is the matrix length, so the shard count is never
+    # written down a second time here either.
+    name: shard ${{ matrix.shard }}/${{ strategy.job-total }}
     runs-on: ubuntu-latest
-    # Raised past 60 minutes on a 2026-08-15 measurement (serial phase ~55 min).
-    # xdist has since cut that: the five successful runs before 2026-09-12 spent
-    # 34, 35, 43, 44 and 48 minutes in "Run tests", well inside this budget. The
-    # headroom is why the previously-deferred bands could simply be switched on
-    # (#721 follow-up) rather than traded against something else.
-    #
-    # Sharding the batches across parallel jobs remains the durable fix if this
-    # tightens again; it was referenced here as "the CI-runtime issue" but never
-    # actually filed, so the numbers above are recorded in the issue that this
-    # change opens.
-    timeout-minutes: 120
+
+    # Per SHARD, not per suite. The unsharded job ran 96 minutes against a
+    # 120-minute cap and #689 was cancelled at 2h00m52s with zero failures, so
+    # the cap was deciding outcomes. The slowest shard is ~33 min of tests plus
+    # ~5 min of install and build; 60 leaves room for a slow runner while still
+    # being a real signal when a shard doubles.
+    timeout-minutes: 60
 
     steps:
       - uses: actions/checkout@v4
@@ -75,6 +110,13 @@ jobs:
         # `pixi run -e dev build` invokes the build task from pixi.toml,
         # which is `pip install . --no-build-isolation` (non-editable —
         # editable installs are project policy violation, see CLAUDE.md).
+        #
+        # Each shard pays this (~2.5 min) rather than downloading a shared
+        # build artefact. That is deliberate: the shards run concurrently, so
+        # the repeated build costs wall-clock ONCE, and the repo is public, so
+        # the extra runner minutes are free. Sharing the artefact would add a
+        # pack/upload/download round trip and a second definition of what
+        # "built" means, to save nothing on the clock.
         run: pixi run -e dev build
 
       # Import smoke test: a hard crash during `import underworld3` (segfault
@@ -89,13 +131,53 @@ jobs:
         run: pixi run -e dev python -u -X faulthandler -c "import faulthandler; faulthandler.enable(); import underworld3; print('import OK', underworld3.__version__)"
 
       - name: Run tests
-        run: pixi run -e dev ./scripts/test.sh --p 2
+        run: pixi run -e dev ./scripts/test.sh --p 2 --shard ${{ matrix.shard }}/${{ strategy.job-total }}
+
+  # One check that says whether the suite passed, so a reader (and anything
+  # querying statusCheckRollup for "test") gets a single answer rather than
+  # having to AND four shard results together. It is named `test` because that
+  # is what the check was called before sharding.
+  test:
+    name: test
+    needs: shards
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Report the shard results
+        run: |
+          echo "shards: ${{ needs.shards.result }}"
+          if [ "${{ needs.shards.result }}" != "success" ]; then
+            echo "At least one shard did not pass — open the shard jobs above."
+            exit 1
+          fi
+
+  # The tier C characterisations are a REPORT, not a gate, so they run on a
+  # push to development rather than on every pull request. They include the
+  # slow level_3 cases and add ~42 min, which a PR should not pay to learn
+  # something that cannot block it. Failures here are expected to be read
+  # and answered, not reverted.
+  #
+  # Its own job, not a step on the sharded one: as a step it would run once per
+  # shard and cost its ~42 minutes four times over for one report.
+  tier-c:
+    name: Tier C characterisations (report only)
+    if: github.event_name == 'push'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install pixi from lockfile
+        uses: prefix-dev/setup-pixi@v0.9.4
+        with:
+          environments: dev
+          frozen: true
+          cache: true
+
+      - name: Build UW3
+        run: pixi run -e dev build
 
-      # The tier C characterisations are a REPORT, not a gate, so they run on a
-      # push to development rather than on every pull request. They include the
-      # slow level_3 cases and add ~42 min, which a PR should not pay to learn
-      # something that cannot block it. Failures here are expected to be read
-      # and answered, not reverted.
       - name: Tier C characterisations (report only)
-        if: github.event_name == 'push'
         run: pixi run -e dev ./scripts/test.sh --tier-c-report

docs/developer/CHANGELOG.md

@@ -4,6 +4,60 @@ This log tracks significant development work at a conceptual level, suitable for
 
 ---
 
+## 2026 Q4 (October – December)
+
+### CI test batches sharded across parallel jobs (October 2026, #730)
+
+The test workflow ran as a single job whose wall-clock tracked the total cost
+of the suite. Measured on 2026-10-28, that was 91 minutes of tests in 16 serial
+batches plus the MPI phase, giving 96-minute jobs against a 120-minute cap, and
+PR #689 was cancelled at 2h00m52s having recorded no failures. The cap was
+deciding outcomes rather than bounding runaway runs.
+
+The serial batches now run as a four-way shard matrix. Each batch is assigned
+to a shard by hand from the measured per-batch times, because the batches
+differ by two orders of magnitude in cost and a round-robin assignment would
+group the three expensive ones together. The assignment and its measurements
+live in `scripts/test.sh` beside the batch definitions, and `--shard K/N`
+selects one; omitting it runs everything, which is what a local full run wants.
+
+Measured on the first sharded run (40638600008): shards of 23.6, 28.3, 17.3 and
+22.6 minutes including install and build, for a wall-clock of 28.7 minutes
+against 96 before. The estimates from the per-batch times did not transfer
+linearly -- shard 1 carried the 33-minute band and finished in 23.6, while
+shard 2 was estimated at 19 minutes of tests and became the long pole at 28.3.
+A batch costs less in a job that runs only its own batches than it did in
+sequence, by more than the assignment assumed. The split is worth revisiting,
+but on more than one sample: runner variance on a shared runner is of the same
+order as the imbalance that remains.
+
+Two parts of the workflow needed moving rather than copying. The tier C
+characterisation report, which costs about 42 minutes, became its own job: as a
+step on a matrix job it would have run once per shard to produce one report.
+The MPI phase belongs to a single shard for the same reason, and because it is
+the only part of the suite holding several ranks at once. A new aggregate check
+named `test` reports the combined result, so a single conclusion still answers
+whether the suite passed.
+
+Each shard rebuilds Underworld3 rather than downloading a shared artefact. The
+shards run concurrently, so the repeated build costs wall-clock once, and the
+repository is public, so the extra runner minutes are free. Sharing the
+artefact would add a transfer round trip and a second definition of what
+"built" means.
+
+`scripts/check_test_coverage.py` reads the batch globs out of `scripts/test.sh`
+to prove every test file is reachable, and it now recognises the sharded call
+site as well. Without that it reported all 355 files as unreachable, which is
+the right direction for the check to fail in.
+
+The remaining cost is concentrated rather than spread. The `test_106*` /
+`test_107*` band takes 33 minutes, and 48% of its 149 tests finish in the first
+two. Because the distribution is `--dist loadfile`, a file is atomic on a
+worker and the band cannot finish before its slowest file does, so splitting it
+further needs the per-file measurements and is left open on #730.
+
+---
+
 ## 2026 Q3 (July – September)
 
 ### Legacy Diffusion Restart Consistency (September 2026, #708)

scripts/check_test_coverage.py

@@ -29,14 +29,24 @@
 
 
 def globs_run_by(script):
-    """Every tests/... pattern passed to $PYTEST on a line that is not commented."""
+    """Every tests/... pattern the script hands to pytest, on an uncommented line.
+
+    Three spellings reach pytest, and missing one of them makes this check
+    report the whole suite dark — loudly, which is the right direction, but it
+    still has to be kept in step with the script:
+
+    * ``batch <shard> tests/...`` — the sharded call site (#730). The shard
+      number decides WHICH CI job runs it, not whether it runs, so coverage
+      does not care which one it is.
+    * ``"${PYTEST[@]}" tests/...`` — the runner is a bash array.
+    * ``$PYTEST tests/...`` — in case a plain one is left behind.
+    """
     text = script.read_text().replace("\\\n", " ")
     patterns = set()
     for line in text.splitlines():
-        # Both spellings: the runner is a bash ARRAY, so call sites read
-        # "${PYTEST[@]}", but a plain $PYTEST is still worth matching in case
-        # one is left behind or reintroduced.
-        if re.search(r"\$\{?PYTEST", line) and not line.lstrip().startswith("#"):
+        if line.lstrip().startswith("#"):
+            continue
+        if re.search(r"\$\{?PYTEST", line) or re.match(r"\s*batch\s+\d+\s", line):
             patterns.update(re.findall(r"tests/[A-Za-z0-9_\[\]*.\-]+", line))
     return patterns