Repository navigation
CI runtime: shard the test batches across parallel jobs when the single job tightens again #730
Description
build_uw3_and_test.yaml referenced "the CI-runtime issue" as the home for the
durable fix to wall-clock. That issue was never actually filed. This is it, with
the measurements.
Where it stands
The 120-minute cap was set on a 2026-08-15 measurement putting the serial phase
alone at ~55 minutes, before xdist distributed the batches. Measured 2026-09-12,
the five most recent successful runs spent this long in "Run tests":
34m19s development
35m23s development
42m32s feature/particle-demos
44m20s docs/tier-c-characterisation-tests
47m34s docs/timestepping-pattern
That headroom is why the previously-deferred bands could simply be switched on
rather than traded against something else.
The fix, for when it is needed
Split the batches across parallel GitHub jobs so wall-clock stops tracking total
test time. The serial phase already falls into natural groups that map onto a
job matrix without restructuring scripts/test.sh much:
test_00*-test_03*test_05*-test_08*test_10*-test_14*- the
--p Nmpirun phase, which runs its own batches
Not yet, though
Doing it now would be machinery ahead of need. The single job fits inside its
budget with room, and every job added pays the ~2.5 min build again unless the
build artefact is shared between them — which is the part that needs designing,
not the matrix.
Worth revisiting when "Run tests" passes roughly 70 minutes, or when the build
step can be cached across jobs.
Underworld development team with AI support from Claude Code
Activity
Stop excluding tests by band, let tier C report instead of gate, and …
Stop excluding tests by band, let tier C report instead of gate, and …
Stop excluding tests by band, let tier C report instead of gate, and …
lmoresi commentedon Oct 28, 2026
The trigger condition is met twice over, so this is now in PR #863.
Measured on run 40623403571 (
development, 2026-10-28, SUCCESS): 91:19 in theserial batches plus 4:38 in the MPI phase, for a 96-minute job against the
120-minute cap. The three most recent successful runs took 96, 97 and 96
minutes. This issue said to revisit at roughly 70. PR #689 was then cancelled
at 2h00m52s having recorded no failures.
#863 implements the matrix this issue sketched, with the batches assigned by
hand from the measured times — the distribution is too skewed for round-robin,
since
test_106*/test_107*alone is 33 minutes against eleven bands underfour. The split is 33 / 19 / 18 / 22 minutes, so wall-clock becomes about 38
minutes including install and build.
On the part this issue called out as needing design: each shard rebuilds rather
than fetching a shared artefact. The shards run concurrently, so the repeated
~2.5 minute build costs wall-clock once, and the repository is public, so the
runner minutes are free. Sharing it would buy nothing and add a transfer round
trip plus a second definition of what "built" means.
Staying open after #863 merges, for a part this issue did not anticipate.
The
test_106*/test_107*band is a tail rather than a volume: 48% of its 149tests finish in the first two minutes and the remaining 31 go to the rest.
Since the distribution is
--dist loadfile, a file is atomic on a worker, sothe band cannot finish before its slowest file. A per-file measurement is
running and the split will follow it rather than a guess.
Label:
fixed-in-PR→fixed-in-developmentwhen #863 merges.lmoresi commentedon Oct 28, 2026
The per-file measurement is in: two files are 78% of the band
#863 landed the matrix and left this issue open for the one part it could not
split on a guess. The numbers are now in. All 28 files of
test_106*/test_107*, run serially one file per process,-m "not tier_c":Two files are 34.1 of the 43.9 minutes. Five test functions, between them,
are 78% of the band.
Why that sets a floor the matrix cannot lower
The distribution is
--dist loadfile, so a FILE is atomic on a worker. Withfour workers the band cannot finish before its slowest file does, whatever the
shard assignment is. That is consistent with what CI shows: the band measures
33 minutes there with four workers, which is one worker inside
test_1065_log_conformationwhile the other three clear the remaining 27files and idle.
So the split that helps is not "divide the band in half by file count" — it is
separate those two files from each other, putting
test_1065_log_conformationin one shard andtest_1064_stress_history_unitsin another. The band's contribution then drops to roughly the larger of the two
rather than their sum plus the tail.
What this is not
Both files are
level_2, tier_b, so they gate every pull request, and that iscorrect — they are proven tests. The answer here is sharding, not moving them
to
tier_c: dropping the slow ones is what hidtest_1064for two months andconcealed #734, so the cost moves off the critical path rather than the
coverage being cut. Same ruling as the tier C report.
State
PR #863 is green and measured: 28.7 minute wall-clock against 96, shards of
23.6 / 28.3 / 17.3 / 22.6. The estimates in that PR did not rank the same way
as the measurements, so the assignment there is deliberately left as it is
until several runs are available — re-tuning off one sample would fit runner
noise, since the 28.3-against-17.3 imbalance is of the same order as the
variance.
This issue stays open for the two-file split above, which is now specified
rather than inferred.
Merge pull request #863 from underworldcode/ci/shard-test-batches-730
lmoresi commentedon Oct 29, 2026
Fixed by #863 (merged to development, c6ee62c): the test job is split into four shards assigned by measured cost; the first sharded run took 28.7 minutes against 96.
Underworld development team with AI support from Claude Code