[lmoresi]

Closes the trigger condition on #730: the test job's wall-clock had started
deciding outcomes rather than bounding runaway runs.

Why now

#730 said to revisit when "Run tests" passed roughly 70 minutes. Measured on
run 40623403571 (development, 2026-10-28, SUCCESS):

serial batches	91:19 across 16 batches
MPI phase	4:38
job wall-clock	96 min, against a 120-minute cap

Three recent successful runs took 96, 97 and 96 minutes. PR #689 was cancelled
at 2h00m52s having recorded no failures — its per-batch times run about
1.35x #677's throughout, so it cannot fit the box however many times it is
re-rolled.

What this does

The serial batches run as a shard matrix. Each batch is assigned to a shard by
hand from the measured times, which are recorded in scripts/test.sh beside
the batch definitions:

test_106*,test_107*        33:00     <- the floor
test_105*                  11:42
test_110*                  10:23
test_08*                    7:14
test_101*,test_102*         7:05
test_00[0-4]*               5:14
...eleven more, all under 4 min
the --p N mpirun phase      4:38

Round-robin would have been worse than useless on that distribution — it would
group the three expensive bands together. The hand assignment gives
33 / 19 / 18 / 22 minutes, so wall-clock becomes about 38 minutes
including install and build, down from 96.

--shard K/N selects one; omitting it runs everything, which is what a local
full run wants.

Two things had to move rather than be copied

The tier C report costs about 42 minutes and is now its own job. As a step
on a matrix job it would have run once per shard to produce one report.

The MPI phase belongs to one shard, for the same reason and because it is
the only part of the suite holding several ranks at once — the part most likely
to contend on a shared runner.

A new aggregate check named test reports the combined result, so one
conclusion still answers whether the suite passed. development carries no
required status check that the rename could strand; we checked the ruleset
rather than assuming.

Each shard rebuilds

#730 named build-artefact sharing as "the part that needs designing". Measured
against what it saves, it is not worth it: the shards run concurrently, so the
repeated ~2.5 min build costs wall-clock once, and the repository is
public, so the runner minutes are free. Sharing it would add a transfer round
trip and a second definition of what "built" means.

Adversarial review

The review found one defect in this change, and it was the serious kind.

The shard count had three copies — the matrix list, a SHARDS env var, and
the assignments in scripts/test.sh. The failure mode is not a nuisance: a
matrix of [1, 2, 3] against an assignment using 4 runs green while shard
4's batches — 194 files, the largest group — never run, with nothing in the
logs to say so. That is #721 exactly, files going dark behind a pattern that
stopped covering them.

Fixed by deriving the number instead of repeating it. --print-shards reads
the largest shard out of the batch call sites in the script itself; a plan
job emits it; the matrix is built with fromJSON; the shard argument uses
strategy.job-total. Adding a batch 5 ... line now grows CI a fifth shard
with nothing else to edit.

Where a count still has to be compared it is checked, not assumed: --shard K/N refuses an N that disagrees with the assignment, and refuses K outside
1..N, rather than running a partial suite quietly.

Evidence

Verified by stubbing pytest and running each shard, before and after that
refactor:

shard 1:  28 files
shard 2:  53 files
shard 3:  81 files
shard 4: 194 files
            ---
union:   356 files   == the unsharded run (356)
no file in two shards, none in neither

Also checked, because a guard that warns is not a guard:

./test.sh --shard 1/3   -> exit 1 (count disagrees with the assignment)
./test.sh --shard 9/4   -> exit 1 (shard out of range)
a failing batch         -> exit 1 on its own shard, exit 0 on the others
./test.sh --print-shards -> [1,2,3,4]

scripts/check_test_coverage.py reads the batch globs out of test.sh to
prove every test file is reachable. Left alone it reported all 355 files
unreachable — loud, which is the right direction for it to break in, but it
still had to learn the sharded call site. It now passes: all 355 test files
are reachable from scripts/test.sh.

Determinism is untouched. Files were not re-partitioned within batches,
only batches assigned to jobs, so which files share an xdist process is exactly
what it was. That matters because the worker count is what #567 turned on.

Left open on #730

The test_106* / test_107* band is 33 minutes and sets the floor. It is a
tail, not a volume: 48% of its 149 tests finish in the first two minutes,
then the remainder takes 31. Because the distribution is --dist loadfile a
file is atomic on a worker, so the band cannot finish before its slowest file
does. Splitting it further wants the per-file numbers rather than a guess, and
a local per-file measurement is running now — #730 stays open for it.

Issue adjudication

• CI runtime: shard the test batches across parallel jobs when the single job tightens again #730 — addressed by this branch, labelled fixed-in-PR. The sharding it
  designed is in; the per-file split it did not anticipate stays on it.
• Add CitcomS predictor-corrector DDt manager for composed transport #689 — still live, and this is what unblocks it. The PR carries no
  defect: it was cancelled by the cap with zero failures. It needs a re-roll
  once this merges, not a fix.
• scripts/test.sh: eleven test files match no glob, so the integration-point and swarm suites have never run in CI #721 — still fixed; this change extends the checker that guards it rather
  than touching the globs it widened.

Underworld development team with AI support from Claude Code

https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[lmoresi]

First sharded run: measured

Run 40638600008, all green.

job	measured	estimated
plan	0.2 min	—
shard 1/4 (the 33-min band)	23.6 min	~38 min
shard 2/4	28.3 min	~24 min
shard 3/4	17.3 min	~23 min
shard 4/4	22.6 min	~27 min
test (aggregate)	pass	—
Tier C	skipped on a PR, as intended	—
run wall-clock	28.7 min	vs 96 min before

The mechanism works as designed: plan derived [1,2,3,4] from
--print-shards, the four shards ran concurrently, the aggregate test check
ANDed them, and Tier C correctly did not run on a pull request.

The estimates were conservative and ranked differently, which is worth
recording. Shard 1 carries the 33-minute test_106*/test_107* band and
finished in 23.6; shard 2 was estimated at 19 minutes of tests and is the long
pole at 28.3. So a batch costs less in a job that runs only its own batches
than it did in sequence, by more than the assignment assumed — the per-batch
times are not a linear predictor of a shard.

We have recorded this rather than re-tuned on it (11cd3e87 replaces the
prediction in the CHANGELOG and in test.sh with the measurement). Re-assigning
off one sample would be fitting runner noise: the imbalance left, 28.3 against
17.3, is of the same order as the variance on a shared runner. It wants several
runs first.

Follow-up for #730, now measured rather than inferred

The test_106*/test_107* band is a tail, and we have the per-file numbers
for the first eleven of its 28 files (serial, one file per process):

582 s   test_1064_stress_history_units.py
 35 s   test_1061_constrained_freeslip.py
 24 s   test_1063_stress_history_restart.py
 23 s   test_1060_generic_flux_model_update.py
 19 s   test_1063_constrained_traction.py
 19 s   test_1062_constrained_solcx.py
 ...the rest under 11 s

582 s is 79% of the 735 s measured so far, and test_1065_log_conformation.py
has since run past 1380 s on its own. So the band has two dominant files,
not one. Under --dist loadfile a file is atomic on a worker, so the band's
floor is the larger of those two — splitting it means separating them, which is
the concrete follow-up #730 stays open for.