Skip to content

ci: the step budget rests on the step's own clock - #9

Open
noidee-dev wants to merge 2 commits into
developfrom
ci/the-step-budget-rests-on-the-step-clock
Open

noidee-dev wants to merge 2 commits into
developfrom
ci/the-step-budget-rests-on-the-step-clock

Conversation

@noidee-dev

@noidee-dev noidee-dev commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #8, which moved the gate to macos-latest and left three things owed: a step budget resting on a figure stitched from two runs, two citations naming heads the merge rebase rewrote, and a backlog row whose next step that PR had quietly fulfilled.

The budget: 16 -> 20 minutes

Withdrawn, quoted verbatim: "16 minutes (960 s) is 2.04x of 392 s + 78 s = 470 s -- this image's measured compile plus the OTHER image's measured run, because this image's run figure does not exist yet: the step was killed before the suite finished."

That paragraph promised its own replacement by the first green run on this image. Keeping the promise exposed the shape of the error rather than just the missing addend: a step budget bounds a step's wall clock, and adding one run's compile to another run's suite duration bounds a step that neither run is. That wall clock is a field pair of the Actions API, present on every run that reached the step:

run image Unit-Tests step conclusion
38225383495 macos-15 329 s success
40933631584 macos-latest 493 s failure — the 480 s budget's own kill
41045404989 macos-latest 581 s success
41047833837 macos-latest 517 s success
41048445836 macos-latest 640 s success

640 s is the maximum of a step that finished, so 1200 s is 1.88x — the first of this comment's derivations to land under 2, after 2.09x and 2.04x. Neither of those states 2x as a requirement, so the comment now says in as many words that "the 2x floor" is a reading of them and not a quote, and names 22 minutes (1320 s, 2.06x) as the smallest whole-minute budget holding that reading. 20 is the maintainer's decision of 2026-10-31, taken while the best figure on offer was the 558.79 s the withdrawn stitching produced, against which 1200 s read as 2.15x. The series was measured after the decision, not before it.

The spread matters more than the maximum, and one pair isolates it: 517 s against 640 s is one commit apart, and that commit adds 23 lines of CLAUDE.md, which no target compiles. The compile half alone moved 345.71s -> 492.86s across it.

The job budget is unaffected: 753 s and 937 s for the two green runs against its 2100 s, and a step running the full 1200 s would put the worse of those at 1497 s, 0.71x.

Two heads the rebase rewrote

Both citations now name the develop commit beside the head the run actually had. Replacing the hash alone would have claimed a run ran on a commit it never saw. git show <h> | git patch-id --stable gives one id per pair: 9c8d3673/a774ac4d in ci.yml, c8041d19/bcd0bc61 in ConnectMainActorLivenessTests.

docs/BACKLOG.md

Two rows corrected in place with their wording quoted, two opened.

The row "Every layout figure the guards hold was measured on Darwin 27" asked for "the first push or pull request that triggers a run, and then reading the checks' verdict". #8 was that run, and it answers half the row: both green runs passed Suite "Bounded view layout guard" and Suite "Resizable container floors" on macOS 26.6.2, so the local Darwin 27 figures survive a Darwin 26 runner across all 33 bounded views and the resizable floors. By how much is still unmeasured — both reporters are gated, so the runs produced a verdict and no figures, and a pass is not a margin. The macos-15 premise is withdrawn with the quote, as is the now-false title half of the row it was split from.

Opened: the three-way toolchain split (gate on Swift 6.3.3, the maintainer's Mac on 6.4, the retired image on 6.1.2), whose first instance was the allLines warning that reddened run 41045404989 at 26 occurrences on a tree every macos-15 run had built clean. And claude/nice-leakey-5da0b9, surfaced by a local branch sweep in which 26 of 32 local branches were content-identical to develop or main and were deleted: it carries ed0b70af, a finished relaunch fix dated 2026-09-05 that is on neither develop nor main.

Verification

Every command written into the comment and into the rows was extracted from the committed text and run in that form; all six reproduce their stated figures. Two false claims were caught in the new comment before committing — "two commits apart" (it is one) and "one test-function annotation" (it is 23 lines of Markdown) — and the correction strengthens the variance argument rather than weakening it.

No local suite run: the only Swift change is four doc-comment lines, parse-checked with swiftc -parse against a positive control that reported 2 errors on a broken file. This PR is what tests the new budget — ci.yml triggers only on a push to main/develop and on pull_request, so the branch alone starts nothing.


20fbe7f2 — the first run on the new budget went red, and not on the budget

Run 41244948849 took 597 s for Unit-Tests, comfortably inside the new 1200 s, so nothing was killed. It went red on one test: ConnectionDiagnosticsTests.theSSHDialNeverPutsTheSecretInTheReport recorded "a closed port did not fail the SSH dial: .timedOut". The dial never failed — its step was settled by its own deadline before the probe was given a thread. The case handed the runner stepTimeout: .seconds(10), and in that run 8184 of the 8307 cases carrying a duration line reported a wall clock at or above 10 s, against a run total of 116.701 s.

That is docs/BACKLOG.md's "Wall-clock ceilings still in the tree", fifth shape, reaching the test through a production parameter the test itself chooses. It had already been bought one file at a time, twice: 61850a9c (2026-09-19) moved the jump suite's twin off .seconds(10) for exactly this reason — one file, 10 insertions, 2 deletions — and ResolveNamesTests had already named its own 600 s roomy. The five literals in ConnectionDiagnosticsTests were never swept and fell 42 days later.

  • A named roomy of 600 s carrying the measurement and the command that produces it, with all five call sites through it. Raising it costs those five nothing, which is the argument that makes 600 s safe here: every target either answers at once (the rig) or refuses at once (a closed loopback port), and the skipped case never dials.
  • Both diagnostics suites withdraw the clause "it is deliberately far larger than any step budget below", quoted in place. False of this file by construction now; false of the jump file on the day it was written — git blame dates both of that file's 600 s sites to 61850a9c and the sentence's heading to ae19a53a, nine days later.
  • Deliberately not swept, with the command that lists them: six occurrences in ICMPEchoTests and NetworkTraceTests. Five send real echo or trace probes, where a roomy budget's cost is unmeasured; the sixth is NetworkTraceTests.traceStep, whose own comment calls 5 s generous and records the 5.004 s behind it. Two more route through per-file factories shared with cases where a tight budget may be load-bearing. Each needs classifying before any value moves — which is how 2026-09-19 produced this red.

Red first, in two parts

The observed red is the CI run. The mechanism reproduces at a keyboard with the budget planted at .nanoseconds(1), where the deadline cannot lose: the SSH case recorded the identical sentence, and the S3 case its own twin. Two issues in that run and three in the first one, same plant — which case varied is not established, because the earlier run's per-case lines were lost to a tail in the command that captured it. Green at 600 s: 154 tests in 6 suites, 0 issues.

One pre-existing red found by the full local run, not fixed here

ViewTestabilitySpike's "Varying only isRegex leaves the pixels unchanged" fails on this Mac (macOS 27.0.1, Swift 6.4.2) and passed on the gate in run 41244948849 itself, after 1.265 s. The case's doc comment states as measured that a Toggle's checked state "stays out of the bitmap"; on macOS 27 it does not. Recorded in the toolchain-split row, which withdraws its own "nothing has been run to find what Swift 6.4 diagnoses that 6.3.3 does not" as far as "nothing has been run" goes — what was found is a rendering divergence, not a diagnostic one, so the row stays open. Untouched here by construction: the spike is in macSCPAppKitTests and this commit touches only macSCPCoreTests and docs/.

Full local suite on the committed tree: 5091 tests in 408 suites passed, 2884 in 242 suites with that one pre-existing issue.

🤖 Generated with Claude Code

noidee-dev and others added 2 commits October 31, 2026 14:42
The Unit-Tests budget goes 16 -> 20 minutes, the maintainer's decision of
2026-10-31, and the derivation under it is replaced rather than extended:
it had been stitched together from two different runs.

Withdrawn, quoted verbatim: "16 minutes (960 s) is 2.04x of 392 s + 78 s
= 470 s -- this image's measured compile plus the OTHER image's measured
run, because this image's run figure does not exist yet: the step was
killed before the suite finished." That paragraph promised its own
replacement by the first green run on this image, and keeping the promise
exposed the shape of the error rather than just the missing addend: a
step budget bounds a step's wall clock, and adding one run's compile to
another run's suite duration bounds a step that neither run is.

That wall clock is a field pair of the Actions API, present on every run
that reached the step. Measured 2026-10-31:

  38225383495  macos-15      329 s  success
  40933631584  macos-latest  493 s  failure -- the 480 s budget's own kill
  41045404989  macos-latest  581 s  success
  41047833837  macos-latest  517 s  success
  41048445836  macos-latest  640 s  success

640 s is the maximum of a step that finished, so 1200 s is 1.88x. The two
derivations this comment has carried landed at 2.09x and 2.04x, and 1.88x
is the first under 2 -- but neither of them states 2x as a requirement, so
the comment now says in as many words that "the 2x floor" is a reading of
them and not a quote, and gives 22 minutes (1320 s, 2.06x) as the smallest
whole-minute budget that would hold that reading. 20 was chosen while the
best figure on offer was the 558.79 s the withdrawn stitching produced,
against which 1200 s read as 2.15x; the series above was measured after
the choice, not before it.

The spread matters more than the maximum, and one pair isolates it: 517 s
against 640 s is one commit apart, and that commit adds 23 lines of
CLAUDE.md, which no target compiles. The compile half alone moved
345.71s -> 492.86s across it.

The job budget above is unaffected: 753 s and 937 s for the two green runs
against its 2100 s, and a step running the full 1200 s would put the worse
of those at 1497 s, 0.71x.

Two citations named heads that the merge rebase rewrote. Both now name the
develop commit beside the head the run actually had, because replacing the
hash alone would have claimed a run ran on a commit it never saw.
`git show <h> | git patch-id --stable` gives one id per pair:
`9c8d3673`/`a774ac4d` in ci.yml, `c8041d19`/`bcd0bc61` in
ConnectMainActorLivenessTests.

docs/BACKLOG.md: two rows corrected in place with their wording quoted,
and two rows opened.

The row "Every layout figure the guards hold was measured on Darwin 27"
asked for "the first push or pull request that triggers a run, and then
reading the checks' verdict". PR 8 was that run, and it answers half of
the row: both green runs passed `Suite "Bounded view layout guard"` and
`Suite "Resizable container floors"` on macOS 26.6.2, so the local
Darwin 27 figures survive a Darwin 26 runner across all 33 bounded views
and the resizable floors. By how much is still unmeasured -- both
reporters are gated, so the runs produced a verdict and no figures, and a
pass is not a margin. The row's macos-15 premise is withdrawn with the
quote, as is the now-false title half of the row it was split from: the
gate has not run macos-15 since `a774ac4d`.

Opened: the three-way toolchain split -- the gate on Swift 6.3.3, this Mac
on 6.4, the retired image on 6.1.2 -- whose first instance was the
`allLines` warning that reddened run 41045404989 at 26 occurrences on a
tree every macos-15 run had built clean. And `claude/nice-leakey-5da0b9`,
which a local branch sweep surfaced: 26 of 32 local branches were
content-identical to develop or main and were deleted, and that one
carries `ed0b70af`, a finished relaunch fix dated 2026-09-05 that sits on
neither develop nor main.

Every command written into the comment and into the rows was extracted
from the committed text and run in that form. No local suite run: the only
Swift change is four doc-comment lines, parse-checked with `swiftc -parse`
against a positive control that reported 2 errors on a broken file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he runner

CI run 41244948849 (2026-10-31, `e4e5d054`) went red on
`ConnectionDiagnosticsTests.theSSHDialNeverPutsTheSecretInTheReport`
with "a closed port did not fail the SSH dial: .timedOut". The dial
never failed: its step was settled by its own deadline before the probe
was given a thread. The case handed the runner
`stepTimeout: .seconds(10)`, and in that run 8184 of the 8307 cases
carrying a duration line reported a wall clock AT OR ABOVE 10 s (the
pipeline compares `>= 10` on the integer seconds) against a run total of
116.701 s -- so the budget bounded scheduling, not the dial.

This is `docs/BACKLOG.md`'s "Wall-clock ceilings still in the tree",
fifth shape, arriving through a production parameter the test chooses.
The same shape had already been bought one file at a time, twice:
`61850a9c` (2026-09-19) moved the jump suite's twin off `.seconds(10)`
for exactly this reason, in a commit touching one file, and
`ResolveNamesTests` had already named its own 600 s `roomy`. The five
literals in this file were never swept and fell 42 days later.

- A named `roomy` of 600 s carrying the measurement and the command
  that produces it, with all five call sites through it. Raising it
  costs those five nothing, which is what makes 600 s safe here: every
  target either answers at once (the rig) or refuses at once (a closed
  loopback port), and the skipped case never dials, so none can park on
  the larger budget.
- Both diagnostics suites' notes withdraw the clause "it is
  deliberately far larger than any step budget below", quoted in place.
  It is false of this file by construction now, and it was false of the
  jump file on the day it was written: `git blame` dates both of that
  file's 600 s sites to `61850a9c` and the sentence's heading to
  `ae19a53a`, nine days later.
- `docs/BACKLOG.md` records the shape as the eighth, what was fixed,
  and the six occurrences in `ICMPEchoTests` and `NetworkTraceTests`
  deliberately NOT swept -- five send real echo or trace probes where a
  roomy budget's cost is unmeasured, and the sixth is
  `NetworkTraceTests.traceStep`, whose own comment calls 5 s generous
  and records the 5.004 s behind it.

A second `docs/BACKLOG.md` row, "The gate's toolchain and the
maintainer's are two versions apart", is answered in part by the full
local run this commit was verified with. It withdraws "nothing has been
run to find what Swift 6.4 diagnoses that 6.3.3 does not" as far as
"nothing has been run" goes, and records what was found instead: not a
diagnostic but a rendering divergence.
`ViewTestabilitySpike`'s "Varying only isRegex leaves the pixels
unchanged" fails on this Mac (macOS 27.0.1, Swift 6.4.2) and passed on
the gate in run 41244948849 itself. Pre-existing and environmental: the
spike is in `macSCPAppKitTests` and nothing here touches that target.
Not fixed in this commit, and the row says so.

Red first, in two parts. The observed red is the CI run above. The
mechanism reproduces at a keyboard with the budget planted at
`.nanoseconds(1)`, where the deadline cannot lose: the SSH case
recorded the identical sentence, `a closed port did not fail the SSH
dial: .timedOut`, and the S3 case its own twin. Two issues in that run
and three in the FIRST one, same plant -- which case varied is not
established, because that earlier run's per-case lines were lost to a
`tail` in the command that captured it. Green at
600 s: `swift test --filter
'ConnectionDiagnostics|ICMPEcho|NetworkTrace|ResolveNames'` is 154
tests in 6 suites, 0 issues.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant