Repository navigation
devkit ship --wait-ci names gh's error on an unavailable verdict (sc-7637) - #283
Merged
Merged
Conversation
…7637) ## Problem On #1389, `devkit ship --wait-ci` tallied `ci: 3 pass`, then printed `ci: gh-failed — retrying (1/3)`, `(2/3)`, and ended `ship: ci-outcome=unavailable pr=1389 reason=gh-failed` on a PR whose checks were all green. The gh stderr was captured in `WaitCiResult.unavailableMessage` but printed nowhere: not on the verdict line, not on the retry lines, not in the `ship_ci` row. So the failing call could not be identified or classified. Under `--wait-ci-strict` the run exits 5. ## Change - An `unavailable` verdict line now ends with `detail=<gh's error>`. It comes after `reason=`, so token-wise readers are unaffected, and is left off when gh said nothing. - The per-strike line names the same text: `ci: gh-failed — retrying (1/3): <detail>`. The gh-unreachable first-sighting line now uses the same helper; it was previously unredacted and capped at 120 chars. - The `ship_ci` telemetry row gains `detail` (null unless unavailable). - All three go through one helper, `failureDetail` in `cli/lib/ship/wait-ci/classify.mts`. It collapses whitespace, applies `redactSecrets`, then caps at 200 code points. The two pre-redaction `raw.slice(0, 200)` cuts in `ghChecksPoller` and head-runs `readApi` are removed. With pretty-printed output, they could leave a `ghp_` stub too short for the redaction regex inside the visible detail. - Docs: `ship --help`, `skills/using-devkit/SKILL.md` (synced), and a dated note on the `ship-ci-wait-is-observation-not-intent` decision axis. ## Not built: "return passed once checks were terminal-green" This acceptance criterion is declined because it contradicts ruling (4) of `docs/decisions/ship-ci-wait-is-observation-not-intent.md`: a terminal tally must hold for the wall-clock settle grace, not merely for N polls. Once the grace holds, the wait already returns passed, so no later gh failure can override it. The remaining case is two green polls (0s, 10s) followed by three gh failures. In that case nothing was observed after the 30s grace, and returning passed would be a false pass that a `--wait-ci-strict && merge` script acts on. The wait stays fail-closed. The fix for a real transient is to classify its now-visible text, so the wait keeps polling to a real confirmation. A regression test locks this: `stays unavailable on a green set the settle grace never confirmed, and names gh's words`. ## Tests - `wait.test.mts`: - the incident sequence: fail-closed, with detail on the verdict and retry lines - one line, `ghp_` redaction, 200 cap, detail last - a token past 200 chars of unparseable output stays redacted - an emoji is never split at the cap - whitespace-only stderr adds no detail - `telemetry.test.mts`: `detail` is redacted, and null when not unavailable. - `ship-wait-ci.test.mts`: the spawned ship asserts the full `detail=` on the real verdict line. - Updated exact-line expectations in `ci-verdict.test.mts` and `head-runs.test.mts`. - Revert proof: dropping `detail=` fails the new verdict tests. coverage-diff: 9/9 added executable lines covered. ## Follow-ups (not in this PR) - Classify the real gh error text once a recurrence prints it: add it to TRANSPORT, or move to status-code classification in the octokit plugin-retry style. - `redactSecrets`' assignment pattern over-redacts diagnostic text after words like `secret`/`token`, e.g. a check named `secret-scan: …`. - A spawned test that reads the `ship_ci` row's `detail` end-to-end. Story: sc-7637 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: NO — return passed when a gh failure follows a terminal green tally: declined, it contradicts ruling (4) of ship-ci-wait-is-observation-not-intent (settle grace, not N polls) and needs an owner decision to reopen
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 1 minute. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…7637) ## Problem On #1389, `devkit ship --wait-ci` tallied `ci: 3 pass`, then printed `ci: gh-failed — retrying (1/3)`, `(2/3)`, and ended `ship: ci-outcome=unavailable pr=1389 reason=gh-failed` on a PR whose checks were all green. The gh stderr was captured in `WaitCiResult.unavailableMessage` but printed nowhere: not on the verdict line, not on the retry lines, not in the `ship_ci` row. So the failing call could not be identified or classified. Under `--wait-ci-strict` the run exits 5. ## Change - An `unavailable` verdict line now ends with `detail=<gh's error>`. It comes after `reason=`, so token-wise readers are unaffected, and is left off when gh said nothing. - The per-strike line names the same text: `ci: gh-failed — retrying (1/3): <detail>`. The gh-unreachable first-sighting line now uses the same helper; it was previously unredacted and capped at 120 chars. - The `ship_ci` telemetry row gains `detail` (null unless unavailable). - All three go through one helper, `failureDetail` in `cli/lib/ship/wait-ci/classify.mts`. It collapses whitespace, applies `redactSecrets`, then caps at 200 code points. The two pre-redaction `raw.slice(0, 200)` cuts in `ghChecksPoller` and head-runs `readApi` are removed. With pretty-printed output, they could leave a `ghp_` stub too short for the redaction regex inside the visible detail. - Docs: `ship --help`, `skills/using-devkit/SKILL.md` (synced), and a dated note on the `ship-ci-wait-is-observation-not-intent` decision axis. ## Not built: "return passed once checks were terminal-green" This acceptance criterion is declined because it contradicts ruling (4) of `docs/decisions/ship-ci-wait-is-observation-not-intent.md`: a terminal tally must hold for the wall-clock settle grace, not merely for N polls. Once the grace holds, the wait already returns passed, so no later gh failure can override it. The remaining case is two green polls (0s, 10s) followed by three gh failures. In that case nothing was observed after the 30s grace, and returning passed would be a false pass that a `--wait-ci-strict && merge` script acts on. The wait stays fail-closed. The fix for a real transient is to classify its now-visible text, so the wait keeps polling to a real confirmation. A regression test locks this: `stays unavailable on a green set the settle grace never confirmed, and names gh's words`. ## Tests - `wait.test.mts`: - the incident sequence: fail-closed, with detail on the verdict and retry lines - one line, `ghp_` redaction, 200 cap, detail last - a token past 200 chars of unparseable output stays redacted - an emoji is never split at the cap - whitespace-only stderr adds no detail - `telemetry.test.mts`: `detail` is redacted, and null when not unavailable. - `ship-wait-ci.test.mts`: the spawned ship asserts the full `detail=` on the real verdict line. - Updated exact-line expectations in `ci-verdict.test.mts` and `head-runs.test.mts`. - Revert proof: dropping `detail=` fails the new verdict tests. coverage-diff: 9/9 added executable lines covered. ## Follow-ups (not in this PR) - Classify the real gh error text once a recurrence prints it: add it to TRANSPORT, or move to status-code classification in the octokit plugin-retry style. - `redactSecrets`' assignment pattern over-redacts diagnostic text after words like `secret`/`token`, e.g. a check named `secret-scan: …`. - A spawned test that reads the `ship_ci` row's `detail` end-to-end. Story: sc-7637 ## Reviewer advisories Fixed: completeness (LOW): `detail=` also carries devkit's own reason (`gh-missing`, `merge-conflict`), so `ship --help` and the using-devkit skill now say "gh's own error, or devkit's reason" instead of "gh's error". Reviewer notes declined: - backend-performance: `failureDetail` collapses and redacts a large unparseable gh output on every failed poll. Measured on 0.6 MB of pretty-printed JSON: about 6 ms per call. A wait makes at most 3 strikes plus 1 verdict call and 1 telemetry call, against a 10 s poll interval. Capping before redaction would bring back the half-token leak this PR removes. - commit-guard: `failureDetail` import order in `wait.test.mts`. The ship's lint and oxfmt passes ran green, and the file already imported `./telemetry.mts` after `./wait.mts`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: NO — return passed when a gh failure follows a terminal green tally: declined, it contradicts ruling (4) of ship-ci-wait-is-observation-not-intent (settle grace, not N polls) and needs an owner decision to reopen
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On #1389,
devkit ship --wait-citalliedci: 3 pass, then printedci: gh-failed — retrying (1/3),(2/3), and endedship: ci-outcome=unavailable pr=1389 reason=gh-failedon a PR whose checks were all green. The gh stderr was captured inWaitCiResult.unavailableMessagebut printed nowhere: not on the verdict line, not on the retry lines, not in theship_cirow. So the failing call could not be identified or classified. Under--wait-ci-strictthe run exits 5.Change
unavailableverdict line now ends withdetail=<gh's error>. It comes afterreason=, so token-wise readers are unaffected, and is left off when gh said nothing.ci: gh-failed — retrying (1/3): <detail>. The gh-unreachable first-sighting line now uses the same helper; it was previously unredacted and capped at 120 chars.ship_citelemetry row gainsdetail(null unless unavailable).failureDetailincli/lib/ship/wait-ci/classify.mts. It collapses whitespace, appliesredactSecrets, then caps at 200 code points. The two pre-redactionraw.slice(0, 200)cuts inghChecksPollerand head-runsreadApiare removed. With pretty-printed output, they could leave aghp_stub too short for the redaction regex inside the visible detail.ship --help,skills/using-devkit/SKILL.md(synced), and a dated note on theship-ci-wait-is-observation-not-intentdecision axis.Not built: "return passed once checks were terminal-green"
This acceptance criterion is declined because it contradicts ruling (4) of
docs/decisions/ship-ci-wait-is-observation-not-intent.md: a terminal tally must hold for the wall-clock settle grace, not merely for N polls. Once the grace holds, the wait already returns passed, so no later gh failure can override it. The remaining case is two green polls (0s, 10s) followed by three gh failures. In that case nothing was observed after the 30s grace, and returning passed would be a false pass that a--wait-ci-strict && mergescript acts on. The wait stays fail-closed. The fix for a real transient is to classify its now-visible text, so the wait keeps polling to a real confirmation. A regression test locks this:stays unavailable on a green set the settle grace never confirmed, and names gh's words.Tests
wait.test.mts:ghp_redaction, 200 cap, detail lasttelemetry.test.mts:detailis redacted, and null when not unavailable.ship-wait-ci.test.mts: the spawned ship asserts the fulldetail=on the real verdict line.ci-verdict.test.mtsandhead-runs.test.mts.detail=fails the new verdict tests. coverage-diff: 9/9 added executable lines covered.Follow-ups (not in this PR)
Client.Timeout exceeded, a TRANSPORT miss and a likely cause of #1389).redactSecrets' assignment pattern over-redacts diagnostic text after words likesecret/token, e.g. a check namedsecret-scan: …(sc-8958).Not filed
ship_cirow'sdetailend-to-end:telemetry.test.mts'recordCiEvent detail' already makes this check, andmain()'s call torecordCiEventis unchanged by this PR.Story: sc-7637
Reviewer advisories
Fixed: completeness (LOW):
detail=also carries devkit's own reason (gh-missing,merge-conflict), soship --helpand the using-devkit skill now say "gh's own error, or devkit's reason" instead of "gh's error".Reviewer notes declined:
failureDetailcollapses and redacts a large unparseable gh output on every failed poll. Measured on 0.6 MB of pretty-printed JSON: about 6 ms per call. A wait makes at most 3 strikes plus 1 verdict call and 1 telemetry call, against a 10 s poll interval. Capping before redaction would bring back the half-token leak this PR removes.detail=always appears, but it is left off when the message is empty. The wait's own messages are never empty:classifyChecksFailurefalls back toe.message/String(e), and head-runs/required prefix theirs. Onlyci-verdict's pre-resolution line (reason=pr-not-found) has none, andci-verdict.test.mtslocks that line exactly.failureDetailimport order inwait.test.mts. The ship's lint and oxfmt passes ran green, and the file already imported./telemetry.mtsafter./wait.mts.🤖 Generated with Claude Code
Complete against acceptance criteria: NO — return passed when a gh failure follows a terminal green tally (sc-8957): declined, it contradicts ruling (4) of ship-ci-wait-is-observation-not-intent (settle grace, not N polls) and needs an owner decision to reopen
Automated evidence
Merge verdict: 🟡 human-review · advisory, at
a7601ad3f0acHead
a7601ad3f0ac· base unknownCaptured runs, not proof of causality. Dependencies are the shipping checkout's installed packages. Stale unless the PR head is
a7601ad3f0aca1e4a496a54e1a6a008ca397cfe2.💬 Reviewer notes · 1 note at
a7601ad3f0ac· 3 at earlier commitsskills/using-devkit/SKILL.mdunavailableline always ends withdetail=<why>, butwait.mts:168omits detail when gh's stderr is empty, so a bare gh failure has no detail. Say "when gh gives one" instead. · correctness3 notes at earlier commit
39387e944a62cli/lib/ship/wait-ci/wait.mtsfailureDetailruns on every transient failed poll, so unparseable gh output that is very large (pre-caprawis no longer sliced) gets whitespace-collapsed and redacted in full each time. · backend-performancecli/lib/ship/wait-ci/wait.test.mtsfailureDetailis imported after./wait.mts, which breaks the alphabetical import order and may trip the import-order lint. · commit-guardNot tied to a line
detail=also appears forreason=gh-missingandreason=merge-conflict, where the text is devkit's own message and not gh's error. The docs call it "gh's error … naming the call that failed", so the wording is a little off for those two reasons. | skills/using-devkit/SKILL.md, cli/commands/ship.mts · completeness