Skip to content

devkit ship --wait-ci names gh's error on an unavailable verdict (sc-7637) - #283

Merged
norvalbv merged 2 commits into
mainfrom
sc-7637-wait-ci-green-pr-gh-failed-unavailable
Nov 20, 2026
Merged

norvalbv merged 2 commits into
mainfrom
sc-7637-wait-ci-green-pr-gh-failed-unavailable

Conversation

@norvalbv

@norvalbv norvalbv commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Problem

On #1389, devkit ship --wait-ci tallied ci: 3 pass, then printed ci: gh-failed — retrying (1/3), (2/3), and ended ship: ci-outcome=unavailable pr=1389 reason=gh-failed on a PR whose checks were all green. The gh stderr was captured in WaitCiResult.unavailableMessage but printed nowhere: not on the verdict line, not on the retry lines, not in the ship_ci row. So the failing call could not be identified or classified. Under --wait-ci-strict the run exits 5.

Change

  • An unavailable verdict line now ends with detail=<gh's error>. It comes after reason=, so token-wise readers are unaffected, and is left off when gh said nothing.
  • The per-strike line names the same text: ci: gh-failed — retrying (1/3): <detail>. The gh-unreachable first-sighting line now uses the same helper; it was previously unredacted and capped at 120 chars.
  • The ship_ci telemetry row gains detail (null unless unavailable).
  • All three go through one helper, failureDetail in cli/lib/ship/wait-ci/classify.mts. It collapses whitespace, applies redactSecrets, then caps at 200 code points. The two pre-redaction raw.slice(0, 200) cuts in ghChecksPoller and head-runs readApi are removed. With pretty-printed output, they could leave a ghp_ stub too short for the redaction regex inside the visible detail.
  • Docs: ship --help, skills/using-devkit/SKILL.md (synced), and a dated note on the ship-ci-wait-is-observation-not-intent decision axis.

Not built: "return passed once checks were terminal-green"

This acceptance criterion is declined because it contradicts ruling (4) of docs/decisions/ship-ci-wait-is-observation-not-intent.md: a terminal tally must hold for the wall-clock settle grace, not merely for N polls. Once the grace holds, the wait already returns passed, so no later gh failure can override it. The remaining case is two green polls (0s, 10s) followed by three gh failures. In that case nothing was observed after the 30s grace, and returning passed would be a false pass that a --wait-ci-strict && merge script acts on. The wait stays fail-closed. The fix for a real transient is to classify its now-visible text, so the wait keeps polling to a real confirmation. A regression test locks this: stays unavailable on a green set the settle grace never confirmed, and names gh's words.

Tests

  • wait.test.mts:
    • the incident sequence: fail-closed, with detail on the verdict and retry lines
    • one line, ghp_ redaction, 200 cap, detail last
    • a token past 200 chars of unparseable output stays redacted
    • an emoji is never split at the cap
    • whitespace-only stderr adds no detail
  • telemetry.test.mts: detail is redacted, and null when not unavailable.
  • ship-wait-ci.test.mts: the spawned ship asserts the full detail= on the real verdict line.
  • Updated exact-line expectations in ci-verdict.test.mts and head-runs.test.mts.
  • Revert proof: dropping detail= fails the new verdict tests. coverage-diff: 9/9 added executable lines covered.

Follow-ups (not in this PR)

  • Classify the real gh error text once a recurrence prints it: add it to TRANSPORT, or move to status-code classification in the octokit plugin-retry style (sc-5899, which covers Go's Client.Timeout exceeded, a TRANSPORT miss and a likely cause of #1389).
  • redactSecrets' assignment pattern over-redacts diagnostic text after words like secret/token, e.g. a check named secret-scan: … (sc-8958).
  • Owner decision on the declined acceptance criterion below (sc-8957).

Not filed

  • A spawned test that reads the ship_ci row's detail end-to-end: telemetry.test.mts 'recordCiEvent detail' already makes this check, and main()'s call to recordCiEvent is unchanged by this PR.

Story: sc-7637

Reviewer advisories

Fixed: completeness (LOW): detail= also carries devkit's own reason (gh-missing, merge-conflict), so ship --help and the using-devkit skill now say "gh's own error, or devkit's reason" instead of "gh's error".

Reviewer notes declined:

  • backend-performance: failureDetail collapses and redacts a large unparseable gh output on every failed poll. Measured on 0.6 MB of pretty-printed JSON: about 6 ms per call. A wait makes at most 3 strikes plus 1 verdict call and 1 telemetry call, against a 10 s poll interval. Capping before redaction would bring back the half-token leak this PR removes.
  • correctness (raised on the advisory re-ship): the skill row reads as if detail= always appears, but it is left off when the message is empty. The wait's own messages are never empty: classifyChecksFailure falls back to e.message/String(e), and head-runs/required prefix theirs. Only ci-verdict's pre-resolution line (reason=pr-not-found) has none, and ci-verdict.test.mts locks that line exactly.
  • commit-guard: failureDetail import order in wait.test.mts. The ship's lint and oxfmt passes ran green, and the file already imported ./telemetry.mts after ./wait.mts.

🤖 Generated with Claude Code

Complete against acceptance criteria: NO — return passed when a gh failure follows a terminal green tally (sc-8957): declined, it contradicts ruling (4) of ship-ci-wait-is-observation-not-intent (settle grace, not N polls) and needs an owner decision to reopen

Automated evidence

Merge verdict: 🟡 human-review · advisory, at a7601ad3f0ac

  • decision-record
  • over-size: 48/50 lines, 10/4 files, default limit
  • no-captured-proof: inconclusive
⚠️ Regression evidence · inconclusive: evidence could not be produced: the tests did not finish within 300s (setup 9s, red 221s, green 71s unfinished)

Head a7601ad3f0ac · base unknown

Captured runs, not proof of causality. Dependencies are the shipping checkout's installed packages. Stale unless the PR head is a7601ad3f0aca1e4a496a54e1a6a008ca397cfe2.

💬 Reviewer notes · 1 note at a7601ad3f0ac · 3 at earlier commits

skills/using-devkit/SKILL.md

  • L61 — The row says an unavailable line always ends with detail=<why>, but wait.mts:168 omits detail when gh's stderr is empty, so a bare gh failure has no detail. Say "when gh gives one" instead. · correctness
3 notes at earlier commit 39387e944a62

cli/lib/ship/wait-ci/wait.mts

  • L283 — failureDetail runs on every transient failed poll, so unparseable gh output that is very large (pre-cap raw is no longer sliced) gets whitespace-collapsed and redacted in full each time. · backend-performance

cli/lib/ship/wait-ci/wait.test.mts

  • L23 — failureDetail is imported after ./wait.mts, which breaks the alphabetical import order and may trip the import-order lint. · commit-guard

Not tied to a line

  • LOW: detail= also appears for reason=gh-missing and reason=merge-conflict, where the text is devkit's own message and not gh's error. The docs call it "gh's error … naming the call that failed", so the wording is a little off for those two reasons. | skills/using-devkit/SKILL.md, cli/commands/ship.mts · completeness

…7637)

## Problem

On #1389, `devkit ship --wait-ci` tallied `ci: 3 pass`, then printed `ci: gh-failed — retrying (1/3)`, `(2/3)`, and ended `ship: ci-outcome=unavailable pr=1389 reason=gh-failed` on a PR whose checks were all green. The gh stderr was captured in `WaitCiResult.unavailableMessage` but printed nowhere: not on the verdict line, not on the retry lines, not in the `ship_ci` row. So the failing call could not be identified or classified. Under `--wait-ci-strict` the run exits 5.

## Change

- An `unavailable` verdict line now ends with `detail=<gh's error>`. It comes after `reason=`, so token-wise readers are unaffected, and is left off when gh said nothing.
- The per-strike line names the same text: `ci: gh-failed — retrying (1/3): <detail>`. The gh-unreachable first-sighting line now uses the same helper; it was previously unredacted and capped at 120 chars.
- The `ship_ci` telemetry row gains `detail` (null unless unavailable).
- All three go through one helper, `failureDetail` in `cli/lib/ship/wait-ci/classify.mts`. It collapses whitespace, applies `redactSecrets`, then caps at 200 code points. The two pre-redaction `raw.slice(0, 200)` cuts in `ghChecksPoller` and head-runs `readApi` are removed. With pretty-printed output, they could leave a `ghp_` stub too short for the redaction regex inside the visible detail.
- Docs: `ship --help`, `skills/using-devkit/SKILL.md` (synced), and a dated note on the `ship-ci-wait-is-observation-not-intent` decision axis.

## Not built: "return passed once checks were terminal-green"

This acceptance criterion is declined because it contradicts ruling (4) of `docs/decisions/ship-ci-wait-is-observation-not-intent.md`: a terminal tally must hold for the wall-clock settle grace, not merely for N polls. Once the grace holds, the wait already returns passed, so no later gh failure can override it. The remaining case is two green polls (0s, 10s) followed by three gh failures. In that case nothing was observed after the 30s grace, and returning passed would be a false pass that a `--wait-ci-strict && merge` script acts on. The wait stays fail-closed. The fix for a real transient is to classify its now-visible text, so the wait keeps polling to a real confirmation. A regression test locks this: `stays unavailable on a green set the settle grace never confirmed, and names gh's words`.

## Tests

- `wait.test.mts`:
  - the incident sequence: fail-closed, with detail on the verdict and retry lines
  - one line, `ghp_` redaction, 200 cap, detail last
  - a token past 200 chars of unparseable output stays redacted
  - an emoji is never split at the cap
  - whitespace-only stderr adds no detail
- `telemetry.test.mts`: `detail` is redacted, and null when not unavailable.
- `ship-wait-ci.test.mts`: the spawned ship asserts the full `detail=` on the real verdict line.
- Updated exact-line expectations in `ci-verdict.test.mts` and `head-runs.test.mts`.
- Revert proof: dropping `detail=` fails the new verdict tests. coverage-diff: 9/9 added executable lines covered.

## Follow-ups (not in this PR)

- Classify the real gh error text once a recurrence prints it: add it to TRANSPORT, or move to status-code classification in the octokit plugin-retry style.
- `redactSecrets`' assignment pattern over-redacts diagnostic text after words like `secret`/`token`, e.g. a check named `secret-scan: …`.
- A spawned test that reads the `ship_ci` row's `detail` end-to-end.

Story: sc-7637

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Complete against acceptance criteria: NO — return passed when a gh failure follows a terminal green tally: declined, it contradicts ruling (4) of ship-ci-wait-is-observation-not-intent (settle grace, not N polls) and needs an owner decision to reopen
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b75548c4-cbaf-45fb-a1e1-3f3617c6ed53
Commits

Reviewing files that changed from the base of the PR and between e7bfe42 and a7601ad.

Files selected for processing: 16

  • .agents/skills/using-devkit/SKILL.md
  • .claude/skills/using-devkit/SKILL.md
  • .cursor/skills/using-devkit/SKILL.md
  • .devkit/skills-manifest.json
  • cli/__tests__/ci-verdict.test.mts
  • cli/__tests__/ship-wait-ci.test.mts
  • cli/commands/ship.mts
  • cli/lib/ship/wait-ci/classify.mts
  • cli/lib/ship/wait-ci/head-runs.mts
  • cli/lib/ship/wait-ci/head-runs.test.mts
  • cli/lib/ship/wait-ci/telemetry.mts
  • cli/lib/ship/wait-ci/telemetry.test.mts
  • cli/lib/ship/wait-ci/wait.mts
  • cli/lib/ship/wait-ci/wait.test.mts
  • docs/decisions/ship-ci-wait-is-observation-not-intent.md
  • skills/using-devkit/SKILL.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…7637)

## Problem

On #1389, `devkit ship --wait-ci` tallied `ci: 3 pass`, then printed `ci: gh-failed — retrying (1/3)`, `(2/3)`, and ended `ship: ci-outcome=unavailable pr=1389 reason=gh-failed` on a PR whose checks were all green. The gh stderr was captured in `WaitCiResult.unavailableMessage` but printed nowhere: not on the verdict line, not on the retry lines, not in the `ship_ci` row. So the failing call could not be identified or classified. Under `--wait-ci-strict` the run exits 5.

## Change

- An `unavailable` verdict line now ends with `detail=<gh's error>`. It comes after `reason=`, so token-wise readers are unaffected, and is left off when gh said nothing.
- The per-strike line names the same text: `ci: gh-failed — retrying (1/3): <detail>`. The gh-unreachable first-sighting line now uses the same helper; it was previously unredacted and capped at 120 chars.
- The `ship_ci` telemetry row gains `detail` (null unless unavailable).
- All three go through one helper, `failureDetail` in `cli/lib/ship/wait-ci/classify.mts`. It collapses whitespace, applies `redactSecrets`, then caps at 200 code points. The two pre-redaction `raw.slice(0, 200)` cuts in `ghChecksPoller` and head-runs `readApi` are removed. With pretty-printed output, they could leave a `ghp_` stub too short for the redaction regex inside the visible detail.
- Docs: `ship --help`, `skills/using-devkit/SKILL.md` (synced), and a dated note on the `ship-ci-wait-is-observation-not-intent` decision axis.

## Not built: "return passed once checks were terminal-green"

This acceptance criterion is declined because it contradicts ruling (4) of `docs/decisions/ship-ci-wait-is-observation-not-intent.md`: a terminal tally must hold for the wall-clock settle grace, not merely for N polls. Once the grace holds, the wait already returns passed, so no later gh failure can override it. The remaining case is two green polls (0s, 10s) followed by three gh failures. In that case nothing was observed after the 30s grace, and returning passed would be a false pass that a `--wait-ci-strict && merge` script acts on. The wait stays fail-closed. The fix for a real transient is to classify its now-visible text, so the wait keeps polling to a real confirmation. A regression test locks this: `stays unavailable on a green set the settle grace never confirmed, and names gh's words`.

## Tests

- `wait.test.mts`:
  - the incident sequence: fail-closed, with detail on the verdict and retry lines
  - one line, `ghp_` redaction, 200 cap, detail last
  - a token past 200 chars of unparseable output stays redacted
  - an emoji is never split at the cap
  - whitespace-only stderr adds no detail
- `telemetry.test.mts`: `detail` is redacted, and null when not unavailable.
- `ship-wait-ci.test.mts`: the spawned ship asserts the full `detail=` on the real verdict line.
- Updated exact-line expectations in `ci-verdict.test.mts` and `head-runs.test.mts`.
- Revert proof: dropping `detail=` fails the new verdict tests. coverage-diff: 9/9 added executable lines covered.

## Follow-ups (not in this PR)

- Classify the real gh error text once a recurrence prints it: add it to TRANSPORT, or move to status-code classification in the octokit plugin-retry style.
- `redactSecrets`' assignment pattern over-redacts diagnostic text after words like `secret`/`token`, e.g. a check named `secret-scan: …`.
- A spawned test that reads the `ship_ci` row's `detail` end-to-end.

Story: sc-7637

## Reviewer advisories

Fixed: completeness (LOW): `detail=` also carries devkit's own reason (`gh-missing`, `merge-conflict`), so `ship --help` and the using-devkit skill now say "gh's own error, or devkit's reason" instead of "gh's error".

Reviewer notes declined:
- backend-performance: `failureDetail` collapses and redacts a large unparseable gh output on every failed poll. Measured on 0.6 MB of pretty-printed JSON: about 6 ms per call. A wait makes at most 3 strikes plus 1 verdict call and 1 telemetry call, against a 10 s poll interval. Capping before redaction would bring back the half-token leak this PR removes.
- commit-guard: `failureDetail` import order in `wait.test.mts`. The ship's lint and oxfmt passes ran green, and the file already imported `./telemetry.mts` after `./wait.mts`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Complete against acceptance criteria: NO — return passed when a gh failure follows a terminal green tally: declined, it contradicts ruling (4) of ship-ci-wait-is-observation-not-intent (settle grace, not N polls) and needs an owner decision to reopen
@norvalbv
norvalbv merged commit f3371a6 into main Nov 20, 2026
8 checks passed
@norvalbv
norvalbv deleted the sc-7637-wait-ci-green-pr-gh-failed-unavailable branch November 20, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant