Skip to content

ISS-13947: Check PR body claims and comment leftovers - #419

Merged
shafty023 merged 4 commits into
mainfrom
fix/iss-13947-takeover-review-classes
Nov 10, 2026
Merged

shafty023 merged 4 commits into
mainfrom
fix/iss-13947-takeover-review-classes

Conversation

@shafty023

@shafty023 shafty023 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

ISS-13947. This PR adds prevention for the five mistake classes the engineer takeover review of symphony-alpha PR #9732 (ISS-13897, head daaf94eff8) found after vibe's own checks passed. The PR adds the strongest check that fits each class, and shows each one failing on PR #9732's real body, diff or tsc output. Class 5 follows the team lead's 2026-11-10 ruling, recorded in ticket version 2. A ready PR with a not-run E2E lane stays allowed (ISS-13321), and the not-run reason wording is unchanged. Only honest reporting is enforced.

Changes

  • pr-preflight.mjs (classes 1, 2, 3 and 5). With --worktree it now checks four body claims:
    • A named commit must be the pushed head, or an ancestor of it whose content equals the head on every path the PR changes, measured from the merge base with origin/main. A commit on the base or origin/main counts as a main citation and passes.
    • A check claimed for a backticked workspace package needs that package to have a script with that name.
    • A promised follow-up ticket needs its slug.
    • An E2E lane described as passed needs a passing pre-pr-lanes.mjs receipt at the current source, a not-run lane is never described as passed, and every not-run lane is named as not run.
    • --worktree no longer needs --base, so env-ctl.mjs pr-update runs these checks too.
  • story-typecheck.mjs plus a pre-pr-lanes.mjs lane (class 2). A changed story whose package has no typecheck script now selects a lane. The lane runs tsc --noEmit in that package and fails on errors on lines the diff adds. It only counts errors elsewhere: symphony-alpha's storybook tsconfig already reports about 150 on main, so a whole-package verdict could never pass, and recorded as main-preexisting it would hide the new error.
  • comment-rules.mjs, wired into diff-rules.mjs (classes 3 and 4).
    • excuse-comment (blocking): an added comment that excuses a deviation with words such as "pre-existing", "defect", "on main", "workaround" or "known issue" must cite a ticket.
    • guarded-constant (blocking): a value used only under C ? or C && that reads !C itself.
    • Citation and removed-name candidates print from the CLI for the guardrails reviewer to judge. They fired on 13 of 40 recent main commits, mostly on legitimate comments, so they never block.
  • Guardrails reviewer (class 4). The leftovers check now covers removals: constant props and dead conditions, plus a comment-rules.mjs run whose candidates the reviewer judges.
  • env-ctl.mjs pr --draft. Optional, never the default.
  • Environment worker text; vibe 1.1.78 in both manifests, README, CHANGELOG and the release-pin tests.

Affected: plugins/vibe/skills/vibe/scripts/{pr-preflight,story-typecheck,pre-pr-lanes,env-ctl}.mjs and their tests, plugins/vibe/skills/handoff/scripts/{comment-rules,diff-rules}.mjs, comment-rules.test.mjs, plugins/vibe/agents/{vibe-guardrails-reviewer,vibe-environment-worker}.md, CHANGELOG.md, and the version sites. Untouched, by coordination: ISS-13846's files (PR #422, now merged: pr-facts, pr-evidence-*, handoff-inventory, test-integrity, e2e-tier, handoff/SKILL.md, references/guardrails.md), plugins/vibe-core (PR #418, now merged), pr-body-claims.mjs, handoff-gate.mjs and criteria-audit.mjs (ISS-13944, PR #421, now merged).

Validation

Head 1ef1868 merges origin/main 00e62e6 (vibe 1.1.77 and vibe-core 1.0.2, after #418 ISS-13869, #422 ISS-13846, #420 ISS-13899 and #421 ISS-13944/ISS-13949) into 4320540. vibe moves to 1.1.78 per the merge order #420, #421, #419, #423. vibe-core is untouched by this PR. pr-preflight.mjs and its test load tested-head.mjs through vibeCore, since its vibe path is now a command wrapper with no exports. None of #421's changes imports or changes a module this PR changes; its only shared dependency is session-record.mjs.

On 1ef1868, on this laptop (targeted only; the full suites run in CI):

  • Pre-push hook: Ruff and Pyright pass.
  • node --test, one file at a time: pr-preflight (9), pre-pr-lanes (11), diff-rules (41), comment-rules (5), env-ctl (23), story-typecheck (3); 92 of 92 pass.
  • pytest on test_pr_takeover_skill_contract.py and test_vibe_ws8_load_contract.py: 30 passed. vibe-bench reply-checker.test.ts: 4 of 4.
  • tools/vibe-workers npm run build: dist unchanged from the merged bundles.
  • instruction-words.mjs --base origin/main --changelog CHANGELOG.md --version 1.1.78: added 341, removed 148, net 193; matches the CHANGELOG.
  • GitHub CI on 1ef1868: 12 of 12 pass (run 42900725123), including Node Tests, Tests, Vibe Eval T0 and all five TypeScript lanes with the dist check.

On 4320540, before the merge: uv run pytest plugins/ -q 2768 passed, 3 skipped; 178 of 180 node test files pass, the other two (docker-disk-admission.test.mjs, e2e-lock.test.mjs) failing the same way on origin/main because this machine's Docker probe fails; tools/vibe-bench 198 of 198.

Failing first, against PR #9732's real artifacts:

  • pr-preflight.mjs --worktree <PR #9732 checkout> --base 6f7018c8b3 on the PR's real body: main's script returns {"ok":true,"problems":[]}. This branch returns three problems: commit e8696cbc10 cannot be resolved, storybook has no typecheck script, and the follow-up ticket promise names no slug. Citing the PR's own ancestor ea180edef3 instead reports the 6 changed paths the merge commits altered.
  • A body that says "Desktop E2E passed" with no receipt: main's script returns ok. This branch refuses it.
  • story-typecheck.mjs on PR #9732's head tsc output reports exactly the shipped TS2345 at v2-issue-detail.stories.tsx:522 and counts the 155 other diagnostics without failing. The new pre-pr-lanes test fails on main's pre-pr-lanes.mjs, which selects no story lane.
  • comment-rules on the PR #9732 diff (6f7018c8b3 to daaf94eff8) blocks the excuse comment at v2-issue-detail.stories.tsx:431 and the constant readOnly at document-artifact-canvas.tsx:117, and lists the DI-13 citation candidate at issue-v2-canvas.tsx:51 (lines 51 and 93). Removing the diff-rules.mjs call fails the wiring test. Both blocking rules fired on none of 80 recent symphony-alpha main commits.

Prevention

Prevention
- Class 1: PR body evidence names a tested commit that is not the pushed head, or that later merge commits changed.
- Already prevented? Partly. pr-facts.mjs binds the verifier runs file to the source, but no check reads commit SHAs in the body text.
- Rung: static check, pr-preflight.mjs testedCommitProblems (run by env-ctl pr and pr-update).
- Proof: PR #9732's body passes on main and is refused here (e8696cbc10 unresolvable; ea180edef3 with 6 changed paths differing).
- Follow-up: ISS-13953 (wire pr-preflight into a direct gh pr create; open-PR state, version range and bare :NNN shorthand claims).

- Class 2: a check is claimed as passed for a package with no script for it, and no lane typechecks Storybook stories.
- Already prevented? No.
- Rung: static check, pr-preflight.mjs claimedCheckProblems; plus a lane, story-typecheck.mjs, which pre-pr-lanes.mjs selects.
- Proof: the "typecheck (`storybook`)" line is refused here and passes on main; the new pre-pr-lanes test fails on main; the lane reports exactly TS2345 at :522 on PR #9732.
- Why not whole-file errors: main's storybook tsconfig already reports about 150 errors, so the lane fails only on added lines.

- Class 3: a promised follow-up ticket is never filed, and a workaround comment cites no ticket.
- Already prevented? No. The repo gate no-unticketed-workaround-comment matches other words.
- Rung: static checks, pr-preflight.mjs followUpTicketProblems and the excuse-comment diff rule.
- Proof: both fire on PR #9732's body and story comment and pass on main.
- Follow-up: ISS-13953 (confirm that the cited slug exists).

- Class 4: unreachable expressions and stale comments are left after a removal.
- Already prevented? A rule existed (guardrails reviewer, "leftovers after an extraction") and missed it.
- Rung: static check, guarded-constant (blocking); stale citations and removed names become reviewer candidates (comment-rules.mjs CLI) plus a tightened reviewer rule.
- Proof: guarded-constant blocks :117 on PR #9732 and passes on main; the candidates list the DI-13 lines.
- Why not higher for citations: they fired on 13 of 40 main commits, mostly correctly, so a human has to judge.
- Not mechanized: the wrong liveAppShellRoutes description and the duplicated route spread stay with the reviewer's adjacent-comments rule.

- Class 5: E2E lanes are reported dishonestly or ambiguously.
- Already prevented? Partly. pr-body-claims.mjs, at the handoff gate, needs a not-run statement for not-run verifier runs. Nothing tied a pass claim to a lane receipt.
- Rung: static check, pr-preflight.mjs e2eClaimProblems against pre-pr-lanes receipts at the current source.
- Proof: "Desktop E2E passed" with no receipt passes on main and is refused here; a not-run lane named as not run still opens a ready PR (ISS-13321).
- Moved: "a runtime criterion is never MET by typecheck alone" goes to ISS-13944, which owns criteria-audit.mjs.

- Event recorded: no lessons log is defined for claude-plugins.

- pr-preflight refuses a body that names a commit other than the pushed
  head or an ancestor equal to it on the changed paths, a check claimed
  for a package with no such script, a promised follow-up ticket with
  no slug, and E2E statements the session's lane receipts do not back
- story-typecheck.mjs typechecks changed stories whose package has no
  typecheck script, failing only on errors on added lines;
  pre-pr-lanes selects it when the diff needs it
- comment-rules.mjs adds the excuse-comment and guarded-constant diff
  rules and prints citation and removed-name candidates for the
  guardrails reviewer, whose leftovers check now covers removals
- env-ctl pr takes an optional --draft
- vibe 1.1.74

Testing: node --test on every plugin test file run alone, uv run pytest
plugins/, ruff, pyright, vibe-bench typecheck, test and validate; each
new check run against PR #9732's real body, diff and tsc output

Risks: a preflight claim check could refuse a body phrasing it did not
anticipate; every refusal names the line and the fix
@shafty023
shafty023 merged commit ffd0750 into main Nov 10, 2026
12 checks passed
@shafty023
shafty023 deleted the fix/iss-13947-takeover-review-classes branch November 10, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant