From d950f979f78dde04e4a648101039eed4be6f3aec Mon Sep 17 00:00:00 2001 From: norvalbv Date: Fri, 20 Nov 2026 04:00:46 +0000 Subject: [PATCH 1/2] related-tests selects the tests that read a staged doc (sc-5831) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary A commit that stages only `AGENTS.md`, `CLAUDE.md` or a `skills/*/SKILL.md` now runs the tests that read those docs in the self-host related-tests gate. It used to run none, so a broken routing doc passed every local gate and went red only in CI. Fixes [sc-5831] Fixes [sc-8769] ## Why the edge is a never-run glob The doc tests read their inputs with `readFileSync`, and vitest's import graph cannot see that. Each one now carries this line next to its reads: ```ts void (() => import.meta.glob([...], { query: '?raw' })); ``` vitest 5.0.5's `related` walk treats it as an edge. Other shapes do not work: - A real `?raw` import or an eager glob cannot run, because the suite uses Node's loader (`viteModuleRunner: false`): "Unknown file extension .md". - A string `import('x.md?raw')` crashes test-graph-reach's closure walker. - Running every test that names the staged file was rejected in sc-4734 (53 test files name SKILL.md). ## Changes - `root-routing-doc`, `doc-gate-claims`, `command-skill-routing` and `feature-critique-guidance` each declare every file they read. - `runRelatedTests` returns the tests it ran. The "not run here, CI will" note subtracts them, where before it subtracted only forced tests (the sc-8769 bug). - `PARITY_INPUT_RE` now also forces `agent-asset-projection-parity` for `agents/` sources, `.{claude,codex,cursor}/agents/` and `.devkit/{agents,agent-hooks}-manifest.json`. Before this, no commit gate checked agent projections. Skill projections stay with the existing `skill-projection` gate. This widening is deliberate. - To keep `related-tests.mts` under its 500-line cap (505 → 430), the unmapped-disclosure functions move to `related-tests-selection.mts` unchanged, apart from the `ran` parameter, and their tests move with them. - A dated note on decision axis `consumer-extra-gates`. ## Proof | Run | Result | |---|---| | A dangling backticked path added to AGENTS.md, then `node cli/lib/husky/gate-policy/related-tests.mts AGENTS.md`, before | exit 0, "no test relates to the 1 staged path(s)" | | Same, after | 4 test files, **exit 1**: `root-routing-doc > AGENTS.md names no dangling path` | | Selection for AGENTS.md / CLAUDE.md / skills/commit-gates/SKILL.md | before 0 / 0 / 0, after 3 / 3 / 3 | | `noteUnmapped` for `cli/lib/ship/ship-branch.sh` after a `_ship-branch-fixture.mts` selection | before it names ship-branch.test.mts and the 2 siblings sc-8769 reports; after those are gone and only unselected tests remain | | Edges removed, tests kept | the real-root pin `selects devkit's own doc tests …` fails | | 10 affected test files | 221/221 pass | | Added-line coverage | 43/43 (100%) | A fixture test pins the vitest behaviour under `viteModuleRunner: false`, including a newly added `SKILL.md`, because upstream vitest #11917 is still open. ## Acceptance criteria - Staging only AGENTS.md runs root-routing-doc and doc-gate-claims: yes, and command-skill-routing. - Staging only skills/commit-gates/SKILL.md runs doc-gate-claims (and command-skill-routing). Projection parity for that path is enforced by the existing `skill-projection` extraGate, which calls the same `projectionDrift(kind: 'skills')` on the commit index, rather than by forcing the parity test. - "Never 'No test files found'": the gate can no longer print it, because vitest always gets an explicit file list, or the gate returns early with "no test relates". The broader reading, running every test that names the file, is declined per sc-4734. Note that `noteUnmapped` gives no disclosure for a basename several files share, such as SKILL.md. ## Follow-ups - Other tests that read docs from disk with no edge yet: `testing-skill-runner-agnostic`, `review-asset-runtime`, `ship-reviewer-assets` (reads `agents/*.md` via `new URL`). - Deleting or renaming a file AGENTS.md names does not select root-routing-doc, because `{staged}` carries ACMR paths only. - `noteUnmapped` says nothing for shared basenames such as SKILL.md (existing behaviour). - A check that binds each test's glob to the files it reads. ## Reviewer notes declined - fallow-advisory, `gitGrepTests` complexity at `related-tests-selection.mts:176` (cyclomatic 12, CRAP 43.1 estimated): it moved unchanged from `related-tests.mts` to keep that file under its size cap. Fallow counts the moved lines as new, and its CRAP is a static estimate (Istanbul matched 0 functions). The spawned-entrypoint cases in related-tests-gate.test.mts run it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: NO — AC 3 broader reading (run every test that names a staged path): rejected by sc-4734 as too slow; the literal criterion is met --- cli/__tests__/command-skill-routing.test.mts | 13 ++ cli/__tests__/doc-gate-claims.test.mts | 13 ++ cli/__tests__/related-tests-gate.test.mts | 144 ++++++++---------- .../related-tests-selection.test.mts | 119 +++++++++++++++ cli/__tests__/root-routing-doc.test.mts | 2 + .../gate-policy/related-tests-selection.mts | 85 ++++++++++- cli/lib/husky/gate-policy/related-tests.mts | 92 ++--------- docs/decisions/consumer-extra-gates.md | 1 + .../feature-critique-guidance.test.mts | 11 ++ 9 files changed, 316 insertions(+), 164 deletions(-) diff --git a/cli/__tests__/command-skill-routing.test.mts b/cli/__tests__/command-skill-routing.test.mts index deb4e710e..580f25f5b 100644 --- a/cli/__tests__/command-skill-routing.test.mts +++ b/cli/__tests__/command-skill-routing.test.mts @@ -14,6 +14,19 @@ import { CLI, testSpawnSync } from './_helpers.mts'; const ROOT = fileURLToPath(new URL('../..', import.meta.url)); const COMMANDS_DIR = join(ROOT, 'cli', 'commands'); const SKILLS_DIR = join(ROOT, 'skills'); +// Never called: declares the docs read below to vitest's graph, so `vitest related` selects this file. +void (() => + import.meta.glob( + [ + '../../skills/**/*.md', + '../../AGENTS.md', + '../../CLAUDE.md', + '../../cli/lib/ship/commit-with-gate-capture.sh', + ], + { + query: '?raw', + }, + )); /** * Lower bound on the registry, so a parse that silently matches nothing cannot make this whole diff --git a/cli/__tests__/doc-gate-claims.test.mts b/cli/__tests__/doc-gate-claims.test.mts index 1ce4e226c..4fba47290 100644 --- a/cli/__tests__/doc-gate-claims.test.mts +++ b/cli/__tests__/doc-gate-claims.test.mts @@ -31,6 +31,19 @@ function gateCalls(text: string): string[] { return [...new Set(text.replace(WHITESPACE_RE, ' ').match(GATE_CALL_RE) ?? [])]; } +// Never called: declares sourceDocs() to vitest's graph, so `vitest related` selects this file. +void (() => + import.meta.glob( + [ + '../../AGENTS.md', + '../../CLAUDE.md', + '../../docs/troubleshooting.md', + '../../skills/*/SKILL.md', + '../../agents/*.md', + ], + { query: '?raw' }, + )); + /** The source docs an agent reads, not the `.claude` / `.cursor` projections of them. */ function sourceDocs(): string[] { const skills = readdirSync(join(ROOT, 'skills')) diff --git a/cli/__tests__/related-tests-gate.test.mts b/cli/__tests__/related-tests-gate.test.mts index af9a8f5a9..e7582ae09 100644 --- a/cli/__tests__/related-tests-gate.test.mts +++ b/cli/__tests__/related-tests-gate.test.mts @@ -11,7 +11,8 @@ import { import { tmpdir } from 'node:os'; import { dirname, join } from 'node:path'; import { stripVTControlCharacters } from 'node:util'; -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from 'vitest'; +import { createVitest } from 'vitest/node'; import { UNHANDLED_REPORTER } from '../../gate-engine/coverage/failures.mts'; import type { UnhandledError } from '../../gate-engine/coverage/unhandled-reporter.mts'; import { @@ -23,11 +24,9 @@ import { import { configExtraGates } from '../../gate-engine/deterministic/command-gates.mts'; import { listRelated, - noteUnmapped, relatedTestsReason, runRelatedTests, timingLine, - unmappedPaths, } from '../lib/husky/gate-policy/related-tests.mts'; import { triggerLine } from '../lib/husky/gate-policy/related-tests-selection.mts'; @@ -79,7 +78,7 @@ describe('runRelatedTests', () => { const log = vi.spyOn(console, 'log').mockImplementation(() => {}); const list = vi.fn(async () => ['/repo/a.test.mts']); const exec = fakeExec(0); - const none = { code: 0, interrupted: false }; + const none = { code: 0, interrupted: false, ran: [] }; expect(await runRelatedTests('/repo', { staged: [], list, exec })).toEqual(none); expect(list).not.toHaveBeenCalled(); expect(exec).not.toHaveBeenCalled(); @@ -94,6 +93,7 @@ describe('runRelatedTests', () => { expect(await runRelatedTests('/repo', { staged: ['package.json'], list, exec })).toEqual({ code: 0, interrupted: false, + ran: [], }); expect(list).not.toHaveBeenCalled(); expect(exec).not.toHaveBeenCalled(); @@ -217,6 +217,7 @@ describe('runRelatedTests', () => { expect(await runRelatedTests('/repo', { staged: ['cli/a.mts'], list, exec })).toEqual({ code: 143, interrupted: true, + ran: ['cli/a.test.mts', REACH], }); }); @@ -227,88 +228,11 @@ describe('runRelatedTests', () => { expect(await runRelatedTests('/repo', { staged: ['cli/a.mts'], list, exec })).toEqual({ code: 1, interrupted: true, + ran: ['cli/a.test.mts', REACH], }); }); }); -describe('unmappedPaths', () => { - it('keeps the paths vite cannot hold in its module graph', () => { - expect( - unmappedPaths([ - 'cli/lib/ship/reship.sh', - 'docs/x.md', - 'templates/x.hbs', - 'cli/a.mts', - 'src/b.ts', - 'dist/c.mjs', - 'src/d.tsx', - 'fixtures/e.json', - ]), - ).toEqual(['cli/lib/ship/reship.sh', 'docs/x.md', 'templates/x.hbs']); - }); -}); - -describe('noteUnmapped', () => { - type Hits = ReadonlyMap>; - function logged(paths: string[], grep: (root: string, needles: string[]) => Hits): string[] { - const log = vi.spyOn(console, 'log').mockImplementation(() => {}); - noteUnmapped('/repo', paths, grep); - return log.mock.calls.map((c) => String(c[0])); - } - - it('names each unmapped path with the tests that mention it, capped at three', () => { - const tests = ['a/w.test.mts', 'x.test.mts', 'reship-anchor.test.mts', 'b/reship.test.mts']; - const grep = vi.fn( - (_root: string, _needles: string[]): Hits => new Map([['reship.sh', tests]]), - ); - expect(logged(['cli/a.mts', 'cli/lib/ship/reship.sh', 'docs/x.md'], grep)).toEqual([ - 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', - ' cli/lib/ship/reship.sh ← reship.test.mts, reship-anchor.test.mts, w.test.mts (+1)', - ]); - }); - - it('does not report a forced test as not run', () => { - const grep = vi.fn( - (_root: string, _needles: string[]): Hits => - new Map([['reship.sh', [SCRUB, 'cli/__tests__/reship.test.mts']]]), - ); - expect(logged(['cli/lib/ship/reship.sh'], grep)).toEqual([ - 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', - ' cli/lib/ship/reship.sh ← reship.test.mts', - ]); - }); - - // A nested script forces nothing, so a test naming it really was not run and stays listed. - it('still lists the scrub test for a staged path that did not force it', () => { - const grep = vi.fn((_root: string, _needles: string[]): Hits => new Map([['x.sh', [SCRUB]]])); - expect(logged(['cli/lib/ship/review/x.sh'], grep)).toEqual([ - 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', - ' cli/lib/ship/review/x.sh ← vitest-setup-env-scrub.test.mts', - ]); - }); - - // One git lookup per run, however many paths: a docs commit stages dozens. - it('looks every unmapped basename up in one batch, once each', () => { - const grep = vi.fn((_root: string, _needles: string[]): Hits => new Map()); - logged(['cli/lib/ship/reship.sh', 'dist/cli/lib/ship/reship.sh', 'docs/x.md'], grep); - expect(grep.mock.calls).toEqual([['/repo', ['reship.sh', 'x.md']]]); - }); - - it('prints nothing for module paths or for paths no test names', () => { - const grep = vi.fn((_root: string, _needles: string[]): Hits => new Map()); - expect(logged(['cli/a.mts', 'src/b.json'], grep)).toEqual([]); - expect(grep).not.toHaveBeenCalled(); - expect(logged(['docs/x.md'], grep)).toEqual([]); - }); - - it('stays silent when the lookup fails', () => { - const grep = (): Hits => { - throw new Error('git grep failed'); - }; - expect(logged(['cli/lib/ship/reship.sh'], grep)).toEqual([]); - }); -}); - describe('related-tests entrypoint', () => { // devkit never spawns a {staged} gate with nothing staged, so no paths means a dropped token. it('blocks when it is given no paths', () => { @@ -407,6 +331,62 @@ describe('listRelated', () => { const root = graphRepo(); expect(await listRelated(root, ['a.test.mts'])).toEqual([join(root, 'a.test.mts')]); }, 60_000); + + // Pins the vitest behaviour the doc tests' never-run raw globs rely on, under the native runner. + it('selects a test whose never-run raw glob names a staged doc, new files included', async () => { + const root = graphRepo(); + writeFileSync( + join(root, 'vitest.config.mjs'), + 'export default { test: { experimental: { viteModuleRunner: false } } };\n', + ); + writeFileSync(join(root, 'doc.md'), '# doc\n'); + writeFileSync( + join(root, 'doc.test.mts'), + "import { it } from 'vitest';\nvoid (() => import.meta.glob(['./doc.md', './skills/*/SKILL.md'], { query: '?raw' }));\nit('d', () => {});\n", + ); + expect(await listRelated(root, ['doc.md'])).toEqual([join(root, 'doc.test.mts')]); + mkdirSync(join(root, 'skills', 'new'), { recursive: true }); + writeFileSync(join(root, 'skills', 'new', 'SKILL.md'), '# new\n'); + expect(await listRelated(root, ['skills/new/SKILL.md'])).toEqual([join(root, 'doc.test.mts')]); + }, 60_000); + + it("selects devkit's own doc tests for a staged AGENTS.md or SKILL.md", async () => { + const root = join(import.meta.dirname, '..', '..'); + // One context for every path, as relatedDrivers walks it: a whole-repo walk costs seconds each. + const ctx = await createVitest({ root, related: [], watch: false, run: true }); + onTestFinished(() => ctx.close()); + const rel = async (path: string) => { + ctx.config.related = [join(root, path)]; + const specs = await ctx.getRelevantTestSpecifications(); + return specs.map((s) => s.moduleId.slice(root.length + 1)); + }; + expect(await rel('AGENTS.md')).toEqual( + expect.arrayContaining([ + 'cli/__tests__/command-skill-routing.test.mts', + 'cli/__tests__/doc-gate-claims.test.mts', + 'cli/__tests__/root-routing-doc.test.mts', + ]), + ); + expect(await rel('skills/commit-gates/SKILL.md')).toEqual( + expect.arrayContaining([ + 'cli/__tests__/command-skill-routing.test.mts', + 'cli/__tests__/doc-gate-claims.test.mts', + ]), + ); + expect(await rel('agents/feature-critique.md')).toEqual( + expect.arrayContaining([ + 'cli/__tests__/doc-gate-claims.test.mts', + 'gate-engine/critique/__tests__/feature-critique-guidance.test.mts', + ]), + ); + // Each edge covers every file its test reads, not only the doc it opens by name. + const guidance = 'gate-engine/critique/__tests__/feature-critique-guidance.test.mts'; + expect(await rel('skills/testing/SKILL.md')).toContain(guidance); + expect(await rel('.claude/agents/feature-critique.md')).toContain(guidance); + expect(await rel('cli/lib/ship/commit-with-gate-capture.sh')).toContain( + 'cli/__tests__/command-skill-routing.test.mts', + ); + }, 60_000); }); describe('related-tests gate process', () => { diff --git a/cli/__tests__/related-tests-selection.test.mts b/cli/__tests__/related-tests-selection.test.mts index c05d0bb7c..118467697 100644 --- a/cli/__tests__/related-tests-selection.test.mts +++ b/cli/__tests__/related-tests-selection.test.mts @@ -14,12 +14,14 @@ import { GATE_REASON_ENV } from '../../gate-engine/deterministic/reason.mts'; import { runRelatedTests } from '../lib/husky/gate-policy/related-tests.mts'; import { forcedTests, + noteUnmapped, previousSelection, recordSelection, relatedDrivers, relatedPaths, selectionLines, triggerLine, + unmappedPaths, } from '../lib/husky/gate-policy/related-tests-selection.mts'; const dirs: string[] = []; @@ -321,6 +323,18 @@ describe('forcedTests', () => { expect(forcedTests(['vitest.setup.mjs', '.claude/hooks/x.mjs'])).toEqual([SCRUB, PARITY]); }); + // No other commit gate checks agent projections; skill projections have their own gate. + it('forces the parity test for an agent source, projection or manifest, not for skills', () => { + expect(forcedTests(['agents/correctness-reviewer.md'])).toEqual([PARITY]); + expect(forcedTests(['.cursor/agents/correctness-reviewer.md'])).toEqual([PARITY]); + expect(forcedTests(['.devkit/agents-manifest.json'])).toEqual([PARITY]); + expect(forcedTests(['.devkit/agent-hooks-manifest.json'])).toEqual([PARITY]); + expect(forcedTests(['.devkit/skills-manifest.json'])).toEqual([]); + expect( + forcedTests(['dist/agents/x.md', 'skills/a/SKILL.md', '.claude/skills/a/SKILL.md']), + ).toEqual([]); + }); + it('names forced tests that exist, so a rename cannot silently select nothing', () => { const root = join(import.meta.dirname, '..', '..'); const tests = forcedTests(['cli/lib/ship/reship.sh', 'agents-hooks/prior-art-gate.mjs']); @@ -328,3 +342,108 @@ describe('forcedTests', () => { for (const test of tests) expect(existsSync(join(root, test)), test).toBe(true); }); }); + +const REACH = 'cli/__tests__/test-graph-reach.test.mts'; + +describe('unmappedPaths', () => { + it('keeps the paths vite cannot hold in its module graph', () => { + expect( + unmappedPaths([ + 'cli/lib/ship/reship.sh', + 'docs/x.md', + 'templates/x.hbs', + 'cli/a.mts', + 'src/b.ts', + 'dist/c.mjs', + 'src/d.tsx', + 'fixtures/e.json', + ]), + ).toEqual(['cli/lib/ship/reship.sh', 'docs/x.md', 'templates/x.hbs']); + }); +}); + +describe('noteUnmapped', () => { + type Hits = ReadonlyMap>; + function logged( + paths: string[], + grep: (root: string, needles: string[]) => Hits, + ran: string[] = [], + ): string[] { + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + noteUnmapped('/repo', paths, ran, grep); + return log.mock.calls.map((c) => String(c[0])); + } + + it('names each unmapped path with the tests that mention it, capped at three', () => { + const tests = ['a/w.test.mts', 'x.test.mts', 'reship-anchor.test.mts', 'b/reship.test.mts']; + const grep = vi.fn( + (_root: string, _needles: string[]): Hits => new Map([['reship.sh', tests]]), + ); + expect(logged(['cli/a.mts', 'cli/lib/ship/reship.sh', 'docs/x.md'], grep)).toEqual([ + 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', + ' cli/lib/ship/reship.sh ← reship.test.mts, reship-anchor.test.mts, w.test.mts (+1)', + ]); + }); + + it('does not report a forced test as not run', () => { + const grep = vi.fn( + (_root: string, _needles: string[]): Hits => + new Map([['reship.sh', [SCRUB, 'cli/__tests__/reship.test.mts']]]), + ); + expect(logged(['cli/lib/ship/reship.sh'], grep, [SCRUB, REACH])).toEqual([ + 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', + ' cli/lib/ship/reship.sh ← reship.test.mts', + ]); + }); + + // A doc test selected through its glob edge ran, so only the tests naming it that did not stay. + it('does not report a graph-selected doc test as not run', () => { + const doc = 'cli/__tests__/root-routing-doc.test.mts'; + const other = 'gate-engine/judge/__tests__/codex-home.test.mts'; + const grep = vi.fn( + (_root: string, _needles: string[]): Hits => new Map([['AGENTS.md', [doc, other]]]), + ); + expect(logged(['AGENTS.md'], grep, [doc, REACH])).toEqual([ + 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', + ' AGENTS.md ← codex-home.test.mts', + ]); + }); + + it('prints nothing when every test naming the path ran', () => { + const doc = 'cli/__tests__/root-routing-doc.test.mts'; + const grep = vi.fn( + (_root: string, _needles: string[]): Hits => new Map([['AGENTS.md', [doc]]]), + ); + expect(logged(['AGENTS.md'], grep, [doc, REACH])).toEqual([]); + }); + + // A nested script forces nothing, so a test naming it really was not run and stays listed. + it('still lists the scrub test for a staged path that did not force it', () => { + const grep = vi.fn((_root: string, _needles: string[]): Hits => new Map([['x.sh', [SCRUB]]])); + expect(logged(['cli/lib/ship/review/x.sh'], grep)).toEqual([ + 'related-tests: 1 staged path(s) have no import-graph tests — not run here, CI will:', + ' cli/lib/ship/review/x.sh ← vitest-setup-env-scrub.test.mts', + ]); + }); + + // One git lookup per run, however many paths: a docs commit stages dozens. + it('looks every unmapped basename up in one batch, once each', () => { + const grep = vi.fn((_root: string, _needles: string[]): Hits => new Map()); + logged(['cli/lib/ship/reship.sh', 'dist/cli/lib/ship/reship.sh', 'docs/x.md'], grep); + expect(grep.mock.calls).toEqual([['/repo', ['reship.sh', 'x.md']]]); + }); + + it('prints nothing for module paths or for paths no test names', () => { + const grep = vi.fn((_root: string, _needles: string[]): Hits => new Map()); + expect(logged(['cli/a.mts', 'src/b.json'], grep)).toEqual([]); + expect(grep).not.toHaveBeenCalled(); + expect(logged(['docs/x.md'], grep)).toEqual([]); + }); + + it('stays silent when the lookup fails', () => { + const grep = (): Hits => { + throw new Error('git grep failed'); + }; + expect(logged(['cli/lib/ship/reship.sh'], grep)).toEqual([]); + }); +}); diff --git a/cli/__tests__/root-routing-doc.test.mts b/cli/__tests__/root-routing-doc.test.mts index fa6f8fe7c..6cd8b22c5 100644 --- a/cli/__tests__/root-routing-doc.test.mts +++ b/cli/__tests__/root-routing-doc.test.mts @@ -21,6 +21,8 @@ import { const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..'); const ROUTING_DOCS = ['CLAUDE.md', 'AGENTS.md'] as const; +// Never called: declares the docs to vitest's graph, so `vitest related` selects this file. +void (() => import.meta.glob(['../../CLAUDE.md', '../../AGENTS.md'], { query: '?raw' })); /** The KEYWORD arm of conventions-reviewer's bar (agents/conventions-reviewer.md:34-35). Its other * arm — a flat imperative like "use X, not Y" — has no mechanical detector here. */ diff --git a/cli/lib/husky/gate-policy/related-tests-selection.mts b/cli/lib/husky/gate-policy/related-tests-selection.mts index 90d02826d..cce0c5e16 100644 --- a/cli/lib/husky/gate-policy/related-tests-selection.mts +++ b/cli/lib/husky/gate-policy/related-tests-selection.mts @@ -1,12 +1,15 @@ /** * What the related-tests gate prints about its selection before vitest starts: the staged paths that * pull in the most test files, the change against the previous attempt, and the slowest selected files. + * After the run it names the staged paths no selected test covers. */ +import { spawnSync } from 'node:child_process'; import { readFileSync } from 'node:fs'; -import { join, relative } from 'node:path'; +import { basename, extname, join, relative } from 'node:path'; import { z } from 'zod'; import { emitGateEvent } from '../../../../gate-engine/judge/gate-events.mts'; import { telemetrySink } from '../../../../gate-engine/judge/run-context.mts'; +import { commitIndexEnv } from '../../../../gate-engine/ratchets/commit-index.mts'; import { scanBackward } from '../../ship/digest/tail.mts'; const SELECTION_EVENT = 'related_tests_selection'; @@ -136,10 +139,11 @@ export function recordSelection(report: string, root: string, files: number): vo const SCRUB_TEST = 'cli/__tests__/vitest-setup-env-scrub.test.mts'; const SCRUB_INPUT_RE = /^cli\/lib\/ship\/[^/]+\.sh$|^vitest\.setup\.mjs$|^vitest(\.e2e)?\.config\.mjs$/; -// The agent-asset parity test reads hook sources, their projections and the selection from disk. +// The agent-asset parity test reads agent and hook sources, their projections and the selection +// from disk. Skill projections are left to the skill-projection gate. const PARITY_TEST = 'cli/__tests__/agent-asset-projection-parity.test.mts'; const PARITY_INPUT_RE = - /^agents-hooks\/|^\.(claude|codex|cursor)\/hooks\/|^\.devkit\/config\.json$/; + /^agents(-hooks)?\/|^\.(claude|codex|cursor)\/(hooks|agents)\/|^\.devkit\/(config|agents-manifest|agent-hooks-manifest)\.json$/; const FORCED: readonly (readonly [RegExp, string])[] = [ [SCRUB_INPUT_RE, SCRUB_TEST], @@ -151,3 +155,78 @@ export function forcedTests(staged: Iterable): string[] { const paths = [...staged]; return FORCED.filter(([re]) => paths.some((p) => re.test(p))).map(([, test]) => test); } + +// Extensions vite can hold in its module graph; any other staged path never selects a test. +const MODULE_RE = /\.[cm]?[jt]sx?$|\.json$/; + +/** Staged paths `vitest related` cannot map to a test, such as shell scripts and templates. */ +export function unmappedPaths(paths: string[]): string[] { + return paths.filter((p) => !MODULE_RE.test(p)); +} + +type Grep = (root: string, needles: string[]) => ReadonlyMap>; + +// `reship.test.mts` first, then `reship-*.test.mts`, then any other test naming reship.sh. +function nameRank(test: string, stem: string): number { + if (test === `${stem}.test.mts`) return 0; + return test.startsWith(stem) ? 1 : 2; +} + +/** The commit-index test files naming each needle: one ls-files and one grep however many there are. */ +function gitGrepTests(root: string, needles: string[]): Map> { + const env = commitIndexEnv(root); + const ls = spawnSync('git', ['ls-files', '--cached', '-z'], { cwd: root, env, encoding: 'utf8' }); + const counts = new Map(); + for (const f of ls.stdout.split('\0')) { + if (f && !f.startsWith('dist/')) counts.set(basename(f), (counts.get(basename(f)) ?? 0) + 1); + } + // A basename several sources share (README.md, check.sh) names none of them; dist/ only mirrors. + const named = needles.filter((n) => (counts.get(n) ?? 0) < 2); + const hits = new Map>(); + if (!named.length) return hits; + // -o prints `file\0match` per hit; -w keeps a test naming reship.sh from naming ship.sh. + const args = ['grep', '--cached', '-zowF', ...named.flatMap((n) => ['-e', n])]; + const r = spawnSync('git', [...args, '--', ':(glob)**/*.test.mts'], { + cwd: root, + env, + encoding: 'utf8', + }); + if (r.status === 1) return hits; + if (r.status !== 0) throw new Error(r.stderr || 'git grep failed'); + for (const [file, match] of r.stdout.split('\n').map((l) => l.split('\0'))) { + if (match) hits.set(match, (hits.get(match) ?? new Set()).add(file)); + } + return hits; +} + +/** Names each unmapped staged path and the tests that mention it, minus the repo-relative `ran`. */ +export function noteUnmapped( + root: string, + paths: string[], + ran: Iterable, + grep: Grep = gitGrepTests, +): void { + try { + const unmapped = unmappedPaths(paths); + if (!unmapped.length) return; + const hits = grep(root, [...new Set(unmapped.map((p) => basename(p)))]); + const done = new Set(ran); + const lines = unmapped.flatMap((p) => { + const stem = basename(p, extname(p)); + const tests = [...(hits.get(basename(p)) ?? [])] + .filter((t) => !done.has(t)) + .map((t) => basename(t)) + .sort((a, b) => nameRank(a, stem) - nameRank(b, stem)); + if (!tests.length) return []; + const more = tests.length > 3 ? ` (+${tests.length - 3})` : ''; + return [` ${p} ← ${tests.slice(0, 3).join(', ')}${more}`]; + }); + if (!lines.length) return; + console.log( + `related-tests: ${lines.length} staged path(s) have no import-graph tests — not run here, CI will:`, + ); + console.log(lines.join('\n')); + } catch { + // Disclosure only: a failed lookup never changes the gate's verdict. + } +} diff --git a/cli/lib/husky/gate-policy/related-tests.mts b/cli/lib/husky/gate-policy/related-tests.mts index 125ef22fe..41f229a2e 100644 --- a/cli/lib/husky/gate-policy/related-tests.mts +++ b/cli/lib/husky/gate-policy/related-tests.mts @@ -6,7 +6,7 @@ import { spawn, spawnSync } from 'node:child_process'; import { mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; -import { basename, dirname, extname, isAbsolute, join, relative } from 'node:path'; +import { dirname, isAbsolute, join, relative } from 'node:path'; import { pathToFileURL } from 'node:url'; import { z } from 'zod'; import { runVitestDetailed, type VitestRun } from '../../../../gate-engine/coverage/vitest-cli.mts'; @@ -25,7 +25,6 @@ import { writeGateReason, } from '../../../../gate-engine/deterministic/reason.mts'; import { emitGateEvent } from '../../../../gate-engine/judge/gate-events.mts'; -import { commitIndexEnv } from '../../../../gate-engine/ratchets/commit-index.mts'; import { type Comparison, label, renderComparison } from '../../baseline-status/compare.mts'; import { assertVitestReport, @@ -36,6 +35,7 @@ import { import { type BaselineAnswer, resolveRef } from '../../baseline-status/query.mts'; import { forcedTests, + noteUnmapped, previousSelection, recordSelection, relatedDrivers, @@ -54,76 +54,6 @@ const NAMED_TESTS = 5; // Source modules whose reachability the reach test guards. const SOURCE_RE = /\.[cm]?[jt]sx?$/; -// Extensions vite can hold in its module graph; any other staged path never selects a test. -const MODULE_RE = /\.[cm]?[jt]sx?$|\.json$/; - -/** Staged paths `vitest related` cannot map to a test, such as shell scripts and templates. */ -export function unmappedPaths(paths: string[]): string[] { - return paths.filter((p) => !MODULE_RE.test(p)); -} - -type Grep = (root: string, needles: string[]) => ReadonlyMap>; - -// `reship.test.mts` first, then `reship-*.test.mts`, then any other test naming reship.sh. -function nameRank(test: string, stem: string): number { - if (test === `${stem}.test.mts`) return 0; - return test.startsWith(stem) ? 1 : 2; -} - -/** The commit-index test files naming each needle: one ls-files and one grep however many there are. */ -function gitGrepTests(root: string, needles: string[]): Map> { - const env = commitIndexEnv(root); - const ls = spawnSync('git', ['ls-files', '--cached', '-z'], { cwd: root, env, encoding: 'utf8' }); - const counts = new Map(); - for (const f of ls.stdout.split('\0')) { - if (f && !f.startsWith('dist/')) counts.set(basename(f), (counts.get(basename(f)) ?? 0) + 1); - } - // A basename several sources share (README.md, check.sh) names none of them; dist/ only mirrors. - const named = needles.filter((n) => (counts.get(n) ?? 0) < 2); - const hits = new Map>(); - if (!named.length) return hits; - // -o prints `file\0match` per hit; -w keeps a test naming reship.sh from naming ship.sh. - const args = ['grep', '--cached', '-zowF', ...named.flatMap((n) => ['-e', n])]; - const r = spawnSync('git', [...args, '--', ':(glob)**/*.test.mts'], { - cwd: root, - env, - encoding: 'utf8', - }); - if (r.status === 1) return hits; - if (r.status !== 0) throw new Error(r.stderr || 'git grep failed'); - for (const [file, match] of r.stdout.split('\n').map((l) => l.split('\0'))) { - if (match) hits.set(match, (hits.get(match) ?? new Set()).add(file)); - } - return hits; -} - -/** Names each unmapped staged path and the tests that mention it, since vitest related ran none. */ -export function noteUnmapped(root: string, paths: string[], grep: Grep = gitGrepTests): void { - try { - const unmapped = unmappedPaths(paths); - if (!unmapped.length) return; - const hits = grep(root, [...new Set(unmapped.map((p) => basename(p)))]); - const forced = new Set(forcedTests(paths)); - const lines = unmapped.flatMap((p) => { - const stem = basename(p, extname(p)); - const tests = [...(hits.get(basename(p)) ?? [])] - .filter((t) => !forced.has(t)) - .map((t) => basename(t)) - .sort((a, b) => nameRank(a, stem) - nameRank(b, stem)); - if (!tests.length) return []; - const more = tests.length > 3 ? ` (+${tests.length - 3})` : ''; - return [` ${p} ← ${tests.slice(0, 3).join(', ')}${more}`]; - }); - if (!lines.length) return; - console.log( - `related-tests: ${lines.length} staged path(s) have no import-graph tests — not run here, CI will:`, - ); - console.log(lines.join('\n')); - } catch { - // Disclosure only: a failed lookup never changes the gate's verdict. - } -} - interface SpawnResult { status: number | null; error?: Error; @@ -416,7 +346,10 @@ const runGrouped: Exec = (bin, args, cwd, env) => runVitestDetailed(bin, args, cwd, env, { group: true }); /** The related-tests run: code 0 when nothing is selected, vitest's outcome otherwise. */ -export async function runRelatedTests(root: string, opts: RunOpts): Promise { +export async function runRelatedTests( + root: string, + opts: RunOpts, +): Promise { const staged = [...opts.staged]; const paths = relatedPaths(staged); // From the raw set: relatedPaths drops vitest.setup.mjs, which the env-scrub test binds. @@ -424,21 +357,22 @@ export async function runRelatedTests(root: string, opts: RunOpts): Promise SOURCE_RE.test(p))) { console.log(`related-tests: no test relates to the ${paths.length} staged path(s)`); - return { code: 0, interrupted: false }; + return { code: 0, interrupted: false, ran: [] }; } // listRelated returns absolute paths, so absolutise the rest for the set to dedupe. const tests = [...new Set([...files, ...[...forced, REACH_TEST].map((t) => join(root, t))])]; + const ran = tests.map((t) => relative(root, t)); console.log( `related-tests: ${tests.length} test file(s) for ${paths.length} staged path(s)${forced.length ? `, ${forced.length} forced` : ''}`, ); @@ -469,7 +403,7 @@ export async function runRelatedTests(root: string, opts: RunOpts): Promise import.meta.glob([...], { query: '?raw' })). vitest's related walk treats it as an edge, so a staged doc now selects each test that reads it: AGENTS.md or CLAUDE.md selects root-routing-doc, doc-gate-claims and command-skill-routing, and a skills/*/SKILL.md selects doc-gate-claims, command-skill-routing and feature-critique-guidance (before, none). A real ?raw import or eager glob was rejected: the suite runs on Node's loader (viteModuleRunner false), which cannot load .md. A string import('x.md?raw') edge was rejected: test-graph-reach's closure walker parses the .md as JS. FORCED regex rows were rejected for these tests: a glob also mirrors its test's reads by hand, but it sits beside them in the same file, while a FORCED row lives in another module. related-tests-gate pins the vitest behaviour (upstream vitest #11917 is open). The 'not run here, CI will' note now subtracts every test the run selected, not only forced ones. The parity test is now also forced for agents/ sources, their projections and .devkit/agents-manifest.json, which no other commit gate checks; skill projections stay with the skill-projection gate. diff --git a/gate-engine/critique/__tests__/feature-critique-guidance.test.mts b/gate-engine/critique/__tests__/feature-critique-guidance.test.mts index f1f2bd829..7cccee7fd 100644 --- a/gate-engine/critique/__tests__/feature-critique-guidance.test.mts +++ b/gate-engine/critique/__tests__/feature-critique-guidance.test.mts @@ -19,6 +19,17 @@ const ROOT = join(import.meta.dirname, '..', '..', '..'); const agent = readFileSync(join(ROOT, 'agents', 'feature-critique.md'), 'utf8'); const skill = readFileSync(join(ROOT, 'skills', 'feature-critique', 'SKILL.md'), 'utf8'); const brainstorming = readFileSync(join(ROOT, 'skills', 'brainstorming', 'SKILL.md'), 'utf8'); +// Never called: declares every doc read below to vitest's graph, so `vitest related` selects this file. +void (() => + import.meta.glob( + [ + '../../../agents/**/*.md', + '../../../skills/**/*.md', + '../../../.claude/agents/*.md', + '../../../.cursor/agents/*.md', + ], + { query: '?raw' }, + )); const EDGE_CASES_ARTIFACT_MARKERS = { EDGE_CASES_ID: /\bEDGE_CASES_ID\b/u, flowId: /\bflowId\b/u, From 5f1112e5ccb29fd0e0fc81c99e87f30d85787df9 Mon Sep 17 00:00:00 2001 From: norvalbv Date: Fri, 20 Nov 2026 04:20:11 +0000 Subject: [PATCH 2/2] related-tests selects the tests that read a staged doc (sc-5831) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary A commit that stages only `AGENTS.md`, `CLAUDE.md` or a `skills/*/SKILL.md` now runs the tests that read those docs in the self-host related-tests gate. It used to run none, so a broken routing doc passed every local gate and went red only in CI. Fixes [sc-5831] Fixes [sc-8769] ## Why the edge is a never-run glob The doc tests read their inputs with `readFileSync`, and vitest's import graph cannot see that. Each one now carries this line next to its reads: ```ts void (() => import.meta.glob([...], { query: '?raw' })); ``` vitest 5.0.5's `related` walk treats it as an edge. Other shapes do not work: - A real `?raw` import or an eager glob cannot run, because the suite uses Node's loader (`viteModuleRunner: false`): "Unknown file extension .md". - A string `import('x.md?raw')` crashes test-graph-reach's closure walker. - Running every test that names the staged file was rejected in sc-4734 (53 test files name SKILL.md). ## Changes - `root-routing-doc`, `doc-gate-claims`, `command-skill-routing` and `feature-critique-guidance` each declare every doc they read (root-routing-doc also reads `package.json`, which is a full-suite trigger that CI runs anyway). - `runRelatedTests` returns the tests it ran. The "not run here, CI will" note subtracts them, where before it subtracted only forced tests (the sc-8769 bug). - `PARITY_INPUT_RE` now also forces `agent-asset-projection-parity` for `agents/` sources, `.{claude,codex,cursor}/agents/` and `.devkit/{agents,agent-hooks}-manifest.json`. Before this, no commit gate checked agent projections. Skill projections stay with the existing `skill-projection` gate. This widening is deliberate. - To keep `related-tests.mts` under its 500-line cap (505 → 430), the unmapped-disclosure functions move to `related-tests-selection.mts` unchanged, apart from the `ran` parameter, and their tests move with them. - A dated note on decision axis `consumer-extra-gates`. ## Proof | Run | Result | |---|---| | A dangling backticked path added to AGENTS.md, then `node cli/lib/husky/gate-policy/related-tests.mts AGENTS.md`, before | exit 0, "no test relates to the 1 staged path(s)" | | Same, after | 4 test files, **exit 1**: `root-routing-doc > AGENTS.md names no dangling path` | | Selection for AGENTS.md / CLAUDE.md / skills/commit-gates/SKILL.md | before 0 / 0 / 0, after 3 / 3 / 3 | | `noteUnmapped` for `cli/lib/ship/ship-branch.sh` after a `_ship-branch-fixture.mts` selection | before it names ship-branch.test.mts and the 2 siblings sc-8769 reports; after those are gone and only unselected tests remain | | Edges removed, tests kept | the real-root pin `selects devkit's own doc tests …` fails | | 10 affected test files | 221/221 pass | | Added-line coverage | 43/43 (100%) | A fixture test pins the vitest behaviour under `viteModuleRunner: false`, including a newly added `SKILL.md`, because upstream vitest #11917 is still open. ## Acceptance criteria - Staging only AGENTS.md runs root-routing-doc and doc-gate-claims: yes, and command-skill-routing. - Staging only skills/commit-gates/SKILL.md runs doc-gate-claims (and command-skill-routing). Projection parity for that path is enforced by the existing `skill-projection` extraGate, which calls the same `projectionDrift(kind: 'skills')` on the commit index, rather than by forcing the parity test. - "Never 'No test files found'": the gate can no longer print it, because vitest always gets an explicit file list, or the gate returns early with "no test relates". The broader reading, running every test that names the file, is declined per sc-4734. Note that `noteUnmapped` gives no disclosure for a basename several files share, such as SKILL.md. ## Follow-ups - Other tests that read docs from disk with no edge yet: `testing-skill-runner-agnostic`, `review-asset-runtime`, `ship-reviewer-assets` (reads `agents/*.md` via `new URL`). - Deleting or renaming a file AGENTS.md names does not select root-routing-doc, because `{staged}` carries ACMR paths only. - `noteUnmapped` says nothing for shared basenames such as SKILL.md (existing behaviour). - A check that binds each test's glob to the files it reads. ## Reviewer notes declined - fallow-advisory, `gitGrepTests` complexity at `related-tests-selection.mts:176` (cyclomatic 12, CRAP 43.1 estimated): it moved unchanged from `related-tests.mts` to keep that file under its size cap. Fallow counts the moved lines as new, and its CRAP is a static estimate (Istanbul matched 0 functions). The spawned-entrypoint cases in related-tests-gate.test.mts run it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Complete against acceptance criteria: NO — AC 3 broader reading (run every test that names a staged path): rejected by sc-4734 as too slow; the literal criterion is met --- docs/decisions/consumer-extra-gates.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/consumer-extra-gates.md b/docs/decisions/consumer-extra-gates.md index 2c5fd4e24..b9c4aad42 100644 --- a/docs/decisions/consumer-extra-gates.md +++ b/docs/decisions/consumer-extra-gates.md @@ -44,4 +44,4 @@ created: 2026-10-08 - 2026-10-17 — related-tests runs vitest through runVitestDetailed with { group: true }: vitest leads its own process group and the wrapper forwards SIGHUP, SIGINT, SIGQUIT and SIGTERM to it. A blocking spawnSync could not forward, so a signal to the wrapper alone orphaned vitest, and forwarding to vitest's main process alone still orphans its fork workers, since vitest exits without closing its pool. The cost: a SIGKILL of the hook's group no longer reaches vitest, so a deadline must SIGTERM the wrapper and let it forward. Under ship the gate supervisor still reaps the detached group through the ownership token. - 2026-11-18 — Every extraGates command now runs with DEVKIT_GATE_LABEL set to its label, beside DEVKIT_GATE_REASON_FILE. Base attribution no longer reruns a failed command containing {staged}: the staged set does not exist at the cut commit, and the literal placeholder made a gate like 'oxlint {staged}' print a false 'also exits 1' (sc-8567). The exception is a gate that publishes its own narrower rerun as a gate_base_rerun telemetry event keyed by that label; devkit's self-host related-tests is the only one that does. - 2026-11-18 — Before vitest starts, the self-host related-tests gate now prints three things beside its selection count. First, the staged paths that select the most test files: one vitest context, walked only for 20 or more selected files across 2 or more paths. Second, the previous attempt's selection size for the same repo and branch, when it differs. Third, the previous attempt's slowest files among this selection. Each finished run inside a ship records a related_tests_selection telemetry row (count plus its 10 slowest files) for the next attempt. There is no total time estimate: whole-gate duration does not track file count (13 files took 607 s, 179 files took 1120 s). A pre-queue preview was rejected (sc-7280; the owner chose this gate-time shape on 2026-11-18). The pre-queue pass discards its output on success, and it cannot resolve the paths of a --resume or --from-branch ship. Consumer gates are unaffected. -- 2026-11-20 — sc-5831 — A test that reads a doc from disk declares that input to the self-host related-tests gate with a never-run lazy glob beside its read: void (() => import.meta.glob([...], { query: '?raw' })). vitest's related walk treats it as an edge, so a staged doc now selects each test that reads it: AGENTS.md or CLAUDE.md selects root-routing-doc, doc-gate-claims and command-skill-routing, and a skills/*/SKILL.md selects doc-gate-claims, command-skill-routing and feature-critique-guidance (before, none). A real ?raw import or eager glob was rejected: the suite runs on Node's loader (viteModuleRunner false), which cannot load .md. A string import('x.md?raw') edge was rejected: test-graph-reach's closure walker parses the .md as JS. FORCED regex rows were rejected for these tests: a glob also mirrors its test's reads by hand, but it sits beside them in the same file, while a FORCED row lives in another module. related-tests-gate pins the vitest behaviour (upstream vitest #11917 is open). The 'not run here, CI will' note now subtracts every test the run selected, not only forced ones. The parity test is now also forced for agents/ sources, their projections and .devkit/agents-manifest.json, which no other commit gate checks; skill projections stay with the skill-projection gate. +- 2026-11-20 — sc-5831 — A test that reads a doc from disk declares that input to the self-host related-tests gate with a never-run lazy glob beside its read: void (() => import.meta.glob([...], { query: '?raw' })). vitest's related walk treats it as an edge, so a staged doc now selects each test that reads it: AGENTS.md or CLAUDE.md selects root-routing-doc, doc-gate-claims and command-skill-routing, and a skills/*/SKILL.md selects doc-gate-claims, command-skill-routing and feature-critique-guidance (before, none). A real ?raw import or eager glob was rejected: the suite runs on Node's loader (viteModuleRunner false), which cannot load .md. A string import('x.md?raw') edge was rejected: test-graph-reach's closure walker parses the .md as JS. FORCED regex rows were rejected for these tests: a glob also mirrors its test's reads by hand, but it sits beside them in the same file, while a FORCED row lives in another module. related-tests-gate pins the vitest behaviour (upstream vitest #11917 is open). The 'not run here, CI will' note now subtracts every test the run selected, not only forced ones. The parity test is now also forced for agents/ sources, their projections and .devkit/agents-manifest.json and .devkit/agent-hooks-manifest.json, which no other commit gate checks; skill projections stay with the skill-projection gate.