Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
#
# Two jobs, split by cost. Roughly a third of the suite drives a real browser,
# and downloading Chrome dominates that job's runtime — so the Node-only tests
# run on their own and report in a couple of minutes, which is the feedback
# that matters for almost every change. See docs/specs/CI-SPEC.md.
# (test/unit) run on their own and report in a couple of minutes, which is the
# feedback that matters for almost every change. test/browser and test/e2e run
# in the browser job. See docs/specs/CI-SPEC.md.
#
# No secrets are needed. The tests that would use an LLM API key skip when one
# is absent, so this stays useful on pull requests from forks.
Expand Down Expand Up @@ -43,6 +44,8 @@ jobs:
# `tsc` with noEmit off is the type check — it is the same compile the
# published package depends on, so a separate lint step would be noise.
- run: npm run build
# tsx runs the tests without checking their types; this does.
- run: npm run typecheck:tests
- run: npm run test:fast

browser:
Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1404,10 +1404,11 @@ npm run build
| `npm run build` | Compile TypeScript to `dist/`. Required before the installed MCP server picks up changes — it loads `dist/index.js`. |
| `npm run dev` | Run the server directly via `tsx` for local iteration. Does not affect the registered MCP server. |
| `npm run viewer [port]` | Start the standalone viewer (default auto-picks from 3001). |
| `npm test` | Run every test that needs only Node (~2 min). Alias for `test:fast`. |
| `npm run test:full` | The above plus the browser-driven tests. Needs Chrome — `npx puppeteer browsers install chrome`. |
| `npx tsx scripts/run-tests.ts <substring>` | Run just the tests whose filename matches. |
| `npx tsx test-*.ts` | Run a single test script directly. |
| `npm test` | Run `test/unit`, every test that needs only Node (~2 min). Alias for `test:fast`. |
| `npm run test:full` | The above plus `test/browser` and `test/e2e` (the tests that drive the MCP server or CLI). Needs Chrome — `npx puppeteer browsers install chrome`. |
| `npm run typecheck:tests` | Type-check the tests (`tsx` runs them without checking types). |
| `npx tsx scripts/run-tests.ts <substring>` | Run just the tests whose path matches, such as `cliche` or `unit/`. |
| `npx tsx test/unit/test-cliche.ts` | Run a single test script directly, from the repo root. |

### Env vars

Expand Down
8 changes: 4 additions & 4 deletions VISION.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ framesmith catches slop but doesn't yet proactively produce taste: from a blank
- [x] Pattern library — expanded `src/structures.ts` to 11 vetted page archetypes (added dashboard, auth, pricing, settings, onboarding; landing≈marquee-hero, data-screen≈split-workbench/catalogue) + elevated the existing ones to the bar
- [x] Craft guidance — a "Designing with taste" section (focal point, hierarchy, type + spacing scale, restraint, pattern-first) on the agent surfaces (GUIDELINES + INSTRUCTIONS + cheatsheet), paired with the anti-slop tells
- [x] Relentless loop — the core loop on every agent surface now reads generate → evaluate → fix/revise → repeat until > 95 with no cliché tells before presenting
- [x] Library quality gate — `test-patterns.ts`: every page structure regression-tested to > 95 with zero cliché tells across all 5 presets (default/dark/light/material/minimal)
- [x] Library quality gate — `test/unit/test-patterns.ts`: every page structure regression-tested to > 95 with zero cliché tells across all 5 presets (default/dark/light/material/minimal)

_Full spec-driven breakdown in [`docs/specs/PHASE-20-SPEC.md`](docs/specs/PHASE-20-SPEC.md). Built across PRs; Slices A (library + gate) and B/D (guidance + loop) shipped._

Expand Down Expand Up @@ -551,7 +551,7 @@ The mobile attempt (2026-09-07) built one iOS screen end to end on v2.2.0 and go
- [x] Slice D — vocabulary: `fixed` / `sticky` positioning with the renderer reserving room for pinned chrome, a root that grows with long content (closing slice C's long-screen gap), the `status-bar`, `home-indicator`, `nav-bar`, `tab-bar`, `button`, `list-row` and `sheet` primitives, the primitives joining the 44px target rule, `mobile-tab-bar`, and pinned bars in the safe-area rule (`safeArea` on the root landed in slice B)
- [x] Slice E — starting points: anatomy and realisation on structures, resolution by canvas type, six mobile realisations, and the pattern gate extended to them
- [x] Slice F — the viewer: thumbnails at the artboard's real proportions and a device frame with a safe-area overlay
- [x] Slice G — the acceptance re-run: Ridgeline Today rebuilt from `dashboard-mobile` in 33 nodes (93 by hand), 100/100 and READY in fast and detailed mode with no dark-theme finding, `canvas_stress` CLEAN, every tap target a primitive, pinned as `test-ridgeline.ts`; the re-run surfaced two render defects, fixed here: large type now scales against the artboard's own width (a phone's titles rendered at 60%, under the body), and chart tick labels no longer stretch with the plot
- [x] Slice G — the acceptance re-run: Ridgeline Today rebuilt from `dashboard-mobile` in 33 nodes (93 by hand), 100/100 and READY in fast and detailed mode with no dark-theme finding, `canvas_stress` CLEAN, every tap target a primitive, pinned as `test/e2e/test-ridgeline.ts`; the re-run surfaced two render defects, fixed here: large type now scales against the artboard's own width (a phone's titles rendered at 60%, under the body), and chart tick labels no longer stretch with the plot

All seven slices are merged and the acceptance re-run passed. The phase is closed.

Expand All @@ -575,7 +575,7 @@ Run on its own viewer (2026-11-08), framesmith scored every page's usability 100
- [ ] Follow-up — a disabled control inside a component instance renders undimmed, because instances render after variable resolution (`resolveInstance`); resolve instance subtrees too, so `disabled` and `state` dim inside components as they do outside
- [x] Slice H — state checks: focus rings, active indicators, disabled renders and control boundaries checked analytically on every interactive primitive, without being drawn, in each theme; text inside a disabled target is exempt from text contrast on every type
- [ ] Follow-up — `select` is not in the boundary check, though its `$border` is its only boundary too; and `tree-row` and the contained `tabs` draw no indicator mark when active (a tint and a heavier weight only), so the active-indicator check has nothing to measure on them
- [x] Slice I — the acceptance re-run: the viewer's gallery, detail with inspector and archive rebuilt as canvases, with the comment popover and delete-confirm states and the empty archive, all six CLEAN in fast and detailed mode in both themes with a CLEAN stress run, and the six baseline defects fixed (`test-viewer-rebuild.ts`); detailed evaluate now lays out in the design system's own fonts, and a drawer or a scrolling popover reads as scrolling under stress
- [x] Slice I — the acceptance re-run: the viewer's gallery, detail with inspector and archive rebuilt as canvases, with the comment popover and delete-confirm states and the empty archive, all six CLEAN in fast and detailed mode in both themes with a CLEAN stress run, and the six baseline defects fixed (`test/e2e/test-viewer-rebuild.ts`); detailed evaluate now lays out in the design system's own fonts, and a drawer or a scrolling popover reads as scrolling under stress
- [ ] Follow-up — a misspelt `$token` in a non-colour prop (`gap: "$space-xxs"`, which does not exist) resolves to nothing and evaluate says nothing; the slice I rebuild caught one only through its own resolved-tree check. Warn on any `$ref` the merged tokens do not define
- [ ] Follow-up — `nav-item.count` is not a stress slot, so a sidebar count of "999+" is never exercised; add it (and `tabs.items[].count`) to the big-numbers perturbation

Expand All @@ -587,7 +587,7 @@ _Full spec-driven breakdown in [`docs/specs/PHASE-31-SPEC.md`](docs/specs/PHASE-

Phases 30 and 31 closed with seven follow-ups open, and five of them are quiet false passes: evaluate says READY on a disabled button that renders undimmed inside a component, on a misspelt `$token` the renderer drops, on a `select` border at 1.46:1, and on a phone screen whose links, text inputs and dialogs are never measured. Reproducing them found more: contrast and spacing never look inside a component instance at all, and seeded dark tints and `danger` survive a light preset. This phase clears every follow-up, and first moves the 140 root-level tests into folders that set their tier, so the fixes land in the new layout.

- [ ] Slice A — tests in folders: `test/unit`, `test/browser`, `test/e2e` and `test/manual` set the tier, a Chrome guard on `test/unit` proves it at run time, tests are type-checked under `strict`, and files are named by subject, not phase or issue
- [x] Slice A — tests in folders: `test/unit`, `test/browser`, `test/e2e` and `test/manual` set the tier, a Chrome guard on `test/unit` proves it at run time, tests are type-checked under `strict`, and files are named by subject, not phase or issue
- [ ] Slice B — one tree: instances expanded and resolved once, so the renderer and every rule (contrast, spacing and cliché included) see the same thing inside a component, and findings there name the instance
- [ ] Slice C — tokens: an `unknown-token` warning for any `$ref` the merged tokens do not define or that is the wrong kind, with a rename autofix when only one name is close; copy props never resolve; tints and `danger` derive from the applied palette, on new stamps and on canvases holding the old seeds
- [ ] Slice D — desktop-app checks: `select` gets a `$border-strong` boundary and joins the boundary check, an active `tree-row` draws a bar the indicator check measures, and stress widens `nav-item` and `tabs` counts
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
"bench": "tsx benchmark/run.ts",
"test": "npm run test:fast",
"test:fast": "tsx scripts/run-tests.ts",
"test:full": "tsx scripts/run-tests.ts --full"
"test:full": "tsx scripts/run-tests.ts --full",
"typecheck:tests": "tsc -p tsconfig.test.json"
},
"dependencies": {
"@anthropic-ai/sdk": "^0.96.0",
Expand Down
134 changes: 69 additions & 65 deletions scripts/run-tests.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,70 +2,65 @@
//
// Every test in this repo is a plain script: it prints PASS/FAIL lines and
// exits 0 or 1. That convention works and this runner does not change it — it
// just discovers the scripts, runs them, and reports. Before it existed the 102
// just discovers the scripts, runs them, and reports. Before it existed the
// test files could only be run one at a time by hand, which is how two of them
// came to be failing at the v2.0.0 tag without anyone noticing.
//
// Tiers exist so the common case is fast. Roughly a third of the suite drives a
// real browser; those tests are worth having but they should not stand between
// you and finding out that a pure-logic change broke something.
// The folder a test lives in sets its tier:
//
// npm run test:fast — everything that needs nothing but Node
// npm run test:full — the above plus the browser tests
// test/unit Node only. Run with Chrome made unlaunchable, so a unit test
// that reaches for a browser fails instead of passing quietly.
// test/browser launches Chrome
// test/e2e spawns the MCP server or the CLI
// test/manual needs a human or a running viewer; never run here
//
// npm run test:fast — test/unit
// npm run test:full — test/unit, test/browser and test/e2e
// npm test — test:fast
//
// Pass names or substrings to run a subset: `npx tsx scripts/run-tests.ts cliche`
import { readdirSync, readFileSync } from 'node:fs';
// Pass substrings of a test's path to run a subset: `npx tsx scripts/run-tests.ts cliche`
// or `unit/` for a folder. `--root <dir>` reads the folders from another directory
// (the runner's own tests use it).
import { existsSync, readdirSync } from 'node:fs';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { spawn, type ChildProcess } from 'node:child_process';

/** Tests that never terminate on their own and are not batch-runnable. */
const INTERACTIVE = new Set([
// Starts a viewer server and waits for a human to look at it. Running it
// here would hang the suite forever, and it also collides with a live viewer
// on the same port.
'test-viewer.ts',
// Expects a viewer already serving on a port passed as argv (its header says
// `npx tsx test-viewer-navbar.ts <port>`) and writes screenshots for a human
// to look at. Nothing here can supply that, so it stays a manual check.
'test-viewer-navbar.ts',
]);

/** Tests that need a browser but don't say so in their imports. */
const CHROME_ANYWAY = new Set([
// Shells out to the CLI, whose check-drift path launches Chrome. Nothing in
// this file's own imports reveals that.
'test-cli.ts',
// Drives the MCP server over stdio, and the server renders in Chrome for
// detailed evaluate, canvas_stress and screenshots.
'test-ridgeline.ts',
// Phase 31's acceptance re-run, the same shape as test-ridgeline.ts.
'test-viewer-rebuild.ts',
// Drives canvas_evaluate's detailed mode over MCP (S11's font path).
'test-evaluate-fonts.ts',
]);

/** Direct use of the browser is the honest signal — these are the entry points
* in screenshot.ts that every browser-driven test reaches for. */
const CHROME_MARKERS = /takeScreenshot|computeLayout|withPage|withIsolatedPage|puppeteer|from '\.\/src\/screenshot\.js'/;

// FRAMESMITH_TEST_TIMEOUT_MS overrides it (test-runner-timeout.ts uses 3s).
const REPO = resolve(dirname(fileURLToPath(import.meta.url)), '..');

// FRAMESMITH_TEST_TIMEOUT_MS overrides it (test/unit/test-runner-timeout.ts uses 3s).
const TIMEOUT_MS = Number(process.env.FRAMESMITH_TEST_TIMEOUT_MS) || 180_000;

/** A Chrome path that does not exist. A unit test that launches Chrome, through
* src/screenshot.ts or puppeteer.launch directly, fails on it. */
const NO_CHROME = join(REPO, 'no-chrome-in-unit-tests');

/** What a failed launch prints: src/screenshot.ts's message, then Puppeteer's own. */
const LAUNCH_FAILURE = /Failed to launch Chrome for rendering|Tried to find the browser at the configured path/;

type Tier = 'fast' | 'full';
const tier: Tier = process.argv.includes('--full') ? 'full' : 'fast';
const filters = process.argv.slice(2).filter((a) => !a.startsWith('--'));
const TIER_FOLDERS: Record<Tier, string[]> = { fast: ['unit'], full: ['unit', 'browser', 'e2e'] };

const needsChrome = (file: string): boolean =>
CHROME_ANYWAY.has(file) || CHROME_MARKERS.test(readFileSync(file, 'utf-8'));
const args = process.argv.slice(2);
const rootAt = args.indexOf('--root');
const root = rootAt >= 0 ? resolve(args[rootAt + 1]) : join(REPO, 'test');
if (rootAt >= 0) args.splice(rootAt, 2);
const tier: Tier = args.includes('--full') ? 'full' : 'fast';
const filters = args.filter((a) => !a.startsWith('--'));

const all = readdirSync('.')
.filter((f) => f.startsWith('test-') && f.endsWith('.ts') && f !== 'test-env.ts')
.filter((f) => !INTERACTIVE.has(f))
.sort();
// Tests read paths relative to the repo root (`resolve('src/index.ts')`).
process.chdir(REPO);

const selected = all
.filter((f) => tier === 'full' || !needsChrome(f))
.filter((f) => filters.length === 0 || filters.some((q) => f.includes(q)));
/** Test paths under the root, as `<folder>/<file>`. */
const discover = (folder: string): string[] => {
const dir = join(root, folder);
if (!existsSync(dir)) return [];
return readdirSync(dir).filter((f) => f.endsWith('.ts')).sort().map((f) => `${folder}/${f}`);
};

const selected = TIER_FOLDERS[tier]
.flatMap(discover)
.filter((t) => filters.length === 0 || filters.some((q) => t.includes(q)));

/** The test running now. Its own process group doesn't get the terminal's
* Ctrl-C, so the runner passes an interrupt on before exiting. */
Expand All @@ -77,13 +72,19 @@ for (const signal of ['SIGINT', 'SIGTERM'] as const) {
});
}

function run(file: string): Promise<{ file: string; ok: boolean; why: string; ms: number }> {
return new Promise((resolve) => {
type Result = { test: string; ok: boolean; why: string; ms: number };

function run(test: string): Promise<Result> {
const unit = test.startsWith('unit/');
const env = unit
? { ...process.env, FRAMESMITH_CHROME_PATH: NO_CHROME, PUPPETEER_EXECUTABLE_PATH: NO_CHROME }
: process.env;
return new Promise((done) => {
const started = Date.now();
// Its own process group, so a timeout kills npx AND the tsx under it.
// Killing npx alone orphaned tsx, which kept the pipes open, and 'close'
// never fired: one hung test stalled the whole run.
const child = spawn('npx', ['tsx', file], { stdio: ['ignore', 'pipe', 'pipe'], detached: true });
const child = spawn('npx', ['tsx', join(root, test)], { stdio: ['ignore', 'pipe', 'pipe'], detached: true, env });
current = child;
let out = '';
child.stdout.on('data', (d) => { out += d; });
Expand All @@ -92,33 +93,36 @@ function run(file: string): Promise<{ file: string; ok: boolean; why: string; ms
child.on('close', (code, signal) => {
clearTimeout(timer);
const ms = Date.now() - started;
if (signal === 'SIGKILL') return resolve({ file, ok: false, why: `timed out after ${TIMEOUT_MS / 1000}s`, ms });
if (code === 0) return resolve({ file, ok: true, why: '', ms });
if (signal === 'SIGKILL') return done({ test, ok: false, why: `timed out after ${TIMEOUT_MS / 1000}s`, ms });
if (code === 0) return done({ test, ok: true, why: '', ms });

const first = out.split('\n').filter((l) => /^FAIL|Error|error TS/.test(l))[0] ?? `exit ${code}`;
resolve({ file, ok: false, why: first.trim().slice(0, 160), ms });
const launched = unit && LAUNCH_FAILURE.test(out)
? '\n this test launches Chrome: move it to test/browser'
: '';
done({ test, ok: false, why: first.trim().slice(0, 160) + launched, ms });
});
});
}

console.log(`${selected.length} tests (${tier} tier)${filters.length ? ` matching ${filters.join(', ')}` : ''}\n`);

const failures: { file: string; why: string }[] = [];
const failures: Result[] = [];
let passed = 0;
for (const file of selected) {
const r = await run(file);
if (r.ok) { passed++; console.log(` ok ${file} ${(r.ms / 1000).toFixed(1)}s`); }
else { failures.push(r); console.log(` FAIL ${file} ${r.why}`); }
for (const test of selected) {
const r = await run(test);
if (r.ok) { passed++; console.log(` ok ${test} ${(r.ms / 1000).toFixed(1)}s`); }
else { failures.push(r); console.log(` FAIL ${test} ${r.why}`); }
}

console.log(`\n${passed}/${selected.length} passed`);
if (tier === 'fast') {
const skipped = all.filter(needsChrome).length;
console.log(`${skipped} browser tests not run in this tier — use npm run test:full`);
const skipped = discover('browser').length + discover('e2e').length;
if (skipped) console.log(`${skipped} browser and e2e tests not run in this tier — use npm run test:full`);
}
if (INTERACTIVE.size) console.log(`${[...INTERACTIVE].join(', ')} excluded (interactive)`);

if (failures.length) {
console.log(`\n${failures.length} failing:`);
for (const f of failures) console.log(` ${f.file} ${f.why}`);
for (const f of failures) console.log(` ${f.test} ${f.why}`);
process.exit(1);
}
2 changes: 1 addition & 1 deletion src/color-system.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// perceptual ramps, a matched neutral, status colors, and a dual-theme
// semantic mapping. Hand-rolled conversions (Björn Ottosson's OKLab math,
// zero dependencies — validated against Chrome's own oklch() parsing in
// test-color-system-readback.ts). Pure: token WRITING is the tool's job.
// test/browser/test-color-system-readback.ts). Pure: token WRITING is the tool's job.
//
// Gamut policy (spec ANALYZE): out-of-gamut colors clip toward LOWER CHROMA
// at fixed lightness and hue — never hue-shift, never lightness-shift.
Expand Down
Loading
Loading