Skip to content

docs: design for front-end test output compression - #19

Open
jazelly wants to merge 4 commits into
docs/test-output-compression-brieffrom
docs/test-compression-design
Open

jazelly wants to merge 4 commits into
docs/test-output-compression-brieffrom
docs/test-compression-design

Conversation

@jazelly

@jazelly jazelly commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Design doc docs/test-compression.md, step 1 of the process in the brief (#18). Docs only, nothing is implemented. Implementation waits for review.

This PR is stacked on #18 so the brief link resolves. GitHub retargets it to main when #18 merges.

What the doc covers

  • Purpose, positioning, and design principles. The main one: agentic-react rewrites output where it is produced (testing-library errors, Playwright/Cypress failure messages, the page console, bundler error reports) and leaves the runner's reporter alone. That is how it composes with agentic-node without either one detecting the other.
  • Integration points for Playwright, Cypress, RTL (Jest/Vitest), Vitest browser mode, and React warnings.
  • Integration points for Vite, webpack, Next (webpack and Turbopack), Rsbuild/Rspack, Astro, TanStack Start, and React Router v7, with the order they ship in and what "supported" means.
  • Source mapping, reusing the existing bippy, owner-stack, and findComponentSourceInProject code.
  • Before/after output from the benchmark repos (section 8).
  • The boundary with agentic-node.
  • Benchmark: four arms plus a Vitest agent-reporter comparison, the browser-tool fairness proposal, fixtures, 16 new tasks, fixes for the known gaps, and exact definitions for the 20% target and the pass-rate floor.
  • Risks.

Evidence and token report

  • docs/test-compression/REPORT.md has per-case and grouped token tables for 51 real cases on the 8 pinned benchmark repos.
  • Two measured arms, run back to back:
    • control (stock repo);
    • agentic-node (0.2.0 at 4e604d3, preloaded through NODE_OPTIONS per its README template).
  • Run conditions: agent markers set, latest browser only (Playwright 1.65.1 / Chrome for Testing 159). Older browser builds were removed from the host.
  • agentic-react targets and "both" estimates are produced by runs/scripts/targets.mjs from those captures. They are targets, not measurements.
  • Headline:
    • agentic-node measured on 48 cases: 240,113 → 78,391 tokens, i.e. -67.4% of the total, -26.0% without the 896-error webpack build. The median case saves 0.7%, and the 13 browser-console cases see 0% (one gets worse).
    • On the 6 cases that have a target: agentic-node -81.1% (measured), agentic-react -98.3% (target), both -89.2% (estimate).
  • Everything is committed under docs/test-compression/runs/: every capture, .meta.txt, agentic-node log, warm-up run and script.
  • Section 8.5 lists three task checks that don't behave as written.

Decisions requested (section 13 of the doc)

  1. Package layout: one @agentic-react/test package with subpath exports.
  2. Browser tool: playwright-cli in every arm, control included.
  3. Pass-rate floor: at most a 3-point drop in pooled pass rate, and no task fully lost.
  4. Scenario weights: 0.4 / 0.4 / 0.2.
  5. Cypress: none of the 8 benchmark repos uses it. Proposed: add Cypress e2e to vite-shadcn-admin through setup edits.
  6. Output is compact by default once installed; AGENTIC_REACT_OUT=raw opts out.
  7. Turbopack terminal output stays untouched in V1.
  8. React warnings in passing tests are kept as one deduped line each.
  9. All runs, the report and the scripts stay committed under docs/test-compression/ (~4 MB).
  10. Web-server output when both are installed: on rr7 e2e, agentic-node keeps 17,812 tokens where agentic-react's target keeps 269. Proposed: keep the current split and leave this to agentic-node.

No paid model runs. No secrets.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VHE7EnFiaEt3pFiQGG7oux

jazelly and others added 4 commits November 19, 2026 15:57
Design for the brief in docs/briefs/test-output-compression.md (step 1 of
its process): integration points per test framework and bundler, source
mapping reuse, boundary with agentic-node, browser-tool fairness proposal,
metric definitions for the 20% target and the pass-rate floor, and risks.

Section 8 quotes real stock-tool output captured from a fixture app
(committed unmodified under docs/test-compression/before/) next to
hand-written target digests (docs/test-compression/after/). The targets
are not output of any implementation; nothing is implemented yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHE7EnFiaEt3pFiQGG7oux
The before/after examples came from a throwaway fixture app, not from the
pinned benchmark repos. Remove them and the claims derived from them;
section 8 will be filled from captures on benchmark/repos/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHE7EnFiaEt3pFiQGG7oux
Section 8 now quotes real output captured on the 8 pinned repos in
benchmark/repos/ (repo test suites for happy paths, existing tasks'
setup.edits for failures), committed unmodified with per-capture meta
under docs/test-compression/before/<repo>/. Six hand-written target
digests (not output of any implementation) are in after/<repo>/.

Design updates from the captures: alias-grouped webpack error digest,
agent-mode behaviour of Next 16.4 / Astro 7.4 / Vitest 4.1 / Vite 8 /
Rstest in the arm definition, fixtures defined on benchmark repos, three
task checks that do not behave as written, Cypress decision reworded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHE7EnFiaEt3pFiQGG7oux
…en report

Re-run all 51 benchmark cases in two measured arms back to back:
control and agentic-node 0.2.0 (4e604d3, NODE_OPTIONS preload per its
README template), agent markers set, latest browser only (Playwright
1.65.1 / Chrome for Testing 159). agentic-react targets and "both"
estimates are produced by runs/scripts/targets.mjs from those captures
and are labelled as targets, not measurements.

docs/test-compression/REPORT.md has the per-case and grouped token
tables and findings. Design updates: log directory moves under
node_modules/.cache (an in-project log dir triggered a Vite reload),
new decision on web-server output in the "both" setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHE7EnFiaEt3pFiQGG7oux
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant