Repository navigation
feat(dev): measure per-stage model and effort on the defect probe set (CORE-112) - #456
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…E-90) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it (CORE-90) Six probes over a small fixture project, each with one planted defect (four PLAN probes, two SPEC probes), pinned by sha256 in manifest.json. The scorer counts a run as caught when a valid finding's file matches an anchor and the valid triage does not give it proceed; it reads no free text. The kit builds one fresh project and one claude -p command line per run, with model and effort per stage, for the user to run. The catch count on main 7.19.0 is pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ORE-112) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et, and make kit overrides take effect (CORE-112) A directly invoked /pharn-<stage> runs on its command's model:/effort: frontmatter, so the kit's --model flag alone did not set a stage's model. The kit now writes an overridden stage into each run project's frontmatter and models.stages and refuses the build unless check-model-config agreement passes. stage-model-pairs.mjs pairs a variant arm with the control and gives it one closed reading. The record pre-registers three arms; results pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-112) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(CORE-112) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
🛡️ pharn-ci ·
|
| Verdict | Diff coverage | Test integrity | Agent setup |
|---|---|---|---|
| 100% (threshold 100%) | ℹ️ 10 findings | 106 files |
🧪 Diff coverage
▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰ 100%
100% of changed lines covered (1104/1104, threshold 100%)
Changed files (3)
| File | Coverage | Covered | Uncovered lines |
|---|---|---|---|
.dev/floor/defect-probe-kit.mjs |
100% | 323/323 | |
.dev/floor/defect-probe-score.mjs |
100% | 491/491 | |
.dev/floor/stage-model-pairs.mjs |
100% | 290/290 |
🔍 Test integrity
- ℹ️ TI15
.dev/floor/defect-probe-kit.mjs:103— The condition at ".dev/floor/defect-probe-kit.mjs:103" compares with a 7-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-kit.mjs:104— The condition at ".dev/floor/defect-probe-kit.mjs:104" compares with a 7-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-kit.mjs:104— The condition at ".dev/floor/defect-probe-kit.mjs:104" compares with a 8-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-kit.mjs:107— The condition at ".dev/floor/defect-probe-kit.mjs:107" compares with a 7-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-kit.mjs:108— The condition at ".dev/floor/defect-probe-kit.mjs:108" compares with a 8-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-kit.mjs:109— The condition at ".dev/floor/defect-probe-kit.mjs:109" compares with a 7-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-score.mjs:151— The condition at ".dev/floor/defect-probe-score.mjs:151" compares with a 8-character string literal that the test file ".dev/floor/defect-probe-score.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-score.mjs:281— The condition at ".dev/floor/defect-probe-score.mjs:281" compares with a 7-character string literal that the test file ".dev/floor/defect-probe-kit.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-score.mjs:297— The condition at ".dev/floor/defect-probe-score.mjs:297" compares with a 6-character string literal that the test file ".dev/floor/defect-probe-score.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
- ℹ️ TI15
.dev/floor/defect-probe-score.mjs:298— The condition at ".dev/floor/defect-probe-score.mjs:298" compares with a 14-character string literal that the test file ".dev/floor/defect-probe-score.test.mjs" also adds, so the code may special-case the test's input.- Hint: Check that the code handles the general case and not only the test's value, and add a test with another input.
🧬 Content
⚠️ PI03.dev/features/stage-model-tuning/REVIEW.md:9— phrase PH01- Hint: Remove the phrase. If the file documents prompt injection on purpose, keep such samples in test files or under
fixtures/.
- Hint: Remove the phrase. If the file documents prompt injection on purpose, keep such samples in test files or under
📦 Dependencies
0 added or changed dependencies checked · registry lookups off
No dependency findings.
⚠️ Not checked: 1 manifest or lockfile file could not be read, so dependency safety did not check it: `.dev/probes/defect-probes/fixture/package.json` (unavailable).- Hint: Keep manifests and lockfiles valid and under 5 MiB, and run on a checkout of the pull request head with fetch-depth: 0, so they can be compared.
💬 Messages (2)
- ℹ️
COVERAGE_PATHS_PARTIALLY_UNMATCHED126 file paths in the coverage report are not tracked files in the repository; their coverage is not used.- Hint: Nothing to do if those files are generated or outside the repo; if changed files are missing from the table, check the coverage root.
⚠️ AGENT_CONFIG_SECRET_UNSCANNEDAgent config files were not scanned for committed secrets: `pr-number mode: the checkout is not the pull request head, so its committed agent config files were not read`.- Hint: Keep agent config files text, under 512 KiB and inside the repository, and run on a checkout of the pull request head (not PR-number mode or a compare-API diff), so they can be scanned for committed secrets.
🤖 Agent setup (106 files)
Agent setup: 106 files · claude-code
Agent setup files (106)
| Path | Kind |
|---|---|
.claude/agents/pharn-review-lens.md |
claude-agent |
.claude/agents/pharn-stage.md |
claude-agent |
.claude/commands/pharn-build.md |
claude-command |
.claude/commands/pharn-dev-build.md |
claude-command |
.claude/commands/pharn-dev-eval.md |
claude-command |
.claude/commands/pharn-dev-grill.md |
claude-command |
.claude/commands/pharn-dev-memory-promote.md |
claude-command |
.claude/commands/pharn-dev-plan.md |
claude-command |
.claude/commands/pharn-dev-regress.md |
claude-command |
.claude/commands/pharn-dev-review.md |
claude-command |
.claude/commands/pharn-dev-ship.md |
claude-command |
.claude/commands/pharn-dev-verify.md |
claude-command |
.claude/commands/pharn-grill.md |
claude-command |
.claude/commands/pharn-loop-close.md |
claude-command |
.claude/commands/pharn-loop-quick.md |
claude-command |
.claude/commands/pharn-loop.md |
claude-command |
.claude/commands/pharn-memory-promote.md |
claude-command |
.claude/commands/pharn-plan.md |
claude-command |
.claude/commands/pharn-regress.md |
claude-command |
.claude/commands/pharn-review.md |
claude-command |
.claude/commands/pharn-ship-close.md |
claude-command |
.claude/commands/pharn-ship-quick.md |
claude-command |
.claude/commands/pharn-ship-stages.md |
claude-command |
.claude/commands/pharn-ship.md |
claude-command |
.claude/commands/pharn-spec.md |
claude-command |
.claude/commands/pharn-supervise.md |
claude-command |
.claude/commands/pharn-test.md |
claude-command |
.claude/commands/pharn-triage.md |
claude-command |
.claude/commands/pharn-verify.md |
claude-command |
.claude/hooks/enforce-writes-scope.cjs |
claude-hook |
.claude/hooks/enforce-writes-scope.test.cjs |
claude-hook |
.claude/hooks/hook-wiring.test.cjs |
claude-hook |
.claude/hooks/protect-trusted-paths.cjs |
claude-hook |
.claude/hooks/protect-trusted-paths.test.cjs |
claude-hook |
.claude/hooks/require-loop-record.cjs |
claude-hook |
.claude/hooks/require-loop-record.test.cjs |
claude-hook |
.claude/hooks/set-writes-scope.cjs |
claude-hook |
.claude/hooks/set-writes-scope.test.cjs |
claude-hook |
.claude/hooks/writes-scope-release.test.cjs |
claude-hook |
.claude/settings.json |
claude-settings |
.claude/skills/phargus-diagnose/SKILL.md |
claude-skill |
.claude/skills/phargus-diagnose/agents/openai.yaml |
claude-skill |
.claude/skills/phargus-diagnose/references/compatibility.md |
claude-skill |
.claude/skills/phargus-diagnose/references/data.md |
claude-skill |
.claude/skills/phargus-diagnose/references/diagnosis.md |
claude-skill |
.claude/skills/phargus-diagnose/references/tools.md |
claude-skill |
.claude/skills/phargus-diagnose/scripts/capture.mjs |
claude-skill |
.claude/skills/phargus-diagnose/scripts/operations.mjs |
claude-skill |
.claude/skills/phargus-diagnose/scripts/phargus.mjs |
claude-skill |
.claude/skills/phargus-diagnose/scripts/records.mjs |
claude-skill |
| …and 56 more files |
pharn-ci 0.10.0 · report schema v2 · coverage format lcov
…stopped-at-plan-unanchored (CORE-90) Both p5 plan sessions refused to plan, naming the planted contradiction, and left no PLAN.md; the pre-registered rule could only score that findings-missing. A plan-stage stop is now caught-at-plan when the plan session's result cites an anchored SPEC.md line, else stopped-at-plan-unanchored (neither a catch nor a miss). Both read model free text and are advisory; the floor part is the line overlap. The catch count is caught + caught-at-plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto feat/core-112-stage-model-tuning
…ORE-112) caught-at-plan is a catch; stopped-at-plan-unanchored is neither, so a pair holding one is costed but not comparable, and the six-pair, control-catch and catch-lost rules count comparable pairs only. Grill and triage sessions are optional for a plan-stopped run. The record states this amendment before any pairing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odel-tuning # Conflicts: # CHANGELOG.md
grill model sonnet: supports-change on the probe chain (8 comparable pairs, 0 lost, chain -43% USD). plan model sonnet: cannot-decide (4 comparable pairs). grill effort medium: does-not-support (1 catch lost). No shipped default changes; each needs E1's loop pairs. Run files and the reader's pair JSON are committed; session files are not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…E-112) Added with -f: the repo-wide runs/ ignore pattern also matches this feature's runs/ directory. Findings, triage, a SPEC probe's PLAN.md and the reader's JSON; no session files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odel-tuning # Conflicts: # CHANGELOG.md
Summary
Per-stage model and effort arms for the defect probe set, the pairing reader that scores them, and a kit fix without which a model arm would not have run the model it names. Apparatus only: nothing under the product surface changes,
SKILLS_VERSIONis unchanged, the CHANGELOG entry is[Unreleased], and no shipped default changes.Depends on #455 (CORE-90) and must merge after it. This branch is cut from
feat/core-90-defect-probe-set. When #455 merges, mergeorigin/maininto this branch and drop nothing..dev/floor/defect-probe-kit.mjs). Each kit session is a directclaude -p '/pharn-<stage> …', and Claude Code takes a directly invoked command's model and effort from that command'smodel:/effort:frontmatter for the turn (pharn/floor/check-model-config.mjs, "TURN SCOPE")./pharn-grilland/pharn-plansaymodel: opus, so--model grill=sonnetwould most likely have run opus and been scored as sonnet. A stage named by--model/--effortis now written into each run project's command frontmatter andpharn.config.jsonmodels.stages. The build then refuses unless this repository'scheck-model-config.mjs agreementpasses over the project; the arm's own copy is never executed. A control kit changes no byte..dev/floor/stage-model-pairs.mjs, new). It pairs the i-th variant run with the i-th control run for each probe. Per pair it reports the catch (via the scorer'sscoreRun), stage minutes, USD, weighted units and the served model, read from each<stage>.session.json. It then gives the arm one reading from a closed set by a fixed first-match rule:cannot-decide(arm-not-applied,fewer-than-six-pairsorcontrol-caught-none),does-not-support(catch-lostorno-saving), orsupports-change. A test shows the control arm's own sessions cannot satisfy the served-model check..dev/measurements/stage-model-tuning-2026-11-17.md, pending). Pre-registers four kits before any run:plan-sonnetwithcontrol-spec, thengrill-sonnet, thengrill-medium, which runs only if at least USD 29 of the USD 70 budget is left. Also pre-registered: the manifest sha256, the arm commit, the exact kit and pairing commands, the reading rule, the budget and stop rule, and the bounds. Those bounds are: no randomization, served effort not recorded, all cache writes weighted as 5-minute writes, and E1 out of scope..dev/guides/floor-checks.md. Ignore lists: the record's futureruns/directory.Results (the user's runs, USD 29.94 of USD 70)
One reading per arm, on the probe chain only:
supports-changecannot-decide(fewer-than-six-pairs)does-not-support(catch-lost)Every variant session on the varied stage was served the expected model, so the frontmatter override took effect. No
shipped default changes: each default needs E1's whole-loop pairs in the measured project and the human's call. The
record's "Defaults" table says so row by row.
caught-at-plan/stopped-at-plan-unanchoredoutcomes are merged from feat(dev): defect probe set, per-finding scorer and run kit (CORE-90) #455, and the reader pairs plan-stage stops on a valid plan session. A pair holding an unanchored stop is costed but is not comparable for the catch rules. The record states this amendment as written before any pairing.origin/main(7.21.0) is merged. CORE-90's[Unreleased]entry is re-dated to 2026-11-17 so it is not older than the 7.21.0 section..dev/features/stage-model-tuning/runs/, added with-fbecause the repo-wideruns/ignore pattern matches this directory. Session files are not committed: they carry session ids.Pipeline
/pharn-dev-ship stage-model-tuning: plan, grill, build, regress, verify, review. GATE 1 was decided by the orchestrator under the user's delegation; GATE 2 was a model decision under delegation. See.dev/features/stage-model-tuning/SHIP.md.Not covered
The whole-loop experiment E1 on the measured project.
Linear: CORE-112
🤖 Generated with Claude Code
Summary by CodeRabbit