Repository navigation
Conversation
…e protocol churn.py splits history into a trailer-marked arm and the baseline arm and compares source lines touched against net change, files rewritten 10+ times, and evidence/docs volume. run_arm.py starts an agent blind from a task's frozen snapshot with a goal and budget; score.py scores the run on the goal metric, preserved tests, structure, bloat, and auditable claims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a dependency) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9a56c73 to
af15812
Compare
|
Census #590 BLOCK at af15812; actual merge-base 1e328f8 (not baseRefOid bfff711). Full diff +435/-0 across 5 bench files: README.md +73, churn.py +126, run_arm.py +63, score.py +125, tasks.toml +48. Bench-root census: foreign_absence_probe +2, string_dispatch +2, string_key_subscript +25, string_equality +11, json_loads +1, class_definition +1, code lines +258; all other measures +0. Production src/toad unchanged. New type switches/isinstance chains/4+-term boolean chains: 0. IMPL-1/IMPL-5: new bucket consumer dispatches at score.py:63-70 and churn.py:91-99 independently switch on source/evidence/docs; bucket() is declared at churn.py:25-32, imported by score.py:25. A new bucket needs at least 3 edits (owner + both consumers); baseline had no such family. IMPL-12: same rewrite-ratio formula appears in Arm.rewrite_ratio (churn.py:56-57) and structure() (score.py:78); edited-10-times policy appears at churn.py:102 and score.py:79. Docs metrics differ deliberately by name (touched versus added), so I do not classify that alone as an accidental contradiction. Lifecycle risk: score.py:32-35 creates git worktrees with tempfile.mkdtemp (volatile location); main :113 retains two such worktrees with no removal/finally cleanup. run_arm.py:34/:41 uses a relative default runs/ location, not the required persistent /home/ts/wt namespace. These paths were inspected, not executed. Goal metrics are empty in tasks.toml, so prospective goal acceptance is not implemented. No agent or benchmark run was launched. Fix owner: bench author / this branch, coordinated with the already-blocked matching Core #742. Acceptance: declaration-owned bucket aggregation, one owner for shared metric formulas, and owned persistent worktree lifecycle with cleanup on success/failure. No replaced behavior to delete in this added-only checkpoint; competing decisions must be removed before PASS. |
Adds
bench/agent-recovery/with a README,churn.py,run_arm.py,score.pyandtasks.toml.churn.pycomputes commit counts, source line changes, rewrite ratio and files edited many times from git history.run_arm.pystarts an agent CLI on a frozen task snapshot with a goal and time budget, andscore.pyscores the resulting branch. The change adds files only.