Skip to content

Add a benchmark for agent progress and recovery - #590

Open
trissim wants to merge 2 commits into
mainfrom
bench/agent-recovery
Open

trissim wants to merge 2 commits into
mainfrom
bench/agent-recovery

Conversation

@trissim

@trissim trissim commented Oct 10, 2026 •

Copy link
Copy Markdown

Adds bench/agent-recovery/ with a README, churn.py, run_arm.py, score.py and tasks.toml. churn.py computes commit counts, source line changes, rewrite ratio and files edited many times from git history. run_arm.py starts an agent CLI on a frozen task snapshot with a goal and time budget, and score.py scores the resulting branch. The change adds files only.

Copilot AI balanced review requested due to automatic review settings October 10, 2026 06:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@trissim trissim changed the title Add agent recovery bench Add a benchmark for agent progress and recovery Oct 11, 2026
trissim and others added 2 commits October 11, 2026 17:02
…e protocol

churn.py splits history into a trailer-marked arm and the baseline arm and
compares source lines touched against net change, files rewritten 10+ times,
and evidence/docs volume. run_arm.py starts an agent blind from a task's
frozen snapshot with a goal and budget; score.py scores the run on the goal
metric, preserved tests, structure, bloat, and auditable claims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a dependency)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trissim
trissim force-pushed the bench/agent-recovery branch from 9a56c73 to af15812 Compare October 11, 2026 21:02
@trissim

trissim commented Oct 13, 2026

Copy link
Copy Markdown
Author

Census #590 BLOCK at af15812; actual merge-base 1e328f8 (not baseRefOid bfff711). Full diff +435/-0 across 5 bench files: README.md +73, churn.py +126, run_arm.py +63, score.py +125, tasks.toml +48. Bench-root census: foreign_absence_probe +2, string_dispatch +2, string_key_subscript +25, string_equality +11, json_loads +1, class_definition +1, code lines +258; all other measures +0. Production src/toad unchanged. New type switches/isinstance chains/4+-term boolean chains: 0.

IMPL-1/IMPL-5: new bucket consumer dispatches at score.py:63-70 and churn.py:91-99 independently switch on source/evidence/docs; bucket() is declared at churn.py:25-32, imported by score.py:25. A new bucket needs at least 3 edits (owner + both consumers); baseline had no such family. IMPL-12: same rewrite-ratio formula appears in Arm.rewrite_ratio (churn.py:56-57) and structure() (score.py:78); edited-10-times policy appears at churn.py:102 and score.py:79. Docs metrics differ deliberately by name (touched versus added), so I do not classify that alone as an accidental contradiction.

Lifecycle risk: score.py:32-35 creates git worktrees with tempfile.mkdtemp (volatile location); main :113 retains two such worktrees with no removal/finally cleanup. run_arm.py:34/:41 uses a relative default runs/ location, not the required persistent /home/ts/wt namespace. These paths were inspected, not executed. Goal metrics are empty in tasks.toml, so prospective goal acceptance is not implemented. No agent or benchmark run was launched. Fix owner: bench author / this branch, coordinated with the already-blocked matching Core #742. Acceptance: declaration-owned bucket aggregation, one owner for shared metric formulas, and owned persistent worktree lifecycle with cleanup on success/failure. No replaced behavior to delete in this added-only checkpoint; competing decisions must be removed before PASS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants