Skip to content

bench: fresh_clone leaks every upstream fix branch into the cell, so real-issue runs can copy the gold patch #1028

Description

@santoshkumarradha

What happened

bench/run.sh's fresh_clone does a full clone and then checkout BASE, so every upstream branch — including the fix branch for the very issue under test — sits in the cell's .git. In the 2026-09-14 dev vs #993 vs #1017 grid (four issues on MALIBA-AI/bambara-text-normalization @ 6c978ffa1c, deepseek/deepseek-v4.1-flash), all three binaries found and read them: git show origin/feat/… appears in the transcripts, one dev cell notes its diff "reproduces the gold index line". Issue 21's diff is the upstream patch verbatim in all three runs, issue 20's in two of three. Only issue 22 was uncontaminated. The grid's quality column is therefore not a measurement.

bench/canary/lib/cell.sh already does this right: a fetch-only tree at the base commit, no other refs.

Replication a stranger can run

AFORGE_MODE=chat bench/run.sh <issue 21 of that corpus>      # then, in the cell:
git -C <cell> branch -r | grep -c feat/                        # > 0 today, must be 0
grep -c 'git show origin/' <cell>/transcript.jsonl             # > 0 today

Acceptance

End to end first: a cell's git branch -r lists only the base ref and git log --all reaches no commit newer than BASE; a run's transcript contains no git show origin//git log origin/ of a branch that does not exist in the cell. Then fresh_clone shares bench/canary/lib/cell.sh's fetch-only path rather than a second clone recipe.

Evidence: spark:~/af-bench-evidence/REPORT.md and the per-cell transcript.jsonl files; the same table is posted on #993 and #1017.

🤖 Generated with Claude Code

https://claude.ai/code/session_0182eyxXP8Xe7ADNKpYDe4wv


Drafted with CodeAF · reviewed and owned by the author

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueSmall and isolated, the acceptance already written, no ruling owed; a safe first pull request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions