Skip to content

test(engineering): replay the Rakazo checkout fix offline - #20

Draft
christso wants to merge 6 commits into
mainfrom
test/rakazo-replay
Draft

christso wants to merge 6 commits into
mainfrom
test/rakazo-replay

Conversation

@christso

@christso christso commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What

Add an opt-in implementation replay that gives Codex the same historical Rakazo checkout bug twice, with and without scope guidance in AGENTS.md. Each run starts from the pinned pre-fix repository and stages the same Compound Engineering skill through oh-my-promptfoo.

Why

The evidence-packet eval measures review judgment. This replay lets us assess what the agent actually changes and whether the resulting code works in Chromium. The browser check covers the shopper note and the existing payment approval requirement.

Verification

  • Promptfoo configuration validation, catalogue validation, and all 62 repository tests passed. The browser check reproduced the false approval on the pinned helper and passed with the merged helper. The order remained guarded in both controls.
  • Both live GPT-6.1-Sol variants passed the skill-read and independent Chromium checks. The baseline added geometry reads for every click. The guided run limited the wait to controls that needed scrolling and added a Chromium regression plus unit coverage. The agents reported that command networking blocked their own browser runs. The independent grader ran separately.
  • The guided row was retried after a launcher change disrupted the original pair. It ran with the same model and Codex binary as the baseline. This is one pilot comparison. Both changes remained staged because the workspaces lacked Git author identity. The guided capture retained the changed Rakazo files and hit the text diff limit.
  • GPT-6.2-Sol passed a separate Promptfoo smoke test using Codex CLI 0.166.0. The replay selects the executable explicitly because the SDK-bundled CLI 0.162.1 returned model_not_found with the same login.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant