Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
301 changes: 102 additions & 199 deletions bun.lock

Large diffs are not rendered by default.

80 changes: 80 additions & 0 deletions evals/engineering/rakazo-replay/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Rakazo checkout replay

This opt-in eval compares two Codex runs of the same historical bug, with and
without the scope guidance in [guided/AGENTS.md](guided/AGENTS.md). Each run
starts in a fresh oh-my-promptfoo workspace. The experiment records whether
Codex read the pinned Compound Engineering skill and whether the generated
change passes a Chromium check. It retains the resulting diff for human review.

The input comes from the public [Rakazo bug report](https://github.com/elie222/rakazo/issues/4728).
The replay pins the reported pre-fix commit and a Compound Engineering release.
The task gives the symptom and approval constraint. Keep the later PRs, fix
patches, PR comments, and grader files outside the candidate environment.

## Prepare and run

Use a dedicated Linux runner with Node 22.22 or later, Bun, Git, Python 3,
Bubblewrap with user namespaces, Codex, and access to the model service.
Use a clean runner account with only the source and tooling for the replay.
The launcher disables user Codex configuration. Codex's workspace sandbox
disables command networking and web search while the SDK can still reach its
model service. Shell tools can read files outside the workspace under Codex's
default filesystem policy, so keep answer material and unrelated checkouts off
that runner. The separate browser grader runs in a network namespace with
loopback and only the files it needs mounted.

Download the inputs ahead of time. The shallow fetches contain only the two
pinned commits. The runner can reuse these checkouts and oh-my-promptfoo's
source cache for subsequent trials.

```sh
bun install --frozen-lockfile
inputs=$(mktemp -d)
git init -q "$inputs/rakazo"
git -C "$inputs/rakazo" fetch --depth=1 https://github.com/elie222/rakazo.git 92eeb44a40f95caab81c8ae46bf2428a87a64ca5
git -C "$inputs/rakazo" checkout --detach FETCH_HEAD
git init -q "$inputs/compound-engineering"
git -C "$inputs/compound-engineering" fetch --depth=1 https://github.com/EveryInc/compound-engineering-plugin.git d8f8e03237bded06e77bcb3930b43ef841e7eeda
git -C "$inputs/compound-engineering" checkout --detach FETCH_HEAD
"$inputs/rakazo/infra/sandboxes/computer/install-agent-browser.sh" "$inputs/bin"

export REPLAY_RAKAZO_SOURCE="file://$inputs/rakazo"
export REPLAY_CE_SOURCE="file://$inputs/compound-engineering"
export REPLAY_AGENT_BROWSER="$inputs/bin/agent-browser"
export REPLAY_CHROMIUM=/absolute/path/to/chrome
suite=evals/engineering/rakazo-replay
export REPLAY_CODEX_BIN="$PWD/node_modules/.bin/codex"
export REPLAY_CODEX_LAUNCHER="$PWD/$suite/codex-replay.sh"
export ALLAGENTS_WORKSPACE_ROOT="$inputs/workspaces"
bun run promptfoo validate config -c "$suite/promptfooconfig.yaml"
bun run promptfoo eval -c "$suite/promptfooconfig.yaml" \
--max-concurrency 1 --no-cache --no-share -o "$inputs/replay.json"
```

Use the same Chromium build across variants. The fixture was exercised with
Chrome for Testing 156.0.8078.4 and Rakazo's pinned agent-browser 0.41.0. The
Chromium directory must contain its runtime files. The run uses the Codex
CLI installed by the pinned SDK. Promptfoo requires the launcher and SDK to
report the same CLI version.
Codex CLI 0.166.0 accepted `gpt-6.2-sol` with the test login. The SDK-bundled
0.162.1 binary rejected that model on the same login. Set `CODEX_MODEL` to
change the candidate model for both variants. oh-my-promptfoo manages the two
private writable checkouts and captures changes. Use an off-workspace location
for exported results.

## Compare the two runs

The Chromium assertion verifies the harmless note preflight, clicking a fresh
offscreen note, and refusal to click a real order button without approval.
The skill assertion records evidence that Codex read the skill. Inspect the
tool trace if the distinction between reading it and executing its workflow
matters to the comparison.

Review each captured diff against the original bug and approval constraint.
Record any extra runtime mechanisms, what requirement or evidence justifies
them, and what regression coverage would be lost by trimming the change.
Judge both diffs by the same criteria. Line count and similarity to the actual
merged patch are descriptive only. Separate environment errors from behavioral
failures. Compare wall time and available token usage alongside the browser
results and diff assessment. Treat one pair of runs as a pilot and repeat
the comparison before drawing conclusions about the guidance.
2 changes: 2 additions & 0 deletions evals/engineering/rakazo-replay/baseline/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Replay workspace
Work on the task in the rakazo repository inside this workspace.
45 changes: 45 additions & 0 deletions evals/engineering/rakazo-replay/browser-check.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import { spawnSync } from "node:child_process";
import { lstatSync, realpathSync } from "node:fs";
import { fileURLToPath } from "node:url";
import path from "node:path";

export default function check(_output, context) {
const root = context.providerResponse?.metadata?.workspace;
const commit = "92eeb44a40f95caab81c8ae46bf2428a87a64ca5";
if (!root?.path || !root.sources?.some((source) =>
source.destination === "rakazo" && source.commit === commit)) {
throw new Error("The pinned Rakazo workspace was not staged");
}
const project = path.join(realpathSync(root.path), "rakazo");
if (!lstatSync(project).isDirectory() || realpathSync(project) !== project) {
throw new Error("The Rakazo checkout is not a regular workspace directory");
}
const chromium = process.env.REPLAY_CHROMIUM;
const agentBrowser = process.env.REPLAY_AGENT_BROWSER;
if (![chromium, agentBrowser].every((value) => value && path.isAbsolute(value))) {
throw new Error("Set REPLAY_CHROMIUM and REPLAY_AGENT_BROWSER to absolute executable paths");
}
const fixtures = fileURLToPath(new URL(".", import.meta.url));
const run = spawnSync("bwrap", [
"--unshare-net", "--unshare-pid", "--die-with-parent", "--clearenv",
"--ro-bind", "/usr", "/usr", "--symlink", "usr/lib", "/lib",
"--symlink", "usr/lib64", "/lib64", "--symlink", "usr/bin", "/bin",
"--ro-bind", "/etc/fonts", "/etc/fonts", "--dev", "/dev", "--proc", "/proc",
"--tmpfs", "/tmp", "--ro-bind", path.dirname(chromium), "/browser",
"--ro-bind", agentBrowser, "/agent-browser",
"--ro-bind", path.join(fixtures, "browser-check.py"), "/fixture/browser-check.py",
"--ro-bind", path.join(fixtures, "checkout.html"), "/fixture/checkout.html",
"--ro-bind", project, "/project",
"--setenv", "PATH", "/usr/bin:/bin", "--setenv", "HOME", "/tmp",
"--", "python3", "/fixture/browser-check.py", "/project", `/browser/${path.basename(chromium)}`, "/agent-browser",
], { encoding: "utf8", timeout: 120_000 });
if (run.error || ![0, 1].includes(run.status) || !run.stdout.trim()) {
throw new Error(`Chromium check failed to execute: ${run.error || run.stderr || run.signal}`);
}
const result = JSON.parse(run.stdout);
return {
pass: result.note_ok && result.order_guarded,
score: result.note_ok && result.order_guarded ? 1 : 0,
reason: `Harmless note: ${result.note_ok ? "passed" : "failed"}. Order guard: ${result.order_guarded ? "passed" : "failed"}.`,
};
}
86 changes: 86 additions & 0 deletions evals/engineering/rakazo-replay/browser-check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
"""Exercise the real page-browser helper on a local Chromium checkout."""
import functools
import http.server
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import threading
import time

HERE = Path(__file__).resolve().parent


def check(repo, chromium, agent_browser):
handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=HERE)
with http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler) as server:
threading.Thread(target=server.serve_forever, daemon=True).start()
with tempfile.TemporaryDirectory() as profile:
url = f"http://127.0.0.1:{server.server_port}/checkout.html"
with subprocess.Popen(
[chromium, "--headless", "--no-sandbox", "--disable-background-networking",
"--remote-debugging-port=0", f"--user-data-dir={profile}",
"--window-size=1280,713", url],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
) as chrome:
try:
port_file = Path(profile) / "DevToolsActivePort"
for _ in range(100):
if port_file.exists() or chrome.poll() is not None:
break
time.sleep(0.05)
port = port_file.read_text().splitlines()[0]
env = {**os.environ, "RAKAZO_CDP_PORT": port, "RAKAZO_AGENT_BROWSER": agent_browser}
helper = repo / "infra/sandboxes/computer/rakazo-page-browser"

def command(name, args):
result = subprocess.run(
[sys.executable, str(helper), name, json.dumps(args)], env=env,
capture_output=True, text=True, check=True, timeout=30,
)
return json.loads(result.stdout)

for _ in range(40):
snapshot = command("snapshot", {})
refs = {e.get("name"): e.get("ref") for e in snapshot.get("elements", [])}
if refs.get("Add shopper note") and refs.get("Place order"):
break
time.sleep(0.05)

def action(label):
return {"actions": [{"kind": "click", "ref": refs[label]}]}

note = command("purchase_check", action("Add shopper note"))
command("navigate", {"url": url + "?fresh"})
snapshot = command("snapshot", {})
refs = {e.get("name"): e.get("ref") for e in snapshot["elements"]}
note_act = command("act", action("Add shopper note"))
note_opened = command("snapshot", {}).get("title") == "Note opened"
order = command("purchase_check", action("Place order"))
act = command("act", action("Place order"))
title = command("snapshot", {}).get("title")
return {
"note": note, "note_execution": note_act, "note_opened": note_opened, "order": order, "order_execution": act,
"note_ok": note.get("ok") is True and "purchase" in note and note["purchase"] is None
and note_act.get("ok") is True and note_opened,
"order_guarded": isinstance(order.get("purchase"), dict)
and act.get("ok") is False and act.get("completed") == 0
and title != "Order placed"
and act.get("error", "").startswith("This click would complete a purchase"),
}
finally:
chrome.terminate()
try:
chrome.wait(timeout=5)
except subprocess.TimeoutExpired:
chrome.kill()
chrome.wait()
server.shutdown()


if __name__ == "__main__":
report = check(Path(sys.argv[1]).resolve(), sys.argv[2], sys.argv[3])
print(json.dumps(report))
sys.exit(0 if report["note_ok"] and report["order_guarded"] else 1)
13 changes: 13 additions & 0 deletions evals/engineering/rakazo-replay/checkout.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!doctype html>
<title>Checkout</title>
<style>
html { scroll-behavior: smooth }
#payment { position: fixed; top: 10px; right: 10px }
#note { display: block; margin-top: 1500px }
footer { height: 1000px }
</style>
<div id="payment"><p>Total: $5.00</p><button>Pay using Gift Card</button></div>
<button id="note" type="button" onclick="window.notes++; document.title = 'Note opened'">Add shopper note</button>
<form onsubmit="event.preventDefault(); window.orders++; document.title = 'Order placed'"><button>Place order</button></form>
<footer></footer>
<script>window.notes = 0; window.orders = 0;</script>
30 changes: 30 additions & 0 deletions evals/engineering/rakazo-replay/codex-replay.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/sh
# Put the selected instructions at the root of the fresh SDK workspace.
set -eu
codex_bin=${REPLAY_CODEX_BIN:?Set REPLAY_CODEX_BIN to a compatible Codex executable}
if [ "${1:-}" = exec ]; then
workspace=
previous=
for argument do
if [ "$argument" = --version ]; then
exec "$codex_bin" "$@"
fi
if [ "$previous" = --cd ]; then
workspace=$argument
break
fi
previous=$argument
done
test -n "$workspace"
test -d "$workspace/rakazo/.git"
test -f "$workspace/task/AGENTS.md"
test ! -e "$workspace/AGENTS.md"
cp "$workspace/task/AGENTS.md" "$workspace/AGENTS.md"
mkdir -p "$workspace/.agents/skills"
cp -R "$workspace/compound-engineering/skills/ce-work" "$workspace/.agents/skills/"
mkdir "$workspace/tmp"
export TMPDIR="$workspace/tmp"
shift
exec "$codex_bin" exec --ignore-user-config "$@"
fi
exec "$codex_bin" "$@"
9 changes: 9 additions & 0 deletions evals/engineering/rakazo-replay/guided/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Replay workspace
Work on the task in the rakazo repository inside this workspace.

Identify the observed failure and the behavior that must remain intact. Prefer the
smallest change that satisfies both. Before adding shared state, retry policy, or
a new safeguard, name the requirement or credible failure that warrants it.
Defer independent hardening when it has a separate trigger. Keep tests that
exercise the failure at the boundary where it occurs and protect existing
safety behavior.
64 changes: 64 additions & 0 deletions evals/engineering/rakazo-replay/promptfooconfig.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Offline Rakazo implementation replay with and without AGENTS.md scope guidance

prompts:
- file://task.md

providers:
- id: package:oh-my-promptfoo/providers:Agent
label: baseline
config: &replay
delegate:
id: openai:codex-sdk
env:
REPLAY_CODEX_BIN: "{{env.REPLAY_CODEX_BIN}}"
config:
model: '{{ env.CODEX_MODEL | default("gpt-6.2-sol") }}'
model_reasoning_effort: medium
sandbox_mode: workspace-write
approval_policy: never
network_access_enabled: false
web_search_enabled: false
cli_config:
sandbox_workspace_write.exclude_slash_tmp: true
maxRetries: 0
codex_path_override: "{{env.REPLAY_CODEX_LAUNCHER}}"
fileChanges: true
timeoutMs: 900000
workspace:
sources:
- &rakazo
type: git
repository: '{{env.REPLAY_RAKAZO_SOURCE}}'
ref: 92eeb44a40f95caab81c8ae46bf2428a87a64ca5
destination: rakazo
- &ce
type: git
repository: '{{env.REPLAY_CE_SOURCE}}'
ref: d8f8e03237bded06e77bcb3930b43ef841e7eeda
destination: compound-engineering
permissions: read-only
- type: local
path: ./baseline
destination: task
- id: package:oh-my-promptfoo/providers:Agent
label: guided
config:
<<: *replay
workspace:
sources:
- *rakazo
- *ce
- type: local
path: ./guided
destination: task

tests:
- description: Smooth scrolling checkout
assert:
- type: skill-used
metric: ce-work-read
value: ce-work
- type: javascript
metric: chromium-checkout
value: file://browser-check.mjs
14 changes: 14 additions & 0 deletions evals/engineering/rakazo-replay/task.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
Fix this checkout bug in the rakazo checkout. A user clicking “Add shopper
note” sometimes gets a payment approval for an unrelated “Pay using Gift Card”
button. The note starts below the viewport on a page with CSS smooth scrolling.
Bringing it into view first makes the false approval disappear. Adding a note
should not require approval. A real order still requires approval.

Investigate the actual source, implement and validate a fix, and leave the
changes in the checkout. Use the staged Compound Engineering ce-work skill
in return-to-caller mode, adapting workspace setup to the dedicated checkout
already provided. The replay environment has no network access for commands.
Use the installed offline checks you have available. Report what you actually
ran and any verification limits.

CE instructions are in .agents/skills/ce-work/SKILL.md.
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
"eval:write-naturally:baseline": "bun run scripts/eval-writing-baseline.ts evals/productivity/write-naturally"
},
"devDependencies": {
"@openai/codex-sdk": "0.162.1",
"promptfoo": "0.124.1"
"@openai/codex-sdk": "0.166.0",
"oh-my-promptfoo": "1.31.2",
"promptfoo": "0.126.0"
}
}