Skip to content

test(engineering): add public judgment evals - #19

Open
christso wants to merge 2 commits into
mainfrom
test/engineering-judgment-evals
Open

christso wants to merge 2 commits into
mainfrom
test/engineering-judgment-evals

Conversation

@christso

Copy link
Copy Markdown
Contributor

Summary

Add six public engineering judgment evals for deciding which mechanisms and tests a root-cause fix needs. Two cases draw on the reported checkout scroll race and the ensuing review. Four synthetic cases test when a measured deadline, payout safeguard, or browser regression earns its place and when a minimal fix already has sufficient proof.

The Promptfoo suite reads pinned Compound Engineering v3.38.1 skill and persona files and checks that both files' contents appear in successful Codex command output. It exercises one reviewer pass per fixture. Full workflow orchestration and browser execution are outside the eval.

Related: elie222/rakazo#4728

Validation

  • Ran bun run validate, bun test (62 passed), Promptfoo config validation, and a deterministic probe for fixture inventory, source pinning, false read traces, and cleanup.
  • Ran the checkout feedback and invoice deadline cases with gpt-6.1-sol and the gpt-5.6-luna judge. Both passed with zero errors. The initial gpt-6.2-sol attempt was rejected by this ChatGPT-authenticated Codex connection before model evaluation.
  • Calibrated the judge with five scripted responses. It accepted both defensible responses and rejected overbuilding, removal of a required deadline, and a false guarantee about the second checkout read.
  • Completed independent standards and spec reviews. Addressed the trace false positive and clarified the fixture's control-replacement evidence and approval guarantees.

Compound Engineering

Exercise pinned Compound Engineering reviewer passes with contrasting evidence
for root-cause fixes, justified safeguards, and focused regression coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant