Repository navigation
feat(exp35): harness rebuild for the pre-registered four-factor benchmark - #212
Conversation
…ness review Corrects the Experiment 35 plan before the harness rebuild. Adds amendment A5 to the pre-registration, protocol section 0.14 and plan.json. Raises the readiness change to 145 tasks and adds decision D39.
H8b is a comparison at a fixed pre-assembly evidence budget, with the realised token total reported beside it. A closed answer scores 0 and is not judged; the unjudged-row rule covers existing answers only. Pre-freeze runs append to output/run-records.jsonl with prefreeze: true. The FreshStack export keeps row text byte for byte. The no-page-label placeholder is FinanceBench only. The search stage saves pinned token counts per chunk. Spec scenarios for G1, the tree hash and the ranked-chunk option are made consistent. Protocol 0.14 items 26 to 31.
Adds a typed rate-limit record to the answer pipeline and its transports. A 402 reply from the provider now gets one short retry before the answer fails. The OpenAPI contract and the web answer status show the new record.
…tion and llama.cpp reranker Native sparse queries now read the full-text index without writing to it; ingest prepares the index. Search gains an option that returns ranked chunks without context assembly. Adds a reranker that calls a llama.cpp server, registered beside the existing ones.
… reranker Documents the typed rate-limit record in the answer replies. Explains that ingest now builds the full-text index and queries only read it. Adds the llama.cpp reranker settings to the guides and to .env.example.
…gates Adds the shared run gate, ordered row output and hosted-call helpers to experiments/_lib. Adds the Experiment 35 scorer, bootstrap and cluster statistics, budget, split and label gates, with tests. Old Experiment 35 stage tests fail until the stage rewrite lands in this branch.
…nto feat/exp35-main-run-readiness
Pre-freeze and smoke runs now write their run record to output/run-records.jsonl with a prefreeze or smoke flag. A main-run check can skip the flagged lines. The experiment records guide describes both flags.
The default pytest paths now include experiments/_lib and the Experiment 35 prefreeze folder. A plain pytest run therefore runs the tests that live beside that code.
…four-factor design The setup table now covers the embedder, chunk size, reranker and hybrid factors, with a new command line and a passage preparation step. Dataset loading, the index and answer stages, outputs and the seal follow the same setup names.
… script Adds the Ragas scoring stage, the reranker check, the release kit and a pod test session script, with a stronger identifier check before release. Smaller fixes: the faithfulness pool leaves out refusal claims, the design effect has a floor of 1, and Ragas replies record the model and provider.
…nto feat/exp35-main-run-readiness
Adds the stage modules that search each arm, write answers and run the judges in parallel, plus the closed-book and oracle controls. A shared stage kit holds the common resume and record helpers.
Adds the join of answers and verdicts, the determinism and rebuild gates (G1 and G2), the scoring run with length and cross-table checks, and the claim, segment and faithfulness pool stages that guard half B.
…unner layout The runner, arms, datasets, Ragas and scoring code now refuse plan values that the frozen plan does not pin, and every stage reads and writes the shared runner layout. New end-to-end, wiring and half-B tests cover this.
…nto feat/exp35-main-run-readiness
The fast suite gains 862 test IDs and loses 69 against v3 2c8fb73. One new case skips without the chroma extra, so executed rises by 792. Its file joins the list of files allowed to carry the Chroma skip reason.
… for the server judge Ingest now builds the native sparse index in a worker thread, so the server event loop stays responsive. A typed judge built by the MCP tool or the HTTP route now gets the short 402 retry, as verify_answer gives the other judges. The MCP tools guide and the tests cover both changes.
…dge cap The stage key now holds the run mode, so a smoke or pre-freeze run is never resumed as a main run. Search refuses an index built with another configuration, searches run one at a time, and the judge error cap now closes only the failing claim. Ragas and the ordered-row writer repair cut last lines on resume.
…ber match and half-B writes The analysis uses the registered sign-flip draw. An unjudged run drops its cell, each exclusion is counted, and an unparseable verdict counts not supported. Number matching reads fiscal-year labels and reads a scale letter only with a currency sign. Reports that hold half-B scores now run behind the half-B guard. New tests cover the missing outcomes, the score join and the guard.
The fast suite gains 56 test IDs and loses 3 against e38446a. Every new case runs without the chroma extra, so executed rises by 53.
Protocol 0.14 items 55 to 64 (source: code review R2, R3; fix notes F5, F6). Pins the sign-flip draw, number-match reading rules and the handling of missing, empty, unjudged, not-defined and unparseable rows in section 12. Adds run mode, index configuration check, one search at a time and the corrected run command lines. The unjudged-scored-0 sensitivity now also scores not-defined Ragas runs 0. Co-Authored-By: Claude <noreply@anthropic.com>
The answer sampling record and the claim-judge call record now hold model_returned. It is the model ID the reply names, or null when the reply names none. The OpenAPI contract, the MCP tools guide and the parity fixture follow.
…e a smoke join Answer and judge rows now copy the model the reply names from the OMRS records. write_confirmatory and write_report refuse a missing, failed or smoke join.
The pin moves from 13430 to 13435 executed cases. A test-ID diff against cbc4395 shows 6 added and 1 removed.
…nto feat/exp35-main-run-readiness
|
Important Review skippedToo many files! This PR contains 178 files, which is 78 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## v3 #212 +/- ##
==========================================
+ Coverage 94.91% 94.93% +0.01%
==========================================
Files 464 467 +3
Lines 37437 37601 +164
Branches 4812 4835 +23
==========================================
+ Hits 35533 35695 +162
- Misses 1264 1266 +2
Partials 640 640
|
… refusals The score35 refusal test read the operator data folder, so it passed only on a machine that has one. It now points EXP35_DATA_DIR at a fixture copy. score35.main prints a dataset refusal and returns 2, as prepare35 does.
Protocol 0.14 items 65 to 67 (source: PR 211 review round 2). The H8b sensitivity must keep the sign of the H8b result on both datasets; an empty reply from the answer model is a final empty answer; a smoke join counts its smoke run-record line. Wording fixes: Ragas final-row and resume rules (not-defined metrics, out-of-credit stop, partial row kept), small-collection search test, MDE rows without a SESOI, section 5 run order (half-A selection and OSF time-stamp before answers), target CLI label and the determinism check against the search-stage list. Co-Authored-By: Claude <noreply@anthropic.com>
…n search test - answers35: an EMPTY_REPLY_ERROR result is a returned empty answer. It is saved with empty text, final, never resent (protocol 0.14 item 66). - select35: a main entry point reads the half-A scores, the index fingerprints, halves.json and plan.json, writes selection.json, and exits 2 on a refusal (protocol section 5 step 3b). - factorial35: the H8b sensitivity agrees only with the H8b sign of each row (protocol 0.14 item 65). - A restrictive threshold test for unassembled search (task 0.7a); the code already passed it.
…nto feat/exp35-main-run-readiness
What this is
The Experiment 35 harness, rebuilt for the pre-registered four-factor design, plus the OMRS app changes it needs. It replaces the old six-arm harness, which a code review found could not run the registered design (36 findings, 7 blockers).
This branch sits on top of pull request 211 (the plan corrections, amendment A5). Review and merge 211 first; this diff then shrinks to the code.
What changed
unassembledsearch option onEngine.search; a llama.cpp reranker backend; run-record flags for pre-freeze and smoke runs.How it was checked
Not in this pull request
plan.jsonand are listed inpins.pending; every main stage refuses to start until they are filled.Do not merge without the owner's word.