Skip to content

feat(exp35): harness rebuild for the pre-registered four-factor benchmark - #212

Merged
cmdaltctr merged 35 commits into
v3from
feat/exp35-main-run-readiness
Nov 17, 2026
Merged

cmdaltctr merged 35 commits into
v3from
feat/exp35-main-run-readiness

Conversation

@cmdaltctr

Copy link
Copy Markdown
Owner

What this is

The Experiment 35 harness, rebuilt for the pre-registered four-factor design, plus the OMRS app changes it needs. It replaces the old six-arm harness, which a code review found could not run the registered design (36 findings, 7 blockers).

This branch sits on top of pull request 211 (the plan corrections, amendment A5). Review and merge 211 first; this diff then shrinks to the code.

What changed

  • OMRS app: a typed rate-limit record and a short 402 retry on the answer and judge paths; the model a hosted reply names is recorded; the native full-text query is read-only and the index is built at ingest, off the event loop; an unassembled search option on Engine.search; a llama.cpp reranker backend; run-record flags for pre-freeze and smoke runs.
  • Shared experiment library: a gate for hosted calls, rows that survive a stop with the registered resend rule and error cap, crash-safe trace lines, a refusal of unfrozen plans and dirty trees, and a server-argument check.
  • Experiment 35: the setup tables, command line, folder layout, datasets, ingest, seal, search, answer, judge, closed-book, oracle, Ragas and claim-kind stages; the join, gates G1 and G2, the scoring run with the half-B guard; the statistics, label gates and release kit.

How it was checked

  • Every task was built test-first, and each fix was checked by breaking the code and seeing its test fail.
  • Three independent code reviews (app, run pipeline, analysis) found 35 defects. Every one scored 4 or more (likelihood times impact) is fixed; the rules the fixes needed are registered in pull request 211.
  • Local CI passed: 13,431 Python tests, 2,678 web tests, OpenSpec strict, lint and the import contracts. The clean-base test pin is updated by collected test IDs.
  • Nothing ran against a real model, pod, index or dataset. Every test uses stubs. The pre-freeze pod session (task 9.2) is the first real run.

Not in this pull request

  • No pre-freeze run, no download, no paid call, no label. Values that only a pre-freeze task can fill stay null in plan.json and are listed in pins.pending; every main stage refuses to start until they are filled.
  • Open code tasks: the sensitivity and report tables of task 16.20 and the closed-book and oracle report tables.

Do not merge without the owner's word.

cmdaltctr and others added 29 commits November 16, 2026 23:09
…ness review

Corrects the Experiment 35 plan before the harness rebuild.
Adds amendment A5 to the pre-registration, protocol section 0.14 and plan.json.
Raises the readiness change to 145 tasks and adds decision D39.
H8b is a comparison at a fixed pre-assembly evidence budget, with the realised token total reported beside it. A closed answer scores 0 and is not judged; the unjudged-row rule covers existing answers only. Pre-freeze runs append to output/run-records.jsonl with prefreeze: true. The FreshStack export keeps row text byte for byte. The no-page-label placeholder is FinanceBench only. The search stage saves pinned token counts per chunk. Spec scenarios for G1, the tree hash and the ranked-chunk option are made consistent. Protocol 0.14 items 26 to 31.
Adds a typed rate-limit record to the answer pipeline and its transports.
A 402 reply from the provider now gets one short retry before the answer fails.
The OpenAPI contract and the web answer status show the new record.
…tion and llama.cpp reranker

Native sparse queries now read the full-text index without writing to it; ingest prepares the index.
Search gains an option that returns ranked chunks without context assembly.
Adds a reranker that calls a llama.cpp server, registered beside the existing ones.
… reranker

Documents the typed rate-limit record in the answer replies.
Explains that ingest now builds the full-text index and queries only read it.
Adds the llama.cpp reranker settings to the guides and to .env.example.
…gates

Adds the shared run gate, ordered row output and hosted-call helpers to experiments/_lib.
Adds the Experiment 35 scorer, bootstrap and cluster statistics, budget, split and label gates, with tests.
Old Experiment 35 stage tests fail until the stage rewrite lands in this branch.
Pre-freeze and smoke runs now write their run record to output/run-records.jsonl with a prefreeze or smoke flag.
A main-run check can skip the flagged lines. The experiment records guide describes both flags.
The default pytest paths now include experiments/_lib and the Experiment 35 prefreeze folder.
A plain pytest run therefore runs the tests that live beside that code.
…four-factor design

The setup table now covers the embedder, chunk size, reranker and hybrid factors, with a new command line and a passage preparation step.
Dataset loading, the index and answer stages, outputs and the seal follow the same setup names.
… script

Adds the Ragas scoring stage, the reranker check, the release kit and a pod test session script, with a stronger identifier check before release.
Smaller fixes: the faithfulness pool leaves out refusal claims, the design effect has a floor of 1, and Ragas replies record the model and provider.
Adds the stage modules that search each arm, write answers and run the
judges in parallel, plus the closed-book and oracle controls.
A shared stage kit holds the common resume and record helpers.
Adds the join of answers and verdicts, the determinism and rebuild gates
(G1 and G2), the scoring run with length and cross-table checks, and the
claim, segment and faithfulness pool stages that guard half B.
…unner layout

The runner, arms, datasets, Ragas and scoring code now refuse plan values
that the frozen plan does not pin, and every stage reads and writes the
shared runner layout. New end-to-end, wiring and half-B tests cover this.
The fast suite gains 862 test IDs and loses 69 against v3 2c8fb73.
One new case skips without the chroma extra, so executed rises by 792.
Its file joins the list of files allowed to carry the Chroma skip reason.
… for the server judge

Ingest now builds the native sparse index in a worker thread, so the server event loop stays responsive.
A typed judge built by the MCP tool or the HTTP route now gets the short 402 retry, as verify_answer gives the other judges.
The MCP tools guide and the tests cover both changes.
…dge cap

The stage key now holds the run mode, so a smoke or pre-freeze run is never resumed as a main run.
Search refuses an index built with another configuration, searches run one at a time, and the judge error cap now closes only the failing claim.
Ragas and the ordered-row writer repair cut last lines on resume.
…ber match and half-B writes

The analysis uses the registered sign-flip draw. An unjudged run drops its cell, each exclusion is counted, and an unparseable verdict counts not supported.
Number matching reads fiscal-year labels and reads a scale letter only with a currency sign. Reports that hold half-B scores now run behind the half-B guard.
New tests cover the missing outcomes, the score join and the guard.
The fast suite gains 56 test IDs and loses 3 against e38446a.
Every new case runs without the chroma extra, so executed rises by 53.
Protocol 0.14 items 55 to 64 (source: code review R2, R3; fix notes F5, F6).
Pins the sign-flip draw, number-match reading rules and the handling of
missing, empty, unjudged, not-defined and unparseable rows in section 12.
Adds run mode, index configuration check, one search at a time and the
corrected run command lines. The unjudged-scored-0 sensitivity now also
scores not-defined Ragas runs 0.

Co-Authored-By: Claude <noreply@anthropic.com>
The answer sampling record and the claim-judge call record now hold model_returned.
It is the model ID the reply names, or null when the reply names none.
The OpenAPI contract, the MCP tools guide and the parity fixture follow.
…e a smoke join

Answer and judge rows now copy the model the reply names from the OMRS records.
write_confirmatory and write_report refuse a missing, failed or smoke join.
The pin moves from 13430 to 13435 executed cases.
A test-ID diff against cbc4395 shows 6 added and 1 removed.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Too many files!

This PR contains 178 files, which is 78 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2c8fb73 and d8c5aa2.

Files selected for processing: 178

  • .env.example
  • docs/guides/architecture.md
  • docs/guides/cli-reference.md
  • docs/guides/configuration.md
  • docs/guides/experiment-records.md
  • docs/guides/library-usage.md
  • docs/guides/mcp-tools.md
  • docs/guides/reranker.md
  • docs/proposals/first-publication-benchmark.md
  • experiments/35-full-rag-benchmark-2026-11-01/answers35.py
  • experiments/35-full-rag-benchmark-2026-11-01/arms.py
  • experiments/35-full-rag-benchmark-2026-11-01/bootstrap35.py
  • experiments/35-full-rag-benchmark-2026-11-01/budget35.py
  • experiments/35-full-rag-benchmark-2026-11-01/cells35.py
  • experiments/35-full-rag-benchmark-2026-11-01/claimkinds35.py
  • experiments/35-full-rag-benchmark-2026-11-01/claimstage35.py
  • experiments/35-full-rag-benchmark-2026-11-01/cli35.py
  • experiments/35-full-rag-benchmark-2026-11-01/closedbook35.py
  • experiments/35-full-rag-benchmark-2026-11-01/correct35.py
  • experiments/35-full-rag-benchmark-2026-11-01/crosstable35.py
  • experiments/35-full-rag-benchmark-2026-11-01/datasets35.py
  • experiments/35-full-rag-benchmark-2026-11-01/determinism35.py
  • experiments/35-full-rag-benchmark-2026-11-01/factorial35.py
  • experiments/35-full-rag-benchmark-2026-11-01/faithpool35.py
  • experiments/35-full-rag-benchmark-2026-11-01/join35.py
  • experiments/35-full-rag-benchmark-2026-11-01/joinscores35.py
  • experiments/35-full-rag-benchmark-2026-11-01/judges35.py
  • experiments/35-full-rag-benchmark-2026-11-01/labelpage35.py
  • experiments/35-full-rag-benchmark-2026-11-01/labels35.py
  • experiments/35-full-rag-benchmark-2026-11-01/lengthcheck35.py
  • experiments/35-full-rag-benchmark-2026-11-01/live35.py
  • experiments/35-full-rag-benchmark-2026-11-01/mde35.py
  • experiments/35-full-rag-benchmark-2026-11-01/oracle35.py
  • experiments/35-full-rag-benchmark-2026-11-01/outputs35.py
  • experiments/35-full-rag-benchmark-2026-11-01/plan.json
  • experiments/35-full-rag-benchmark-2026-11-01/prefreeze/pod_test_session.sh
  • experiments/35-full-rag-benchmark-2026-11-01/prefreeze/ragas_stage.py
  • experiments/35-full-rag-benchmark-2026-11-01/prefreeze/test_pod_test_session_script.py
  • experiments/35-full-rag-benchmark-2026-11-01/prepare35.py
  • experiments/35-full-rag-benchmark-2026-11-01/preregistration.md
  • experiments/35-full-rag-benchmark-2026-11-01/protocol.md
  • experiments/35-full-rag-benchmark-2026-11-01/ragas35.py
  • experiments/35-full-rag-benchmark-2026-11-01/ragas35_judge.py
  • experiments/35-full-rag-benchmark-2026-11-01/rebuild35.py
  • experiments/35-full-rag-benchmark-2026-11-01/refusalreport35.py
  • experiments/35-full-rag-benchmark-2026-11-01/refusals35.py
  • experiments/35-full-rag-benchmark-2026-11-01/release35.py
  • experiments/35-full-rag-benchmark-2026-11-01/rerank_check35.py
  • experiments/35-full-rag-benchmark-2026-11-01/run_arms.py
  • experiments/35-full-rag-benchmark-2026-11-01/score35.py
  • experiments/35-full-rag-benchmark-2026-11-01/scoring35.py
  • experiments/35-full-rag-benchmark-2026-11-01/search35.py
  • experiments/35-full-rag-benchmark-2026-11-01/segments35.py
  • experiments/35-full-rag-benchmark-2026-11-01/select35.py
  • experiments/35-full-rag-benchmark-2026-11-01/split35.py
  • experiments/35-full-rag-benchmark-2026-11-01/stagekit35.py
  • experiments/35-full-rag-benchmark-2026-11-01/stages35.py
  • experiments/35-full-rag-benchmark-2026-11-01/verdicts35.py
  • experiments/_lib/hosted_calls.py
  • experiments/_lib/llamacpp_runtime.py
  • experiments/_lib/ordered_rows.py
  • experiments/_lib/release_check.py
  • experiments/_lib/run_guard.py
  • experiments/_lib/run_record.py
  • experiments/_lib/stats_cluster.py
  • experiments/_lib/stats_d6.py
  • experiments/_lib/trace.py
  • openspec/changes/exp35-main-run-readiness/design.md
  • openspec/changes/exp35-main-run-readiness/proposal.md
  • openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md
  • openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md
  • openspec/changes/exp35-main-run-readiness/specs/benchmark-replicate-runs/spec.md
  • openspec/changes/exp35-main-run-readiness/specs/retrieval-context-assembly/spec.md
  • openspec/changes/exp35-main-run-readiness/tasks.md
  • openspec/changes/experiment-35-full-document-rag-benchmark/design.md
  • openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md
  • openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md
  • pyproject.toml
  • src/omrs/compose_judges.py
  • src/omrs/config/rules.py
  • src/omrs/core/answer/graph_adapters.py
  • src/omrs/core/answer/judges/base.py
  • src/omrs/core/answer/pipeline.py
  • src/omrs/core/answer/rate_limits.py
  • src/omrs/core/answer/results.py
  • src/omrs/core/answer/stages.py
  • src/omrs/core/answer/verify.py
  • src/omrs/core/experiment_records/run_record.py
  • src/omrs/core/ingestion/pipeline.py
  • src/omrs/core/retrieval/backend.py
  • src/omrs/core/retrieval/listing.py
  • src/omrs/core/retrieval/pipeline.py
  • src/omrs/core/retrieval/registry.py
  • src/omrs/core/retrieval/reranker_llamacpp.py
  • src/omrs/core/retrieval/sparse_dispatch.py
  • src/omrs/core/vectordb/base.py
  • src/omrs/core/vectordb/lancedb/fts.py
  • src/omrs/engine.py
  • src/omrs/engine_answer.py
  • src/omrs/engine_profiles.py
  • src/omrs/transports/api/openapi.yaml
  • tests/_exp35_e2e_fixtures.py
  • tests/_exp35_fixtures.py
  • tests/_exp35_join_fixtures.py
  • tests/_exp35_label_fixtures.py
  • tests/_exp35_mainrun_fixtures.py
  • tests/_exp35_ragas_fixtures.py
  • tests/fixtures/pipeline_parity/answer/expected.json
  • tests/test_answer_transport_mcp_rate_limit.py
  • tests/test_clean_base_tripwire.py
  • tests/test_experiment35_answer_review.py
  • tests/test_experiment35_arms.py
  • tests/test_experiment35_bootstrap.py
  • tests/test_experiment35_budget.py
  • tests/test_experiment35_claimkinds.py
  • tests/test_experiment35_claimstage.py
  • tests/test_experiment35_closedbook_oracle.py
  • tests/test_experiment35_correct.py
  • tests/test_experiment35_crosstable.py
  • tests/test_experiment35_datasets.py
  • tests/test_experiment35_determinism.py
  • tests/test_experiment35_factorial.py
  • tests/test_experiment35_factorial_missing.py
  • tests/test_experiment35_guard.py
  • tests/test_experiment35_halfb.py
  • tests/test_experiment35_halfb_stages.py
  • tests/test_experiment35_join.py
  • tests/test_experiment35_join_scores.py
  • tests/test_experiment35_labels.py
  • tests/test_experiment35_layout.py
  • tests/test_experiment35_layout_e2e.py
  • tests/test_experiment35_lengthcheck.py
  • tests/test_experiment35_mde.py
  • tests/test_experiment35_one_round.py
  • tests/test_experiment35_outputs.py
  • tests/test_experiment35_parallel_answers.py
  • tests/test_experiment35_parallel_judges.py
  • tests/test_experiment35_ragas.py
  • tests/test_experiment35_ragas_setup.py
  • tests/test_experiment35_rebuild.py
  • tests/test_experiment35_refusal_report.py
  • tests/test_experiment35_refusals.py
  • tests/test_experiment35_release.py
  • tests/test_experiment35_rerank_check.py
  • tests/test_experiment35_run_arms.py
  • tests/test_experiment35_run_mode.py
  • tests/test_experiment35_run_stages.py
  • tests/test_experiment35_score.py
  • tests/test_experiment35_scoring.py
  • tests/test_experiment35_search.py
  • tests/test_experiment35_segments.py
  • tests/test_experiment35_select.py
  • tests/test_experiment35_select_cli.py
  • tests/test_experiment35_split.py
  • tests/test_experiment35_stages.py
  • tests/test_experiment35_stats_cluster.py
  • tests/test_experiment35_wiring.py
  • tests/test_experiment_hosted_calls.py
  • tests/test_experiment_ordered_rows.py
  • tests/test_experiment_release_check.py
  • tests/test_experiment_run_guard.py
  • tests/test_experiment_run_record_lib.py
  • tests/test_experiment_stats_d6.py
  • tests/test_native_sparse_backend.py
  • tests/test_native_sparse_prepare_at_ingest.py
  • tests/test_native_sparse_read_only.py
  • tests/test_vectordb_contract.py
  • tests/unit/test_answer_rate_limit.py
  • tests/unit/test_answer_response_contract.py
  • tests/unit/test_answer_sampling_record.py
  • tests/unit/test_claim_judges.py
  • tests/unit/test_compose_judges.py
  • tests/unit/test_experiment_records_run_record_flags.py
  • tests/unit/test_reranker_llamacpp.py
  • tests/unit/test_search_unassembled.py
  • web/src/api/documents.ts
  • web/src/chat/answerStatus.test.ts
  • web/src/chat/answerStatus.ts

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.87629% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.93%. Comparing base (2c8fb73) to head (d8c5aa2).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/omrs/core/retrieval/listing.py 80.00% 3 Missing and 1 partial ⚠️
src/omrs/core/vectordb/lancedb/fts.py 81.81% 1 Missing and 1 partial ⚠️
src/omrs/engine_answer.py 71.42% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##               v3     #212      +/-   ##
==========================================
+ Coverage   94.91%   94.93%   +0.01%     
==========================================
  Files         464      467       +3     
  Lines       37437    37601     +164     
  Branches     4812     4835      +23     
==========================================
+ Hits        35533    35695     +162     
- Misses       1264     1266       +2     
  Partials      640      640              
Files with missing lines Coverage Δ
src/omrs/compose_judges.py 88.94% <100.00%> (+0.05%) ⬆️
src/omrs/config/rules.py 94.50% <ø> (ø)
src/omrs/core/answer/graph_adapters.py 95.94% <ø> (ø)
src/omrs/core/answer/judges/base.py 96.66% <100.00%> (+0.05%) ⬆️
src/omrs/core/answer/pipeline.py 100.00% <ø> (ø)
src/omrs/core/answer/rate_limits.py 100.00% <100.00%> (ø)
src/omrs/core/answer/results.py 100.00% <100.00%> (ø)
src/omrs/core/answer/stages.py 100.00% <100.00%> (ø)
src/omrs/core/answer/verify.py 99.32% <100.00%> (+0.01%) ⬆️
src/omrs/core/experiment_records/run_record.py 98.03% <100.00%> (+0.05%) ⬆️
... and 12 more

... and 1 file with indirect coverage changes

cmdaltctr and others added 6 commits November 17, 2026 17:32
… refusals

The score35 refusal test read the operator data folder, so it passed only
on a machine that has one. It now points EXP35_DATA_DIR at a fixture copy.
score35.main prints a dataset refusal and returns 2, as prepare35 does.
Protocol 0.14 items 65 to 67 (source: PR 211 review round 2). The H8b
sensitivity must keep the sign of the H8b result on both datasets; an
empty reply from the answer model is a final empty answer; a smoke join
counts its smoke run-record line. Wording fixes: Ragas final-row and
resume rules (not-defined metrics, out-of-credit stop, partial row kept),
small-collection search test, MDE rows without a SESOI, section 5 run
order (half-A selection and OSF time-stamp before answers), target CLI
label and the determinism check against the search-stage list.

Co-Authored-By: Claude <noreply@anthropic.com>
…n search test

- answers35: an EMPTY_REPLY_ERROR result is a returned empty answer. It is
  saved with empty text, final, never resent (protocol 0.14 item 66).
- select35: a main entry point reads the half-A scores, the index
  fingerprints, halves.json and plan.json, writes selection.json, and
  exits 2 on a refusal (protocol section 5 step 3b).
- factorial35: the H8b sensitivity agrees only with the H8b sign of each
  row (protocol 0.14 item 65).
- A restrictive threshold test for unassembled search (task 0.7a); the
  code already passed it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant