From 737678bdd88ed06434a6f67d8e52c9cf299d49c6 Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 17:08:17 +0000 Subject: [PATCH 1/9] docs(exp35): pre-registration and amendment A4 from the publication review - Add preregistration.md as the new governing file for Experiment 35. - Add protocol amendment A4 (section 0.13) with the fixes from the methods review (F1 to F30) and the consistency review (C1 to C24). - Update the first-publication proposal to Draft 7. - Align plan.json and the three OpenSpec changes with the new design. - Main design changes: FreshStack uses the full 49,514-document corpus; one clustered analysis method with no mixed model; fixed Holm families of 19, 19 and 8; H8 split into H8a and H8b; gates for faithfulness and correctness; rules for missing outcomes; a results-blind drop order; a Part 2 estimand rule and a shared reranker. --- docs/proposals/first-publication-benchmark.md | 236 +++++++--- .../plan.json | 117 ++++- .../preregistration.md | 444 ++++++++++++++++++ .../protocol.md | 77 ++- .../exp35-main-run-readiness/design.md | 85 +++- .../exp35-main-run-readiness/proposal.md | 21 +- .../specs/benchmark-answer-scoring/spec.md | 121 ++++- .../specs/benchmark-harness/spec.md | 102 +++- .../changes/exp35-main-run-readiness/tasks.md | 80 +++- .../design.md | 78 +-- .../proposal.md | 21 +- .../specs/codebase-rag-evaluation/spec.md | 21 +- .../specs/document-rag-evaluation/spec.md | 146 ++++-- .../specs/evaluation-observability/spec.md | 100 +++- .../tasks.md | 43 +- .../rag-benchmark-readiness-flow/design.md | 17 +- .../rag-benchmark-readiness-flow/proposal.md | 3 +- .../rag-benchmark-readiness-flow/tasks.md | 54 ++- 18 files changed, 1473 insertions(+), 293 deletions(-) create mode 100644 experiments/35-full-rag-benchmark-2026-11-01/preregistration.md diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index dfc1fa93..2fe4b77e 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -1,21 +1,24 @@ # First publication: a reproducible local RAG benchmark of OMRS -Draft 6, 13 November 2026 (proposal becomes the master plan; Part 1 aligned with Experiment 35 amendments A1–A3; RAGFlow as the single controlled competitor on held-out half B; haiku.rag dropped). Draft 5, 12 November 2026, added a controlled comparison and parked PipesHub for v4. Draft 4, 9 November 2026, added the EnterpriseRAG-Bench protocol. Plan only. No measured run has started. +Draft 7, 14 November 2026 (Experiment 35 amendment A4 after the publication-grade methods review, findings F1 to F30, plus the fixes from the post-fix consistency review, findings C1 to C24 in protocol.md section 0.13: the new pre-registration governs every Experiment 35 design fact; Part 2 gets its estimand, shared reranker, missing-outcome and label rules, and a list of what its protocol must hold; limitations and references fixed). Draft 6, 13 November 2026 (proposal becomes the master plan; Part 1 aligned with Experiment 35 amendments A1–A3; RAGFlow as the single controlled competitor on held-out half B; haiku.rag dropped). Draft 5, 12 November 2026, added a controlled comparison and parked PipesHub for v4. Draft 4, 9 November 2026, added the EnterpriseRAG-Bench protocol. Plan only. No measured run has started. This is the first paper of the fellowship programme. The manṭiq work in `v4` (authority-aware retrieval and reasoning gates) comes after it. This paper does not wait for that work. ## Master plan and source of truth -This proposal is the master plan for the first OMRS publication. It holds what the paper claims and why. Each experiment protocol holds how its part runs: counts, seeds, models, statistical models, gates and costs. +This proposal is the master plan for the first OMRS publication. It holds the scope, the order of work, what the paper claims and why, Part 2 and E1. Each experiment's governing file holds how its part runs: counts, seeds, models, statistical methods, gates and costs. -- Part 1 is Experiment 35. Its protocol is `experiments/35-full-rag-benchmark-2026-11-01/protocol.md`. Section 0 (amendments A1, A2, A2c and A3) is the current design. -- Part 2 (controlled comparison with RAGFlow) gets its own protocol before the freeze, and E1 (EnterpriseRAG-Bench context run) gets one before its run. Until then, this proposal is their only record. +- Part 1 is Experiment 35. Its governing file is the pre-registration `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` (amendment A4, 14 November 2026). It holds the live design only. `protocol.md` sections 0.1 to 0.13 (amendments A1 to A4) and sections 1 to 7 are history. +- Part 2 (controlled comparison with RAGFlow) gets its own protocol before the freeze, and E1 (EnterpriseRAG-Bench context run) gets one before its run. Pre-registration section 18 lists what Experiment 35 fixes for them. -Rule: where this proposal and a protocol disagree, this proposal wins. Fix the protocol with a dated amendment (A3 is the first). Do not copy protocol numbers into this file. This file names datasets, factors and factor levels only, so the two cannot drift. +Rule: for every Experiment 35 design fact, the pre-registration governs. Where this proposal and the pre-registration disagree on such a fact, the pre-registration wins and this proposal is corrected. For Part 2 and E1, pre-registration section 18 wins over this proposal on the points it fixes: the OMRS setup and its hash, the halves, the shared parts, the measurement settings, the C1 and C2 measures and tests, and the Part 2 missing-outcome rules. This proposal governs every other Part 2 and E1 fact until the Part 2 protocol is frozen. After that, the Part 2 protocol governs them. This proposal also governs scope and order of work. This file states Experiment 35 facts in short form and points to the pre-registration section. It does not copy long tables. + +OSF deposit at the freeze: the pre-registration as the main file, with this proposal, the protocol history (`protocol.md`), `plan.json` and the Part 2 protocol. Related records: -- `experiments/35-full-rag-benchmark-2026-11-01/protocol.md` and `plan.json`: Part 1 design and runtime pins. +- `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md`: Part 1 live design (governs). +- `experiments/35-full-rag-benchmark-2026-11-01/protocol.md` and `plan.json`: Part 1 history and runtime pins. - `openspec/changes/experiment-35-full-document-rag-benchmark/`: Part 1 design decisions and their sources. - `openspec/changes/exp35-main-run-readiness/`: Part 1 harness code. - `openspec/changes/rag-benchmark-readiness-flow/`: task order and gates for the whole benchmark. @@ -38,11 +41,11 @@ One hosted answer model is used throughout. This is a stated limitation. The paper answers three research questions: -- **RQ1 (absolute performance):** what retrieval and answer quality does OMRS reach on FinanceBench, FreshStack, BioASQ and (exploratory) Legal RAG Bench under stated models and settings? -- **RQ2 (component effects):** how do embedder size, chunk size, reranker size and hybrid search, and their interactions, change retrieval quality? How does chunk size change answer faithfulness and correctness? -- **RQ3 (external comparison):** with the shared models fixed, how does an OMRS setup chosen on half A compare with RAGFlow on half B (Part 2)? Where does OMRS sit beside published systems on EnterpriseRAG-Bench (E1, descriptive)? +- **RQ1 (absolute performance):** what retrieval and answer quality does OMRS reach on FinanceBench, FreshStack, BioASQ and (exploratory) Legal RAG Bench under stated models and settings? The headline is the A3-selected cell scored on half B (pre-registration section 14.3). The 72-setup table on all questions is descriptive. +- **RQ2 (component effects):** how do embedder size, chunk size, reranker size and hybrid search change retrieval quality? How does chunk size change answer faithfulness and correctness? +- **RQ3 (external comparison):** with the shared models fixed, how does the OMRS setup selected on half A compare on half B with RAGFlow (Part 2)? The step 7.2a decision fixes the RAGFlow side: RAGFlow selected on half A if the matched search runs, else RAGFlow at its defaults. Where does OMRS sit beside published systems on EnterpriseRAG-Bench (E1, descriptive)? -The claims stay testable. OMRS reaches stated scores under stated conditions. Named pipeline choices change those scores by measured amounts. OMRS and RAGFlow differ by measured amounts only where the paired tests support it. +The claims stay testable. OMRS reaches stated scores under stated conditions. Named pipeline choices change those scores by measured amounts. Selected OMRS and RAGFlow (default, or selected on half A if the matched search runs) differ by measured amounts only where the paired tests support it. Every claim holds only within scope S (pre-registration section 2): Qwen3-Embedding and Qwen3-Reranker at Q8_0 on one pinned llama.cpp build, English text, no query instruction, the reranker's default instruction, one hosted answer model, and the datasets as built. ## Why now @@ -52,12 +55,16 @@ The claims stay testable. OMRS reaches stated scores under stated conditions. Na ## Order and gates -1. Freeze Part 1. The operator approves the Experiment 35 protocol, `plan.json` and the run budget (master flow gate G7). Amendment A3 and the Part 2 protocol exist before this point (A3 is written; the Part 2 protocol is not). -2. Deposit on OSF in one deposit: the Experiment 35 protocol and plan, the Part 2 protocol (C1, C2) and amendment A3. Record the link and the commit hash. -3. Run Experiment 35 as its protocol states. -4. Apply the A3 rule on half A as soon as the search stage is scored. Freeze the chosen OMRS setup. Record its configuration hash on OSF before anyone opens half B for Part 2. -5. Run Part 2 on half B, once per system. -6. Run E1 if the budget allows. E1 is the first item to drop when the budget is short. +1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The open items O1 to O6 and the pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The Part 2 protocol exists before this point (it is not written yet). +2. Deposit on OSF in one deposit: the pre-registration as main file, this proposal, the protocol history, `plan.json` and the Part 2 protocol (C1, C2). Record the link and the commit hash. +3. Build and seal the 9 indexes per dataset. The rebuild gate G2 passes before any search writes output. Then run the 72 search setups on all questions (pre-registration section 14.2). +4. Score half A only. Apply the A3 rule. Write `selection.json`. +5. Time-stamp the A3 configuration hash by a file upload to the registered OSF project (open item O1: the user confirms the mechanism). +6. Only then score half B and start the answer stage. "Sealed" means the indexes only: half-B ranked lists exist on disk before step 5, but no half-B score is computed before it. +7. Run Part 2 on half B, once per system. +8. Run E1 if the budget allows. E1 is the first item to drop when the budget is short. + +After the freeze, every change is a dated, numbered deviation. The report lists each one with its reason, its time (before or after seeing the affected data) and its effect on each claim. Each deviation also goes into `plan.json` `amendments` and is uploaded to the OSF project (pre-registration section 20). Part 2 follows the same policy. The master flow `openspec/changes/rag-benchmark-readiness-flow` holds group 7 for Part 2 and E1. @@ -67,17 +74,21 @@ Done and archived on 8 November 2026 (Experiment 48, `openspec/changes/archive/2 ## Step 2: Part 1, Experiment 35 (Confirmatory) -The protocol is the authority for every number in this part. Read section 0 first; it wins over the older sections of that file. +The pre-registration is the authority for every design fact and number in this part. This section is a short guide to it. | Dataset | Domain | Questions | Gold unit | Licence | Status | |---|---|---|---|---|---| -| FinanceBench (open sample) | Finance, real PDFs | 150 | Document and page | CC BY-NC 4.0 | Confirmatory | -| FreshStack, LangChain slice | Software | 203 | Document | CC BY-SA 4.0 | Confirmatory | +| FinanceBench (open sample) | Finance, real PDFs | 150 | Page | CC BY-NC 4.0 | Confirmatory | +| FreshStack, LangChain slice (full public corpus, 49,514 documents) | Software | 203 | Document | CC BY-SA 4.0 | Confirmatory | | BioASQ (`enelpol/rag-mini-bioasq`, test split) | Biomedicine | 707 | Passage | CC BY 2.5 | Confirmatory | | Legal RAG Bench (`isaacus/legal-rag-bench`) | Law | 100 | Passage | CC BY-NC-SA 4.0 | Exploratory | BioASQ passed every data check, so the NFCorpus fallback is retired. RepoProbe moves to a later paper because it has no retrieval labels. No open medical dataset has real PDFs with page-level gold evidence, so FinanceBench is the only PDF dataset. EnterpriseRAG-Bench is not part of Experiment 35 (see E1). +- FreshStack: all 9 indexes are built on the full public LangChain corpus at a pinned revision. The 10,000-document subset of Experiments 9a and 22 (every gold document plus random distractors) is retired. No FreshStack result is compared with the public leaderboard unless the same corpus and metric are used. alpha-nDCG@10 is a secondary metric if the nugget labels allow it (pre-registration sections 4 and 6). +- Halves: seed 20261115. FinanceBench is split by document, so no document sits in both halves. The other datasets are split by question (pre-registration section 4). +- Existing data: earlier looks at these datasets (Experiments 9a, 10 and 22, smoke runs, the pilot) are declared in pre-registration section 5. + ### Search stage: full factorial, no baseline | Factor | Levels | @@ -89,51 +100,59 @@ BioASQ passed every data check, so the NFCorpus fallback is retired. RepoProbe m Every combination runs on every dataset. No setup is the baseline: each hypothesis is a planned contrast across the grid. Chunk size replaces the old token-aware chunking arm. The ONNX MiniLM reranker is gone; every reranker runs on llama.cpp. -The primary metric is nDCG@10 at unit level. Each setup saves its top 100 chunks. The chunks collapse to documents, pages or passages in rank order, and the list is cut at 10 units. Without this, bigger chunks would score higher only because each one covers more units. +The primary metric is nDCG@10 at unit level, with the trec_eval formulas written in pre-registration section 6. Each setup saves its top 100 chunks. The chunks collapse to documents, pages or passages in rank order, and the list is cut at 10 units. Without this, bigger chunks would score higher only because each one covers more units. On FinanceBench a chunk counts for its first page. The H6 FinanceBench endpoint `evidence_recall_20k` gives a chunk credit for every gold page it overlaps (any-page credit); first-page credit is a sensitivity analysis. ### Answer stage -The answer stage varies chunk size, reranker on or off and hybrid on or off. The embedder and reranker size are chosen on half A by the protocol's selection rule. Token-matched setups, one per chunk size (`c512-t20k`, `c1024-t20k`, `c2048-t20k`), hold the evidence at one token budget. A closed-book run (no retrieval) and an oracle run (gold evidence given directly) frame the results. +The answer stage uses the embedder and reranker size chosen on half A (pre-registration section 14.1). It has 15 setups, each run 3 times on all 1,160 questions: 12 crossed setups (chunk size x reranker on or off x hybrid on or off, 10 chunks of evidence each) and 3 token-matched setups (`c512-t20k`, `c1024-t20k`, `c2048-t20k`, reranker and hybrid on, evidence cut at 20,480 Qwen3 tokens). Two closed-book runs (own-knowledge prompt, and the RAG prompt with an empty evidence block) and an oracle run (gold evidence given directly) frame the results (pre-registration section 13). - Faithfulness: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. -- Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. -- Human check: two labellers score samples blind. Each checker and the correctness judge must pass its agreement gate before its scores count. +- Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. +- Human check: two labellers score samples blind (L1 750 labels, L2 225). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. -### Hypotheses H1 to H8 +### Hypotheses H1 to H8b -Full wording, contrasts and decision rules: protocol section 0.4, restated by A2 (section 0.10) and A2c (section 0.11). +Full wording, contrast rows and "holds" rules: pre-registration sections 7 and 8. Each null is a zero effect. Each hypothesis holds only within scope S. -- **H1:** a bigger embedder finds better evidence. -- **H2:** the reranker helps. -- **H3:** hybrid search helps. -- **H4:** reranker plus hybrid beats either alone. +- **H1:** a bigger embedder ranks gold units higher. +- **H2:** the reranker raises nDCG@10. +- **H3:** hybrid search raises nDCG@10. +- **H4:** reranker plus hybrid beats each alone (simple effects, not an interaction test). - **H5:** a bigger reranker ranks better. -- **H6:** chunk size changes search quality (FinanceBench and FreshStack). -- **H7:** bigger embedders gain more from bigger chunks. -- **H8:** chunk size changes answer faithfulness and correctness. +- **H6:** chunk size changes retrieval (the same row, same sign, on FinanceBench and FreshStack). +- **H7:** a bigger embedder gains more from bigger chunks (FinanceBench and FreshStack; expected to be underpowered). +- **H8a:** chunk size changes faithfulness and correctness in the crossed setups, where context length grows with chunk size (the same row, same sign, on FinanceBench and FreshStack; open item O6). +- **H8b:** chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. The paper says "chunk size itself changes X" only when H8b holds for X. ### Methods in short -Detail for each item is in protocol sections 0.3 to 0.11. +Detail for each item is in the pre-registration section named. -- Search runs once per setup on a sealed index. A rebuild check and a repeat check confirm determinism. Answer setups repeat to measure drift in the hosted API. -- Each dataset gets one model of per-question scores with planned contrasts. P-values use cluster-robust errors, clustered by document on FinanceBench. Holm correction runs across the contrasts of each dataset. Cross-dataset rules decide when a hypothesis holds overall. -- Minimum detectable effects come from pilot data before the freeze. No equivalence claim is made. +- Search runs once per setup on a sealed index, and each index is built once. A rebuild check and a repeat check confirm determinism (section 15). Answer setups repeat to measure drift in the hosted API. +- Analysis: no mixed model. One clustered method gives every estimate, interval and p-value: GEE with bias-reduced covariance on question clusters for FreshStack, BioASQ and Legal, and a document-level sign-flip permutation test on FinanceBench (section 9). +- Holm families are fixed now: 19 contrasts on FinanceBench, 19 on FreshStack, 8 on BioASQ (46 in all; BioASQ rows 9 to 19 are sanity checks, open item O6); Legal is exploratory. No gate changes a family size (section 8). H1 to H5 hold on at least 2 of 3 core datasets. H6, H7, H8a and H8b hold only on FinanceBench and FreshStack. A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs (section 7). +- Missing outcomes: every question stays in its denominator, with a rule per outcome (section 12). +- Power: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the minimum detectable effect table before the freeze. It builds at most the four H7 corner indexes per core dataset and does not reuse them. Its cost is projected from the P2 timing and approved before the pod starts; if the user declines, the table is filled from assumed SDs and labelled so. The smallest effect of interest is 0.02 nDCG@10 (open item O2). No equivalence claim is made (section 10). - Every main-run pod uses one RunPod GPU type, chosen by a timing job before the freeze. The Mac scores the results. -- The budget has an estimate and a hard ceiling. The hosted-call key carries a credit limit, so an overrun stops cleanly. -- Pre-registration: the protocol and `plan.json` go to OSF at the freeze commit, with Part 2 and A3. +- Budget: estimate about $730 (an estimate, not measured), ceiling $975, pods and hosted calls together. The hosted-call key carries a credit limit, so an overrun stops cleanly. At the ceiling a results-blind drop order applies: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop. E1 has its own ceiling and is not in this order. A run that finishes every confirmatory item after drops is "complete with registered drops", each drop a numbered deviation. No confirmatory item is dropped by choice (section 17). +- Pre-registration: the pre-registration is the main file of the OSF deposit at the freeze commit, with this proposal, the protocol history, `plan.json` and the Part 2 protocol. ### Runtime and hosted models -- Local models: llama.cpp only. `plan.json` pins the repository, file, SHA-256, build and server arguments of each one. The live-server check passes before the first score. +- Local models: llama.cpp only. `plan.json` pins the repository, file, SHA-256, build and server arguments of each one. The live-server check passes before the first score. Pre-registration section 16 holds one pins table; the items still marked missing there (answer-model host, reasoning mode and output cap, query instruction, hybrid settings, FreshStack revision, prompt hashes) are pre-freeze tasks. - Answer model: `deepseek/deepseek-v4.1-flash-1017`, through the API only, with one pinned upstream host. The open-weight model is too large for one pod, and smaller open models answer worse (protocol A2). - Main claim checker GLM-5.3 and Ragas judge `openai/gpt-oss-120b`: hosted, pinned by model ID. - Limitation: results describe OMRS retrieval with one hosted answer model. Clef-Flash gives a local path for the faithfulness check. The answer step has no local path in this paper. ## Step 3: Part 2, controlled comparison with RAGFlow (Confirmatory) -A separate experiment with its own number and protocol, deposited at the Experiment 35 freeze. It reuses the Experiment 35 corpora, question IDs, half split, prompt, answer model, checkers and judges. +A separate experiment with its own number and protocol, deposited at the Experiment 35 freeze. It reuses the Experiment 35 corpora (FreshStack full corpus), question IDs, half split, prompt, answer model, checkers and judges. The points Experiment 35 fixes for Part 2 are in pre-registration section 18, which wins over this section on those points. This proposal governs every other Part 2 fact until the Part 2 protocol is frozen; after that, the Part 2 protocol governs them. + +**Estimand.** The cost decision at master flow step 7.2a (rule 9) fixes which estimand applies, before any half-B score exists. The estimand is never chosen after seeing results. The shared models are fixed in both cases. OMRS is the A3 cell chosen on half A from the 72-setup search. + +- If the matched RAGFlow search runs: "selected OMRS against selected RAGFlow". The RAGFlow cell is chosen on half A by the same selection rule. Its configuration hash is time-stamped on OSF with the OMRS hash, before any half-B score exists. Default RAGFlow becomes a secondary, descriptive row. +- If the matched search does not run: "selected OMRS against default RAGFlow". RAGFlow runs at its defaults, except the settings shared or set for measurement below. The paper names this tuning gap wherever it states the C1 or C2 result. ### Why RAGFlow alone @@ -148,51 +167,81 @@ Systems left out, and why: ### Which OMRS setup -The main comparison uses one OMRS setup chosen on half A of the Experiment 35 split. A pre-registered rule fixes all four factors: embedder, chunk size, reranker level and hybrid. The exact rule is Experiment 35 protocol amendment A3 (section 0.12, 13 November 2026). It runs in two steps. First, pooled over the three core datasets on half A, it picks the embedder and the reranker size (the Experiment 35 selection rule, amended). Then, at that embedder and reranker size, it picks chunk size, reranker on or off and hybrid on or off by the pooled mean nDCG@10 on half A. Its principles: +The main comparison uses one OMRS setup chosen on half A of the Experiment 35 split. A pre-registered rule fixes all four factors: embedder, chunk size, reranker level and hybrid. The exact rule is pre-registration section 14.1 (amendment A3, as restated by A4). It runs in two steps. First, pooled over the three core datasets on half A, it picks the embedder and the reranker size (the Experiment 35 selection rule, amended). Then, at that embedder and reranker size, it picks chunk size, reranker on or off and hybrid on or off by the pooled mean nDCG@10 on half A. Its principles: - Choose on the three core datasets pooled. Legal RAG Bench takes no part in selection. -- Setups within 0.01 of the best tie. Ties go to the cheaper setup in this order: reranker off, hybrid off, lowest index cost, then smaller chunk size. -- Record the configuration hash on OSF before half B is opened for Part 2. +- Setups within 0.01 of the best tie. Ties go to the cheaper setup in this order: reranker off, hybrid off, lowest index cost, then smaller chunk size. Index cost is the number of stored vector values (chunks times embedding dimension) summed over the three core datasets, known before any score. +- The configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the SHA-256 of `halves.json`. A file upload to the registered OSF project time-stamps it before any half-B score is computed (open item O1). -A **defaults row** sits beside the main comparison: OMRS shipped defaults against RAGFlow documented defaults. Both use the embedder chosen on half A and the same answer model. Each keeps its own shipped retrieval settings. It names the tuning gap, which is the part of any OMRS lead that comes from choosing a setup on half A. The defaults row is secondary. It gets paired confidence intervals and no confirmatory claim. RAGFlow runs at its documented defaults in both rows. +A **defaults row** sits beside the main comparison: OMRS shipped defaults against default RAGFlow. Both use the embedder chosen on half A and the same answer model. OMRS keeps its own shipped retrieval settings. Default RAGFlow is one RAGFlow run: its defaults read from the pinned release, plus the shared parts (the embedder, and the reranker when the A3 cell has it on) and the two measurement settings (cut-off 0, depth 100) of rule 1. When the matched search does not run, the RAGFlow side of the main row and of the defaults row is this same run, and the table shows its scores once. The row names the tuning gap, which is the part of any OMRS lead that comes from choosing a setup on half A. The defaults row is secondary. It gets paired confidence intervals and no confirmatory claim. -A **search-cost figure** plots the best half-A score against the number of setups tried (expected validation performance). It shows how much of the OMRS score comes from selection over the grid. +A **search-cost figure** shows how much of the OMRS score comes from selection over the grid. It plots the expected maximum pooled half-A score over n setups drawn at random, adapted from Dodge et al. 2019 (drawn without replacement), computed exactly from the scores sorted in ascending order. One curve covers the 12 A3 cells and one covers the 72 setups. RAGFlow has no point unless its matched search runs; then it gets its own curve over its half-A cells (pre-registration section 14.3). ### Which questions -Half B only. No selection step in Part 1 or Part 2 reads half B. Each system runs once. The three core datasets are confirmatory; Legal RAG Bench is exploratory. Half B is about half of each dataset, so power is lower than in Part 1. Report the minimum detectable effect computed from the data for each dataset and hypothesis. +Half B only. No selection step in Part 1 or Part 2 reads half B. Every pre-freeze RAGFlow sample (the page-number gate and the repeatability check) comes from half A only, and so does the matched RAGFlow search if it runs. The timing run per dataset indexes the corpus only and sends no question. Each system runs once. The three core datasets are confirmatory; Legal RAG Bench is exploratory. Half B is about half of each dataset, so power is lower than in Part 1. C2 is expected to detect only large correctness differences (pre-registration section 10). The Part 2 protocol states a minimum detectable effect for C1 and C2 per dataset before the freeze, and the report gives it again from the data. ### Rules that keep the comparison fair -1. **Shared parts are fixed.** Both systems use the embedder chosen on half A, the same answer model and prompt, the same checkers and judges, the same corpus snapshot and the same question IDs. Each system keeps its own parser, chunker and retrieval method. Those are what the comparison measures. +1. **Shared parts are fixed.** Both systems use the embedder chosen on half A, the same answer model and prompt, the same 20,480-token budget, the same checkers and judges, the same corpus snapshot and the same question IDs and halves. When the A3 cell has the reranker on, RAGFlow gets the same Qwen3-Reranker file and size through its rerank setting. RAGFlow's similarity cut-off is set to 0 and its retrieval depth to 100, for measurement only. Every other RAGFlow setting is its default, read from the pinned release and not from main-branch documentation (a pre-freeze task). Each system keeps its own parser, chunker and retrieval method. Those are what the comparison measures. The Part 2 protocol gives a settings table with three columns: shared and fixed, default kept, set for measurement. 2. **Equal evidence.** Request retrieval depth 100 from RAGFlow. Score nDCG@10 on the same units as Experiment 35, after the same collapse. Both systems answer from evidence cut to the same 20,480-token budget. Report mean context tokens per question for each system. On FinanceBench the relevance unit is the one fixed in rule 3. -3. **FinanceBench relevance unit.** One unit, fixed before the run, applies to every FinanceBench question and to both systems. The primary unit is the page, as in Part 1 (a chunk counts for its first page). Pre-freeze gate: on a seeded FinanceBench sample, confirm that RAGFlow returns a page number for each chunk. Pre-registered fallback: if the gate fails, the primary unit becomes the document for every question and both systems. If the gate passes but a chunk in the main run has no page number, that chunk counts as not relevant at page level for its system, and the report gives the count of such chunks per system. The other unit is reported as a labelled sensitivity analysis. It is required only when both systems provide page provenance; otherwise the report states it is unavailable and gives the count of questions affected. No question is dropped, and no question changes unit, because of one system's output. +3. **FinanceBench relevance unit.** One unit, fixed before the run, applies to every FinanceBench question and to both systems. The primary unit is the page, as in Part 1 (a chunk counts for its first page). Pre-freeze gate: on a seeded FinanceBench sample from half A, confirm that RAGFlow returns a page number for each chunk. Pre-registered fallback: if the gate fails, the primary unit becomes the document for every question and both systems. If the gate passes but a chunk in the main run has no page number, that chunk counts as not relevant at page level for its system, and the report gives the count of such chunks per system. The other unit is reported as a labelled sensitivity analysis. It is required only when both systems provide page provenance; otherwise the report states it is unavailable and gives the count of questions affected. No question is dropped, and no question changes unit, because of one system's output. 4. **Answers.** Each system's retrieved evidence goes through the common OMRS answer harness, with the same prompt, the answer model `deepseek/deepseek-v4.1-flash-1017` and the same 20,480-token budget. The answer difference then comes from parsing, chunking and retrieval only. RAGFlow's own chat pipeline output may be reported descriptively, and it never enters C1 or C2. -5. **Determinism.** Search runs once per system. Before the run, repeat RAGFlow retrieval on a small seeded sample and report whether the ranked lists match. Rerun OMRS search for the chosen setup on its sealed Experiment 35 index and check that the lists equal the saved ones. -6. **Pin and disclose.** Pin the RAGFlow release and container digest. Record the full configuration, parser settings included. Send the configuration to the RAGFlow maintainers before the run and log every reply. Use RAGFlow's documented defaults for every setting that is not shared. Do not tune it below them. -7. **Human labels.** Part 2 adds its own correctness labels and human-agreement sample, drawn from the answers of both systems, so checker and judge agreement covers RAGFlow answers. They run in Part 2. The Experiment 35 label counts do not change. +5. **Determinism.** Search runs once per system. Before the run, repeat RAGFlow retrieval on a small seeded half-A sample and report whether the ranked lists match. Rerun OMRS search for the chosen setup on its sealed Experiment 35 index and check that the lists equal the saved ones. +6. **Pin and disclose.** Pin the RAGFlow release and container digests. Record the full configuration, parser settings included. Send the configuration to the RAGFlow maintainers before the run and log every reply. Do not tune RAGFlow below its defaults, apart from the settings of rule 1. +7. **Human labels.** Part 2 adds its own correctness labels and human-agreement sample, drawn from the answers of both systems, so judge agreement covers RAGFlow answers. L2 is the primary Part 2 labeller, so the OMRS authors' labels are the check, not the gate. System identity is hidden: both systems' answers are shuffled into one list. Labellers follow a written Part 2 correctness guideline, with worked examples from outside every sample, fixed before the first label. They run in Part 2. The Experiment 35 label counts do not change. 8. **Report.** Map every retrieved chunk to the dataset's unit ID. Give one table per dataset, separate from the Experiment 35 tables. +9. **Matched RAGFlow search (conditional).** A retrieval-only RAGFlow search on half A (the same selection rule, 12 to 24 cells). Its exact cell list (the RAGFlow settings varied and their levels) is fixed and recorded before the cost measurement at master flow step 7.2. It runs only if its projected pod cost, measured at master flow step 7.2, is at most $50. The decision and its cost figure are recorded at master flow step 7.2a, before any half-B score exists. The decision fixes the estimand (see "Estimand" above). If the search runs, the selected RAGFlow cell enters C1 and C2, its configuration hash is time-stamped on OSF with the OMRS hash before any half-B score exists, default RAGFlow becomes a secondary, descriptive row, and the search-cost figure shows a RAGFlow curve. +10. **Missing outcomes.** Every half-B question stays in both systems' denominators. Retries follow the content-blind resend rule of Part 1 only. A RAGFlow timeout or error that remains scores 0 for C1 and gets an answer from empty evidence for C2. A list shorter than 10 units scores the missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. A chunk that does not map to a dataset unit counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. +11. **Conflict of interest.** The authors built OMRS. The paper states this. The safeguards are: the pre-registration and its deviation log; selection on half A only; Holm families fixed before the run; blind labels with L2 as primary labeller; the published RAGFlow settings table and maintainer correspondence; and release of every output, RAGFlow outputs included. ### Hypotheses C1 and C2 -- **C1:** OMRS and RAGFlow differ in retrieval nDCG@10. Two-sided, per core dataset. -- **C2:** OMRS and RAGFlow differ in answer correctness. Two-sided, per core dataset. +"RAGFlow" below means selected RAGFlow if the matched search runs, else default RAGFlow (see "Estimand"). -C1 and C2 form their own Holm family, separate from H1 to H8: Holm runs per dataset across C1 and C2 (two tests per dataset). The cross-dataset rule is the one of Part 1: a hypothesis holds when it holds on at least 2 of the 3 core datasets (FinanceBench, FreshStack, BioASQ). Each test pairs the two systems question by question: +- **C1:** selected OMRS and RAGFlow differ in retrieval nDCG@10. Null: zero difference. Two-sided, per core dataset. +- **C2:** selected OMRS and RAGFlow differ in answer correctness, measured by continuous Ragas `AnswerCorrectness`. Null: zero difference. Two-sided, per core dataset. FinanceBench number match is secondary. -- Continuous metrics: paired permutation test, with a paired bootstrap 95% interval. -- Binary correctness: McNemar's exact test, or another exact paired test. -- FinanceBench: questions share documents, so p-values come from GEE with errors clustered by document, as in the protocol. +C1 and C2 form their own Holm family, separate from H1 to H8b: Holm runs per dataset across C1 and C2 (a family of 2 per core dataset). The cross-dataset rule is the 2-of-3 rule of Part 1 with the same-sign rule: a hypothesis holds when it counts with the same sign on at least 2 of the 3 core datasets (FinanceBench, FreshStack, BioASQ). Opposite signs are reported as a dataset-dependent effect, and the hypothesis does not hold. The report states the direction. Each test pairs the two systems question by question, with the analysis of pre-registration section 9: -C2 uses the correctness measure that passes its human-agreement gate (kappa 0.60, as in the Experiment 35 protocol). If none passes, C2 becomes exploratory. Legal RAG Bench and the defaults row get paired intervals and no confirmatory test. +- FreshStack and BioASQ: GEE on question clusters with bias-reduced covariance (Mancl and DeRouen 2001) and a t reference distribution. +- FinanceBench: questions share documents, so the decision p-value comes from a document-level sign-flip permutation test, and the interval inverts the same test. The randomisation test is the recommended test for comparing two IR systems (Smucker, Allan and Carterette 2007); here it flips signs per document, not per question. +- No McNemar test where documents cluster. Any binary secondary measure is reported descriptively or with a test that allows for clustering (Obuchowski 1998). + +C2 needs its own Part 2 correctness gate: the pre-registration section 11.2 rule (kappa point estimate at least 0.60 and lower 95% bound at least 0.40, per core dataset), on labels from both systems' answers. A failed gate leaves C2 in the family as "not testable" on that dataset. Legal RAG Bench and the defaults row get paired intervals and no confirmatory test. The paper claims superiority only where the tests support it. No TOST margin is planned, so the paper makes no equivalence claim. A non-significant result means no difference was detected at this sample size. -What this comparison can claim: with the shared models fixed, the measured difference between two complete systems on held-out questions. Han et al. 2025 (RAG against GraphRAG) fixed the answer model, chunk size, embedding model and retrieval budget in the same way. BERGEN and FlashRAG fix shared parts inside one toolkit. The half split answers the selection-bias risk (Cawley and Talbot 2010). The defaults row and the maintainer review answer the weak-baseline risk (Yang et al. 2019). +What this comparison can claim: with the shared models fixed, the measured difference between the OMRS setup selected on half A and RAGFlow (at its defaults, or selected on half A by the same rule if the matched search runs), on held-out questions. Without the matched search it does not show that OMRS beats a tuned RAGFlow. Han et al. 2025 (RAG against GraphRAG) fixed the answer model, chunk size, embedding model and retrieval budget in the same way. BERGEN and FlashRAG fix shared parts inside one toolkit. The half split answers the selection-bias risk (Cawley and Talbot 2010). The shared reranker, the settings table, the defaults row, the search-cost figure and the maintainer review answer the weak-baseline risk (Armstrong et al. 2009; Yang et al. 2019; Lin 2019; Dodge et al. 2019). Holm within each dataset answers the multiple-testing risk of many-contrast IR experiments (Carterette 2012). The choice of tests draws on Urbano, Lima and Hanjalic 2019. ### Pre-registration -Part 2 (C1, C2) and the A3 rule go to OSF at the same freeze as Experiment 35, before any half-B score exists. +The Part 2 protocol (C1, C2) goes to OSF in the same deposit as the Experiment 35 pre-registration, at the freeze, before any half-B score exists. The A3 configuration hash is time-stamped on OSF before any half-B score is computed. After the freeze, Part 2 follows the deviation policy of pre-registration section 20. + +### What the Part 2 protocol must hold before its freeze + +Until these items exist, Part 2 cannot freeze. Each can be short in the protocol. They come from the methods review of 13 November 2026 (findings F2, F9, F10, F17 to F21, F27 to F29). + +1. Experiment number, folder and `experiment.json` (purpose `publication`, Confirmatory), from master flow step 7.1. +2. C1 and C2 with nulls, decision rule, Holm family of 2 per core dataset and the 2-of-3 same-sign rule, as above. +3. RAGFlow pin: release tag, container digest of every service (RAGFlow, search engine, database, object store, cache), DeepDoc model versions and API version. +4. Settings table for both systems, three columns (shared and fixed, default kept, set for measurement). Rows cover chunk method and size, tokenizer, parser and layout model, every LLM-based parsing feature (off or stated), embedder, query instruction, reranker and its instruction, keyword and vector weights, similarity cut-off, search depth, page size and depth 100. Defaults are read from the pinned release. +5. Tuning budget and estimand: the exact matched RAGFlow cell list (settings varied and their levels), fixed before the master flow step 7.2 cost measurement; the matched-search decision at master flow step 7.2a (rule 9), with its cost figure measured at step 7.2, recorded before any half-B score. The decision fixes the estimand: "selected OMRS against selected RAGFlow" if the search runs (the RAGFlow configuration hash time-stamped on OSF with the OMRS hash), else "selected OMRS against default RAGFlow". +6. Model serving: RAGFlow calls the same llama.cpp router, build and GGUF files as OMRS, with an environment record and the live-server check before the first call. +7. Evidence mapping: chunk to unit ID per dataset, the FinanceBench page rule, page gate and fallback, and the unmappable-chunk rule. +8. Answer harness input: how a RAGFlow chunk becomes an evidence item (text, title, order), Qwen3 token counting for the 20,480-token budget, and truncation records. +9. Pre-freeze samples from half A only, with their seeds. +10. Repeatability check rule, and what a mismatch means for C1 (report and keep, or block). +11. Missing-outcome rules as rule 10, every half-B question kept in both denominators. +12. Labels: counts, seeds, guidelines, hidden system identity, L1 and L2 roles (L2 primary), and the per-dataset correctness gate. +13. Defaults row: definition (one RAGFlow run as pre-registration section 18) and secondary role; it is a descriptive row when the matched search runs, and shown once when it does not. +14. Budget: Part 2 estimate and ceiling, and the drop order at the ceiling. +15. Conflict-of-interest statement and safeguards (rule 11), a rerun kit for both systems on half B, and the maintainer correspondence log. +16. Release table for RAGFlow configurations, outputs and maintainer replies. +17. The A3 configuration hash definition and the OSF time-stamp before half B is scored. +18. The search-cost figure estimator, with the RAGFlow curve if the matched search runs. +19. Run record and environment record rules, as in Part 1, and the deviation policy. +20. The minimum detectable effect for C1 and C2 per dataset, from half-A data only. ## Step 4: EnterpriseRAG-Bench context run (E1, descriptive) @@ -256,32 +305,51 @@ The journal's own author guide covers ORCID, CRediT, the funding wording, confli 3. Respect the dataset licences: - FinanceBench is CC BY-NC. Do not release its text or derived files under CC BY. - FreshStack is CC BY-SA. Release derived files under CC BY-SA. - - BioASQ is CC BY 2.5. + - BioASQ is CC BY 2.5, released with the NLM terms notice. - Legal RAG Bench is CC BY-NC-SA 4.0. Do not release its text or derived files under CC BY. Release derived files under CC BY-NC-SA. - EnterpriseRAG-Bench is MIT. - - Publish scores, question IDs and scripts that rebuild the data. Do not publish the corpus text. + - Do not publish the corpus text. + - Release each artefact under its rule in the release table of pre-registration section 19: code and label guidelines, ranked lists, answers, cited evidence IDs (never text), verdicts and scores, human labels, run and environment records, the compute table and the deviation log. RAGFlow configurations, outputs and maintainer replies follow the Part 2 protocol. + - Ship a re-judge kit: a script that rebuilds evidence text from the pinned dataset revisions and the released IDs, so a newer model can judge the answers again without redistributing corpus text. 4. Archive code and outputs on Zenodo with a DOI. Keep each dataset's derived files in their own archive with their own licence. 5. Before the repository or a release goes public, remove names and home-folder paths. +6. Report compute in one table (pre-registration section 21): GPU type, pod hours per stage including every pre-freeze pod (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, an energy estimate (GPU hours times board power), `resent_calls`, error rows per stage and role, and every `out_of_credit` stop (Strubell et al. 2019; Lacoste et al. 2019). Part 2 and E1 add their own rows. +7. State the conflict of interest (the authors built OMRS) and the AI assistance used to draft the plan. ## Risks | Risk | Response | |---|---| | A small difference between llama.cpp back ends | Every main-run pod uses one GPU type. The plan states a tolerance per metric (ADR-103). | -| A checker or the correctness judge disagrees with people | Human labels gate each one before its scores count. A measure that fails its gate becomes exploratory. | -| The run costs more than planned | The protocol ceiling stops the run at the next stage, and the operator decides. | +| A checker or the correctness judge disagrees with people | Human labels gate each one before its scores count. A row whose measure fails its gate stays in its Holm family and is reported as "not testable". | +| The run costs more than planned | At the $975 ceiling the results-blind drop order of pre-registration section 17 applies: Legal RAG Bench answer generation, then the length-controlled check, then the closed-book run with the RAG prompt, then stop. A run with every confirmatory item done is "complete with registered drops". No confirmatory item is dropped by choice. Part 2 and E1 get their own estimate and ceiling before their runs; across the project, E1 is dropped first when the budget is short. | | The reranker result changes on llama.cpp | Report it as found. A negative result is still a result. | | Choosing the OMRS setup inflates its score | Choose on half A by a pre-registered rule, test on half B only, and publish the search-cost figure. | | Half B is too small to detect a difference | Report the minimum detectable effect from the data. Make no equivalence claim. | | The single hosted answer model changes behind its ID | Pin one upstream host. Record the returned model ID and provider per call. State it as a limitation. | | RAGFlow cannot use our models, or hides its retrieved chunks | Record the blocker and report it. Do not substitute another model. | -| RAGFlow chunks carry no page number | The pre-freeze gate checks this on a FinanceBench sample. If it fails, the primary unit is the document for every question and both systems. The page-level result is reported as a labelled sensitivity analysis, or as unavailable (with the count) when a system gives no page provenance. No question is dropped. | -| RAGFlow indexing costs too much GPU time | Time one RAGFlow index per dataset before the freeze. Report the cost. | -| A reviewer says we set up RAGFlow badly | Use its documented defaults, pin the release, publish the full configuration, send it to the maintainers and report the defaults row. | +| RAGFlow chunks carry no page number | The pre-freeze gate checks this on a half-A FinanceBench sample. If it fails, the primary unit is the document for every question and both systems. The page-level result is reported as a labelled sensitivity analysis, or as unavailable (with the count) when a system gives no page provenance. No question is dropped. | +| RAGFlow indexing costs too much GPU time | Time one RAGFlow index per dataset before the freeze (corpus only, no question). Report the cost. The matched RAGFlow search runs only if its projected pod cost is at most $50. | +| A reviewer says we set up RAGFlow badly | Share the reranker, set cut-off 0 and depth 100, read every other default from the pinned release, publish the three-column settings table, send it to the maintainers, report the defaults row and state the estimand fixed at master flow step 7.2a ("selected OMRS against default RAGFlow", or "against selected RAGFlow" if the matched search runs). | +| A RAGFlow call fails or a document fails to parse | Rule 10: the question stays in both denominators and the failure is scored and counted, with a sensitivity analysis on questions both systems indexed fully. | +| The authors' labels favour OMRS | L2 is the primary Part 2 labeller and system identity is hidden. The paper states the conflict of interest. | | E1 indexing exceeds the budget | E1 is dropped first. Report the measured cost. | | Public leaderboard results lack per-question scores | E1 is descriptive. Do not infer significance from rank or aggregate scores. | | The journal deal does not cover the article type | Ask the library before writing. Research Article is the fallback. | +## Limitations and claim scope + +The Experiment 35 limitations are in pre-registration section 22. The paper repeats them, and adds the Part 2 and E1 ones. + +- Scope S: one model family at Q8_0 on one llama.cpp build, English only, no query instruction, one dataset per domain, and one hosted answer model that may change behind its ID. No claim extends past scope S. +- The reranker's default instruction is a web-search instruction, applied to finance and legal text. +- Unjudged units count as not relevant. FreshStack labels came from pools that included BM25 and fusion systems, which can favour hybrid search and keyword-weighted RAGFlow. Neither Part 2 system added to any pool. +- H8a and H8b faithfulness is checker-measured, and the checker sees longer passages for bigger chunks. The differential-error rule and the length-controlled check address this only in part. H8b holds only at reranker on and hybrid on with the chosen sizes. +- The answer model may have seen benchmark text in training. The closed-book runs bound this only in part. +- One index build per cell. H7 and C2 are expected to be underpowered. +- L1 is an OMRS author. If the matched RAGFlow search does not run (step 7.2a decision), Part 2 compares selected OMRS with RAGFlow at its defaults, so it does not show that OMRS beats a tuned RAGFlow. If it runs, Part 2 compares selected OMRS with RAGFlow selected on half A over its matched cells only. +- E1 compares whole configurations with a different answer model (GPT-5.4), so it says nothing about OMRS retrieval alone. + ## Next steps 1. Done: pipeline sweep (Step 1), archived 8 November 2026. @@ -289,13 +357,18 @@ The journal's own author guide covers ORCID, CRediT, the funding wording, confli 3. Done: Experiment 35 amendments A1, A2 and A2c. 4. Ask the Napier library to confirm the article type and the deal. 5. Done: Experiment 35 amendment A3 (protocol section 0.12) holds the selection rule for Part 2. -6. Reserve an experiment number for Part 2. Write its protocol: C1, C2, the fairness rules and A3 as reference. -7. Before the freeze, pin the RAGFlow release and container digest and connect it to the llama.cpp models. Time one index per dataset. Check page provenance on a seeded FinanceBench sample (the rule 3 gate: if RAGFlow returns no page for a chunk, the primary unit falls back to the document for both systems) and retrieval repeatability. Send the configuration to the RAGFlow maintainers. -8. Done: group 7 (Part 2 and E1) is in `rag-benchmark-readiness-flow`. -9. Freeze Part 1. Deposit Experiment 35, Part 2 and A3 on OSF together. -10. Before any E1 run, estimate its cost and ceiling. Pin the corpus and leaderboard revisions. Confirm the GPT-5.4 judging or submission procedure with Onyx. +6. Done: amendment A4 (protocol section 0.13) and the pre-registration, after the methods review of 13 November 2026. +7. Close the open items O1 to O6 and the pre-freeze tasks P1 to P16 of pre-registration section 23. +8. Reserve an experiment number for Part 2. Write its protocol, covering every item of "What the Part 2 protocol must hold before its freeze". +9. Before the freeze, pin the RAGFlow release and container digests and connect it to the llama.cpp models. Read the RAGFlow defaults from the pinned release. Time one index per dataset (corpus only). Measure the projected pod cost of the matched search at master flow step 7.2 and record the decision at step 7.2a; it fixes the Part 2 estimand. Check page provenance on a seeded half-A FinanceBench sample (the rule 3 gate: if RAGFlow returns no page for a chunk, the primary unit falls back to the document for both systems) and retrieval repeatability on half A. Send the settings table to the RAGFlow maintainers. +10. Write the Part 2 estimate and ceiling, and the E1 estimate and ceiling (pre-freeze task P16; no figure exists yet). +11. Done: group 7 (Part 2 and E1) is in `rag-benchmark-readiness-flow`. It must take the Part 2 requirements above. +12. Freeze Part 1. Deposit the pre-registration (main file), this proposal, the protocol history, `plan.json` and the Part 2 protocol on OSF together. +13. Before any E1 run, pin the corpus and leaderboard revisions. Confirm the GPT-5.4 judging or submission procedure with Onyx. + +## Sources checked on 27 October, 9, 12, 13 and 14 November 2026 -## Sources checked on 27 October, 9 November, 12 November and 13 November 2026 +The Experiment 35 reference list is in pre-registration section 24. - FinanceBench, arXiv 2311.11944. - FreshStack, arXiv 2504.13128. @@ -303,11 +376,18 @@ The journal's own author guide covers ORCID, CRediT, the funding wording, confli - BEIR, arXiv 2104.08663. - Ragas, arXiv 2309.15217. Version 0.4.3 on PyPI is dated 13 January 2026. - Miller 2024, "Adding Error Bars to Evals", arXiv 2411.00640. -- Smucker, Allan and Carterette 2007, CIKM, doi:10.1145/1321440.1321528 (to verify). -- Urbano, Lima and Hanjalic 2019, SIGIR, arXiv 1905.11096 (choice of statistical test in IR evaluation). -- Cawley and Talbot 2010, JMLR, "On over-fitting in model selection and subsequent selection bias in performance evaluation". -- Yang, Lu, Yang and Lin 2019, SIGIR, doi:10.1145/3331184.3331340 (weak baselines). -- Dodge et al. 2019, EMNLP, "Show your work", arXiv 1909.03004 (expected validation performance; to verify). +- Smucker, Allan and Carterette 2007, "A comparison of statistical significance tests for information retrieval evaluation", CIKM, doi:10.1145/1321440.1321528 (randomisation test for two-system comparisons). +- Urbano, Lima and Hanjalic 2019, "Statistical significance testing in information retrieval: an empirical analysis of type I, type II and type III errors", SIGIR, doi:10.1145/3331184.3331259; arXiv 1905.11096. +- Carterette 2012, "Multiple testing in statistical analysis of systems-based information retrieval experiments", ACM TOIS, doi:10.1145/2094072.2094076. +- Cawley and Talbot 2010, "On over-fitting in model selection and subsequent selection bias in performance evaluation", JMLR 11, . +- Yang, Lu, Yang and Lin 2019, "Critically examining the 'neural hype'", SIGIR, doi:10.1145/3331184.3331340. +- Armstrong, Moffat, Webber and Zobel 2009, "Improvements that don't add up", CIKM, doi:10.1145/1645953.1646031. +- Lin 2019, "The neural hype and comparisons against weak baselines", SIGIR Forum, doi:10.1145/3308774.3308781. +- Dodge et al. 2019, "Show your work: improved reporting of experimental results", EMNLP, arXiv 1909.03004; doi:10.18653/v1/D19-1224 (expected validation performance). +- Mancl and DeRouen 2001, "A covariance estimator for GEE with improved small-sample properties", Biometrics, doi:10.1111/j.0006-341X.2001.00126.x. +- Obuchowski 1998, "On the comparison of correlated proportions for clustered data", Stat. Med. 17(13), 1495. +- OSF Prereg form, . +- Strubell et al. 2019, energy and policy considerations, arXiv 1906.02243. Lacoste et al. 2019, machine learning emissions, arXiv 1910.09700. - RAGPerf, arXiv 2603.10765. - BERGEN, arXiv 2407.01102. - FlashRAG, arXiv 2405.13576 and doi:10.1145/3701716.3715313. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 7dcebe06..0dae682f 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -1,63 +1,69 @@ { "experiment_id": "35-full-rag-benchmark-2026-11-01", "protocol_version": 2, - "question": "How do embedder size, chunk size, the reranker and hybrid search, and their interactions, change retrieval quality on FinanceBench, the FreshStack LangChain slice, BioASQ and Legal RAG Bench with pinned local models, and how does chunk size change answer faithfulness and correctness at the best-value search settings?", + "question": "How do embedder size, chunk size, the reranker and hybrid search, and their interactions, change retrieval quality on FinanceBench, FreshStack LangChain (full public corpus), BioASQ and Legal RAG Bench (exploratory) with pinned local models, and how does chunk size change answer faithfulness and correctness at the embedder and reranker sizes chosen on half A? preregistration.md governs (amendment A4).", "decision": "The first fellowship paper cites these numbers; the experiment changes no production default, and any setting change it suggests goes to a separate proposal.", "hypotheses": [ { "id": "H1", "role": "primary", - "statement": "a bigger embedder finds better documents: marginal nDCG@10 rises from 0.6B to 4B and from 4B to 8B", - "null": "each embedder contrast is zero" + "statement": "within scope S (preregistration.md section 2), a bigger embedder ranks gold units higher: rows 1 (4B minus 0.6B) and 2 (8B minus 4B), nDCG@10, directional; a step holds when its row counts on at least 2 of 3 core datasets; H1 holds in full when both steps hold, in part when one does", + "null": "each contrast is zero" }, { "id": "H2", "role": "primary", - "statement": "the reranker helps: marginal nDCG@10 with the reranker on (mean of three sizes) exceeds reranker off", - "null": "reranker on and off do not differ (zero effect)" + "statement": "within scope S (preregistration.md section 2), the reranker raises nDCG@10: row 3 (mean of the three reranker sizes minus off), directional; holds when row 3 counts on at least 2 of 3 core datasets", + "null": "the contrast is zero" }, { "id": "H3", "role": "primary", - "statement": "hybrid search helps: marginal nDCG@10 with hybrid on exceeds hybrid off", - "null": "hybrid on and off do not differ (zero effect)" + "statement": "within scope S (preregistration.md section 2), hybrid search raises nDCG@10: row 4 (hybrid on minus off), directional; holds as H2", + "null": "the contrast is zero" }, { "id": "H4", "role": "primary", - "statement": "reranker and hybrid together beat either alone: both on exceeds reranker only and exceeds hybrid only", - "null": "at least one of the two contrasts is zero" + "statement": "within scope S (preregistration.md section 2), reranker plus hybrid beats each alone (simple effects, not an interaction test): rows 5 (both on minus reranker only) and 6 (both on minus hybrid only), nDCG@10, directional; holds on a dataset when both rows count there, and overall on at least 2 of 3 core datasets; the interaction is reported descriptively with its interval", + "null": "each contrast is zero" }, { "id": "H5", "role": "primary", - "statement": "a bigger reranker ranks better: marginal nDCG@10 rises from 0.6B to 4B and from 4B to 8B (reranker-on setups)", - "null": "each reranker-size contrast is zero" + "statement": "within scope S (preregistration.md section 2), a bigger reranker ranks better: rows 7 (4B minus 0.6B) and 8 (8B minus 4B), reranker-on setups, nDCG@10, directional; holds as H1", + "null": "each contrast is zero" }, { "id": "H6", "role": "primary", - "statement": "chunk size changes retrieval on FinanceBench (endpoint evidence_recall_20k) and FreshStack (endpoint nDCG@10); both must support it; BioASQ is a sanity check and Legal RAG Bench is exploratory", - "null": "no chunk-size contrast counts on FinanceBench or on FreshStack" + "statement": "within scope S (preregistration.md section 2), chunk size changes retrieval: rows 9 (1,024 minus 512) and 10 (2,048 minus 1,024), two-sided; endpoint evidence_recall_20k with any-page credit on FinanceBench and nDCG@10 on FreshStack; holds when the same row (9 or 10) counts with the same sign on FinanceBench and on FreshStack; opposite signs are reported as a dataset-dependent effect; BioASQ is a sanity check and Legal RAG Bench is exploratory", + "null": "each contrast is zero" }, { "id": "H7", "role": "primary", - "statement": "bigger embedders gain more from bigger chunks: the 8B gain from 512 to 2048 tokens exceeds the 0.6B gain", - "null": "the two gains do not differ (zero effect)" + "statement": "within scope S (preregistration.md section 2), a bigger embedder gains more from bigger chunks: row 11, (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, on the H6 endpoints; holds when row 11 counts on both FinanceBench and FreshStack; BioASQ is a sanity check and Legal RAG Bench is exploratory; expected to be underpowered", + "null": "the difference of differences is zero" }, { - "id": "H8", + "id": "H8a", "role": "answer-stage", - "statement": "chunk size changes GLM-5.3 faithfulness on factual claims and answer correctness (Ragas AnswerCorrectness on all four datasets, FinanceBench number match); faithfulness is expected to fall as chunks grow; tested on all questions", - "null": "each chunk-size contrast is zero" + "statement": "within scope S (preregistration.md section 2), chunk size changes GLM-5.3 faithfulness on factual claims and Ragas AnswerCorrectness in the 12 crossed setups, where context length grows with chunk size: rows 12 to 13 (faithfulness) and 14 to 15 (correctness), 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided, all questions, 3 runs averaged per question; per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack (open item O6); BioASQ rows are sanity checks; faithfulness is expected to fall as chunks grow; Legal RAG Bench is exploratory; FinanceBench number match is secondary; confirmatory", + "null": "each contrast is zero" + }, + { + "id": "H8b", + "role": "answer-stage", + "statement": "within scope S (preregistration.md section 2), chunk size changes faithfulness and correctness when evidence is held at 20,480 Qwen3 tokens: rows 16 to 17 (faithfulness) and 18 to 19 (correctness), c1024-t20k minus c512-t20k and c2048-t20k minus c1024-t20k, two-sided, at reranker on and hybrid on with the chosen sizes (conditional on that cell); holds as H8a (same row, same sign, on FinanceBench and on FreshStack); 'chunk size itself changes X' is claimed only when H8b holds for outcome X; confirmatory", + "null": "each contrast is zero" } ], - "decision_rule": "Per dataset, a linear mixed model (statsmodels MixedLM, REML: ndcg10 ~ C(embedder)*C(chunk)*C(reranker)*C(hybrid) + (1|question)) gives estimates; each planned contrast's p-value comes from a GEE with the same mean model and cluster-robust errors by question (by document on FinanceBench). Search scoring uses the top 100 chunks (rerank pool fixed at 100 with fetch_k 100) collapsed to distinct units and cut at 10. H6 on FinanceBench uses evidence_recall_20k (gold pages within 20,480 real Qwen3 tokens of evidence) as its endpoint, with first-page and any-page nDCG@10 as sensitivity; H6 on FreshStack uses nDCG@10. H8 uses all questions, 3 runs averaged per question. Holm across every H1-H8 contrast of the dataset. A contrast counts when the Holm-adjusted p is below 0.05 and its sign matches. A hypothesis holds overall when supported on at least 2 of the 3 confirmatory datasets (FinanceBench, FreshStack, BioASQ); H6 needs both FinanceBench and FreshStack. Legal RAG Bench is exploratory. Paired permutation (or the document-grouped bootstrap on FinanceBench) is the pre-registered sensitivity check.", + "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks, rows 12 to 19 pending open item O6); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs; opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", "stats": "expert", "seed": 20261101, "baseline": "none: full factorial of embedder x chunk size x reranker x hybrid (72 search setups per dataset); e4b-c1024-roff-hoff is the reference cell of the G2 rebuild check only", - "candidate": "72 search setups e<0.6b|4b|8b>-c<512|1024|2048>-r-h; 15 answer setups (12 crossed plus token-matched c512-t20k, c1024-t20k and c2048-t20k; the third added by A3), chosen on half A, at the embedder and reranker sizes chosen by the pre-registered selection rule (pooled over the three core datasets, amendment A3)", + "candidate": "72 search setups e<0.6b|4b|8b>-c<512|1024|2048>-r-h; 15 answer setups (12 crossed plus token-matched c512-t20k, c1024-t20k and c2048-t20k; the third added by A3), chosen on half A, at the embedder and reranker sizes chosen by the pre-registered selection rule (pooled over the three core datasets, amendment A3); index cost and configuration hash as defined in preregistration.md section 14.1", "metrics": [ { "name": "Recall@10", @@ -122,6 +128,11 @@ { "name": "evidence_recall_20k", "binary": false, + "primary": true + }, + { + "name": "alpha-nDCG@10", + "binary": false, "primary": false }, { @@ -129,6 +140,11 @@ "binary": false, "primary": false }, + { + "name": "closed_book_rag_prompt_correctness", + "binary": false, + "primary": false + }, { "name": "oracle_correctness", "binary": false, @@ -156,6 +172,55 @@ "value": "count", "op": "<=", "threshold": 0 + }, + { + "id": "classifier-agreement", + "metric": "claim_classifier_agreement", + "value": "share on 100 L1 smoke claims", + "op": ">=", + "threshold": 0.9 + }, + { + "id": "classifier-refusal-recall", + "metric": "claim_classifier_refusal_recall", + "value": "share on 100 L1 smoke claims", + "op": ">=", + "threshold": 0.9 + }, + { + "id": "faithfulness-kappa", + "metric": "checker_vs_human_kappa", + "value": "point estimate, per checker, 450 claims from answers to all questions (both halves)", + "op": ">=", + "threshold": 0.6 + }, + { + "id": "faithfulness-kappa-lower", + "metric": "checker_vs_human_kappa", + "value": "lower 95% bound, per checker", + "op": ">=", + "threshold": 0.4 + }, + { + "id": "correctness-kappa", + "metric": "ragas_correctness_vs_human_kappa", + "value": "point estimate, per core dataset, 50 half-B answers", + "op": ">=", + "threshold": 0.6 + }, + { + "id": "correctness-kappa-lower", + "metric": "ragas_correctness_vs_human_kappa", + "value": "lower 95% bound, per core dataset", + "op": ">=", + "threshold": 0.4 + }, + { + "id": "correctness-auc-lower", + "metric": "ragas_correctness_auc", + "value": "lower 95% bound, per core dataset", + "op": ">", + "threshold": 0.5 } ], "frozen": [], @@ -238,11 +303,6 @@ "provider": "openrouter", "model": "openai/gpt-oss-120b" }, - { - "role": "judge-crosscheck", - "provider": "openrouter", - "model": "openai/gpt-6.1-terra" - }, { "role": "judge-glm", "provider": "zai-coding-plan", @@ -310,6 +370,13 @@ "before_first_run": true, "summary": "Selection rule amended: the pooled score for embedder and reranker-size choice uses the three core datasets only (Legal excluded). Adds a four-factor pick on half A for Part 2 (chunk size, reranker on or off, hybrid on or off; pooled mean nDCG@10 over the three core datasets; ties within 0.01 go to reranker off, then hybrid off, then lowest index cost, then smaller chunk). The cell's configuration hash goes to OSF before half B opens. Part 2 (RAGFlow) runs on half B only, once per system, with its own protocol (C1, C2) and its own human labels. Experiment 35 label counts are unchanged. Clarified after the PR 209 review (2026-11-13, no change of design): H1 to H8 nulls are zero effects, the minimum detectable effect is for power planning only; the G2 rebuild check runs before any search writes into the shared output folder. PR 209 clarification (2026-11-13): (a) token-matched control: a 15th answer setup c2048-t20k (20,480-token cut at 2,048-token chunks, reranker on, hybrid on) sits beside c512-t20k and c1024-t20k; the c2048 x 10 factorial cell is unchanged; H8 token-matched comparison uses the three -t20k setups; 15 answer setups x 3 runs; G1 covers 15; answer stage cost rises about 1/14 (about $40: $24 hosted plus about 18 pod hours), estimate about $690, ceiling stays $975. (b) Part 2 FinanceBench C1 unit: page level for every question and both systems; gate that RAGFlow returns page numbers on a seeded sample; fallback document level for every question and both systems if the gate fails; other unit as labelled sensitivity.", "where": "protocol.md section 0.12; docs/proposals/first-publication-benchmark.md; change experiment-35-full-document-rag-benchmark design D0.14" + }, + { + "id": "A4", + "date": "2026-11-14", + "before_first_run": true, + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b by open item O6), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF, then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O1 to O6 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF.", + "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] } diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md new file mode 100644 index 00000000..b5cadf63 --- /dev/null +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -0,0 +1,444 @@ +# Experiment 35 pre-registration: reproducible local RAG benchmark of OMRS + +## 1. Status and governing rule + +- **Date**: 2026-11-14 (amendment A4). Written before the freeze. No measured run has started. +- **This file governs Experiment 35.** It holds the live design only, with no history. Where it and any other file disagree (`protocol.md`, `plan.json`, the OpenSpec changes, the proposal), this file wins. It is the main file of the OSF deposit. +- **The proposal** `docs/proposals/first-publication-benchmark.md` stays the master plan for scope, order of work, Part 2 (RAGFlow) and E1 (EnterpriseRAG-Bench). Its Experiment 35 design facts must match this file. +- **Part 2 and E1.** Section 18 wins over the proposal for the points it fixes: the OMRS setup and its hash, the halves, the shared parts, the measurement settings, the C1 and C2 measures and tests, and the Part 2 missing-outcome rules. The proposal governs every other Part 2 and E1 fact until the Part 2 protocol is frozen. After that, the Part 2 protocol governs them. +- **History**: `protocol.md` sections 0.1 to 0.13 (amendments A1 to A4) and sections 1 to 7 hold earlier plans and their reasons. They are not live design. +- **Study type**: confirmatory benchmark, purpose `publication`. OSF data category (OSF Prereg form): registration before analysis of the outcome data. Some outcome data exist before the freeze. Index builds and search are deterministic (G2), so the MDE pilot's retrieval scores on half A equal main-run outcome data for the pilot cells. Smoke and test-pod answers also exist. Section 5 lists every earlier look at the data. +- **Task pointers.** "RDY task n" is task n of the OpenSpec change `exp35-main-run-readiness`. "C35 task n" is task n of `experiment-35-full-document-rag-benchmark`. "FLOW step n" is step n of `rag-benchmark-readiness-flow`. + +## 2. Research questions + +- **RQ1 (absolute performance).** What retrieval and answer quality does OMRS reach on FinanceBench, FreshStack, BioASQ and (exploratory) Legal RAG Bench, under the stated models and settings? Answered by the headline of section 14.3. +- **RQ2 (component effects).** How do embedder size, chunk size, reranker size and hybrid search change retrieval quality? How does chunk size change answer faithfulness and correctness? Answered by H1 to H8b. +- **RQ3 (external comparison).** Part 2 and E1. Section 18 gives what Experiment 35 fixes for them and wins over the proposal on those points (section 1). + +**Scope S.** Every hypothesis holds only within this scope: Qwen3-Embedding and Qwen3-Reranker models at Q8_0 on llama.cpp build b11941; English text; queries embedded with no query instruction; the reranker's default instruction; one hosted answer model; the four datasets as built in section 4. No claim extends past scope S. + +## 3. Factors, levels and setups + +| Factor | Levels | +| --- | --- | +| Embedder | Qwen3-Embedding 0.6B, 4B, 8B (Zhang et al. 2025) | +| Chunk size | 512, 1,024, 2,048 Qwen3 tokens; overlap 15% (77, 154, 307 tokens) | +| Reranker | Off, Qwen3-Reranker 0.6B, 4B, 8B (llama.cpp `/v1/rerank`) | +| Hybrid | Off (dense only), or BM25 plus dense with RRF fusion | + +- **Search stage.** Full factorial: 9 indexes (`e-c`) and 72 search setups (`e-c-r-h`) per dataset. Each setup runs once on its sealed index. Each index is built once (one-build-per-cell assumption, section 15). +- **Fixed in search.** Every setup saves its top 100 chunks (`top_k` 100, `fetch_k` 100, so the rerank pool is exactly 100 and the reranker returns all 100 in order). `similarity_threshold` 0.0. Metadata extraction `disabled`. All datasets go through the Markdown splitter and count tokens with the pinned Qwen3 tokenizer (revision `5cf2132a`). BioASQ and Legal passages are written as `.md`. Dense search is exact: no approximate vector index is built. +- **Answer stage.** 15 answer setups at the embedder and reranker size chosen on half A (section 14.1): 12 crossed setups (chunk size 3 x reranker on or off x hybrid on or off), each with 10 chunks of evidence; and 3 token-matched setups `c512-t20k`, `c1024-t20k`, `c2048-t20k` (reranker on, hybrid on), each with evidence cut at 20,480 real Qwen3 tokens. The `c2048` x 10 crossed cell keeps its 10 chunks. Each answer setup runs 3 times on all 1,160 questions. +- **Token cut.** Chunks are taken in rank order and their Qwen3 token counts are summed. A chunk is included when the total before it is below 20,480. The chunk that crosses the limit is truncated at the limit, and no later chunk is included. Truncation is recorded per question. +- **Other runs.** Two closed-book runs, one oracle run (section 13), and a length-controlled checker run (section 11.2). + +## 4. Datasets + +| Dataset | Questions | Corpus | Gold unit | Role | +| --- | ---: | --- | --- | --- | +| FinanceBench open sample (`PatronusAI/financebench`; Islam et al. 2023) | 150 | 84 PDFs, 12,013 pages, read to Markdown by `pdf-inspector` 1.17.0 | Page (189 evidence items) | Core, confirmatory | +| FreshStack LangChain (`freshstack/queries-oct-2024`, `freshstack/corpus-oct-2024`, config `langchain`; Thakur et al. 2025) | 203 | Full public corpus, 49,514 documents | Document | Core, confirmatory | +| BioASQ mini (`enelpol/rag-mini-bioasq`, test split) | 707 | 40,181 passages (`text-corpus/test`) | Passage | Core, confirmatory | +| Legal RAG Bench (`isaacus/legal-rag-bench`) | 100 | 4,876 passages | Passage (one per question) | Exploratory | + +- **FreshStack corpus.** All 9 FreshStack indexes are built on the full public LangChain corpus at a pinned revision (pre-freeze task P1). The download needs the user's approval at build time. The 10,000-document subset used in Experiments 9a and 22 (all 1,547 gold documents plus 8,453 random distractors) is retired for Experiment 35. The 20 Experiment 9 continuity queries, 15 continuity documents and the overlay runbook are not used. No result is compared with the FreshStack leaderboard unless the same corpus and metric are used. +- **Revisions and hashes.** FinanceBench revision `e04404e3a97f69f79c14d42f24981a1c9c3bcd18`, PDFs at commit `cc39aeb4afdf33909ee1412188bf89035950c2eb`; BioASQ revision `8a845907dc1cff31d42fa6f7bb9c6eef5f3ae6f6` (`protocol.md` section 4.1). FreshStack and Legal revisions: section 16. +- **Licences.** FinanceBench CC BY-NC 4.0; FreshStack CC BY-SA 4.0; BioASQ CC BY 2.5 with the NLM terms notice; Legal RAG Bench CC BY-NC-SA 4.0. Source checks: `protocol.md` section 4.2. Release rules: section 19. +- **Halves A and B.** Each dataset is split into half A and half B with seed 20261115 (`halves.json`, SHA-256 recorded at the freeze). FinanceBench is split by document: documents are shuffled with the seed and each goes, with all its questions, to the half that has fewer questions so far (ties to A). No FinanceBench document sits in both halves. FreshStack, BioASQ and Legal are split by question: questions are shuffled with the seed, and the first half (rounded up) is half A. +- **Use of the halves.** Half A feeds every selection step (section 14) and every pre-freeze sample drawn after `halves.json` exists (the MDE pilot, RAGFlow checks). Samples drawn earlier (smoke runs, test pod, classifier-gate claims) are listed in section 5 with their question IDs, and with their half once `halves.json` exists (P12). No selection step reads half B, so selection bias stays out of the half-B scores (Cawley and Talbot 2010). Half-B scores are computed only after the A3 configuration hash is time-stamped (section 14.2). H1 to H8b use all questions. + +## 5. Existing data + +| Data seen before the freeze | What was seen | Effect on this study | +| --- | --- | --- | +| FreshStack queries and qrels in Experiments 9a and 22 | Per-query nDCG@10 of Qwen3-Embedding-4B on the retired 10,000-document subset; 223 queries (203 public) | Queries and gold labels were seen; the full corpus was not. Experiment 22 values do not feed the MDE table. | +| Experiment 10 reranker result | The reranker then in use lowered technical-workload retrieval by 19 to 27% (repository `AGENTS.md`) | H2 predicts the reranker helps, set with this result known. Experiment 35 uses Qwen3-Reranker on llama.cpp. | +| Smoke runs (H-CC smoke and FreshStack rerun, 2026-11-11) | Answers and claims for 15 questions: BioASQ 4043, 1372, 604, 2291, 1122; FinanceBench `financebench_id_01476`, `_01964`, `_01484`, `_05718`, `_00476`; FreshStack 76404464, 78048414, 77762811, 77828572, 77536364 | Answers were read; 10 smoke claims were practice-labelled. The half of each ID is listed once `halves.json` exists (P12). | +| Factorial smoke run (FLOW step 6.10) and test pod session | 5 seeded questions per dataset, answers and retrieval | IDs and halves recorded at the freeze (P12). Never scored as results. | +| MDE pilot (section 10) | Retrieval scores of 40 seeded half-A questions per core dataset | Half A only. No selection is made from it. | +| Checker selection runs | LLM-AggreFact test split (external data) and smoke claims | The own-claims guard limits were set after the smoke data were seen (`protocol.md` section 4.4). | +| Experiment 48 timing | Embedding speed on slices of FreshStack, BioASQ and FinanceBench | Timing only, no scores. | + +## 6. Metrics + +**Unit scoring.** Each setup's top 100 chunks map to units (FinanceBench page, FreshStack document, BioASQ or Legal passage). The first occurrence of each unit is kept in rank order, and the list is cut at 10 units. On FinanceBench a chunk counts for its first page (primary); any-page credit is a sensitivity analysis. Relevance is binary. Every question has at least one gold unit. + +For question q with gold set G_q (size R_q) and unit list L_q of up to 10 units, rel_i = 1 when unit i of L_q is in G_q (missing ranks count as 0). Following the trec_eval convention, every metric is a plain mean over the questions of a dataset: + +- DCG@10 = sum over i = 1 to 10 of rel_i / log2(i + 1). +- IDCG@10 = sum over i = 1 to min(R_q, 10) of 1 / log2(i + 1). +- nDCG@10 = DCG@10 / IDCG@10. **Primary metric of the study.** +- Recall@10 = |L_q intersect G_q| / R_q. +- MRR@10 = 1 / (rank of the first gold unit in L_q), else 0. +- hit@page (FinanceBench) = 1 when any gold page is in L_q. +- **`evidence_recall_20k`** = share of G_q covered by the evidence taken from the top 100 chunks under the token cut of section 3. **Any-page credit:** a whole chunk covers every gold page it overlaps; a truncated crossing chunk covers only its first page, because the reader stores no page offsets inside a chunk. On FreshStack, BioASQ and Legal a chunk, whole or truncated, covers its one unit. First-page credit is a registered sensitivity analysis. **Confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11).** `plan.json` flags both nDCG@10 and `evidence_recall_20k` as `primary`, because there the flag marks a metric that a primary hypothesis's decision rests on (the guide asks for a tolerance per primary metric); it does not mark a second headline metric. +- **alpha-nDCG@10** (FreshStack, secondary): computed with the benchmark's own scorer if its nugget labels allow (P3); otherwise reported as not available. + +**Answer metrics.** Per answer, then averaged over the 3 runs per question, then over questions. + +- **Faithfulness** (`faithfulness_glm_factual`): share of the answer's factual claims (classifier kinds fact and mixed) that GLM-5.3 calls supported. Refusal claims are reported apart. Claims and citations come from the answer itself: the answer model cites evidence by ordinal (for example `[1]`), the saved answer is split into claims, only prose segments with at least one valid ordinal go to the checker, and the checker sees one claim with the text of the evidence its ordinals cite (OpenSpec specs `grounded-answer-synthesis` and `benchmark-answer-scoring`). +- **Correctness** (`ragas_answer_correctness`): Ragas 0.4.3 (Es et al. 2023) `AnswerCorrectness`, default weights (factual 0.75, similarity 0.25), judge `openai/gpt-oss-120b`, local 4B embedder. The judge sees the question, the answer and the reference answer, never the contexts. FreshStack's reference is `answer_text` (the accepted Stack Overflow answer). +- **Secondary**: Clef-Flash faithfulness, FinanceBench number match (`protocol.md` section 0.6 rule), refusal rate, the four other Ragas metrics, closed-book and oracle correctness. + +## 7. Hypotheses + +Each null is a zero effect. Every test is two-sided. A directional contrast counts when its Holm-adjusted p is below 0.05 and its estimate has the predicted sign (this is a one-sided test at 0.025). A two-sided contrast counts when its Holm-adjusted p is below 0.05. A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs. Rows that count with opposite signs are reported as a dataset-dependent effect, and the hypothesis does not hold. The report always states the direction. Marginal contrasts weight every cell of the other factors equally (for example, the embedder effect averages over 24 cells, 18 of them reranker-on). Row numbers refer to section 8. + +- **H1 (embedder).** Within scope S, a bigger embedder ranks gold units higher. Rows 1 (4B minus 0.6B) and 2 (8B minus 4B), nDCG@10, directional. Null: each contrast is zero. Holds: each step holds when its row counts on at least 2 of the 3 core datasets. H1 holds in full when both steps hold, in part when one does. +- **H2 (reranker).** Within scope S, the reranker raises nDCG@10. Row 3 (mean of the three reranker sizes minus off), directional. Null: zero. Holds when row 3 counts on at least 2 of 3 core datasets. +- **H3 (hybrid).** Within scope S, hybrid search raises nDCG@10. Row 4, directional. Null: zero. Holds as H2. +- **H4 (combining reranker and hybrid: simple effects).** Within scope S, reranker plus hybrid beats each alone. Row 5 (both on minus reranker only) and row 6 (both on minus hybrid only), directional. These are simple effects, not an interaction test; the interaction is reported descriptively with its interval. Null: each contrast is zero. Holds on a dataset when both rows count there; holds overall on at least 2 of 3 core datasets. +- **H5 (reranker size).** Within scope S, a bigger reranker ranks better (reranker-on setups). Rows 7 (4B minus 0.6B) and 8 (8B minus 4B), directional. Holds as H1. +- **H6 (chunk size, search).** Within scope S, chunk size changes retrieval. Rows 9 (1,024 minus 512) and 10 (2,048 minus 1,024), two-sided. Endpoint: `evidence_recall_20k` on FinanceBench, nDCG@10 on FreshStack. Null: each contrast is zero. Holds when the same row (9 or 10) counts with the same sign on FinanceBench and on FreshStack. BioASQ is a sanity check (most passages are shorter than 512 tokens, so chunk size barely changes its chunks); Legal is exploratory. FinanceBench sensitivity: nDCG@10 with first-page and with any-page credit; the report gives sign agreement per row. +- **H7 (embedder by chunk size).** Within scope S, a bigger embedder gains more from bigger chunks. Row 11: (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, with the H6 endpoints. Null: zero. Holds when row 11 counts on both FinanceBench and FreshStack. BioASQ is a sanity check, Legal exploratory. Expected to be underpowered (section 10). +- **H8a (chunk size with context length, answers).** Within scope S, chunk size changes faithfulness and correctness in the 12 crossed setups, where context length grows with chunk size (long contexts can hide evidence: Liu et al. 2024). Rows 12 to 13 (faithfulness) and 14 to 15 (correctness): 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided. Expected direction for faithfulness: falls as chunks grow (Wang et al. 2024); the report gives the sign. Null: each contrast is zero. Per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack. Legal is exploratory. BioASQ chunks barely change with chunk size, so BioASQ rows 12 to 19 are sanity checks, like rows 9 to 11 (section 8). This rule depends on open item O6. +- **H8b (chunk size at equal tokens, answers).** Within scope S, chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. Rows 16 to 17 (faithfulness) and 18 to 19 (correctness): `c1024-t20k` minus `c512-t20k` and `c2048-t20k` minus `c1024-t20k`, two-sided, at reranker on and hybrid on with the chosen sizes (so H8b is conditional on that cell). Holds as H8a (per outcome, the same row counts with the same sign on FinanceBench and on FreshStack). **Reading fixed now:** "chunk size itself changes X" is claimed only when H8b holds for outcome X. When H8a holds and H8b does not, the claim is "chunk size together with context length changes X". + +**Error rates of the composite rule.** Holm within each dataset plus "at least 2 of 3 datasets" (H1 to H5) or "FinanceBench and FreshStack" (H6 to H8b) is conservative. Under the null on three independent datasets, a false "holds" for one directional contrast has probability about 3 x 0.025 squared, about 0.002, before Holm. Under the FinanceBench-and-FreshStack rule it is about 0.025 squared (about 0.0006) for a directional row, and 2 x 0.025 squared (about 0.001) for a two-sided row under the same-sign rule. Power falls the other way: at a per-dataset power of 0.6, the 2-of-3 rule has about 0.65; at 0.5, about 0.50. + +## 8. Contrast table and Holm families + +Contrasts are per question, marginal over every factor not named. Search rows use all 72 setups; H8a rows use the 12 crossed setups; H8b rows use the 3 token-matched setups. FB is FinanceBench, FS FreshStack, BQ BioASQ. + +| Row | Hyp. | Contrast | Sign | FB endpoint | FS endpoint | BQ | +| ---: | --- | --- | --- | --- | --- | --- | +| 1 | H1 | e4b minus e0.6b | + | nDCG@10 | nDCG@10 | family | +| 2 | H1 | e8b minus e4b | + | nDCG@10 | nDCG@10 | family | +| 3 | H2 | mean(r0.6b, r4b, r8b) minus roff | + | nDCG@10 | nDCG@10 | family | +| 4 | H3 | hon minus hoff | + | nDCG@10 | nDCG@10 | family | +| 5 | H4 | (reranker on, hon) minus (reranker on, hoff) | + | nDCG@10 | nDCG@10 | family | +| 6 | H4 | (reranker on, hon) minus (roff, hon) | + | nDCG@10 | nDCG@10 | family | +| 7 | H5 | r4b minus r0.6b | + | nDCG@10 | nDCG@10 | family | +| 8 | H5 | r8b minus r4b | + | nDCG@10 | nDCG@10 | family | +| 9 | H6 | c1024 minus c512 | two-sided | `evidence_recall_20k` | nDCG@10 | sanity | +| 10 | H6 | c2048 minus c1024 | two-sided | `evidence_recall_20k` | nDCG@10 | sanity | +| 11 | H7 | (e8b c2048 minus e8b c512) minus (e0.6b c2048 minus e0.6b c512) | + | `evidence_recall_20k` | nDCG@10 | sanity | +| 12 | H8a | c1024 minus c512 | two-sided | faithfulness | faithfulness | sanity | +| 13 | H8a | c2048 minus c1024 | two-sided | faithfulness | faithfulness | sanity | +| 14 | H8a | c1024 minus c512 | two-sided | correctness | correctness | sanity | +| 15 | H8a | c2048 minus c1024 | two-sided | correctness | correctness | sanity | +| 16 | H8b | c1024-t20k minus c512-t20k | two-sided | faithfulness | faithfulness | sanity | +| 17 | H8b | c2048-t20k minus c1024-t20k | two-sided | faithfulness | faithfulness | sanity | +| 18 | H8b | c1024-t20k minus c512-t20k | two-sided | correctness | correctness | sanity | +| 19 | H8b | c2048-t20k minus c1024-t20k | two-sided | correctness | correctness | sanity | + +"Reranker on" in rows 5 and 6 is the mean over the three reranker sizes. BioASQ uses nDCG@10 for rows 1 to 8. + +**Family sizes, fixed now.** FinanceBench: 19 (rows 1 to 19). FreshStack: 19 (rows 1 to 19). BioASQ: 8 (rows 1 to 8). Legal RAG Bench: no family; all 19 rows are exploratory. In all, 46 contrasts sit in Holm families. BioASQ rows 9 to 19 are sanity checks, reported with unadjusted 95% intervals and no decision. The reason is the same for all of them: most BioASQ passages are shorter than 512 tokens, so chunk size barely changes BioASQ chunks. Rows 12 to 19 as sanity checks depend on open item O6. + +**Gated rows stay in the family.** No gate changes a family size. A row whose measure fails its gate (section 11), or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as "not testable". With p = 1 it takes the last Holm rank, so the other rows keep the same number of comparisons. + +## 9. Analysis model + +One clustered method gives every estimate, interval and p-value. There is no mixed model. + +1. **Per-question contrast score.** For row k and question q, d_qk is the mean of question q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights). Answer scores are first averaged over the 3 runs. +2. **Estimate.** The plain mean of d_qk over the questions that enter the row (section 12). +3. **FreshStack, BioASQ and Legal.** The decision test is defined by its closed form. Let n be the number of questions that enter the row and r_q = d_qk minus the estimate. Estimate: the mean of d_qk. Variance: (sum of r_q squared) x (n / (n minus 1)) squared / n squared. Reference distribution: t on n minus 1 degrees of freedom, for the p-value and the 95% interval. This is the Mancl and DeRouen (2001) bias-reduced sandwich for an intercept-only model with one row per cluster. It equals the paired t-test variance times n / (n minus 1), so it needs no convergence fallback. With complete data the estimate equals the cell-means contrast of the full-factorial model. **Implementation:** `statsmodels` GEE (Liang and Zeger 1986) of d_qk on an intercept (Gaussian family, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) gives the estimate and the variance. The p-value and the interval are always computed from the t reference above, whatever reference the library reports by default. A known-answer test (RDY task 15.6) checks that the library estimate and variance equal the closed form. If they do not, the closed form is used and the difference is reported. +4. **FinanceBench (questions share documents).** The decision p-value comes from a document-level sign-flip permutation test. The statistic is the question-weighted mean of d_qk. Under the null, the sign of each document's sum of d_qk is flipped at random: 100,000 draws, seed 20261101, p = (1 + number of draws with an absolute statistic at least as large) / 100,001. The 95% interval inverts the same test (the set of shifts not rejected at 0.05, with the same draws). The test assumes each document's sum is symmetric about zero under the null. The interval also assumes a constant additive shift for every question (a location-shift model); the point estimate and the p-value do not need it. +5. **Decision.** Holm (alpha 0.05) within each family (Carterette 2012) on the decision p-values, then the sign rule of section 7, then the 2-of-3 or both-datasets rule of each hypothesis (with the same-sign rule of section 7 for two-sided rows). +6. **Sensitivity checks, registered, never decisive.** FreshStack and BioASQ: paired permutation test by question (10,000 shuffles, seed 20261101; Smucker, Allan and Carterette 2007; Urbano, Lima and Hanjalic 2019). FinanceBench: the GEE of point 3 with document clusters. Every row where the decision and the sensitivity check disagree on "counts" is listed in one table. +7. **Search-stage scores** come from one run per setup. **Intervals** printed beside Holm decisions are unadjusted 95% intervals (Miller 2024). + +A non-significant result means no difference was detected at this sample size. The study makes no equivalence claim and runs no TOST. + +## 10. Power, minimum detectable effect and smallest effect of interest + +- **Smallest effect of interest (SESOI).** 0.02 nDCG@10 (OPEN item O2: the user confirms the value; Lakens 2017). Each row's interval is reported against plus and minus the SESOI. No other endpoint has a SESOI; their intervals are reported against the MDE. +- **Pilot.** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118), on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. The pilot builds at most 4 indexes per core dataset: the four H7 corner cells (`e0.6b-c512`, `e0.6b-c2048`, `e8b-c512`, `e8b-c2048`). Reranker and hybrid setups run on those indexes as the cell list needs. A contrast between adjacent levels (for example 4B minus 0.6B) uses the SD of the extreme-level contrast (8B minus 0.6B) as a stand-in, and the MDE table says so. Pilot indexes are built with the pinned models and settings and are not reused: the main run builds all 9 indexes after the freeze. The pilot gives the SD of per-question contrast scores per dataset and contrast type, and on FinanceBench the design effect of document clustering. Experiment 22 values are not used. +- **Pilot cost.** The pilot cost is projected from the P2 timing, added to the estimate before the freeze and shown in the compute table (section 17). The user approves it before the pod starts. Registered fallback if the user declines: no pilot; the MDE table is filled as a range from assumed SDs of 0.15 to 0.25 for retrieval contrasts, labelled as assumed, and the report says H7 and Part 2 C2 are expected to be underpowered. +- **MDE.** MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha = 0.05 / family size (worst Holm rank: 0.05/19 on FinanceBench and FreshStack, 0.05/8 on BioASQ) and n divided by the design effect on FinanceBench. Answer-stage rows use the SD from named smoke and test-pod answer files (P6) and are labelled rough. + +| Dataset | Row(s) | SD of d (source) | MDE | Below SESOI? | +| --- | --- | --- | --- | --- | +| (filled by P5 and P6 before the freeze) | | | | | + +- **Known weak spots (reviewer estimates, to be replaced by the table).** H7 compares differences of differences of single cells; at plausible SDs its MDE on FreshStack is about 0.08 to 0.12 nDCG@10, so H7 is expected to be underpowered. Part 2 C2 on half B is expected to detect only large correctness differences. The report states both. + +## 11. Judges, checkers and human labels + +### 11.1 Models + +- **Main claim checker**: GLM-5.3 (Z.AI Coding Plan endpoint, `chat`, reasoning `low`, temperature 0, seed 20261109). **Second checker**: Clef-Flash (local GGUF). If Clef-Flash fails its faithfulness gate, its results are reported as not validated and it changes no confirmatory row. The runner never switches checker itself; a failed call stops the stage and `--resume` retries it. If GLM-5.3 is withdrawn before the run ends, that is a deviation (section 20). +- **Checker tolerance.** A local checker may replace the main one only under the rule of `protocol.md` section 4.4. Its 2-point allowance is restated against the H8a and H8b faithfulness MDE once the MDE table is filled: the bar stays only if it is at most half that MDE (P7). The withdrawn "about 4 points" figure is not used. +- **Claim classifier**: `openai/gpt-oss-120b` sorts claims into fact, refusal or mixed. Pre-freeze gate: agreement at least 0.90 and refusal recall at least 0.90 on 100 L1-labelled smoke claims (seed 20261112). +- **Ragas judge**: `openai/gpt-oss-120b` (reasoning `low`, temperature 0, one pinned upstream host). Its seed, and the classifier's seed, are pins still to record (P10). There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. + +### 11.2 Gates + +Gate decisions are computed and recorded, with a time, before any H8 contrast result is computed. + +**Interval method for every gate.** Every gate interval (kappa, the AUC lower bound, the specificity difference) is a percentile bootstrap: 10,000 resamples, resampling by question, so all claims or answers of one question move together. Each interval has its own seed (section 16). + +- **Faithfulness gate, per checker.** Cohen's kappa (Sim and Wright 2005) of the checker against the human label: point estimate at least 0.60 and lower 95% bound at least 0.40. Sample: 450 claims from the crossed setups, 150 per chunk size, 50 per core dataset within each chunk size, at most one claim per answer, seed 20261116. The claims are drawn from answers to all questions (both halves), because H8a and H8b are tested on all questions. The correctness sample below stays on half B. The gate is pooled over the three core datasets because 150 claims per dataset are too few for a per-dataset gate; kappa per checker and per core dataset is reported with its interval as descriptive and decides nothing. "Can't tell" counts as not supported (the binary benchmark scale); kappa is also reported with "can't tell" items left out. If GLM-5.3 fails, rows 12, 13, 16 and 17 are not testable on every dataset. +- **Differential-error rule.** "Unsupported" is the positive class. Specificity is the share of human-supported claims the checker calls supported. The rule uses GLM-5.3, the main checker; Clef-Flash specificity is reported only. If the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, H8a and H8b faithfulness (rows 12, 13, 16, 17) is exploratory and not testable. Sensitivity per chunk size is reported. +- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. The Ragas score is cut at 0.5 (fixed before labelling). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). +- **Length-controlled check (registered sensitivity, no gate).** 300 claims from `c2048` crossed answers (seed 20261119). GLM-5.3 and Clef-Flash judge each claim again against one 512-token window of each cited chunk: the chunk is split into consecutive 512-token windows, and the window that shares the most distinct lower-case claim words is kept (ties go to the earlier window). The report gives the verdict change rate with a Wilson interval. + +### 11.3 Human labels + +- **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller (OPEN item O4: name, role and relation to the project; recorded in the run record, never in public files). L2 labels a 30% subset of each set (seeds 20261122 to 20261124, one per set; section 16). Kappa of L1 against L2 is reported per set with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. +- **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: 225. +- **Guidelines.** One written guideline per set (claim kinds, faithfulness, correctness, and Part 2 correctness), with worked examples drawn from outside every sample, fixed before the first label and released (P8). Faithfulness follows the LLM-AggreFact task definition (MiniCheck): a claim is supported if and only if the cited passages support it, judged as the checker sees it, on meaning not grammar. +- **Blinding.** Labellers never see scores, verdicts, setup names or chunk sizes. Cited passage length can still hint at chunk size; the report says so. A training round on practice items comes first and is not evidence. +- **Ethics.** An ethics query goes to the university before the first label (OPEN item O5); the answer is recorded. +- **Conflict of interest.** The authors built OMRS. Safeguards: this pre-registration and its deviation log; selection on half A only; Holm families fixed now; blind labels with L2 as primary labeller in Part 2; published RAGFlow settings and maintainer correspondence; release of every output (section 19). The paper states the conflict. +- **Optional blind relevance audit** for H3 and C1 (about 300 to 400 L1 labels on unjudged top-3 units, seed 20261120): runs only if the user approves (OPEN item O3). It would be descriptive. + +## 12. Missing outcomes + +Every question stays in its dataset's denominator. Every exclusion is counted per setup and row and printed. + +- **Search.** A list shorter than 10 units scores its missing ranks as not relevant. A search row still missing after `--resume` scores 0 and is listed. +- **Failed answer call.** The resend rule applies: a call is resent only when no result was saved, never because of content, and each resend is counted (`resent_calls`). An answer still missing when the run closes scores 0 for correctness and number match, and counts as an answer with no factual claim. +- **Answer with no factual claim.** Faithfulness is undefined for that run. Per question and setup, faithfulness is the mean over the runs that have at least one factual claim. A question enters a faithfulness row only when every cell used by that row has a value (complete cases). Two registered sensitivity analyses keep every question: no-claim answers scored 1, and scored 0. The refusal rate per setup is reported as a co-outcome. +- **Unjudged rows.** Ragas and checker rows that failed are retried on `--resume`. A row still unjudged when the run closes leaves that row's complete cases and is counted; a sensitivity analysis scores it 0. +- **Unclassified claims.** A claim the classifier still fails on after `--resume` is treated as a factual claim: it stays in the faithfulness denominator and is counted. A sensitivity analysis leaves such claims out. +- **Part 2 rules** are in section 18. + +## 13. Floor, ceiling and closed-book runs + +All are single runs on all 1,160 questions, scored like the main runs, descriptive only, with no hypothesis test. The report says that the main setups average 3 runs and these do not. + +- **Closed-book, own-knowledge prompt (floor).** The answer model with no evidence and a fixed prompt that asks for an answer from its own knowledge. +- **Closed-book, RAG prompt.** The main answer prompt with an empty evidence block, so it differs from the RAG setups only in the evidence. A registered sensitivity analysis. +- **Oracle (ceiling).** Gold units given directly as evidence (Legal gold passage, FinanceBench gold page texts, BioASQ gold passages, FreshStack gold documents), capped at 20,480 Qwen3 tokens with truncation recorded. It has no distractors, so it is an upper bound, not a fair comparison. +- **Contamination sensitivity.** Rows 14, 15, 18 and 19 are re-run on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5). The report gives closed-book correctness per dataset beside every correctness table, and a per-question cross-table on retrieval misses (closed-book correct or not against RAG correct or not). + +## 14. Selection rule, order of work and RQ1 headline + +### 14.1 Selection (A3), half A only + +Pooled score = plain mean of the per-dataset mean nDCG@10 over FinanceBench, FreshStack and BioASQ, on half A. Legal takes no part. + +1. **Embedder.** Pooled score per embedder averaged over all other factors. Choose the smallest embedder within 0.01 of the best (a gap of exactly 0.01 counts as within). +2. **Reranker size.** At the chosen embedder, reranker-on setups averaged over chunk size and hybrid. Choose the smallest size within 0.01 of the best. Equal scores go to the smaller size. +3. **A3 cell (for RQ1 and Part 2).** At the chosen embedder and reranker size, the 12 cells (chunk size x reranker on or off x hybrid on or off). A cell ties the best cell when its pooled score is within 0.01 of it. Among tied cells choose, in order: (1) reranker off, (2) hybrid off, (3) the chunk size with the lowest index cost, (4) the smaller chunk size. Each step applies to the cells left by the step before. + +- **Index cost** = stored vector values: number of chunks times embedding dimension, summed over the three core datasets. It is known from the sealed indexes before any score. +- **Configuration hash** = SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings, with the SHA-256 of `halves.json` as one field. +- `selection.json` records every pooled score, both chosen sizes, the A3 cell and the hash. A sensitivity analysis repeats step 3 with any-page nDCG@10 on FinanceBench and reports whether the cell changes; the registered cell does not change. + +### 14.2 Order of work + +1. Build and seal the 9 indexes; G2 passes before any search writes into the shared output folder. +2. Run the 72 search setups on all questions. +3. Score half A only. Apply 14.1. Write `selection.json`. +4. Time-stamp the configuration hash with a file upload to the registered OSF project (OPEN item O1: the user confirms the mechanism). +5. Only then score half B, and start the replicate pods (answer stage). + +The word "sealed" applies to indexes only. Half B is held out from every selection step; its search lists exist on disk before step 4, but no half-B score is computed before it. + +### 14.3 RQ1 headline + +- **Retrieval headline**: the A3 cell's nDCG@10 on half B, per core dataset, with its 95% interval. The 72-setup table on all questions is descriptive. +- **Answer headline**: the answer setup that matches the A3 cell, on half B, faithfulness and correctness per dataset with intervals. +- **Search-cost figure**: the expected maximum of the pooled half-A score over n setups drawn at random, adapted from Dodge et al. 2019 (drawn without replacement), computed exactly from the scores sorted in ascending order, v(1) to v(N), as the sum over i = n to N of v(i) x C(i minus 1, n minus 1) / C(N, n). One curve over the 12 A3 cells (n = 1 to 12) and one over the 72 setups (n = 1 to 72). RAGFlow has no point unless its matched search runs (section 18); then RAGFlow gets its own curve over its half-A cells. + +## 15. Gates and determinism + +| Gate | Pass when | +| --- | --- | +| G1 | For each of the 15 answer setups, the retrieval of answer runs 1, 2 and 3 equals the single search-stage run of that setup. | +| G2 | The `e4b-c1024` index of each dataset, rebuilt in a fresh process, gives the same ranked lists for `e4b-c1024-roff-hoff`. It runs after the seal and before any search writes into the shared output folder. | +| G3 | `verify_llamacpp_runtime` passes for every pinned GGUF file before the first score of every run. | + +- **One-build-per-cell assumption.** Each index is built once. Build-time variation is assumed small against the MDEs. G2 covers one index per dataset. +- **Optional rebuild check `e8b-c2048`** on FinanceBench: it runs only if the cost projection after the search stage is at or below the estimate (a cost-only decision). It is reported as the share of identical ranked lists and the nDCG@10 change, not as a gate. +- **Tolerance on another back end**: 0.005 absolute on nDCG@10, Recall@10, MRR@10 and hit@page (`plan.json` `runtime.tolerance`); `evidence_recall_20k` tolerance is task P14. +- A determinism mismatch is recorded and reported, never swapped for another run. + +## 16. Pins + +| Pin | Value or rule | Where recorded | +| --- | --- | --- | +| llama.cpp build | b11941 exactly | `plan.json` `runtime.models[].build` | +| Embedders 0.6B, 4B, 8B (GGUF, SHA-256, server args) | Q8_0 files | `plan.json` `runtime.models` (roles `embedding-*`) | +| Rerankers 0.6B, 4B, 8B | Pinned after Spearman at least 0.95 against the transformers reference on 200 pairs per size (seed 20261117) | Missing: C35 task 4A.3, RDY task 15.10 | +| Clef-Flash | Q8_0 file and server args | `plan.json` `runtime.models` (role `judge-clef-flash`) | +| Tokenizer | `Qwen/Qwen3-Embedding-4B` `tokenizer.json`, revision `5cf2132a` | `protocol.md` section 0.3; add to `plan.json` (P10) | +| Answer model | `deepseek/deepseek-v4.1-flash-1017`, OpenRouter, temperature 0, seed 20261101, `answer.context_window` 131,072 characters, `answer.max_rounds` 1 | `plan.json` `runtime.references` (model only) | +| Answer model host, reasoning mode, output cap, context length (at least 65,536 tokens) | To record | Missing: P10 | +| Ragas judge and classifier | `openai/gpt-oss-120b`, reasoning `low`, temperature 0, fixed seed, one host | `plan.json` `runtime.references` (model only); host and settings P10 | +| GLM-5.3 | Z.AI, `chat`, reasoning `low`, temperature 0, seed 20261109 | `protocol.md` section 4.4; `plan.json` model only (P10) | +| Ragas, statsmodels | Ragas 0.4.3; statsmodels in the `stats` dependency group | Ragas: `uv.lock`. statsmodels: Missing: RDY task 15.6 adds a `stats` dependency group and pins the version | +| Query instruction | Empty string | Missing: P11 | +| Reranker instruction | llama-server default Qwen3-Reranker instruction, text recorded | Missing: P11 | +| Hybrid settings | RRF constant, per-side fusion depth, BM25 tokeniser and parameters | Missing: P11 | +| Search settings | Section 3 values | `protocol.md` section 0.3; `plan.json` `decision_rule` | +| FreshStack revision and corpus tree hash | Full 49,514-document corpus | Missing: P1 | +| Legal RAG Bench revision and SHA-256 | To record | Missing: C35 task 4A.4, RDY task 13.4 | +| FinanceBench, BioASQ revisions and hashes | Section 4 | `protocol.md` section 4.1 | +| `halves.json` SHA-256 | Seed 20261115, FinanceBench by document | Missing: P12 | +| Prompt template hashes | Answer, both closed-book, oracle, classifier | Missing: P13 | +| GPU type, image digest, driver, CUDA | Chosen by the `protocol.md` section 0.8a rule | Environment record per pod; GPU choice is a readiness task | +| Seeds | One seed per use (table below) | This file; `plan.json` `seed` holds 20261101 | +| Freeze commit and OSF link | At the freeze | `plan.json` `frozen` (empty today) | + +**Seeds.** Each seed has one use, except 20261101. That seed has three independent uses: the answer model's hosted seed, the FinanceBench sign-flip test and the FreshStack and BioASQ permutation check. This is safe: the hosted seed goes to the provider and feeds no local random draw, and the two tests run on different datasets, so no draw feeds two results. The Ragas judge and classifier seeds are pins still to record (P10). New seeds wait for the user (P9). + +| Seed | Use | +| --- | --- | +| 20261101 | Answer model hosted seed; FinanceBench sign-flip test (section 9 point 4); FreshStack and BioASQ permutation check (section 9 point 6) | +| 20261109 | GLM-5.3 hosted seed | +| 20261112 | Classifier-gate sample (100 smoke claims) | +| 20261113 | Correctness label sample: questions and answer setups (section 11.2) | +| 20261115 | Halves A and B (`halves.json`) | +| 20261116 | Faithfulness label sample (450 claims) | +| 20261117 | Reranker rank check (200 pairs per size) | +| 20261118 | MDE pilot questions (40 per core dataset, half A) | +| 20261119 | Length-controlled check (300 claims) | +| 20261120 | Optional relevance audit (O3) | +| 20261121 | Factorial smoke run and test pod: setups and 5 questions per dataset | +| 20261122 | L2 subset, claim kinds (30 of 100) | +| 20261123 | L2 subset, faithfulness (135 of 450) | +| 20261124 | L2 subset, correctness (60 of 200) | +| 20261125 | Faithfulness-gate kappa bootstrap (each checker, pooled and per dataset) | +| 20261126 | Differential-error rule bootstrap (GLM-5.3 specificity difference) | +| 20261127 | Correctness-gate kappa bootstrap | +| 20261128 | Correctness-gate AUC bootstrap | +| 20261129 | L1 against L2 kappa bootstrap | +| 20261130 | RAGFlow page-number gate sample (FinanceBench, half A) | +| 20261131 | RAGFlow repeatability sample (half A) | +| 20261132 | Part 2 label list order (both systems shuffled into one list) | +| 20261133 | Part 2 correctness-gate sample | +| 20261134 | Part 2 correctness-gate kappa bootstrap | +| 20261135 | Part 2 correctness-gate AUC bootstrap | + +## 17. Budget, ceiling, stop rule and drop order + +- **Estimate**: about **$730** on an RTX 4090 at $0.89 per hour (estimate). It is the A3 estimate of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 extra pod hours, about $30 to $40, not timed: P2), the second closed-book run (about $3 hosted and about 1 pod hour, by analogy with the first) and the pilot. The pilot cost is not yet estimated: it is projected from the P2 timing, added here before the freeze and approved by the user before the pod starts (section 10). Search cost barely changes with corpus size, because the rerank pool stays 100. +- **Ceiling**: **$975**, pods and hosted calls together (1.5 x the A2 estimate, about 1.34 x this one). At 3 times slower reranking than assumed, the total is about $950 (estimate), still under the ceiling with little margin. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. +- **Checks**: before the freeze (test pod extrapolation), on the build pods after the search stage, and at each stage end while the replicate pods run. Each projection uses billing and measured rates only, never scores. +- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. +- **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early when the preflight fails, a gate fails, or a projection reaches the ceiling after the last drop step. +- **Part 2 and E1** each get their own estimate and ceiling before their runs (P16). + +## 18. What Experiment 35 fixes for Part 2 and E1 + +Experiment 35 fixes the points below. On these points this section wins over the proposal. The proposal governs every other Part 2 and E1 fact until the Part 2 protocol is frozen; after that, the Part 2 protocol governs them (section 1). + +- **OMRS setup**: the A3 cell (section 14.1), its hash time-stamped on OSF before half B is scored. Part 2 runs on half B only, once per system. +- **Estimand, fixed before any half-B score.** The cost decision at FLOW step 7.2a fixes which estimand applies. It is recorded before any half-B score exists, so the estimand is never chosen after seeing results. + - If the matched RAGFlow search runs: C1 and C2 compare "selected OMRS against selected RAGFlow". The RAGFlow cell is chosen on half A by the same selection rule. Its configuration hash is time-stamped on OSF with the OMRS hash, before any half-B score exists. Default RAGFlow is then a secondary, descriptive row. + - If the matched search does not run: C1 and C2 compare "selected OMRS against default RAGFlow". + - In both cases the shared models are fixed. +- **Default RAGFlow** is one RAGFlow run: its defaults from the pinned release, plus the shared parts (the embedder, and the reranker when the A3 cell has it on) and the two measurement settings (cut-off 0, depth 100). When the matched search does not run, the RAGFlow side of the main row and of the defaults row is this same run, and the table shows its scores once. +- **Shared and fixed**: the chosen embedder; the reranker when the A3 cell has it on (same Qwen3-Reranker file and size, through RAGFlow's rerank setting); the answer model, prompt and 20,480-token budget; the checkers and judges; the corpus snapshot (FreshStack full corpus); the question IDs and halves. +- **Set for measurement in RAGFlow**: similarity cut-off 0, retrieval depth 100. Every other RAGFlow setting is its default, read from the pinned release, not from main-branch documentation (P16). The Part 2 protocol gives a settings table with three columns: shared and fixed, default kept, set for measurement. +- **Matched RAGFlow search** on half A (same selection rule, 12 to 24 cells, retrieval only) guards against a weak-baseline comparison (Armstrong et al. 2009; Lin 2019; Yang et al. 2019). Its exact cell list (the RAGFlow settings varied and their levels) is fixed and recorded before the cost measurement at FLOW step 7.2. It runs only if its projected pod cost, measured at FLOW step 7.2, is at most $50. The decision is recorded at FLOW step 7.2a, before any half-B score exists, and it fixes the estimand above. +- **Pre-freeze RAGFlow samples** (page-number gate, seed 20261130; repeatability check, seed 20261131) come from half A only. +- **FinanceBench unit**: page for every question and both systems; pre-freeze gate that RAGFlow returns page numbers; fallback to document for every question and both systems if it fails; the other unit as labelled sensitivity (`protocol.md` section 0.12 item 4). +- **C1 and C2**: one Holm family of 2 per core dataset, Legal exploratory. Each holds when it counts with the same sign on at least 2 core datasets; opposite signs are reported as a dataset-dependent effect, and it does not hold. The report states the direction. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`; number match is secondary. Both use the analysis of section 9 (GEE on question clusters for FreshStack and BioASQ, document-level sign-flip on FinanceBench). No McNemar test where documents cluster (Obuchowski 1998). C2 needs its own Part 2 correctness gate (the section 11.2 rule, labels from both systems); a failed gate leaves C2 in the family as not testable. +- **Part 2 missing outcomes**: every half-B question stays in both systems' denominators. A RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2. A short list scores missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. An unmappable chunk counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. +- **Labels**: L2 is the primary Part 2 labeller; system identity is hidden; both systems' answers are shuffled into one list (seed 20261132). The Part 2 correctness-gate sample and its bootstraps use seeds 20261133 to 20261135. +- **Budget**: Part 2 gets its own estimate and ceiling (P16). + +## 19. Release + +| Artefact | Content | Licence or rule | +| --- | --- | --- | +| Code, harness, prompts, analysis scripts, label guidelines | As run, with commit | Repository licence | +| Ranked lists | Unit IDs per setup and question, TREC run format | Per dataset licence (IDs only) | +| Answers | Text per setup, run and question | FinanceBench CC BY-NC 4.0; FreshStack CC BY-SA 4.0; Legal CC BY-NC-SA 4.0; BioASQ CC BY 2.5 with the NLM currency notice | +| Cited evidence | Unit and chunk IDs, never text | As ranked lists | +| Verdicts, classifier rows, Ragas rows, scores | Per answer and claim | Per dataset licence | +| Human labels | L1, L2 and adjudicated, with guidelines | Per dataset licence | +| Run records, environment records, compute table, deviation log | As written | Repository licence | +| RAGFlow configurations, outputs, maintainer replies | Part 2 | Part 2 protocol | +| Not released | Corpus text, PDFs, derived text; LLM-AggreFact data (CC BY-ND, evaluation only) | | + +- **Re-judge kit**: a script that rebuilds evidence text from the pinned dataset revisions and the released IDs, so faithfulness and correctness can be judged again with a newer model without redistributing corpus text. +- Archive on Zenodo with a DOI, one archive per dataset licence. Remove names and home-folder paths before release. + +## 20. Deviation policy + +After the freeze, every change is a dated, numbered deviation. The report lists each one with its reason, its time (before or after seeing the affected data) and its effect on each claim. Each is also added to `plan.json` `amendments` and uploaded to the OSF project. A run on another llama.cpp build needs a deviation first. + +## 21. Reporting + +- Decisions per row and per hypothesis, estimates with unadjusted 95% intervals, SESOI and MDE lines, and the gated rows marked "not testable". +- **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed. +- Realised chunk-length distribution per chunk-size level and dataset. +- Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit. +- **Compute table**: GPU type, pod hours per stage including every pre-freeze pod (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, an energy estimate (GPU hours times board power; Strubell et al. 2019; Lacoste et al. 2019), `resent_calls`, error rows per stage and role, and every `out_of_credit` stop. +- Returned model ID and provider per hosted call, with the count of differences from the requested values. +- AI assistance: the plan was drafted with AI agents; the paper says so. + +## 22. Limitations + +- Scope S: one model family at Q8_0 on llama.cpp, English only, no query instruction, one dataset per domain, one hosted answer model behind an ID that may change. +- The reranker's default instruction is a web-search instruction, applied to finance and legal text. +- Incomplete relevance judgements: unjudged units count as not relevant (Zobel 1998; Buckley and Voorhees 2004). FreshStack qrels came from pools that included BM25 and fusion systems, which can favour hybrid search. The BioASQ mini corpus is built from gold passages of many questions, so a passage gold for another question is unlabelled here. Most FinanceBench questions have one gold page, while figures often repeat on other pages. +- BioASQ is split by question, so some gold passages are shared across halves. +- Chunk size is a cap, not the realised length; BioASQ and Legal chunks barely change with it. +- H8 faithfulness is checker-measured, and the checker sees longer passages at bigger chunks; the differential-error rule and the length-controlled check address this only in part. +- H8b is conditional on reranker on and hybrid on at the chosen sizes. +- The answer model may have seen benchmark text in training; the closed-book runs bound this only in part. +- One index build per cell. +- L1 is an OMRS author. + +## 23. Open items before the freeze + +**OPEN (the user decides):** + +- O1. The OSF mechanism that time-stamps the A3 configuration hash (proposed: a file upload to the registered project). +- O2. The SESOI value, 0.02 nDCG@10. +- O3. Whether the optional blind relevance audit for H3 and C1 runs. +- O4. L2's name, role and relation to the project. +- O5. The ethics query to the university and its answer. +- O6. BioASQ rows 12 to 19 as sanity checks (BioASQ family 8, families 19 / 19 / 8, 46 contrasts), with H8a and H8b holding only on FinanceBench and FreshStack. This changes the earlier 2-of-3 rule for H8. The alternative: keep rows 12 to 19 in the BioASQ family (16 rows, 54 contrasts, alpha 0.05/16 for the BioASQ MDE) and keep 2-of-3 for H8a and H8b. The user confirms one before the freeze. + +**Pre-freeze tasks:** + +- P1. Download the full FreshStack LangChain corpus at a pinned revision (user approval at build time); record the revision, the row count and the tree hash. +- P2. Time one full-corpus FreshStack index build; update the estimate before relying on it. +- P3. Check whether FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's scorer; pin the scorer or record "not available". +- P4. Check, per dataset, whether the qrels list judged non-relevant units. +- P5. Fix the pilot cell list (at most the four H7 corner indexes per core dataset), project the pilot cost from the P2 timing, get the user's approval, run the pilot, and fill the MDE table (or apply the registered fallback of section 10). +- P6. Name the smoke and test-pod answer files that feed the H8 MDE rows. +- P7. Restate the checker 2-point tolerance against the MDE table. +- P8. Write the label guidelines for each set, with worked examples outside every sample. +- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16), and the length-controlled sample size (300). Seed 20261120 (audit) is used only if O3 is yes; confirm it then. +- P10. Record the tokenizer revision, answer-model host, reasoning mode, output cap and context length, and the judge, classifier and GLM-5.3 settings (the Ragas judge and classifier seeds included) in `plan.json`. +- P11. Record the query instruction, the reranker instruction text and the hybrid settings in `plan.json`. +- P12. Write `halves.json`; record its SHA-256; list every smoke and pre-freeze question ID with its half. +- P13. Record the prompt template hashes. +- P14. Set the `evidence_recall_20k` tolerance. +- P15. Check that no file still names `openai/gpt-6.1-terra` as a cross-check judge. +- P16. Part 2 and E1: own estimates and ceilings; RAGFlow defaults read from the pinned release; the settings table; the matched RAGFlow cell list (fixed before FLOW step 7.2), the cost measured at FLOW step 7.2 and the decision at FLOW step 7.2a, which also fixes the Part 2 estimand (section 18). + +Existing readiness tasks also close before the freeze: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. + +## 24. References + +- Armstrong, Moffat, Webber and Zobel 2009, CIKM. doi:10.1145/1645953.1646031. +- Buckley and Voorhees 2004, "Retrieval evaluation with incomplete information", SIGIR. doi:10.1145/1008992.1009000. +- Byrt, Bishop and Carlin 1993, "Bias, prevalence and kappa", J. Clin. Epidemiol. doi:10.1016/0895-4356(93)90018-V. +- Carterette 2012, "Multiple testing in statistical analysis of systems-based information retrieval experiments", ACM TOIS. doi:10.1145/2094072.2094076. +- Cawley and Talbot 2010, "On over-fitting in model selection and subsequent selection bias in performance evaluation", JMLR 11. https://www.jmlr.org/papers/v11/cawley10a.html. +- Dodge et al. 2019, "Show your work: improved reporting of experimental results", EMNLP. arXiv 1909.03004; doi:10.18653/v1/D19-1224. +- Feinstein and Cicchetti 1990, "High agreement but low kappa", J. Clin. Epidemiol. doi:10.1016/0895-4356(90)90158-L. +- FinanceBench, Islam et al. 2023. arXiv 2311.11944. +- FreshStack, Thakur et al. 2025. arXiv 2504.13128. +- Lacoste et al. 2019, machine learning emissions. arXiv 1910.09700. +- Lakens 2017, "Equivalence tests", Soc. Psychol. Personal. Sci. doi:10.1177/1948550617697177. +- Liang and Zeger 1986, Biometrika. doi:10.1093/biomet/73.1.13. +- Lin 2019, "The neural hype and comparisons against weak baselines", SIGIR Forum. doi:10.1145/3308774.3308781. +- Liu et al. 2024, "Lost in the middle", TACL. doi:10.1162/tacl_a_00638. +- Mancl and DeRouen 2001, "A covariance estimator for GEE with improved small-sample properties", Biometrics. doi:10.1111/j.0006-341X.2001.00126.x. +- MiniCheck, Tang, Laban and Durrett 2024. arXiv 2404.10774. +- Miller 2024, "Adding error bars to evals". arXiv 2411.00640. +- Obuchowski 1998, "On the comparison of correlated proportions for clustered data", Stat. Med. 17(13), 1495. +- OSF Prereg form. https://preregr.opens.science/articles/form_OSFprereg_v1.html. +- Qwen3 Embedding, Zhang et al. 2025. arXiv 2506.05176. +- Ragas, Es et al. 2023. arXiv 2309.15217. +- Sakai 2007, "Alternatives to Bpref", SIGIR. doi:10.1145/1277741.1277756. +- Sim and Wright 2005, "The kappa statistic in reliability studies", Physical Therapy. doi:10.1093/ptj/85.3.257. +- Smucker, Allan and Carterette 2007, "A comparison of statistical significance tests for information retrieval evaluation", CIKM. doi:10.1145/1321440.1321528. +- Strubell et al. 2019, energy and policy considerations. arXiv 1906.02243. +- Urbano, Lima and Hanjalic 2019, "Statistical significance testing in information retrieval: an empirical analysis of type I, type II and type III errors", SIGIR. doi:10.1145/3331184.3331259; arXiv 1905.11096. +- Wang et al. 2024, "Searching for best practices in retrieval-augmented generation", EMNLP. doi:10.18653/v1/2024.emnlp-main.981. +- Yang, Lu, Yang and Lin 2019, "Critically examining the 'neural hype'", SIGIR. doi:10.1145/3331184.3331340. +- Zobel 1998, "How reliable are the results of large-scale information retrieval experiments?", SIGIR. doi:10.1145/290941.291014. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index d5d57506..511cf204 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -84,6 +84,10 @@ Estimate about $650 on an RTX 4090 (after A2c; about $690 after A3, section 0.12 - **One GPU type.** Every main-run pod (index build pods, search and replicate pods) uses the chosen GPU type, and every stage runs on it. GPU types are never mixed. The earlier 4090 pre-freeze runs (smoke runs, LLM-AggreFact runs, timing in section 4.5) were preparation, not results. - **Parallel index builds.** The 9 indexes build on 9 parallel pods of the chosen GPU type, each into its own store on the RunPod network volume. Each pod seals its index and runs its 8 search setups. The `e4b-c1024` rebuild check runs beside them (amended A3, section 0.12 clarification 2: it runs after that seal and before any search into the shared output folder). Then the Mac scores the search stage and applies the selection rule, and the three replicate pods start. +### 0.9 Questions decided on 2026-11-13 + +The operator chose: (1) FreshStack corpus without the continuity documents and the overlay runbook; (2) BioASQ and Legal passages as `.md` with Qwen3 counting; (3) Ragas `AnswerCorrectness` on all four datasets with a 100-answer hand-label check, plus the FinanceBench number match; (4) one answer round, `answer.context_window` 131,072 characters; (5) one search run per setup. Still open: the 4B and 8B Qwen3-Reranker GGUF repositories, chosen by the test pod check. + ### 0.10 Pre-freeze amendment A2: methods review (2026-11-13) An independent methods review ran on 2026-11-13; the operator approved every fix. **This section wins over sections 0.3 to 0.9 where they differ.** Detail: change `experiment-35-full-document-rag-benchmark`, design D0.12; code: change `exp35-main-run-readiness`, design D35 and task group 15. @@ -137,9 +141,70 @@ Written before the freeze. No measured run has started. The operator approved th - **Scope.** Comparison with other frameworks is outside Experiment 35. Part 2 compares OMRS with RAGFlow only. - **Clarifications (PR 209 review, 2026-11-13; items 1, 2 and 4 change no Experiment 35 design, item 3 adds one answer setup).** (1) Each null of H1 to H8 is a zero effect, as A2 dropped equivalence tests: a contrast counts by the decision rule (Holm-adjusted p below 0.05, sign matches). The minimum detectable effect is for power planning and reporting only. This replaces the "Null (each)" line in section 0.4 and the null texts in `plan.json`. (2) G2 order: the `e4b-c1024` rebuild check runs after that index is sealed and before any search writes into the shared output folder. A search stage refuses to start without a passing `rebuild.json` for its dataset, so index pods wait for it. This replaces "runs beside them" in section 0.8a. (3) Token-matched control: the factorial cell `c2048` x 10 keeps its 10-chunk evidence, so the control does not change a factorial cell. A separate 15th answer setup, `c2048-t20k`, cuts the evidence at 20,480 real Qwen3 tokens (the section 0.11 N2 rule) at 2,048-token chunks, at the chosen embedder and reranker size, reranker on and hybrid on. It sits beside `c512-t20k` and `c1024-t20k`, so each chunk size has a token-matched cell, and the token-matched comparison of H8 uses the three `-t20k` setups. The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each, and G1 covers all 15. The setup is registered before the freeze. Cost: the answer stage grows by about 1/14 (about 7%). Counted from the readiness design D35 table (two token-matched setups: +36 pod hours, +$47 hosted), one more setup adds about $24 of hosted calls (answers, five Ragas metrics, judges and classifier) and about 18 pod hours (about $16 at $0.89 per hour, about 6 hours per replicate pod), so about $40 in all. The estimate moves from about $650 to about $690. The ceiling stays $975 (1.5 x the old estimate, 1.4 x the new one). At 3 times slower reranking the total is about $910, still under it. (4) FinanceBench unit in Part 2 (decided 2026-11-13): Part 2 scores FinanceBench C1 at page level for every question and for both systems, the unit of this protocol (a chunk counts for its first page). Pre-freeze gate: RAGFlow returns a page number for each chunk on a seeded FinanceBench sample. Pre-registered fallback: if the gate fails, the primary unit is the document for every question and both systems. The other unit is a labelled sensitivity analysis. It is required only when both systems provide page provenance; otherwise the report states it is unavailable and gives the count of questions affected. No question is dropped because of a system's output. The master plan, Step 3 rule 3, holds the text. -### 0.9 Questions decided on 2026-11-13 +### 0.13 Pre-freeze amendment A4: publication review fixes (2026-11-14) -The operator chose: (1) FreshStack corpus without the continuity documents and the overlay runbook; (2) BioASQ and Legal passages as `.md` with Qwen3 counting; (3) Ragas `AnswerCorrectness` on all four datasets with a 100-answer hand-label check, plus the FinanceBench number match; (4) one answer round, `answer.context_window` 131,072 characters; (5) one search run per setup. Still open: the 4B and 8B Qwen3-Reranker GGUF repositories, chosen by the test pod check. +Written before the freeze. No measured run has started. A publication-grade methods review (2026-11-13) gave 30 findings, F1 to F30; the operator approved the recommended fix for each on 2026-11-14. **From A4 on, `preregistration.md` in this folder is the governing live design of Experiment 35.** It wins over this file, `plan.json` and the OpenSpec changes wherever they differ. Sections 0.1 to 0.13 and 1 to 7 of this file are history. The proposal stays the master plan for scope, order and Part 2; pre-registration section 18 wins over it on the Part 2 points it fixes, and the frozen Part 2 protocol governs the rest after its freeze. Section 0.9 now sits after 0.8a, so section 0 reads in number order; its text is unchanged. + +| Finding | Change | Pre-registration section | +| --- | --- | --- | +| F1 | FreshStack indexes rebuilt on the full 49,514-document public corpus; the 10,000-document subset retired; alpha-nDCG@10 secondary if nugget labels allow | 4, 6, 17, 23 (P1 to P3) | +| F2 | Part 2: reranker shared, RAGFlow cut-off 0 and depth 100, three-column settings table, matched RAGFlow search only if at most $50; the step 7.2a decision fixes the estimand ("selected OMRS against selected RAGFlow" if the search runs, else "against default RAGFlow"); default RAGFlow defined as one run | 18 | +| F3 | Faithfulness gate per checker (kappa at least 0.60, lower bound at least 0.40), 450 claims by chunk size from answers to all questions, differential-error rule, length-controlled check | 11.2 | +| F4 | Numbered contrast table; Holm families fixed at 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; was 16, changed by C3 below); gated rows stay in with p = 1 | 8 | +| F5 | H8 split into H8a (crossed) and H8b (token-matched), both confirmatory | 7, 8 | +| F6 | Retrieval-only pilot on 40 half-A questions per core dataset; SESOI 0.02 nDCG@10 (open); underpowered rows named | 10 | +| F7 | MixedLM dropped; one clustered method for estimates, intervals and p-values | 9 | +| F8 | Correctness gate per core dataset, 50 items each, point estimate and lower bound; continuous score validated by AUC | 11.2 | +| F9 | Missing-outcome rules per outcome, Part 1 and Part 2 | 12, 18 | +| F10 | Decision test defined by its closed form (Mancl and DeRouen bias-reduced variance, t on n minus 1), `statsmodels` GEE as the checked implementation; FinanceBench document-level sign-flip; C2 fixed as continuous correctness, no McNemar | 9, 18 | +| F11 | Unjudged share at 10 and condensed-list sensitivity; blind audit optional (open) | 21, 11.3 | +| F12 | `evidence_recall_20k` gives any-page credit; first-page credit is sensitivity | 6 | +| F13 | Second closed-book run with the RAG prompt; H8a and H8b correctness re-run on closed-book-wrong questions | 13 | +| F14 | RQ1 headline is the A3 cell on half B; 72-setup table descriptive | 14.3 | +| F15 | One governing file; conflicts settled there (token cut truncates, zero-effect nulls, Legal exploratory in H8a and H8b, `openai/gpt-6.1-terra` cross-check removed, frozen pins) | 1, 3, 7, 8, 11.1, 16 | +| F16 | Existing data declared | 5 | +| F17 | Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop (E1 left the order, C6 below); "complete with registered drops" defined; "the operator decides" removed; Part 2 budget a pre-freeze task | 17 | +| F18 | Release table per artefact and licence, re-judge kit | 19 | +| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 and ethics open | 11.3 | +| F20 | Pins table; index cost and configuration hash defined | 14.1, 16 | +| F21 | FinanceBench split by document; pre-freeze samples from half A; order: score half A, hash, score half B | 4, 14.2 | +| F22 | "Holds" defined per hypothesis; H7 confirmatory on FinanceBench and FreshStack, as H6; H4 named as simple effects | 7 | +| F23 | Every hypothesis scoped (scope S); limitations extended | 2, 22 | +| F24 | nDCG@10, Recall@10 and MRR@10 formulas (trec_eval convention) | 6 | +| F25 | One-build-per-cell assumption stated; optional `e8b-c2048` rebuild | 15 | +| F26 | Checker 2-point tolerance restated against the MDE table | 11.1, 23 (P7) | +| F27 | Expected-maximum estimator for the search-cost figure | 14.3 | +| F28 | Compute table in the report, pre-freeze pods included | 21 | +| F29 | Deviation policy; OSF file upload time-stamps the A3 hash (open) | 20, 14.2 | +| F30 | Reference fixes | 24 | + +**Second round (C1 to C24).** A post-fix consistency review (2026-11-14) gave 24 findings, C1 to C24. The orchestrator ruled on each the same day. + +| Finding | Change | Pre-registration section | +| --- | --- | --- | +| C1 | OSF category "registration before analysis of the outcome data"; pilot scores on half A are main-run outcome data for the pilot cells | 1 | +| C2 | Pilot: at most the 4 H7 corner indexes per core dataset, not reused; stand-in SD for adjacent levels; cost projected from P2 and approved before the pod; registered fallback without a pilot | 10, 17, 23 (P5) | +| C3 | BioASQ rows 12 to 19 become sanity checks; families 19 / 19 / 8, 46 contrasts; BioASQ MDE alpha 0.05/8; open item O6 | 7, 8, 10, 23 | +| C4 | Two-sided hypotheses hold only when the same row counts with the same sign on every dataset the rule needs (H6, H8a, H8b, C1, C2) | 7, 18 | +| C5 | One gate interval method: percentile bootstrap, 10,000 resamples by question, own seed; differential-error rule on GLM-5.3, either direction | 11.2 | +| C6 | Drop order without E1 (E1 has its own ceiling); contamination check falls back to the own-knowledge run; "complete with registered drops" | 13, 17 | +| C7 | Half-A rule covers samples drawn after `halves.json` exists; earlier samples listed in section 5 | 4 | +| C8 | `statsmodels` pin marked missing: RDY task 15.6 adds the `stats` group | 16 | +| C12 | Search-cost figure: RAGFlow curve only if its matched search runs; "adapted from Dodge et al. 2019", ascending sort | 14.3 | +| C13 | Sign-flip interval assumes a constant additive shift | 9 | +| C14 | Seeds table, one row per seed; new seeds 20261121 to 20261135 wait for P9 | 16, 23 | +| C15 | Task prefixes RDY, C35 and FLOW defined; pointers name the change and task | 1, 16, 18 | +| C16 | P15 is now a check that no file names the cross-check judge | 23 | +| C17 | Clef-Flash gate failure, unclassified claims, correctness sample draw | 11.1, 11.2, 12 | +| C18 | References cited where used; Angelopoulos 2023, Barr 2013 and Kapoor 2024 deleted | 24 | +| C19 | Matched RAGFlow cell list fixed before the FLOW step 7.2 cost measurement | 18 | +| C21 | Where claims and citations come from | 6 | +| C22 | Four fixed search settings added to `plan.json` `decision_rule` | 16 | +| C24 | Why the faithfulness gate is pooled; per-dataset kappa descriptive | 11.2 | + +C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 markers, RQ3 wording, OSF deposit list). + +Cost: estimate about $730 (was about $690) plus the pilot, projected from P2 before the freeze; ceiling $975 unchanged (pre-registration section 17). ## 1. Question @@ -316,7 +381,7 @@ The smoke subset (task 4.1) runs a small slice of each dataset through every arm - **Stop rule**: the run is complete when every search setup has its one run, every answer setup and dataset has 3 runs, the closed-book run, the determinism and rebuild checks, and the Ragas pass. Stop early when the preflight fails, a gate fails, or the projected spend reaches the ceiling of section 0.10. A determinism mismatch is recorded and reported, never swapped for another run. - **What each outcome means**: - - Pass: gates hold; the report states each arm's difference or "no difference larger than the minimum detectable effect". + - Pass: gates hold; the report states each arm's difference or "no difference larger than the minimum detectable effect". (Superseded by A3 and A4: a non-significant result means no difference was detected at this sample size.) - Fail: a gate failed; investigate before any number is cited in the paper. - Unclear: an interval crosses zero; report it as not shown, not as no effect. @@ -331,10 +396,10 @@ The smoke subset (task 4.1) runs a small slice of each dataset through every arm | Embedding 0.6B | `Qwen/Qwen3-Embedding-0.6B-GGUF` | `Qwen3-Embedding-0.6B-Q8_0.gguf` | `06507c7b42688469c4e7298b0a1e16deff06caf291cf0a5b278c308249c3e439` | 11941 | as baseline | | Embedding 8B | `Qwen/Qwen3-Embedding-8B-GGUF` | `Qwen3-Embedding-8B-Q8_0.gguf` | `d20ddc71e8a5c4344f2343481e242233a997dc5eaff442427a945836c97b4deb` | 11941 | as baseline | | Claim checker, Clef-Flash | `ggml-org/Clef-Flash-GGUF` | `Clef-Flash-Q8_0.gguf` | `25f801e2a27610b0ccc0cb1558d7eda3d7a304e34e2c4a04e3079357ff7d906e` | 11941 | `--ctx-size 32768 --batch-size 32768 --ubatch-size 32768 --parallel 1` | -| Claim checker, Granite Guardian | `ibm-granite/granite-guardian-3.3-8b-GGUF` | `granite-guardian-3.3-8b-Q8_0.gguf` | `356fc80d343a699a3d12316de161d4341ee2d11f7057123dabb158804e876bc5` | 11941 | as Clef-Flash | +| Claim checker, Granite Guardian (superseded by A1: dropped) | `ibm-granite/granite-guardian-3.3-8b-GGUF` | `granite-guardian-3.3-8b-Q8_0.gguf` | `356fc80d343a699a3d12316de161d4341ee2d11f7057123dabb158804e876bc5` | 11941 | as Clef-Flash | -The ONNX reranker and the OCR engines do not run through llama.cpp. The freeze records them under the runtime manifest with name, revision and file hash. The embedding tokenizer (`Qwen/Qwen3-Embedding-4B`, `tokenizer.json`, revision `5cf2132a`) is the token-aware chunking tokenizer, cached and hash-checked by sweep task 2.4a. +(Superseded by A1: the ONNX reranker is gone; every reranker runs on llama.cpp.) The ONNX reranker and the OCR engines do not run through llama.cpp. The freeze records them under the runtime manifest with name, revision and file hash. The embedding tokenizer (`Qwen/Qwen3-Embedding-4B`, `tokenizer.json`, revision `5cf2132a`) is the token-aware chunking tokenizer, cached and hash-checked by sweep task 2.4a. -- **Hosted references** (section 3, ADR-119): answer model `deepseek/deepseek-v4.1-flash-1017` (OpenRouter, temperature 0, seed 20261101, one pinned upstream host); main Ragas judge `openai/gpt-oss-120b` (OpenRouter, reasoning `low`, temperature 0, fixed seed, one pinned upstream host); cross-check judge `openai/gpt-6.1-terra` (OpenRouter, fixed seed, 50 hand-checked items); main claim checker GLM-5.3 (Z.AI, section 4.4). A judge never scores its own model family. +- **Hosted references** (section 3, ADR-119): answer model `deepseek/deepseek-v4.1-flash-1017` (OpenRouter, temperature 0, seed 20261101, one pinned upstream host); main Ragas judge `openai/gpt-oss-120b` (OpenRouter, reasoning `low`, temperature 0, fixed seed, one pinned upstream host); cross-check judge `openai/gpt-6.1-terra` (OpenRouter, fixed seed, 50 hand-checked items; superseded by A4: removed); main claim checker GLM-5.3 (Z.AI, section 4.4). A judge never scores its own model family. - **Tolerance**: 0.005 absolute on nDCG@10, the only primary metric, and on the secondary Recall@10, MRR@10 and hit@page (A2; nDCG@10 only is primary). The largest change that still counts as the same result on another back end. Confirm or change at freeze. - **Live check**: the runner calls `verify_llamacpp_runtime` before the first score of each run and passes `runtime=found.as_run_record()` to `record_run_start`. A mismatch stops the run. diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index 6d8ee680..3ba73aae 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -63,7 +63,7 @@ See proposal.md (Why) for the motivation and specs/ for the requirements. This s **Non-Goals:** - Several GPUs or several routers on one pod. The router port and the CUDA device are not settings. -- Any change to models, checkers, prompts, metrics or the statistics plan beyond what the 2026-11-12 redesign set (Exp 35 design D0). +- Any change to models, checkers, prompts, metrics or the statistics plan beyond what the 2026-11-12 redesign set (Exp 35 design D0), amendments A2 to A4 changed, and `preregistration.md` holds (D38). - Any change under `src/omrs/` or `web/` other than the two app fixes D25 and D26 and the llama.cpp reranker backend D27 (operator decisions, 2026-11-12). - The Exp 35 summary script (Exp 35 task 4.2). This change gives it the join gate, the rebuild report and the refusal figures. - Parallel local judges. @@ -233,7 +233,7 @@ Reason: faithfulness and H-CC are meant to measure whether checked facts are bac The smoke data (5 questions per dataset, direction only) suggest that the FreshStack gap moves from +19.3 points (all claims) to +16.3 points (factual claims only). That is close to the 15-point line. The rule is chosen for what H-CC means, not for this outcome. The protocol amendment says this openly, and the all-claims figure stays in the report. -H-CC was registered on 2026-11-11, before the freeze. The change of its claim population is a dated pre-freeze amendment in `protocol.md` section 4.4 and an ADR-119 amendment note. The other H-CC rules stay. A question with no factual claim in any arm or run leaves that dataset's bootstrap, as a question with no claims did before. +H-CC was registered on 2026-11-11, before the freeze. The change of its claim population is a dated pre-freeze amendment in `protocol.md` section 4.4 and an ADR-119 amendment note. The other H-CC rules stay. A question with no factual claim in any arm or run leaves that dataset's bootstrap, as a question with no claims did before. (History: H-CC was dropped on 2026-11-12. A4 replaces this rule with the missing-outcome rules of D38: complete cases per row, plus sensitivity analyses that score no-claim answers 1 and 0.) Mechanics, in a new `refusals35.py`: @@ -375,23 +375,37 @@ These are estimates. The local judge rates come from LLM-AggreFact rows, which h | GLM-5.3 claim checks | Z.AI Coding Plan, no per-token cost | | $0 | | **Total** | | **about 341** | **about $555** (A1b total; A2 total in D35) | -Rounded estimate after A1b: **$555, ceiling $830** (history). Final estimate after A2, the oracle run and A2c: **$650, ceiling $975 (1.5 x)**; see D35 and D37. +Rounded estimate after A1b: **$555, ceiling $830** (history). Estimate after A2, the oracle run and A2c: **$650, ceiling $975 (1.5 x)**; see D35 and D37 (history). Live estimate after A4: about **$730**, ceiling **$975**; see D38. + +Additions after the A2 total of $650: A3 (to about $690) and A4 (to about $730). Estimates, not measured; pre-registration section 17: + +| Item | Basis | Pod hours | Cost | +| --- | --- | ---: | ---: | +| A3 token-matched `c2048-t20k` setup | Protocol section 0.12 item 3 | about 18 | about $40 with hosted calls | +| FreshStack full corpus (49,514 documents) | Reviewer estimate, not timed (task 16.2, P2) | about 30 to 40 | about $30 to $40 | +| Second closed-book run (RAG prompt) | By analogy with the first closed-book run | about 1 | about $3 hosted plus about $1 | +| MDE pilot, retrieval only | 40 half-A questions per core dataset (task 16.5) | a few | a few dollars | +| Length-controlled check | 300 claims, GLM-5.3 (no per-token cost) and Clef-Flash on a replicate pod | small | small | + +The compute table of the report (task 16.20) gives measured pod hours per stage, every pre-freeze pod included (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, and an energy estimate. The largest unknown is reranker speed. llama.cpp issue #30422 reports rerank mode about 6 times slower than it should be. At 3 times slower than assumed, the search stage and the answer-stage retrieval add about 257 pod hours (about $229), and the total is about $870, still under the ceiling. At 6 times slower, the total passes the ceiling, and the stop rule triggers before the freeze. The test pod session measures the real speed. -Stop rule, checked three times: +Stop rule, checked three times (A4: each projection uses billing and measured rates only, never scores): -1. **Before the freeze**, in the test pod session (D24): the same sum with the measured build hours per chunk size, reranker throughput per size, answer and Ragas seconds and hosted cost per answer. Above $975, the operator decides before the freeze. -2. **On the build pods, after the search stage**: real pod cost so far plus the projected replicate cost. Above $975, the replicate pods do not start and the operator decides. -3. **While the replicate pods run**: RunPod billing plus OpenRouter spend, read at each stage end. At $975 the next stage does not start. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling (about $450), so an overrun ends as a clean `out_of_credit` stop (exit code 5, D2). +1. **Before the freeze**, in the test pod session (D24): the same sum with the measured build hours per chunk size (and the full-corpus FreshStack build time, task 16.2), reranker throughput per size, answer and Ragas seconds and hosted cost per answer. When the projection reaches $975, the results-blind drop order of D38 applies, step by step, before the freeze. +2. **On the build pods, after the search stage**: real pod cost so far plus the projected replicate cost. When the projection reaches $975, the drop order of D38 applies before the replicate pods start. +3. **While the replicate pods run**: RunPod billing plus OpenRouter spend, read at each stage end. When the projection reaches $975, the drop order of D38 applies before the next stage starts. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling (about $450), so an overrun ends as a clean `out_of_credit` stop (exit code 5, D2). -Older text (the $250 cap): +Each drop step is recorded with the projection that triggered it. When the drop order is used up, the run stops. A run that finishes every confirmatory item after drop steps is "complete with registered drops" (each drop a numbered deviation); it is incomplete only when a confirmatory item is missing (A4 second round, D38). No confirmatory item is dropped by choice. ("The operator decides" in the earlier text of this decision is withdrawn by A4.) + +Older text (the $250 cap; history, replaced by the stop rule above and D38): The whole run has one cap: $250, for pods and hosted calls together. There are no per-pod hour caps. Task 10.2 writes the cap into `protocol.md` as Exp 35 task 1.11. The cap is checked twice: -1. **Before the freeze**, in the test pod session (D24). The extrapolated total is: the measured index-build hours (protocol section 4.5 until the build pod replaces them) times the pod price, plus three replicate pods at the measured D22 hours times the pod price, plus the measured hosted cost per answer (reported `cost_usd` of answer, judge, Ragas and classifier calls) times 19,440. Above $250, the operator decides before the freeze. +1. **Before the freeze**, in the test pod session (D24). The extrapolated total is: the measured index-build hours (protocol section 4.5 until the build pod replaces them) times the pod price, plus three replicate pods at the measured D22 hours times the pod price, plus the measured hosted cost per answer (reported `cost_usd` of answer, judge, Ragas and classifier calls) times 19,440. Above the $975 ceiling (pre-registration section 17), the freeze stops and the user decides at G7 (the $250 cap is history). 2. **After the indexes, on the build pod** (a light re-check, no calibration). The same sum, with the real build pod cost so far in place of the index estimate. Above $250, the procedure stops before the replicate pods start and asks the operator. ### D24. Pre-freeze test pod session (operator approval, 2026-11-12) @@ -510,7 +524,7 @@ Fallback, only if every 4B or 8B file fails the pod check: the existing `reranke - Legal RAG Bench loader in `datasets35.py`: `qa` and `corpus` configs, split `test`, pinned HF revision and file SHA-256. Each passage is written once as `.md` holding the title, the text and the footnotes when present; gold is `relevant_passage_id`. Questions keep the frozen seeded order. - BioASQ and Legal passages as `.md` (operator decision, 2026-11-13): `datasets35.py` writes `.md` in place of `.txt` (today `datasets35.py:293-296`). The passages hold no headings, so the Markdown path makes one section per passage and splits it with the pinned Qwen3 tokenizer through `markdown_chunk_size`; the only change from `.txt` is the token counter (Qwen3 in place of the LlamaIndex default cl100k). The writer escapes a `#` at the start of any line so no text reads as a heading. The gold mapping strips the suffix, so gold ids do not change. -- FreshStack (operator decision, 2026-11-13): the loader drops the 20 Experiment 9 continuity queries (203 remain) and the corpus drops the 15 continuity documents and the overlay runbook. The harness references that add them leave `datasets35.py`: the `_FS_OVERLAY` and `_FS_OVERLAY_ID` constants and their SHA-256 entry (`:42-43`, `:62`), the overlay id mapping (`:221`), and the `continuity/*.md` glob and overlay append (`:262-263`). A new corpus tree hash is recorded at the freeze. +- FreshStack (operator decision, 2026-11-13): the loader drops the 20 Experiment 9 continuity queries (203 remain) and the corpus drops the 15 continuity documents and the overlay runbook. The harness references that add them leave `datasets35.py`: the `_FS_OVERLAY` and `_FS_OVERLAY_ID` constants and their SHA-256 entry (`:42-43`, `:62`), the overlay id mapping (`:221`), and the `continuity/*.md` glob and overlay append (`:262-263`). A new corpus tree hash is recorded at the freeze. **A4 (2026-11-14, D38):** the corpus is the full public `freshstack/corpus-oct-2024` config `langchain` (49,514 documents) at a pinned revision; the 10,000-document subset of Experiments 9a and 22 is retired. The loader checks the revision and the row count. - Token-count check: one test ingests a small fixture of each of the four datasets at each chunk size and asserts that every chunk's Qwen3 token count is at most the level, that the splitter path is the model-token-aware Markdown path, and that the cl100k counter is never called. - FinanceBench number match in `scoring35.py` (or a new `numbers35.py` if `scoring35.py` nears 500 lines): extract numbers with their scale words and signs, normalise thousands, millions, billions, percent, currency and parentheses, and pass when every reference number matches after rounding to its decimal places or within 1% relative difference. @@ -527,7 +541,7 @@ At 2,048 tokens x `top_k` 10, the evidence is about 20,000 tokens (70,000 to 80, ### D33. Answer correctness and its hand-label check (operator decision, 2026-11-13) - The Ragas stage (D6) adds a fifth metric: `AnswerCorrectness` from `ragas.metrics.collections`, Ragas 0.4.3, called as `ascore(user_input=question, response=answer, reference=reference_answer)`. It needs the LLM (the frozen judge `openai/gpt-oss-120b`) and embeddings (the local 4B embedder, as for `AnswerRelevancy`). Weights stay at the library default (factual 0.75, similarity 0.25) and are written into the protocol. References come from each dataset's `answer` field (FreshStack: its reference answer). A question without a reference is recorded as not judged. -- Hand-label check before the metric counts: `correct35.py sample` draws 100 answers with seed 20261113 from the main run's half B (A2, D35; was the test pod session's answers), 25 per dataset, spread evenly over the three chunk sizes. A labelling page (as in task 8.4) shows the question, the reference and the answer, with no Ragas score, and records correct or not correct. `correct35.py agree` binarises the Ragas score at 0.5 (fixed before labelling) and gives Cohen's kappa (`cohen_kappa`, task 8.8) per dataset and pooled. At pooled kappa of at least 0.60, correctness is an H8 outcome; below, it is exploratory. +- Hand-label check before the metric counts (revised by A4, D38; pre-registration section 11.2): `correct35.py sample` draws 50 half-B questions per dataset with seed 20261113, 200 in all, each with the run-1 answer of an answer setup the seed picks uniformly at random. A labelling page (as in task 8.4) shows the question, the reference and the answer, with no Ragas score, and records correct, not correct or "can't tell". `correct35.py agree` cuts the Ragas score at 0.5 (fixed before labelling) and gives Cohen's kappa (`cohen_kappa`, task 8.8) per core dataset with a percentile bootstrap 95% interval (10,000 resamples by question, own seed), once with "can't tell" as correct and once as not correct. The gate passes on a core dataset when both give a point estimate of at least 0.60 and a lower bound of at least 0.40, and the continuous score's AUC against the human label has a lower 95% bound above 0.5. A failed gate leaves rows 14, 15, 18 and 19 in that dataset's Holm family at p = 1, marked "not testable". Legal's 50 labels are reported with no gate. No pooled kappa decides anything. (The earlier rule, 100 answers and a pooled kappa of 0.60, is withdrawn.) - Cost: about 3 more calls per answer without the contexts, about $0.0004 and 23 s per answer (estimate; measured in the test pod). Ragas goes from $83 to about $100, and the replicate pods from about 59 h to about 65 h each. - The FinanceBench number match (D29) stays as a deterministic second check. @@ -538,7 +552,7 @@ Each of the 72 search setups runs once (run 1). Query-time retrieval on a sealed ### D31. Selection rule and factorial analysis (redesign, 2026-11-12) - `select35.py` reads the saved search scores (Mac only, saved files, half A only) and applies the pre-registered rule of protocol section 0.12 (amendment A3): pooled score = plain mean of the per-dataset mean nDCG@10 over the three core datasets (FinanceBench, FreshStack, BioASQ); Legal RAG Bench takes no part in selection. Smallest embedder within 0.01 of the best (inclusive), then smallest reranker size within 0.01 at that embedder; equal scores pick the smaller size. A second step, at the chosen embedder and reranker size, picks chunk size, reranker on or off and hybrid on or off from the 12 cells by the same pooled score; cells within 0.01 of the best (inclusive) tie, and ties go to reranker off, then hybrid off, then the chunk size with the lowest index cost, then the smaller chunk size. It writes `selection.json` with every pooled score, the chosen sizes and the chosen cell. Tests cover a gap of exactly 0.01, a tie, a missing setup (refused), the four-factor tie order, and Legal scores that change neither choice. -- `factorial35.py` builds per-question marginal scores and runs the contrasts of H1 to H8 with the existing `paired_permutation`, `paired_bootstrap`, `holm`, `wilson`, `mcnemar_exact` and `min_detectable_effect`. It adds no statistics function. Tests plant a known effect in a fixture and check that only that contrast counts, that Holm runs within one dataset, and that a wrong-sign result does not count. +- `factorial35.py` builds per-question contrast scores for rows 1 to 19 of pre-registration section 8 (H1 to H8b) and gets every decision p-value from the clustered method of D38 (`stats_cluster.py`), then `holm` within each fixed family; `paired_permutation` is used only for the registered sensitivity check, and `wilson` and `paired_bootstrap` only for descriptive intervals. `mcnemar_exact` is not used (A4: no McNemar test where documents cluster). Tests plant a known effect in a fixture and check that only that row counts, that Holm runs within one dataset over the fixed family, and that a wrong-sign result does not count. ### D32. Parallel index builds and search (operator instruction, 2026-11-12) @@ -549,7 +563,7 @@ The 9 indexes (3 embedders x 3 chunk sizes) are independent, so they build on 9 - The rebuild check (D20) runs as soon as the shared `e4b-c1024` index is sealed: on a tenth pod, or on the `e4b-c1024` pod after its seal and before its search, into a separate store. Its two searches use `--index-from` and separate output folders, so the D20 guard does not block them. No pod runs its 8 canonical searches into the shared output folder until the dataset's `rebuild.json` passes (D20 guard); a pod that seals earlier waits for it. - The GGUF files sit on the network volume once, so each pod loads them without a new download. - Wall time on a 4090: about 4 h for the slowest build (8B, all four corpora) plus up to about 14 h of search on a 2,048-token index (three reranker sizes, both hybrid levels): about 18 h for the whole build and search phase, against about 133 h on one pod. Pod hours stay about the same, plus about 0.5 h of start-up per pod (D23). -- Then on the Mac: score the search stage and apply the selection rule (D31). Then the three replicate pods (about 65 h each, D22). End to end, about 4 days of pod wall time. +- Then on the Mac (A4 order of work, D38): score half A only, apply the selection rule (D31) and write `selection.json`, time-stamp the configuration hash on OSF, and only then score half B. Then the three replicate pods (about 65 h each, D22). End to end, about 4 days of pod wall time. ### D35. Methods-review fixes in the harness (pre-freeze amendment A2, 2026-11-13) @@ -557,12 +571,12 @@ Decided by the operator on 2026-11-13 (A2c, D37, revises parts of this section): The operator approved every fix of an independent methods review (Exp 35 change design D0.12). Harness consequences: -- **Unit scorer (B1).** `scoring35.ranked_units` changes from "cut 10 chunks, then map" to "map, keep first occurrence of each unit, then cut 10 units" (`scoring35.py:39-51`). Search stages save the top 100 chunks: `top_k` 100 and `fetch_k` 100 for the search call (A2c N1: without `fetch_k`, `policy.py:63-67` would size the rerank pool at 300), and the reranker returns the whole 100-candidate pool in its order. FinanceBench page credit is the chunk's first page; the "any page" credit (today's `scoring35.py:49-50`) is kept only as a sensitivity column. New secondary metric `evidence_recall_20k`: share of gold units in the chunks taken in rank order until their real Qwen3 token total reaches 20,480 (A2c N2; the crossing chunk is truncated at the limit and the truncation is recorded). The answer stage still sends 10 chunks in the 12 main setups; the token-matched setups send chunks up to the same 20,480-token budget, taken from the same saved list. +- **Unit scorer (B1).** `scoring35.ranked_units` changes from "cut 10 chunks, then map" to "map, keep first occurrence of each unit, then cut 10 units" (`scoring35.py:39-51`). Search stages save the top 100 chunks: `top_k` 100 and `fetch_k` 100 for the search call (A2c N1: without `fetch_k`, `policy.py:63-67` would size the rerank pool at 300), and the reranker returns the whole 100-candidate pool in its order. FinanceBench page credit is the chunk's first page; the "any page" credit (today's `scoring35.py:49-50`) is kept only as a sensitivity column. New metric `evidence_recall_20k`: share of gold units in the chunks taken in rank order until their real Qwen3 token total reaches 20,480 (A2c N2; the crossing chunk is truncated at the limit and the truncation is recorded). It is the confirmatory H6 and H7 endpoint on FinanceBench (A2c N3). **A4 (D38):** on FinanceBench it uses any-page credit (a whole chunk covers every gold page it overlaps; the truncated chunk covers only its first page); first-page credit is a registered sensitivity analysis. The answer stage still sends 10 chunks in the 12 main setups; the token-matched setups send chunks up to the same 20,480-token budget, taken from the same saved list. - **Setups (M5).** Three token-matched answer setups, `c512-t20k`, `c1024-t20k` and (added by A3, 2026-11-13) `c2048-t20k`, at the chosen embedder and reranker size, reranker on and hybrid on; the setup name gains `-t20k` when evidence is cut by the token budget instead of 10 chunks. 15 answer setups in all (14 before A3). -- **Closed-book run (M6).** A `closedbook` stage: one run, all 1,160 questions, the answer model with no evidence (a fixed prompt that asks for an answer from the model's own knowledge), then `AnswerCorrectness` and the number match. It runs on replicate pod 1. -- **Split halves (M3).** `split35.py` writes `halves.json` per dataset (seed 20261115). `select35.py` reads half A only. The report reads half B for answer-stage headlines. -- **Mixed model (M2; p-values revised by A2c N4, D37).** `experiments/_lib/stats_mixed.py` wraps `statsmodels` `MixedLM` for estimates, and `GEE` with cluster-robust errors for contrast p-values; (formula `ndcg10 ~ C(embedder)*C(chunk)*C(reranker)*C(hybrid)`, random intercept per question, REML) and returns planned contrasts with standard errors. It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock`. The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. -- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling). +- **Closed-book run (M6).** A `closedbook` stage: one run, all 1,160 questions, the answer model with no evidence (a fixed prompt that asks for an answer from the model's own knowledge), then `AnswerCorrectness` and the number match. It runs on replicate pod 1. **A4 (D38):** a second closed-book run uses the main RAG answer prompt with an empty evidence block, and the contamination sensitivity re-runs rows 14, 15, 18 and 19 on the questions both closed-book runs answer wrongly. +- **Split halves (M3).** `split35.py` writes `halves.json` per dataset (seed 20261115). `select35.py` reads half A only. The report reads half B for answer-stage headlines. **A4 (D38):** FinanceBench is split by document, so no document sits in both halves; the other datasets are split by question. +- **Clustered analysis (M2, A2c N4; replaced by A4, D38).** No MixedLM. For FreshStack, BioASQ and Legal the decision test is defined by its closed form (pre-registration section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster; the paired t-test variance times n / (n minus 1)); t reference on n minus 1 degrees of freedom. `experiments/_lib/stats_cluster.py` implements it with `statsmodels` `GEE` on per-question contrast scores (intercept only, Gaussian, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) and computes the p-value and interval from the t reference itself, because GEE results report a normal reference by default (`use_t=False` in the statsmodels source). The known-answer test of task 15.6 checks the GEE estimate and variance against the closed form; on a mismatch the closed form is used and the difference reported. The same module runs the document-level sign-flip permutation test (100,000 draws, seed 20261101) for FinanceBench; one method gives every estimate, interval and p-value (D38). It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock`. The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. +- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750, L2 225. - **Provenance (M7).** Each hosted call row records `model_returned` and `provider_returned` from the reply (OpenRouter `model` and `provider`; Z.AI `model`), beside the requested values. The join counts differences per role. - **Reranker check (M8).** `rerank_check35.py` scores 200 seeded pairs (seed 20261117) per size with llama-server and with the transformers recipe, and reports Spearman correlation; below 0.95 the file is not pinned. @@ -573,12 +587,12 @@ The operator approved every fix of an independent methods review (Exp 35 change | Depth 100 retrieval with `fetch_k` 100 passed, so the rerank pool stays 100 (without it, 300 and three times the rerank hours); dense and hybrid top 100 cost seconds | about 0 | $0 | | Two token-matched answer setups (6,960 answers at 2,048-level context: answers, five Ragas metrics, classifier; judges and Ragas on the replicate pods) | +36 (about +12 per replicate pod) | +$47 | | Closed-book run (1,160 short answers plus correctness) | +1 | +$3 | -| Human labels (400 L1, 120 L2) | 0 | $0 | +| Human labels (400 L1, 120 L2; A4 raises them to 750 L1 and 225 L2, D38) | 0 | $0 | | **A2 total** | **about 378 pod hours, about $350 with the volume** | **about $290** | | Oracle run (D36) and A2c (no cost) | +4 | +$8 | | **Final total** | **about 382 pod hours, about $354 with the volume** | **about $298** | -Final estimate after the oracle run and A2c: about **$650**, ceiling **$975** (1.5 x); see D37. At 3 times slower reranking than assumed, about $870. Replicate pods run about 77 h each. +Estimate after the oracle run and A2c: about **$650**, ceiling **$975** (1.5 x); see D37 (history; live estimate about $730 after A4, D38). At 3 times slower reranking than assumed, about $870. Replicate pods run about 77 h each. **A3 addition, `c2048-t20k` (PR 209 review, 2026-11-13).** The factorial cell `c2048` x 10 keeps its 10-chunk evidence. A separate setup, `c2048-t20k`, takes the token cut of D37 N2, so each chunk size has a token-matched cell and the token-matched comparison of H8 uses the three `-t20k` setups. The setup is registered before the freeze. Cost: the answer stage grows by about 1/14 (about 7%). Counted from the readiness design D35 table (two token-matched setups: +36 pod hours, +$47 hosted), one more setup adds about $24 of hosted calls (answers, five Ragas metrics, judges and classifier) and about 18 pod hours (about $16 at $0.89 per hour, about 6 hours per replicate pod), so about $40 in all. The estimate moves from about $650 to about $690. The ceiling stays $975 (1.5 x the old estimate, 1.4 x the new one). At 3 times slower reranking the total is about $910, still under it. The replicate pods then run about 83 h each. @@ -593,12 +607,31 @@ Final estimate after the oracle run and A2c: about **$650**, ceiling **$975** (1 - **N1. Rerank pool stays 100.** With `top_k` 100 for depth-100 scoring, OMRS would size the rerank pool as max(`rerank_max_fetch`, `top_k` x `rerank_fetch_multiplier`) = max(100, 300) = 300 (`src/omrs/core/retrieval/policy.py:63-67`, `settings.py:61-62`). The harness passes `fetch_k=100` to `Engine.search` (it reaches the existing `fetch_k_override`, `pipeline.py:279-284`), so every reranker setup scores a pool of exactly 100 and returns all 100 in order. Dense and hybrid setups also fetch 100. - **N2. Equal evidence by real tokens.** The equal-evidence metric `evidence_recall_20k` and the equal-text answer setups no longer cut by chunk count. They take chunks in rank order and add each chunk's real Qwen3 token count to a running total; a chunk is included when the total before it is below 20,480; the chunk that crosses the limit is truncated at the limit (20,480 is a hard limit, and the truncation is recorded per question), and no later chunk is. Short documents and short last chunks therefore no longer under-fill the budget. The setup names become `c512-t20k` and `c1024-t20k`. (Amended by A3: the factorial cell `c2048` x 10 keeps its 10-chunk evidence, and a separate setup `c2048-t20k` takes the token cut.) - **N3. H6 endpoint on FinanceBench.** First-page credit penalises big chunks. H6 on FinanceBench is tested on `evidence_recall_20k` (gold pages within the 20,480-token evidence), its confirmatory endpoint for H6. nDCG@10 with first-page credit and with any-page credit are sensitivity results. H6 counts on FinanceBench only if the 20k measure supports it, and the report gives whether the two page rules agree in sign per contrast. FreshStack keeps unit-level nDCG@10 for H6. -- **N4. p-values from cluster-robust errors.** The mixed model with a random intercept per question stays for estimates and description. Each planned contrast's p-value comes from a GEE with the same mean model (`statsmodels` `GEE`, Gaussian, exchangeable working correlation) and cluster-robust standard errors clustered by question; on FinanceBench clustered by document. Holm runs across the contrasts of each dataset. The paired permutation test, or on FinanceBench the document-grouped bootstrap, is the pre-registered sensitivity check. +- **N4. p-values from cluster-robust errors.** (History: replaced by A4, D38. There is no mixed model; the GEE uses independence working correlation on per-question contrast scores with bias-reduced covariance; FinanceBench uses the document-level sign-flip test; the sensitivity checks are those of D38.) The mixed model with a random intercept per question stays for estimates and description. Each planned contrast's p-value comes from a GEE with the same mean model (`statsmodels` `GEE`, Gaussian, exchangeable working correlation) and cluster-robust standard errors clustered by question; on FinanceBench clustered by document. Holm runs across the contrasts of each dataset. The paired permutation test, or on FinanceBench the document-grouped bootstrap, is the pre-registered sensitivity check. - **N5. H8 on all questions.** The selection rule averages over chunk size, so choosing on half A barely biases the chunk-size contrasts. H8 is tested on all questions. Descriptive headline answer numbers stay on half B. - **Minor.** Only nDCG@10 is primary (the tolerance line names the others as secondary). FreshStack's reference answer is `answer_text` of `freshstack/queries-oct-2024`, config `langchain` (the accepted Stack Overflow answer). Oracle evidence is capped at 20,480 Qwen3 tokens for every dataset (FinanceBench, BioASQ and FreshStack; a Legal passage is at most 512), with truncation recorded. The one-round test covers the oracle and closed-book stages too. Floor (closed-book) and ceiling (oracle) are single runs, while main setups average 3 runs; the report says so. The report adds a per-question cross-table on retrieval misses: closed-book correct or not against RAG correct or not. Oracle retrieval is now part of the plan (an oracle run, approved 2026-11-13). -- **Cost.** N1 to N5 add no pod hours or calls (the pool stays 100). Final estimate: about **$650** (pods and network volume about $354 for about 382 pod hours, hosted calls about $298), ceiling **$975** (1.5 x). +- **Cost.** N1 to N5 add no pod hours or calls (the pool stays 100). Estimate then (history; about $730 after A4, D38): about **$650** (pods and network volume about $354 for about 382 pod hours, hosted calls about $298), ceiling **$975** (1.5 x). + +Harness consequences: `run_arms.py` passes `fetch_k=100` on every search and answer retrieval; `scoring35.py` gains a token-budget cut (`evidence_by_tokens(chunks, budget=20480)`, Qwen3 counts from the pinned tokenizer); `answers35.py` builds `c512-t20k`, `c1024-t20k` and `c2048-t20k` with the same cut, and leaves the `c2048` x 10 cell on 10 chunks; `stats_cluster.py` (A4, D38; was `stats_mixed.py`) returns the clustered estimates, intervals and p-values; `factorial35.py` uses `evidence_recall_20k` for H6 and H7 on FinanceBench and all questions for H8a and H8b; `oracle35.py` caps every oracle context at 20,480 tokens. + +### D38. Publication-review fixes; `preregistration.md` governs (pre-freeze amendment A4, 2026-11-14) + +Written before the freeze; no measured run has started. A publication-grade methods review (2026-11-13) gave 30 findings, F1 to F30. The user approved every recommended fix on 2026-11-14 (amendment A4, `protocol.md` section 0.13). The file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` now governs Experiment 35 and is the main file of the OSF deposit. Where this change and that file disagree, the file wins. Section numbers below ("PR 9") are sections of that file. Earlier text in D23, D31, D33, D35 and D37 that A4 changes is marked as history in place. This change builds the design below. + +- **Deviation policy (F15, F29; PR 20).** After the freeze, every change is a dated, numbered deviation. The report gives its reason, its time (before or after seeing the affected data) and its effect on each claim. Each goes into `plan.json` `amendments` and is uploaded to the OSF project. +- **FreshStack full corpus (F1; PR 4).** All 9 FreshStack indexes are built on the full public LangChain corpus (49,514 documents) at a pinned revision (P1). The download needs the user's approval at build time. The 10,000-document subset of Experiments 9a and 22 is retired for Experiment 35, and the continuity queries, continuity documents and overlay runbook are not used. No result is compared with the FreshStack leaderboard unless the corpus and the metric are the same. alpha-nDCG@10 is a secondary FreshStack metric if the nugget labels allow it (P3). One full-corpus build is timed before the estimate is relied on (P2). +- **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3, O6: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. +- **Analysis (F7, F10; PR 9).** No MixedLM. The per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells; the estimate is its plain mean. FreshStack, BioASQ and Legal: the closed-form test of PR 9 point 3 (mean of d_qk; Mancl and DeRouen bias-reduced variance, sum of squared residuals x (n / (n minus 1)) squared / n squared; t on n minus 1 degrees of freedom), implemented with `statsmodels` GEE on an intercept (independence working correlation, question clusters, `cov_type="bias_reduced"`) and checked against the closed form by the known-answer test of task 15.6. FinanceBench: document-level sign-flip permutation test, 100,000 draws, seed 20261101, interval by inverting the test. Registered sensitivity checks, never decisive: a paired permutation test by question (10,000 shuffles) on FreshStack and BioASQ, and the GEE with document clusters on FinanceBench; disagreements go in one table. No McNemar test and no equivalence test. The type I error simulation test now covers this method (task 15.6). +- **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). +- **Missing outcomes (F9; PR 12).** Every question stays in its dataset's denominator, and every exclusion is counted per setup and row. A short list scores missing ranks as not relevant. A failed answer call follows the resend rule (D3); an answer still missing at run close scores 0 for correctness and number match and has no factual claim. Faithfulness uses complete cases per row, with two sensitivity analyses (no-claim answers scored 1, and 0). A row still unjudged at run close is counted and scored 0 in a sensitivity analysis. +- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 225, with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. +- **Closed-book runs (F13; PR 13).** Two single runs: own-knowledge prompt (floor) and the RAG prompt with an empty evidence block. Contamination sensitivity: rows 14, 15, 18 and 19 re-run on the questions both runs answer wrongly. +- **Halves, order of work and selection (F14, F20, F21, F27; PR 4, 14).** FinanceBench is split by document. Every pre-freeze sample drawn after `halves.json` exists (the pilot, the RAGFlow checks) comes from half A; earlier samples are listed with their IDs and halves (PR 5). Order: build and seal the 9 indexes and pass G2; run the 72 search setups on all questions; score half A only; apply the selection rule and write `selection.json`; time-stamp the configuration hash on OSF (O1); only then score half B and start the replicate pods. The harness enforces it (task 16.17). "Sealed" applies to indexes only. Index cost = chunks times embedding dimension, summed over the three core datasets. Configuration hash = SHA-256 of the canonical JSON of the selected cell's resolved settings, with the `halves.json` SHA-256 as one field. RQ1 retrieval headline = the A3 cell on half B; the 72-setup table is descriptive. The search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, without replacement, ascending sort). +- **Pilot and MDE (F6; PR 10).** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118) on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. SESOI 0.02 nDCG@10 (O2). Experiment 22 values are not used. +- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. +- **Pins, release, determinism, existing data (F16, F18, F20, F25; PR 5, 15, 16, 19).** One pins table (P10 to P14). A release table per artefact and licence, and a re-judge kit. One build per cell is assumed; an optional `e8b-c2048` rebuild check on FinanceBench runs only when the cost projection after the search stage is at or below the estimate, and is reported, not gated. The existing-data table lists every earlier look; P12 lists each smoke and pre-freeze question ID with its half. -Harness consequences: `run_arms.py` passes `fetch_k=100` on every search and answer retrieval; `scoring35.py` gains a token-budget cut (`evidence_by_tokens(chunks, budget=20480)`, Qwen3 counts from the pinned tokenizer); `answers35.py` builds `c512-t20k`, `c1024-t20k` and `c2048-t20k` with the same cut, and leaves the `c2048` x 10 cell on 10 chunks; `stats_mixed.py` returns MixedLM estimates and GEE contrast p-values; `factorial35.py` uses `evidence_recall_20k` for H6 on FinanceBench and all questions for H8; `oracle35.py` caps every oracle context at 20,480 tokens. +Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O6 are group 16. ## Risks / Trade-offs @@ -630,10 +663,10 @@ Harness consequences: `run_arms.py` passes `fetch_k=100` on every search and ans 1. Build and test the code (tasks 1 to 8) on the feature branch. The default behaviour stays serial-equivalent: concurrency 1 gives today's files. 2. Run the classifier gate (tasks 8.1 to 8.7). The classifier calls run on a 4090 pod; labelling and scoring run on the Mac. 3. Resolve and record the image digest (task 9.1). -4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. +4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the user's decisions on O1 to O6. 5. Write the procedure, the measured values, the GPU choice, the G2 stop rule and the budget with its ceiling into `protocol.md` (group 10). -6. At G7, freeze `protocol.md` and `plan.json` with the measured values. -7. Index pods (D32, 9 in parallel on the chosen GPU type): each builds one index for all four datasets and seals it. The `e4b-c1024` rebuild check (D20) runs after that seal, and each pod then runs its 8 search setups once `rebuild.json` passes. A failed check stops here: find the cause, fix, build again. Then the FreshStack recheck on the real indexes with zero errors in every answer setup, scoring and the selection rule on the Mac (D31), and the budget re-check (D23). Above $975, stop and ask the operator. +6. At G7, freeze `preregistration.md` (the governing file, D38), `protocol.md` and `plan.json` with the measured values, after every group 16 task is done. +7. Index pods (D32, 9 in parallel on the chosen GPU type): each builds one index for all four datasets and seals it. The `e4b-c1024` rebuild check (D20) runs after that seal, and each pod then runs its 8 search setups once `rebuild.json` passes. A failed check stops here: find the cause, fix, build again. Then the FreshStack recheck on the real indexes with zero errors in every answer setup; on the Mac, half-A scoring, the selection rule (D31) and `selection.json`; the OSF time-stamp of the configuration hash; then half-B scoring (D38); and the budget re-check (D23). When the projection reaches $975, apply the drop order of D38. 8. Replicate pods: run 1, run 2 and run 3 of the 15 answer setups at the same time. Each pod runs answer (with its own retrieval), the two judges (`clef_flash`, `chat`), `ragas` and `claimkinds` on its own run. 9. Copy back, then on the Mac run only steps on saved files: `join35.py`, determinism, the refusal report and statistics. diff --git a/openspec/changes/exp35-main-run-readiness/proposal.md b/openspec/changes/exp35-main-run-readiness/proposal.md index 690d1fa2..8051320f 100644 --- a/openspec/changes/exp35-main-run-readiness/proposal.md +++ b/openspec/changes/exp35-main-run-readiness/proposal.md @@ -8,7 +8,7 @@ On 2026-11-12 the operator agreed a full-factorial redesign of Experiment 35 (pr - Search stage: embedder (Qwen3-Embedding 0.6B, 4B, 8B) x chunk size (512, 1,024, 2,048 tokens, 15% overlap) x reranker (off, Qwen3-Reranker 0.6B, 4B, 8B) x hybrid (off, BM25 on). 9 indexes and 72 search setups per dataset, run once each. Setups are named by their settings. - Answer stage: 12 setups (chunk size x reranker on or off x hybrid on or off) at the embedder and reranker sizes chosen by a pre-registered rule, 3 runs each: 41,760 answers. - Claim checkers GLM-5.3 (main) and Clef-Flash (second). Granite Guardian and hypothesis H-CC are dropped. The refusal rule and its classifier gate stay. -- The $250 cap is replaced by an estimate of about $650 and a stop rule at a $975 ceiling. +- The $250 cap is replaced by an estimate (about $650 then, about $730 after A4) and a stop rule at a $975 ceiling. Older text in this change that names six arms, `documents-profile-4b`, `chunking-off`, Granite Guardian, H-CC or the $250 cap was rewritten on 2026-11-12. The earlier plan stays readable in git history and in the dated design notes. @@ -26,11 +26,12 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - App fix B (design D26): a native full-text query only reads the store. The ingest builds or refreshes the full-text index at its end when the resolved sparse backend is `native`. A query on a collection with no index falls back to BM25 with a warning that names the ingest remedy. - App addition C (design D27): a llama.cpp reranker backend. One new file `src/omrs/core/retrieval/reranker_llamacpp.py` sends query-document pairs to the local router's `/v1/rerank` endpoint for a model whose preset sets `pooling: rank`. One `register()` line, one backend map entry and the accepted value `llamacpp` for `retrieval.rerank_backend`. Scores are normalised to (0, 1) like the other backends. - Harness setups (design D28): the six arms become 72 search setups and 12 crossed answer setups (15 answer setups with the token-matched ones of A2 and A3) named by their settings (`e-c-r-h`). Every process writes `chunking.chunk_size`, `chunking.markdown_chunk_size` and `chunking.chunk_overlap` for its chunk-size level, `retrieval.rerank_backend: llamacpp` and the reranker model, and `retrieval.hybrid_sparse_backend: bm25` for hybrid-on setups. The search stage needs the embedder and the reranker loaded together on the router. -- Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set, and a FinanceBench number-match check. -- Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against 100 operator labels by Cohen's kappa (design D33); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget: about $650, ceiling $975. -- Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; one `statsmodels` mixed model per dataset (new experiment-only dependency, approved); two labellers with 400 L1 items and a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. Budget: about $650, ceiling $975. +- Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set on the full public corpus (A4), and a FinanceBench number-match check. +- Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against human labels by Cohen's kappa (design D33; A4 makes it a gate per core dataset on 50 labels each, D38); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget then: about $650, ceiling $975. +- Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; `statsmodels` as an experiment-only dependency (approved); two labellers with a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. (The A2 mixed model, the 400 L1 items and the $650 estimate are replaced by A4 below.) +- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3: BioASQ rows 9 to 19 are sanity checks, open item O6) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O1 to O6 are tasks. - Answer context budget (design D30): a test proves that the largest evidence set of every answer setup synthesises without the earlier context overflow, and each answer row records its rounds, evidence characters and truncation. -- Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8 with the existing statistics functions. +- Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8b with the existing statistics functions. - The harness reads the `rate_limited` record instead of parsing error text, and stops at once on `out_of_credit`. Every hybrid-on setup pins `retrieval.hybrid_sparse_backend: bm25`, as the protocol defines it. - The existing `hybrid-retrieval` wording on unindexed rows is corrected to match the contract tests. The seal step creates no full-text index; its fingerprint stays as a check that no replicate stage writes the shared store. - Add a per-provider concurrency limit for hosted calls: answers (OpenRouter), the `chat` claim judge (GLM-5.3 on Z.AI), the Ragas judge and the claim classifier (OpenRouter). Each limit is a command-line setting with a frozen default. The run record holds the values used. The local judge (`clef_flash`) stays at one call at a time. @@ -49,7 +50,7 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - Gate the classifier before the freeze: operator hand labels on a seeded sample of about 100 smoke claims from the three smoke datasets, with pass thresholds fixed before labelling (approved: refusal-against-not agreement of at least 0.90 and refusal recall of at least 0.90, weighted, seed 20261112). Add a small labelling page for the operator. - Add one tested `cohen_kappa` function to the shared statistics code (approved 2026-11-12). It gives the GLM-5.3 against Clef-Flash kappa. - Run one pre-freeze test pod session on the standard 4090 pod with the pinned image, after the code is merged and tested (about 6 h, about $5 of pod time plus a few dollars of hosted calls). It measures the concurrency values and the costs; Qwen3-Reranker throughput at each size; the VRAM of the 8B embedder and the 8B reranker loaded together; and the index build time per chunk size. It checks each reranker GGUF file, runs the first budget check, rechecks FreshStack questions `77536364` and `77762811` with zero errors, and runs every new stage end to end once on its own small indexes. The frozen protocol holds the measured values. The operator gives the go after a price quote, and the pod is deleted when done. -- Write the run procedure, the measured values and the claim-kind rule into `protocol.md`. Write the budget into `protocol.md` (section 0.8, replacing Exp 35 task 1.11): an estimate of about $650 and a ceiling of $975 for pods and hosted calls together, checked before the freeze, on the build pod after the search stage, and while the replicate pods run. +- Write the run procedure, the measured values and the claim-kind rule into `protocol.md`. Write the budget into `protocol.md` (section 0.8, replacing Exp 35 task 1.11): an estimate of about $650 (A4: about $730) and a ceiling of $975 for pods and hosted calls together, checked before the freeze, on the build pod after the search stage, and while the replicate pods run. Why factual claims only: faithfulness is meant to measure whether checked facts are backed by the sources. A refusal is a statement about retrieval, not a fact from a source. The checkers also grade refusals very differently, so with refusals in, faithfulness would partly measure refusal grading. @@ -60,11 +61,11 @@ Evidence (smoke data, 5 questions per dataset, direction only): refusal claims a ### New Capabilities - `benchmark-replicate-runs`: one environment for every stage, building the indexes once, checking a rebuild of the reference index, running the repetitions of a benchmark on separate machines against one shared read-only index, and joining the runs before determinism and statistics. -- `benchmark-answer-scoring`: answer-level scoring of a benchmark run: parallel answers with serial retrieval, the Ragas stage, the claim-kind classifier with its gate and its refusal reporting, and the answer context record. +- `benchmark-answer-scoring`: answer-level scoring of a benchmark run: parallel answers with serial retrieval, the Ragas stage, the claim-kind classifier with its gate and its refusal reporting, the correctness and faithfulness gates, the length-controlled check, the closed-book runs, the missing-outcome rules, and the answer context record. ### Modified Capabilities -- `benchmark-harness`: adds requirements for per-provider concurrency of hosted calls, rate-limit retry and clean stop, row files that are the same for parallel and serial runs, setups named by settings with pinned chunk, reranker and sparse settings, and the answer-stage selection rule. +- `benchmark-harness`: adds requirements for per-provider concurrency of hosted calls, rate-limit retry and clean stop, row files that are the same for parallel and serial runs, setups named by settings with pinned chunk, reranker and sparse settings, the answer-stage selection rule with index cost and configuration hash, halves split by document on FinanceBench, the half-B guard, the full FreshStack corpus, any-page evidence recall, and the clustered analysis with fixed Holm families. - `grounded-answer-synthesis`: adds a typed `rate_limited` record for a rate-limited generation, the `out_of_credit` kind, the short HTTP 402 retry, and a typed `verification_rate_limited` record for a rate-limited claim judge. - `answer-sampling`: the answer contract declares `rate_limited` and `verification_rate_limited`. - `hybrid-retrieval`: the full-text index is built and refreshed at ingest, never by a query; a native query never writes to the store; the wording on unindexed rows matches the contract tests. @@ -74,7 +75,7 @@ These capabilities add to the requirements that the open change `experiment-35-f ## Impact -- Harness code (under `experiments/`): `35-full-rag-benchmark-2026-11-01/` (`run_arms.py`, `stages35.py`, `outputs35.py`, `arms.py` rewritten as a setup table, `datasets35.py` with the Legal loader and the 203-query FreshStack set, `scoring35.py` with the number match, `live35.py`; new modules for answers, Ragas, join, rebuild comparison, the claim-kind classifier, the selection rule and the factorial analysis), `_lib/` (new shared modules for concurrency, rate-limit retry and ordered rows; `cohen_kappa` in `stats_d6.py`), and `prefreeze/ragas_stage.py` (reused, not changed). +- Harness code (under `experiments/`): `35-full-rag-benchmark-2026-11-01/` (`run_arms.py`, `stages35.py`, `outputs35.py`, `arms.py` rewritten as a setup table, `datasets35.py` with the Legal loader and the 203-query FreshStack set, `scoring35.py` with the number match, `live35.py`; new modules for answers, Ragas, join, rebuild comparison, the claim-kind classifier, the selection rule, the split, the factorial analysis and the length-controlled check), `_lib/` (new shared modules for concurrency, rate-limit retry, ordered rows and the clustered analysis `stats_cluster.py`; `cohen_kappa` in `stats_d6.py`), and `prefreeze/ragas_stage.py` (reused, not changed). - App code (group 0 and group 12): - fix A: `src/omrs/core/answer/rate_limits.py` (new), `src/omrs/core/answer/stages.py`, `src/omrs/core/answer/results.py`, `src/omrs/core/answer/verify.py` (judge record), `src/omrs/core/answer/pipeline.py`, `src/omrs/core/answer/graph_adapters.py`, `src/omrs/transports/api/openapi.yaml` (`AnswerResponse` gains `rate_limited`), `web/src/api/documents.ts`, `web/src/chat/answerStatus.ts`; - fix B: `src/omrs/core/vectordb/lancedb/fts.py`, `src/omrs/core/vectordb/base.py`, `src/omrs/core/ingestion/pipeline.py` (or `writer.py`), `src/omrs/core/retrieval/sparse_dispatch.py` (warning text); @@ -85,4 +86,4 @@ These capabilities add to the requirements that the open change `experiment-35-f - Tests: new files under `tests/` (and one under `web/src/chat/`) for each new behaviour. The clean-base tripwire count changes. - Documents: `protocol.md` sections 0, 4.4, 4.5, 5 and 6 of Experiment 35. ADR-119 gains an amendment note for the claim-kind rule and for the removal of Granite Guardian and H-CC. - Operations: 9 parallel index pods (design D32) and three replicate pods, all on the one chosen GPU type and pod spec, on one RunPod network volume in one data centre. The Mac pod-creation script sets the pinned image. The paid calls run at a higher rate. The pre-freeze test pod session sets the concurrency values and measures the reranker; the build pod re-checks the budget before the replicate pods start. -- No change to the answer model, Ragas judge, metrics or statistics functions. The checker set shrinks to GLM-5.3 and Clef-Flash. +- No change to the answer model or the Ragas judge. The checker set shrinks to GLM-5.3 and Clef-Flash. A4 adds one experiment-only statistics module (`experiments/_lib/stats_cluster.py`) that calls `statsmodels`, and changes no existing statistics function. diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md index 86116ead..d248afb4 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md @@ -115,6 +115,12 @@ A `claimkinds` stage SHALL give each judged claim of one run folder one kind fro - **WHEN** the classification stopped part-way and runs again with `--resume` - **THEN** it MUST classify only the claims without a saved kind +#### Scenario: Claim still unclassified after resume + +- **WHEN** the classifier still fails on a claim after `--resume` +- **THEN** the claim MUST count as a factual claim, stay in the faithfulness denominator and be counted +- **AND** a sensitivity analysis MUST leave such claims out + ### Requirement: The classifier passes a hand-label gate before the freeze Before the freeze, the operator SHALL label a seeded sample of about 100 smoke claims from the three smoke datasets, blind to the classifier's output and to every verdict. The pass thresholds SHALL be fixed before labelling starts. The classifier SHALL be frozen only when it passes; a failed gate SHALL block the freeze of the classifier. @@ -170,29 +176,128 @@ For each checker separately, the report SHALL recompute faithfulness on all clai - **WHEN** the bounds run for Clef-Flash - **THEN** only Clef-Flash's refusal verdicts MUST change, and the report MUST give the lowest and highest faithfulness per answer setup and dataset from the two settings -### Requirement: Answer correctness is checked against hand labels +### Requirement: Answer correctness passes a gate per core dataset -A sampling step SHALL draw 100 answers with seed 20261113, 25 per dataset, even over the chunk sizes, and SHALL write them without any Ragas score. An agreement step SHALL binarise `AnswerCorrectness` at 0.5 and SHALL give Cohen's kappa against the operator labels per dataset and pooled, and whether the pooled kappa reaches 0.60. +A sampling step SHALL draw 50 half-B questions per dataset with seed 20261113, each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score shown. A core dataset passes only when Cohen's kappa of `AnswerCorrectness` cut at 0.5 against the human labels has a point estimate of at least 0.60 and a lower 95% bound of at least 0.40, with "can't tell" scored both ways (both must pass), and the AUC of the continuous score has a lower 95% bound above 0.5. #### Scenario: Same sample twice - **WHEN** the sample is drawn twice with the same seed - **THEN** both samples MUST hold the same answers in the same order, with no score shown -#### Scenario: Kappa below the rule +#### Scenario: Gate interval method + +- **WHEN** the agreement step computes any gate interval (kappa, AUC or specificity difference) +- **THEN** it MUST use a percentile bootstrap with 10,000 resamples by question and the seed listed for that interval in pre-registration section 16 + +#### Scenario: Point estimate passes but lower bound fails + +- **WHEN** kappa on FinanceBench is 0.65 with a lower 95% bound of 0.35 +- **THEN** the gate MUST fail on FinanceBench + +#### Scenario: Continuous score fails + +- **WHEN** kappa on FreshStack passes and the lower 95% bound of the AUC is 0.48 +- **THEN** the gate MUST fail on FreshStack + +### Requirement: A failed correctness gate keeps its rows in the family + +A failed correctness gate SHALL mark rows 14, 15, 18 and 19 "not testable" on that dataset, with p = 1 in its Holm family. Legal RAG Bench labels SHALL be reported with no gate. Prevalence, raw agreement and PABAK SHALL be reported beside kappa. The gate SHALL be recorded with its time before any H8a or H8b contrast result is computed. + +#### Scenario: Gate fails on one dataset only + +- **WHEN** the gate fails on BioASQ and passes on FinanceBench and FreshStack +- **THEN** rows 14, 15, 18 and 19 MUST stay testable on FinanceBench and FreshStack, and the BioASQ result MUST change no family, because BioASQ rows 12 to 19 are sanity checks + +### Requirement: Faithfulness checkers pass a gate per checker + +A sampling step SHALL draw 450 claims from crossed-setup answers to all questions (both halves) with seed 20261116: 150 per chunk size, 50 per core dataset within each, at most one claim per answer, with no verdict shown. A checker's gate SHALL pass when Cohen's kappa against the human label, with "can't tell" as not supported, has a point estimate of at least 0.60 and a lower 95% bound of at least 0.40. If GLM-5.3 fails, rows 12, 13, 16 and 17 SHALL be "not testable" on every dataset. + +#### Scenario: Faithfulness sample spans both halves + +- **WHEN** the sampling step draws the faithfulness sample +- **THEN** it MUST draw from answers to questions in both half A and half B, because H8a and H8b are tested on all questions +- **AND** the correctness sample MUST still come from half B only + +#### Scenario: Second checker fails its gate + +- **WHEN** Clef-Flash fails its kappa gate +- **THEN** its results MUST be reported as not validated and MUST change no confirmatory row + +#### Scenario: Kappa also without can't tell + +- **WHEN** the agreement step reports a checker's kappa +- **THEN** it MUST also report kappa with the "can't tell" items left out, and that value MUST NOT decide the gate + +### Requirement: A differential-error rule guards H8a and H8b faithfulness + +The agreement step SHALL give each checker's specificity (the share of human-supported claims it calls supported) and sensitivity per chunk size. When the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 SHALL be "not testable", with p = 1 in their families. The gate record SHALL carry its time, and no H8a and H8b faithfulness row SHALL be computed before it exists. + +#### Scenario: Specificity differs by chunk size -- **WHEN** the pooled kappa is 0.55 -- **THEN** the agreement report MUST mark answer correctness exploratory +- **WHEN** GLM-5.3 passes its kappa gate and the 95% interval of its specificity difference (2,048 minus 512) excludes zero, whether the difference is below or above zero +- **THEN** rows 12, 13, 16 and 17 MUST be "not testable", and the family sizes MUST NOT change -### Requirement: A closed-book run checks contamination +#### Scenario: Second checker specificity differs -A `closedbook` stage SHALL answer every question once with no retrieval and SHALL score it with `AnswerCorrectness` and the FinanceBench number match. The report SHALL show RAG correctness split by retrieval hit or miss beside closed-book correctness. +- **WHEN** only the Clef-Flash specificity difference excludes zero +- **THEN** it MUST be reported only and MUST NOT change any row + +#### Scenario: Gate decided before H8a and H8b results + +- **WHEN** the analysis starts an H8a or H8b faithfulness row and no gate record with a time exists +- **THEN** it MUST refuse and MUST name the gate step + +### Requirement: A length-controlled check reruns long-chunk claims + +A `lengthcheck` stage SHALL draw 300 claims from the `c2048` crossed answers with seed 20261119. It SHALL split each cited chunk into consecutive 512-token windows, keep the window that shares the most distinct lower-case claim words, and have GLM-5.3 and Clef-Flash judge the claim again against it. The report SHALL give the verdict change rate with a Wilson interval, as a registered sensitivity analysis with no gate. + +#### Scenario: Window choice tie + +- **WHEN** two windows share the same number of claim words +- **THEN** the stage MUST keep the earlier window + +### Requirement: Closed-book runs check contamination + +A `closedbook` stage SHALL answer every question once with no retrieval under two prompts: an own-knowledge prompt (the floor) and the main RAG answer prompt with an empty evidence block. It SHALL score both with `AnswerCorrectness` and the FinanceBench number match. Closed-book results SHALL be descriptive, with no hypothesis test. #### Scenario: No search call -- **WHEN** the closed-book stage runs +- **WHEN** either closed-book run runs - **THEN** it MUST send no search call and MUST pass an empty evidence list +#### Scenario: RAG prompt with empty evidence + +- **WHEN** the closed-book RAG-prompt run builds its prompt +- **THEN** the prompt MUST equal the main answer prompt with an empty evidence block, and its template hash MUST be recorded + +### Requirement: A contamination sensitivity re-runs the correctness rows + +The report SHALL show closed-book correctness per dataset beside every correctness table, and a per-question cross-table on retrieval misses (closed-book correct or not against RAG correct or not). A registered sensitivity analysis SHALL re-run rows 14, 15, 18 and 19 on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5). + +#### Scenario: Contamination subset + +- **WHEN** a question is answered correctly by one closed-book run and wrongly by the other +- **THEN** it MUST NOT enter the contamination subset + +### Requirement: Missing answers and unjudged rows keep every question + +The analysis SHALL keep every question in its dataset's denominator and SHALL count every exclusion per setup and row. An answer still missing when its run closes SHALL score 0 for correctness and number match and SHALL count as an answer with no factual claim. A Ragas or checker row still unjudged when its run closes SHALL leave that row's complete cases, SHALL be counted, and SHALL be scored 0 in a sensitivity analysis. + +#### Scenario: Answer missing at run close + +- **WHEN** an answer is still missing when its run closes +- **THEN** its correctness and number match MUST be 0 and it MUST count as an answer with no factual claim + +### Requirement: Faithfulness uses complete cases with registered bounds + +Faithfulness per question and setup SHALL be the mean over the runs with at least one factual claim. A question SHALL enter a faithfulness row only when every cell the row uses has a value. Two sensitivity analyses SHALL keep every question, with no-claim answers scored 1 and scored 0. The refusal rate per setup SHALL be reported as a co-outcome. + +#### Scenario: Answer with no factual claim + +- **WHEN** a question's answers in one cell of a faithfulness row have no factual claim in any run +- **THEN** the question MUST leave that row's complete cases, the exclusion MUST be counted, and both sensitivity analyses MUST include it + ### Requirement: Every answer is one synthesis round Each answer setup, the closed-book run and the oracle run SHALL run with `answer.context_window` 131,072 characters and `answer.max_rounds` 1. diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md index f6cd9409..b11d2c4f 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md @@ -148,13 +148,90 @@ A selection step SHALL read the saved half-A search scores of FinanceBench, Fres ### Requirement: The selection step picks one cell for Part 2 -At the chosen embedder and reranker size, the selection step SHALL pick chunk size, reranker on or off and hybrid on or off from the 12 cells by the same three-dataset pooled half-A score. Cells within 0.01 of the best, inclusive, SHALL tie. Ties SHALL go to reranker off, then hybrid off, then the chunk size with the lowest index cost, then the smaller chunk size (amendment A3). It SHALL write the cell scores and the chosen cell to the selection file. +At the chosen embedder and reranker size, the selection step SHALL pick chunk size, reranker on or off and hybrid on or off from the 12 cells by the same three-dataset pooled half-A score. Cells within 0.01 of the best, inclusive, SHALL tie. Ties SHALL go to reranker off, then hybrid off, then the chunk size with the lowest index cost, then the smaller chunk size (amendment A3). Index cost SHALL be chunks times embedding dimension over the three core datasets (A4). #### Scenario: Four-factor tie - **WHEN** the best cell has reranker on and hybrid on, and a cell with reranker off and hybrid on pools 0.008 lower - **THEN** the selection MUST choose the cell with reranker off +#### Scenario: Index cost decides a chunk-size tie + +- **WHEN** two tied cells differ only in chunk size and the 512-token indexes store more vector values than the 1,024-token indexes +- **THEN** the selection MUST choose the 1,024-token cell + +### Requirement: The selection file holds a configuration hash + +The selection step SHALL write the cell scores, the chosen cell and the configuration hash: the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings, with the SHA-256 of `halves.json` as one field. A sensitivity step SHALL repeat the cell choice with any-page nDCG@10 on FinanceBench and SHALL record whether the cell changes; the registered cell SHALL NOT change. + +#### Scenario: Same hash twice + +- **WHEN** the selection runs twice on the same scores and the same `halves.json` +- **THEN** both selection files MUST hold the same configuration hash + +### Requirement: Half B is scored only after the hash is time-stamped + +The harness SHALL follow pre-registration section 14.2: score half A only; apply the selection rule and write the selection file; time-stamp the configuration hash with a file upload to the registered OSF project; then score half B and start the replicate pods. A scoring step SHALL refuse any half-B score while the selection file or the time-stamp record is missing. A replicate pod's first stage SHALL refuse to start under the same condition. + +#### Scenario: Half-B score before the time-stamp + +- **WHEN** the scoring step is asked for half-B scores and no time-stamp record exists +- **THEN** it MUST stop before it computes any half-B score and MUST name the missing record + +#### Scenario: Half-A scoring + +- **WHEN** the scoring step is asked for half-A scores and no selection file exists +- **THEN** it MUST compute them + +### Requirement: Halves keep FinanceBench documents apart + +A split step SHALL write `halves.json` with seed 20261115. It SHALL split FinanceBench by document: documents are shuffled with the seed and each goes, with all its questions, to the half that has fewer questions so far, ties to half A. It SHALL split FreshStack, BioASQ and Legal RAG Bench by question: questions are shuffled with the seed and the first half, rounded up, is half A. + +#### Scenario: No FinanceBench document in both halves + +- **WHEN** the split runs on FinanceBench +- **THEN** every document's questions MUST sit in one half only + +#### Scenario: Same halves twice + +- **WHEN** the split runs twice with the same seed +- **THEN** both `halves.json` files MUST be byte-identical + +### Requirement: Planned contrasts use one clustered method + +The analysis SHALL compute rows 1 to 19 of pre-registration section 8 from per-question contrast scores. On FreshStack, BioASQ and Legal RAG Bench it SHALL use the closed-form test of pre-registration section 9 point 3 (mean, bias-reduced variance, t on n minus 1), computed through GEE and checked against the closed form. On FinanceBench it SHALL use a document-level sign-flip permutation test (100,000 draws, seed 20261101). It SHALL NOT fit a mixed model or use a McNemar or equivalence test. + +#### Scenario: GEE matches the closed form + +- **WHEN** the analysis runs on a fixed known-answer fixture of n contrast scores +- **THEN** the estimate MUST equal their mean, and the variance MUST equal the sum of squared residuals times (n / (n minus 1)) squared, divided by n squared +- **AND** the p-value and 95% interval MUST use t on n minus 1 degrees of freedom, whatever reference the library reports by default +- **AND** if the `statsmodels` GEE output differs from the closed form, the analysis MUST use the closed form and record the difference + +#### Scenario: FinanceBench document clusters + +- **WHEN** two FinanceBench questions share a document +- **THEN** the sign-flip test MUST flip their contrast scores together, as one document sum, and the 95% interval MUST come from inverting the same test on the same draws + +#### Scenario: Type I error under the null + +- **WHEN** the analysis runs on 200 simulated datasets with no planted effect and unequal error variance across cells +- **THEN** the share of rejections at 0.05 MUST stay near 0.05 for the GEE and for the sign-flip test + +#### Scenario: Sensitivity check disagrees + +- **WHEN** the paired permutation test by question on FreshStack disagrees with the GEE on whether a row counts +- **THEN** the decision MUST follow the GEE, and the row MUST appear in the disagreement table + +### Requirement: Holm families are fixed before the run + +The analysis SHALL apply Holm within each dataset over families fixed at 19 rows (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8). BioASQ rows 9 to 19 SHALL be sanity checks with unadjusted intervals. Legal RAG Bench SHALL have no family. A row that fails a gate SHALL stay in its family with p = 1 and SHALL be reported "not testable". + +#### Scenario: Gated row keeps the family size + +- **WHEN** rows 12, 13, 16 and 17 are "not testable" on FreshStack +- **THEN** Holm MUST still run over 19 rows on FreshStack, with those rows at p = 1 + ### Requirement: The answer stage has a token-matched setup for each chunk size The answer stage SHALL run 15 setups: the 12 crossed setups on 10 chunks each and three token-matched setups (`c512-t20k`, `c1024-t20k` and `c2048-t20k`) at the chosen embedder and reranker size with reranker on and hybrid on. A token-matched setup SHALL cut its evidence at a hard limit of 20,480 real Qwen3 tokens. The chunk that crosses the limit SHALL be truncated at the limit, and the truncation SHALL be recorded per question. The `c2048` x 10 factorial cell SHALL keep its 10-chunk evidence. @@ -169,6 +246,29 @@ The answer stage SHALL run 15 setups: the 12 crossed setups on 10 chunks each an - **WHEN** the chosen embedder and reranker retrieve short 2,048-token chunks - **THEN** `c2048` x 10 MUST send exactly 10 chunks, and `c2048-t20k` MUST send chunks until the 20,480-token limit, truncating the chunk that crosses it and recording the truncation +### Requirement: FreshStack uses the full public corpus at a pinned revision + +The FreshStack loader SHALL load the full public `freshstack/corpus-oct-2024` corpus, config `langchain`, at the revision pinned in `plan.json`, and the 203 public queries. It SHALL refuse a corpus whose revision or row count differs from the pin. It SHALL NOT sample a subset, add distractors, load the 20 continuity queries or the continuity documents, or append the overlay runbook. + +#### Scenario: Subset on disk + +- **WHEN** the corpus folder holds the 10,000-document subset of Experiments 9a and 22 +- **THEN** the loader MUST stop before any ingest and MUST name the expected and the found row counts + +### Requirement: Evidence recall gives any-page credit on FinanceBench + +`evidence_recall_20k` SHALL be the share of a question's gold units covered by the evidence that the token cut takes from the top 100 chunks. On FinanceBench a whole chunk SHALL cover every gold page it overlaps, and the truncated crossing chunk only its first page; first-page credit SHALL be a sensitivity column. Elsewhere a chunk SHALL cover its one unit. nDCG@10, Recall@10, MRR@10 and hit@page SHALL keep first-page credit as primary. + +#### Scenario: Chunk across two pages + +- **WHEN** a whole FinanceBench chunk spans pages 41 and 42 and the gold page is 42 +- **THEN** `evidence_recall_20k` MUST count page 42 as covered, and the first-page column MUST NOT + +#### Scenario: Truncated chunk across two pages + +- **WHEN** the crossing chunk spans pages 41 and 42, is truncated, and the gold page is 42 +- **THEN** `evidence_recall_20k` MUST NOT count page 42 as covered by that chunk + ### Requirement: Parallel and serial runs write the same row files A stage SHALL write its row file and its trace rows in frozen question order, whatever order the calls finish in. With a fixed clock and deterministic call results (stub calls), a run at any concurrency SHALL give a row file with the same bytes as a run at concurrency 1. Its trace file SHALL be the same after the concurrency values in the trace settings are removed; the trace keeps the value each role used. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index 259e02dc..f21cfac0 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -306,9 +306,10 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d - it writes each checker's unsupported rate on refusal claims, and the Clef-Flash against GLM-5.3 kappa on refusal and on factual claims per dataset; - it writes faithfulness on all claims, marked secondary robustness; - it writes per-checker bounds where only that checker's refusal verdicts change, with the lowest and highest faithfulness per answer setup and dataset; - - a question with no factual claim leaves that dataset's bootstrap and is counted. + - missing outcomes follow pre-registration section 12 (A4): every question stays in the denominator; faithfulness per question and setup is the mean over runs with at least one factual claim; a question enters a faithfulness row only when every cell the row uses has a value (complete cases), and the excluded count is printed per setup and row; two sensitivity tables keep every question, with no-claim answers scored 1 and scored 0; the refusal rate per setup is reported as a co-outcome; + - an answer still missing when its run closes scores 0 for correctness and number match and counts as an answer with no factual claim; a Ragas or checker row still unjudged when its run closes leaves that row's complete cases, is counted, and a sensitivity table scores it 0. - Prove they fail, then implement with `experiments/_lib/stats_d6.py` (`wilson`, `paired_bootstrap`). Verify: the tests pass. + Prove they fail, then implement with `experiments/_lib/stats_d6.py` (`wilson`, `paired_bootstrap` for the descriptive intervals). Verify: the tests pass. ## 9. Image pin and pre-freeze test pod session (before G7) @@ -319,7 +320,7 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d 2a. Reranker files (design D24 revision, item 1): for each Qwen3-Reranker GGUF candidate, a relevant pair scores near 1 and an irrelevant pair near 0, and the ranking of 20 query-passage sets matches the model card's transformers recipe. Record the result per file. 2b. Reranker throughput at 0.6B, 4B and 8B on 100-candidate pools at 512-, 1,024- and 2,048-token passages; the 8B embedder and the 8B reranker loaded together (peak VRAM, any swap); index build time per chunk size on a fixed slice of each corpus for each embedder. 3. Calibration: answers at 16, `chat` at 4, Ragas at 16, `claimkinds` at 16. Record per role the 429 and 402 counts, the waits, the stops and the calls per minute. Halve a value that hit a rate-limit stop and repeat that role once. Check that each answer's evidence chunk IDs equal the same run's search IDs. Measure wall seconds per answer per stage and the reported `cost_usd` per answer. - 4. The first budget check (design D23 revision): measured build hours per chunk size times 9 indexes, plus the search stage from the measured reranker throughput, plus three replicate pods at the measured hours, all times the pod price, plus the hosted cost per answer times 41,760. Above the $975 ceiling, the operator decides before the freeze. + 4. The first budget check (design D23 revision): measured build hours per chunk size times 9 indexes, plus the search stage from the measured reranker throughput, plus three replicate pods at the measured hours, all times the pod price, plus the hosted cost per answer times 41,760, plus the full-corpus FreshStack builds from task 16.2, the two closed-book runs, the oracle run, the length-controlled check and the pilot. When the projection reaches the $975 ceiling, apply the results-blind drop order of design D38 (pre-registration section 17) in order and record each step with the projection that triggered it; no confirmatory item is dropped by choice. 5. Copy the session outputs to the Mac, then delete the pod. Verify: steps 1 and 2 pass with zero errors; a dated calibration note in `protocol.md` gives the measured values, the counts, the hours per replicate pod and the extrapolated total; design.md D22 holds the measured rates; the `arms.py` defaults equal the measured values; the pod no longer exists in the RunPod pod list. A failure in step 1 or 2 is fixed in code and the session runs again. @@ -344,8 +345,8 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d - the G2 stop rule in section 6: if a rebuild check fails, the replicate pods do not start; find the cause, fix it, build the indexes again, then seal and check again; - the zero-error recheck of FreshStack `77536364` and `77762811` in every answer-setup shape (`--qids`, `--index-from`, its own output folder); the pass rule: every `answer-summary.json` has `failed` 0, and every judge summary has `errors` 0; - the search-stage time of the recheck on the volume, recorded as a check on read speed; - - on the Mac: score the 72 search setups and run the selection rule (`select35.py`), and write the dated selection note into `protocol.md` section 0.5; - - the budget re-check after the search stage, with the real pod cost so far and the measured rates; above the ceiling, stop before the replicate pods and ask the operator; + - the order of work of pre-registration section 14.2 (A4): on the Mac, score half A only of the 72 search setups, run the selection rule (`select35.py`) and write `selection.json`; time-stamp the configuration hash with a file upload to the registered OSF project (mechanism: open item O1, task 16.23); only then score half B and start the replicate pods; the scorer refuses a half-B score while the time-stamp record is missing (task 16.17); + - the budget re-check after the search stage, with the real pod cost so far and the measured rates; when the projection reaches the ceiling, apply the results-blind drop order of design D38 and record each step; no confirmatory item is dropped by choice; - the three replicate pods with `--run 1`, `2` and `3`, `--local-dir` on the container disk, and their own checkouts; each pod runs answer (with its own retrieval), the two judges, `ragas` and `claimkinds` for the 15 answer setups on its own run, one stage at a time, with the budget read at each stage end; - the expected pod hours from design D22 and D32, as measured by tasks 9.2 and 9.3; - the copy-back without `lancedb/`; @@ -353,7 +354,7 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d - stopping each pod as soon as its work is copied. Verify: every command in the procedure runs with `--dry-run` or `--help` as written. -- [ ] 10.2 Write the budget into `protocol.md` section 0.8 (replaces Exp 35 task 1.11 and the $250 cap of the first 2026-11-12 decision): the estimate of design D23 updated with the measured values of tasks 9.2 and 9.3, and the ceiling of 1.5 x the estimate ($975 before measurement), pods and hosted calls together. State the three checks (design D23): before the freeze, on the index pods after the search stage (stop before the replicate pods), and at each stage end during the replicate pods. Set the OpenRouter key's credit limit to the hosted share of the ceiling. Verify: `protocol.md` states the estimate, the ceiling, the checks and the date, and Exp 35 tasks 1.11 and 4A.5 are ticked with a pointer to it. +- [ ] 10.2 Write the budget into `preregistration.md` section 17 (A4; it replaces `protocol.md` section 0.8, Exp 35 task 1.11 and the $250 cap of the first 2026-11-12 decision): the estimate of design D38 (about $730, from the D23 figures plus the A4 items) updated with the measured values of tasks 9.2, 9.3 and 16.2, and the ceiling of $975, pods and hosted calls together. State the three checks (design D23): before the freeze, on the index pods after the search stage, and at each stage end during the replicate pods. Each projection uses billing and measured rates only, never scores. State the results-blind drop order of design D38. Set the OpenRouter key's credit limit to the hosted share of the ceiling. Verify: `preregistration.md` states the estimate, the ceiling, the checks, the drop order and the date, and Exp 35 tasks 1.11 and 4A.5 are ticked with a pointer to it. - [ ] 10.3 Write the dated pre-freeze amendment (2026-11-12) into `protocol.md` section 4.4, and add a matching amendment note to ADR-119. It must give: - the three claim kinds, with mixed sentences and negative findings counted as factual claims; - primary faithfulness on factual claims only, with the reason (a refusal is a statement about retrieval, and the checkers grade refusals very differently); @@ -372,7 +373,7 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d - [ ] 11.2 Run `./scripts/local_ci.sh` (after the operator agrees to a full run). Verify: the final line reports success, and no step prints `SKIPPED`. - [ ] 11.3 Run `uv run --no-sync omrs experiments check` and `openspec validate exp35-main-run-readiness --strict`. Verify: both pass. -Group 11 runs last, after groups 12 to 15 below. +Group 11 runs last, after groups 12 to 16 below (the code tasks of group 16; its pre-freeze tasks close before G7). ## 12. App addition C: llama.cpp reranker backend (redesign, 2026-11-12; design D27) @@ -413,7 +414,7 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 13.2 Replace `--arm` with `--setup NAME|all` and `--index NAME` in `run_arms.py` (or `cli35.py`). Write failing tests: `--setup all --stage search` plans 72 setups in a fixed order; `--index e4b-c1024 --stage ingest` plans one ingest per dataset; `--stage search --index e4b-c1024` plans that index's 8 setups; an unknown name exits with code 2. Verify: the tests pass and `--help` shows the options. - [ ] 13.3 Per-index stores (design D32). Write failing tests: `Layout` puts each index's store under `/stores//lancedb`; an ingest writes only there; the seal fingerprint is `///index-fingerprint.json`; two ingests of different indexes into one output folder at the same time leave each store written by one process only. Prove they fail, then implement in `outputs35.Layout`. Verify: the tests pass. - [ ] 13.4 Legal RAG Bench loader (design D29). Write failing tests with a small fixture of both configs: 100-row shape, gold `relevant_passage_id`, passages written once as `.md` holding title, text and footnotes, with a leading `#` on any line escaped, frozen seeded order, file SHA-256 check, and a stop that names a missing gold passage. Prove they fail, then implement in `datasets35.py`. Verify: the tests pass. -- [ ] 13.5 FreshStack 203 queries and public corpus (operator decision, 2026-11-13). Write failing tests: the loader drops the 20 continuity queries; the corpus holds no `continuity/` file and no overlay runbook; `datasets35.py` has no `_FS_OVERLAY` or `_FS_OVERLAY_ID` reference and no expected hash for the overlay; questions `77536364` and `77762811` stay. Prove they fail, then remove the overlay constants, its hash entry, its id mapping and the `continuity/*.md` glob. Verify: the tests pass, the loader reports 203 queries, and the new corpus tree hash is written into `protocol.md` section 4.1. +- [ ] 13.5 FreshStack 203 queries on the full public corpus (amendment A4, 2026-11-14; pre-registration section 4; design D38). Write failing tests: the loader drops the 20 continuity queries; the corpus is the full `freshstack/corpus-oct-2024` config `langchain` at the pinned revision (49,514 documents on the real data; a fixture stands in for tests), with no `continuity/` file, no overlay runbook and no subset or distractor sampling; `datasets35.py` has no `_FS_OVERLAY` or `_FS_OVERLAY_ID` reference and no expected hash for the overlay; the loader refuses a corpus whose revision or row count differs from the pin; questions `77536364` and `77762811` stay. Prove they fail, then remove the overlay constants, its hash entry, its id mapping and the `continuity/*.md` glob, and add the revision and row-count check. Verify: the tests pass, the loader reports 203 queries, and the revision, row count and corpus tree hash from task 16.1 (P1) are in `plan.json`. - [ ] 13.5a BioASQ passages as `.md` (operator decision, 2026-11-13). Write failing tests: `datasets35.py` writes `.md` (not `.txt`), escapes a leading `#` on any line, and maps gold ids without the suffix. Prove they fail, then implement. Verify: the tests pass. - [ ] 13.5b Qwen3 token counts on all four datasets. Write a failing test that ingests a small fixture of FinanceBench (Markdown PDF text), FreshStack, BioASQ and Legal at 512, 1,024 and 2,048 tokens and asserts: every chunk has at most the level's Qwen3 token count (pinned tokenizer revision); every dataset goes through the model-token-aware Markdown path; the cl100k counter is never called (patched to raise). Prove it fails on the `.txt` passages, then pass it after 13.4 and 13.5a. Verify: the test passes. - [ ] 13.6 FinanceBench number match (design D29). Write failing tests with known pairs: "$1,577 million" against "$1.58 billion" passes; "12.5%" against "12.5 percent" passes; "(3.2)" against "-3.2" passes; a missing number fails; two reference numbers need both; a 2% difference fails. Prove they fail, then implement in `scoring35.py` (or `numbers35.py`). Verify: the tests pass. @@ -424,10 +425,10 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 14.1 Answer context budget, one round (design D30 revision). Write failing tests: `settings_values` of every answer setup holds `answer.context_window` 131072 and `answer.max_rounds` 1; for each chunk size, the largest evidence set (10 passages at the level, 2,048-token fixture of about 9,200 characters each) synthesises through a recording stub seam with exactly one completion call, no context overflow and no truncation; the test fails and names the setup when a second call happens (prove this with `max_rounds` 4 and `context_window` 8192). Prove they fail on today's settings, then set the values. Verify: the tests pass for all three chunk sizes. - [ ] 14.1a Record the answer model's context length. Write a failing test for a `context35.py` check that reads `context_length` of `deepseek/deepseek-v4.1-flash-1017` from a stubbed OpenRouter models reply and refuses below 65,536 tokens. Prove it fails, then implement. Run it once live in the test pod session (task 9.2). Verify: the test passes and `protocol.md` section 0.5 records the value and the date. - [ ] 14.2 Write failing tests that each answer row records `completion_calls`, the evidence characters, and the truncated passages and characters dropped, computed with OMRS's own `_fit_and_measure` on the same rows and settings. Prove they fail, then implement in `answers35.py`. Verify: the tests pass, and a row with cut evidence is still saved. -- [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. +- [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score, both chosen sizes, the A3 cell and the configuration hash. Then (A4, pre-registration section 14.1) tests that: index cost is the number of chunks times the embedding dimension, summed over the three core datasets, read from the sealed indexes with no score; the configuration hash is the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings with the SHA-256 of `halves.json` as one field, and the same inputs give the same hash twice; the any-page sensitivity repeats step 3 with any-page nDCG@10 on FinanceBench, records whether the cell changes, and never changes the registered cell. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. - [ ] 14.5 Ragas `AnswerCorrectness` (design D33). Write failing tests with a stub LLM and stub embeddings: the `ragas` stage scores five metrics, `AnswerCorrectness` gets the question, the answer and the dataset's reference answer (never the contexts), uses the frozen judge and the local 4B embedder, and records the weights; an answer without a reference records "not judged" and sends no call; resume skips saved rows. Prove they fail, then implement in `ragas35.py` with Ragas 0.4.3. Verify: the tests pass, and the version check still refuses another Ragas version. -- [ ] 14.6 Correctness hand-label check (design D33). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 100 answers twice, 25 per dataset and even over chunk sizes, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` binarises at 0.5, gives `cohen_kappa` per dataset and pooled, and marks correctness exploratory below a pooled kappa of 0.60. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (A2, task 15.8). Verify: the tests pass, `protocol.md` holds the threshold and the kappa rule in a commit before labelling, and then the kappa result (Exp 35 task 4A.4a). -- [ ] 14.4 Factorial analysis (`factorial35.py`, design D31; A2c: H6 on FinanceBench uses `evidence_recall_20k` with both page rules as sensitivity and their sign agreement reported; H8 uses all questions, descriptive headlines half B). Write failing tests on a fixture with a planted embedder effect: only H1's contrasts count; Holm runs across every H1 to H8 contrast within one dataset (A2); a significant wrong-sign contrast does not count; H4 holds only when both of its contrasts count; H6 is tested on FinanceBench and FreshStack only (A2), and BioASQ chunk-size rows carry `sanity_check: true`; H8 averages the 3 runs per question first; Legal rows carry `exploratory: true`. Prove they fail, then implement with the existing statistics functions only. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). +- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B, each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). +- [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome (O6); a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). ## 15. Methods-review fixes (pre-freeze amendment A2, 2026-11-13; design D35) @@ -436,24 +437,63 @@ Each test is proved to fail on the current code before the fix. - [ ] 15.1 Unit scorer (B1). Write failing tests in `tests/test_experiment35_scoring.py`: - 100 ranked chunks where chunks 1 to 3 share one FreshStack document give that document rank 1 once, and the 10-unit list reaches past chunk 10; - the same gold set scores the same nDCG@10 whether the chunks are 512 or 2,048 tokens when the unit order is the same (the bias case of `scoring35.py:39-51` fails this today); - - a FinanceBench chunk spanning pages 41 and 42 with gold page 42 counts for page 41 (first page); the "any page" sensitivity column counts it for 42; + - for nDCG@10, Recall@10, MRR@10 and hit@page, a FinanceBench chunk spanning pages 41 and 42 with gold page 42 counts for page 41 (first page, primary); the "any page" sensitivity column counts it for 42; + - nDCG@10, Recall@10 and MRR@10 follow the formulas of pre-registration section 6 (trec_eval convention, ideal DCG over min(R_q, 10) gold units, plain mean over questions); a known-answer fixture with 12 gold units checks the cap; - fewer than 10 distinct units in 100 chunks scores the short list, missing ranks as misses; - - `evidence_recall_20k` takes chunks in rank order until their real Qwen3 token total reaches 20,480 (the chunk that crosses the limit is truncated at the limit, nothing after it), as in task 15.14; a test with short chunks proves it does not count by chunk number. + - `evidence_recall_20k` takes chunks in rank order while the Qwen3 token total before a chunk is below 20,480 (the chunk that crosses the limit is truncated at the limit, nothing after it), as in task 15.14; a test with short chunks proves it does not count by chunk number; + - `evidence_recall_20k` uses any-page credit on FinanceBench (pre-registration section 6): a whole chunk covers every gold page it overlaps, and the truncated crossing chunk covers only its first page; first-page credit is a separate sensitivity column; on FreshStack, BioASQ and Legal a chunk, whole or truncated, covers its one unit. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.2 Depth 100. Write failing tests: every search stage saves 100 chunks per question; at depth 100 every reranker setup reranks a pool of exactly 100 (the harness passes `fetch_k=100`; without it `policy.py:63-67` gives 300, which the test must show failing first); a reranker-on search returns the whole reranked pool (100) in order; the answer stage takes its 10 (or 40 or 20) chunks from the head of that list. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.3 Token-matched answer setups (M5, revised by A2c N2). Write failing tests: the answer setup table has 15 setups (A3): the 12 crossed setups on 10 chunks each, plus `c512-t20k`, `c1024-t20k` and `c2048-t20k` at the chosen sizes with reranker on and hybrid on; the evidence of the three `-t20k` setups is cut by real Qwen3 tokens with the rule of task 15.14, while the `c2048` x 10 factorial cell keeps its 10-chunk evidence (a test with short fixture chunks proves `c2048` x 10 still sends 10 chunks and `c2048-t20k` sends more); the cost line of D35 (about +$40, ceiling $975) is checked in the test pod budget sum; with one answer round (task 14.1) the largest 40-chunk set still fits. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.4 Closed-book stage (M6). Write failing tests: `--stage closedbook` sends no search call, uses the frozen answer settings with an empty evidence list and a fixed prompt, writes one row per question for all four datasets, runs `AnswerCorrectness` and the number match, and resumes without repeat calls. Then a report test: correctness split by retrieval hit or miss beside closed-book correctness. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.5 Split halves (M3). Write failing tests: `split35.py` with seed 20261115 gives the same halves twice, splits each dataset about evenly, and `select35.py` reads half A only (a planted half-B effect does not change the choice). Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.6 Mixed model (M2; A2c N4). Tests must also show: each planned contrast's p-value comes from `GEE` (same mean model, Gaussian, exchangeable working correlation) with cluster-robust errors by question, by document on FinanceBench; the MixedLM gives the estimates; Holm runs on the GEE p-values; a fixture with unequal error variance across cells keeps the GEE type I error near 0.05 over 200 simulations. Add `statsmodels` with `uv add --group stats statsmodels`, and add `{include-group = "stats"}` to the `dev` group (decided by the operator, 2026-11-13: a dependency group like `ragas`, kept out of the app install). Write a test that `uv export --no-dev --no-group stats` lists no `statsmodels`. Write failing tests for `experiments/_lib/stats_mixed.py` on a simulated dataset with a known embedder effect and a question random effect: the fitted contrast is within its standard error of the truth; contrasts for H1 to H7 come out with p values that `holm` adjusts across all contrasts of one dataset; the cross-dataset rule gives "holds" at 2 of 3 confirmatory datasets and ignores Legal; H6 needs both FinanceBench and FreshStack. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. -- [ ] 15.7 Grouped bootstrap for FinanceBench (M2). Write a failing test that `factorial35.py` passes `groups=doc_name` to `paired_bootstrap` for FinanceBench and no groups elsewhere. Prove it fails, then implement. Verify: the test passes. -- [ ] 15.8 Labelling for two labellers (M4). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the stated sizes (100, 200, 100) and strata (dataset; dataset and chunk size; dataset); the L2 subset is 30% of each set with seed plus 1; no judge score or verdict reaches the page; kappa per checker, per dataset and L1 against L2 comes with bootstrap 95% intervals; a "can't tell" after discussion leaves the kappa against the judge and is counted. Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.4 Closed-book stages (M6; A4, pre-registration section 13). Write failing tests: `--stage closedbook` runs two prompts, each once on all 1,160 questions: the own-knowledge prompt (floor) and the main RAG answer prompt with an empty evidence block; neither sends a search call; both use the frozen answer settings, write one row per question for all four datasets, run `AnswerCorrectness` and the number match, and resume without repeat calls. Then report tests: closed-book correctness per dataset beside every correctness table; the per-question cross-table on retrieval misses; and the registered contamination sensitivity, which re-runs rows 14, 15, 18 and 19 on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5). Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.5 Split halves (M3; A4, pre-registration section 4). Write failing tests: `split35.py` with seed 20261115 gives the same halves twice; FinanceBench is split by document (documents shuffled with the seed, each going with all its questions to the half that has fewer questions so far, ties to A), so no document sits in both halves; FreshStack, BioASQ and Legal are split by question (shuffled, first half rounded up is half A); `halves.json` lists every question ID with its half; `select35.py` reads half A only (a planted half-B effect does not change the choice). Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.6 Clustered analysis (A4 replaces M2 and A2c N4; pre-registration section 9). No MixedLM. Write failing tests for `experiments/_lib/stats_cluster.py` and `factorial35.py`: + - the per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells (equal cell weights), with answer scores averaged over the 3 runs first; the estimate is the plain mean of d_qk; on FreshStack, BioASQ and Legal the decision test is the closed form of PR 9 point 3: variance = (sum of squared residuals) x (n / (n minus 1)) squared / n squared, and the p-value and 95% interval use t on n minus 1 degrees of freedom, computed by our code whatever reference the library reports by default (`statsmodels` GEE results use a normal reference unless told otherwise); the implementation is `statsmodels` `GEE` of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`); a known-answer test on a fixed fixture checks that the GEE estimate and variance equal the closed form and that the closed form equals the paired t-test variance times n / (n minus 1); if the library does not match, the code uses the closed form and the report states the difference; + - FinanceBench uses the document-level sign-flip permutation test: the statistic is the question-weighted mean d_qk, each document's sum is flipped at random, 100,000 draws with seed 20261101, p = (1 + draws at least as large in absolute value) / 100,001, and the 95% interval inverts the same test on the same draws; + - registered sensitivity checks, never decisive: a paired permutation test by question on FreshStack and BioASQ (10,000 shuffles, seed 20261101, `paired_permutation`), and the GEE above with document clusters on FinanceBench; every row where decision and sensitivity disagree on "counts" goes into one table; + - type I error simulation, now for this method: with no planted effect and unequal error variance across cells, the rejection rate at 0.05 stays near 0.05 over 200 simulations for the GEE (FreshStack-sized data) and for the sign-flip test (84 documents, about 1.8 questions each); a fixture with complete data gives the same estimate as the cell-means contrast of the full-factorial model. + - Add `statsmodels` in a new `stats` dependency group (decided by the operator, 2026-11-13; not yet in `pyproject.toml` or `uv.lock`), with `{include-group = "stats"}` in the `dev` group; a test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. +- [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8; O6); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the L2 subset is 30% of each set with seeds 20261122, 20261123 and 20261124 (225 in all); no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.9 Hosted-call provenance (M7). Write failing tests: each answer, Ragas, GLM-5.3 and classifier row records `model_returned` and `provider_returned` from the stubbed reply; the join counts rows whose returned model differs from the requested one per role. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.10 Reranker rank check (M8). Write failing tests for `rerank_check35.py`: 200 pairs drawn with seed 20261117 are the same twice; Spearman below 0.95 marks the file "not pinned". Prove they fail, then implement. Run it on the test pod for each size (task 9.2 step 2a). Verify: the tests pass and `protocol.md` lists the correlation per file. -- [ ] 15.11 MDE from data (M1). Write `mde35.py` on saved files only: per dataset and contrast, the SD of per-question paired differences from Experiment 22 per-query rows (FreshStack) and the smoke and test-pod rows (all four datasets), and `min_detectable_effect` at alpha 0.05 / number of contrasts per dataset, power 0.80; plus the H7 difference-of-differences. Write failing tests on a fixture first. Verify: the tests pass and `protocol.md` section 0.10 holds the table with its sources before the freeze. +- [ ] 15.11 MDE from the pilot (M1; A4, pre-registration section 10). Write `mde35.py` on saved files only: per core dataset and contrast type, the SD of the per-question contrast scores from the retrieval-only pilot (task 16.5, P5; an adjacent-level contrast uses the extreme-level SD as a stand-in, marked so), and on FinanceBench the design effect of document clustering; MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha 0.05/19 on FinanceBench and FreshStack and 0.05/8 on BioASQ, and n divided by the design effect on FinanceBench; answer-stage rows use the SD from the smoke and test-pod answer files named in task 16.6 (P6) and are labelled rough; each row is shown against the SESOI (0.02 nDCG@10, open item O2). Experiment 22 values are not used. Write failing tests on a fixture first. Verify: the tests pass and the MDE table of pre-registration section 10 is filled before the freeze. - [ ] 15.12 Oracle contexts (design D36; A2c caps). Write failing tests for `oracle35.py` on fixtures of all four datasets: every oracle context is capped at 20,480 Qwen3 tokens with truncation recorded (FinanceBench and BioASQ too); Legal gives the one gold passage; FinanceBench gives the gold page text(s) from the read PDF pages; BioASQ gives the gold corpus passages by relevant passage id; FreshStack gives the gold documents in qrels order, capped at 20,480 Qwen3 tokens, and records the truncated documents and tokens dropped; a missing gold unit stops with a setup error naming it; no search call is sent. Prove they fail, then implement. - [ ] 15.13 Oracle stage. Write failing tests: `--stage oracle` runs once (run 1), uses the frozen answer prompt and settings with the oracle evidence, and feeds the judge, `ragas` (`AnswerCorrectness`) and number-match stages; the report puts closed-book (floor) and oracle (ceiling) beside the RAG setups, marked descriptive with the no-distractor caveat. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.14 Equal evidence by real tokens (A2c N2). Write failing tests for `evidence_by_tokens`: chunks are taken in rank order while the running Qwen3 token total before a chunk is below 20,480; the chunk that crosses is truncated at 20,480 tokens, the truncation is recorded per question, and no later chunk is taken; a list of short chunks fills to the budget instead of stopping at a fixed count; `evidence_recall_20k` uses this cut; the token counts come from the pinned tokenizer. Prove they fail on the chunk-count rule, then implement. Verify: the tests pass. +- [ ] 15.14 Equal evidence by real tokens (A2c N2; A4 page credit). Write failing tests for `evidence_by_tokens`: chunks are taken in rank order while the running Qwen3 token total before a chunk is below 20,480; the chunk that crosses is truncated at 20,480 tokens, the truncation is recorded per question, and no later chunk is taken; a list of short chunks fills to the budget instead of stopping at a fixed count; `evidence_recall_20k` uses this cut with any-page credit on FinanceBench (a whole chunk covers every gold page it overlaps; the truncated chunk covers only its first page) and reports first-page credit as a sensitivity column; the token counts come from the pinned tokenizer. Prove they fail on the chunk-count rule, then implement. Verify: the tests pass. - [ ] 15.15 One round for closed-book and oracle (A2c). Extend the task 14.1 test: the largest oracle context of each dataset and every closed-book prompt synthesise in exactly one completion call. Prove it fails before the caps of task 15.12, then pass it. Verify: the test passes. - [ ] 15.16 Floor, ceiling and the miss cross-table (A2c). Write failing report tests: floor and ceiling rows are marked single-run beside 3-run averages; a per-question cross-table on retrieval misses counts closed-book correct or not against RAG correct or not, per dataset. Prove they fail, then implement. Verify: the tests pass. +## 16. Publication-review fixes (pre-freeze amendment A4, 2026-11-14; design D38) + +`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 are code. Tasks 16.23 to 16.28 are the user's decisions O1 to O6; record each answer, never guess it. Every task closes before G7. + +- [ ] 16.1 P1. FreshStack full corpus. Ask the user to approve the download of `freshstack/corpus-oct-2024`, config `langchain`, at a pinned revision. Then download it into `~/Development/DATA/omrs/` and record the revision, the row count (49,514 expected) and the corpus tree hash in `plan.json` and in the pins table (PR 16). Verify: the loader of task 13.5 accepts the corpus, and its check refuses a changed row count. +- [ ] 16.2 P2. Time one full-corpus FreshStack index build on the chosen GPU type (part of task 9.2 or 9.3 when possible). Update the estimate in PR 17 from the measured time before relying on it. Verify: the compute table and the estimate name the measured hours; the "30 to 40 pod hours" reviewer figure is no longer the basis. +- [ ] 16.3 P3. Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's own scorer. If yes, pin the scorer (version or commit) and write failing tests that a FreshStack secondary column holds it, then implement. If no, record "not available" in PR 6. Verify: PR 6 and `plan.json` say which. +- [ ] 16.4 P4. Check, per dataset, whether the qrels list judged non-relevant units. Record the answer. Then write failing report tests: the unjudged share at 10 per setup and dataset; condensed-list nDCG@10 (unjudged units removed) as a sensitivity analysis for H1 to H5 where non-relevant judgements exist; and the statement "neither can be computed" where the qrels hold positives only. Prove they fail, then implement. Verify: the tests pass and PR 21 matches the recorded answer. +- [ ] 16.5 P5. MDE pilot. Fix the pilot cell list before the pilot runs: at most 4 indexes per core dataset, the H7 corner cells, built with the pinned settings and not reused. Project the pilot cost from the P2 timing, add it to the PR 17 estimate and get the user's approval; if declined, apply the PR 10 fallback (assumed SDs 0.15 to 0.25). Draw 40 questions per core dataset from half A with seed 20261118 (after task 16.12). Run retrieval only on the full-corpus indexes, with no hosted call, after a price quote and the user's go. Run `mde35.py` (task 15.11) and fill the MDE table of PR 10. Verify: every pilot question ID is in half A, the cell list has a commit dated before the pilot, the table is filled, and the pilot pod hours are in the compute table. +- [ ] 16.6 P6. Name the smoke and test-pod answer files that feed the H8a and H8b MDE rows, with their SHA-256. Verify: PR 10 names them and marks those rows rough. +- [ ] 16.7 P7. Restate the checker 2-point tolerance against the H8a and H8b faithfulness MDE: the bar stays only if it is at most half that MDE. Verify: PR 11.1 and `protocol.md` section 4.4 give the restated rule; the "about 4 points" figure is not used. +- [ ] 16.8 P8. Write one label guideline per set (claim kinds, faithfulness, correctness, Part 2 correctness), with worked examples drawn from outside every sample. Faithfulness follows the LLM-AggreFact task definition. Fix each guideline before its first label. Verify: the guidelines are committed before any label, and no worked example is in a sample (a test of `labels35.py` checks the IDs). +- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Seed 20261120 (audit) is used only if O3 (task 16.25) is yes; confirm it then. Verify: PR 16 and `plan.json` hold the confirmed values. +- [ ] 16.10 P10. Record in `plan.json`: the tokenizer revision; the answer-model host, reasoning mode, output cap and context length (task 14.1a); the Ragas judge and classifier host and settings; the GLM-5.3 settings. Verify: each "Missing: P10" cell of PR 16 points to a `plan.json` key. +- [ ] 16.11 P11. Record in `plan.json`: the query instruction (empty string), the reranker instruction text, and the hybrid settings (RRF constant, per-side fusion depth, BM25 tokeniser and parameters). Write a failing test that the run record of a search stage holds the same values, then implement. Verify: the test passes. +- [ ] 16.12 P12. Run `split35.py` (task 15.5), write `halves.json`, record its SHA-256 in `plan.json`, and list every smoke and pre-freeze question ID with its half in the existing-data table (PR 5). Verify: the hash in `plan.json` equals the file, and every listed ID has a half. +- [ ] 16.13 P13. Record the SHA-256 of the prompt templates: answer, both closed-book prompts, oracle, classifier. Verify: `plan.json` holds them and a test recomputes each one. +- [ ] 16.14 P14. Set the `evidence_recall_20k` tolerance on another back end, with its reason, in `plan.json` `runtime.tolerance`. Verify: PR 15 names it. +- [ ] 16.15 P15. Check that no file still names `openai/gpt-6.1-terra` as a cross-check judge (A4 removed it from `plan.json`). This change never planned it. The `experiment-35-full-document-rag-benchmark` design marks its cross-check text as removed by A4. Verify: a search for `gpt-6.1-terra` finds it only in history text. +- [ ] 16.16 P16. Part 2 and E1 inputs (readiness flow group 7): own estimates and ceilings for Part 2 and for E1; RAGFlow defaults read from the pinned release, not main-branch documentation; the three-column settings table; the matched RAGFlow cell list fixed before FLOW step 7.2; the cost measured at FLOW step 7.2 and the decision at FLOW step 7.2a (at most $50 projected pod cost), recorded before any half-B score exists. Verify: the Part 2 protocol holds each item and FLOW steps 7.1, 7.2 and 7.2a are done. +- [ ] 16.17 Half-B guard (PR 14.2). Write failing tests: the scorer refuses any half-B score while `selection.json` or the OSF time-stamp record (upload time and file hash, task 16.23) is missing, and names both; the replicate pods' first stage refuses to start under the same condition; half-A scoring needs neither. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.18 Faithfulness gate and differential-error rule (PR 11.2). Write failing tests for `labels35.py`: per checker, kappa against the human label on the 450-claim sample (drawn from answers to all questions, both halves), with "can't tell" as not supported (also reported with those items left out), passes at a point estimate of at least 0.60 and a lower bound of at least 0.40 (percentile bootstrap by question, seed 20261125); a failed Clef-Flash gate marks its results not validated and changes no row; specificity per chunk size, and when the 95% interval of the GLM-5.3 2,048 minus 512 difference (seed 20261126) excludes zero in either direction, rows 12, 13, 16 and 17 become "not testable" (p = 1, task 15.7); sensitivity per chunk size is reported; the gate file records its time and the analysis refuses to compute an H8a or H8b row before it exists. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.19 Length-controlled check (PR 11.2). Write failing tests for a `lengthcheck` stage: 300 claims from `c2048` crossed answers with seed 20261119; each cited chunk split into consecutive 512-token windows, the window that shares the most distinct lower-case claim words kept (ties to the earlier window); GLM-5.3 and Clef-Flash judge the claim again; the report gives the verdict change rate with a Wilson interval; no gate. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.20 Report items (PR 21). Write failing report tests: the realised chunk-length distribution per chunk-size level and dataset; one sensitivity table set (clustered method against sensitivity check, FinanceBench page rules, missing-outcome bounds, closed-book subset, length-controlled check, A3 cell under any-page credit); the compute table (GPU type, pod hours per stage with every pre-freeze pod, hosted tokens and cost per role, energy estimate, `resent_calls`, error rows per stage and role, every `out_of_credit` stop); the returned model ID and provider per hosted call with the count of differences; every row with SESOI and MDE lines and "not testable" marks. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.21 Release kit (PR 19). Write failing tests for a re-judge script that rebuilds evidence text from the pinned dataset revisions and the released unit and chunk IDs, on a fixture; and for an export that writes ranked lists in TREC run format, never corpus text. Prove they fail, then implement. Verify: the tests pass, and the release table of PR 19 names each artefact and its licence. +- [ ] 16.22 Optional `e8b-c2048` rebuild check (PR 15). Run it on FinanceBench only when the cost projection after the search stage is at or below the estimate (a cost-only decision, recorded). Report the share of identical ranked lists and the nDCG@10 change; it is not a gate. Verify: the decision and, if run, the result are in the run record. +- [ ] 16.23 O1 (user decides). The OSF mechanism that time-stamps the A3 configuration hash (proposed: a file upload to the registered project). Record the answer in PR 14.2 and PR 23. Verify: the answer is recorded before G7. +- [ ] 16.24 O2 (user decides). The SESOI value (proposed: 0.02 nDCG@10). Record the answer in PR 10. Verify: recorded before the pilot results are read. +- [ ] 16.25 O3 (user decides). Whether the optional blind relevance audit for H3 and C1 runs (about 300 to 400 L1 labels, seed 20261120, descriptive). Record the answer. If yes, add its sample and page to `labels35.py` with tests first. Verify: the answer is recorded before G7. +- [ ] 16.26 O4 (user decides). L2's name, role and relation to the project. Record it in the run record only, never in public files. Verify: no public file holds the name. +- [ ] 16.27 O5 (user decides). The ethics query to the university and its answer, before the first label. Verify: the answer is recorded before any label. +- [ ] 16.28 O6 (user decides). BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack), or the alternative: rows 12 to 19 in the BioASQ family (16 rows) with 2-of-3 for H8. Record the answer in PR 7, 8 and 23. Verify: recorded before G7. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md index 714807be..47bce1d0 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md @@ -4,7 +4,9 @@ DRAFT. Measured execution must not begin until the protocol and plan are committed (Confirmatory) and the pipeline sweep has passed. -Source of the agreed decisions: `docs/proposals/first-publication-benchmark.md` (Draft 1, 27 October 2026). Where that document and an earlier version of this change disagree, that document wins. +Source of the agreed decisions: `docs/proposals/first-publication-benchmark.md` (Draft 1, 27 October 2026). Where that document and an earlier version of this change disagree, that document wins. From A4, `preregistration.md` wins on every design fact; the proposal stays the master plan for scope, order of work and Part 2. + +**Pre-freeze amendment A4 (2026-11-14, D0.15): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs the live design of Experiment 35.** Where it and this design differ, it wins. D0 to D0.14 and D1 to D10 are history wherever they differ from it. Text that A4 changes carries the marker "(superseded by A4, D0.15)". **Pre-freeze amendments A3 (D0.14), A2c (D0.13) and A2 (D0.12), all 2026-11-13, win over A1 where they differ; A3 wins over A2c on the selection rule, and A2c wins over A2. Pre-freeze amendment A1 (2026-11-12) wins over every older section.** Section D0 below holds the full-factorial redesign agreed with the operator on 2026-11-12. Sections D1 to D10 keep the earlier plan for history. Where they disagree with D0, D0 wins. No measured run had started when A1 was written: only smoke runs, which are never scored. @@ -31,7 +33,7 @@ Every dataset has gold passages (or gold documents or pages) and human reference | Dataset | Questions | Corpus | Gold unit | Reference answers | Licence | Status | |---|---:|---|---|---|---|---| | FinanceBench open sample (`PatronusAI/financebench`) | 150 | 84 PDFs, 12,013 pages | Document and page | Yes, plus a number-match check | CC BY-NC 4.0 | Confirmatory | -| FreshStack LangChain slice (`freshstack/queries-oct-2024`, `freshstack/corpus-oct-2024`) | 203 | LangChain documents (D0.2 note) | Documents | Nuggets, usable for context recall | CC BY-SA 4.0 | Confirmatory | +| FreshStack LangChain (`freshstack/queries-oct-2024`, `freshstack/corpus-oct-2024`) | 203 | Full public LangChain corpus, 49,514 documents (A4, D0.15) | Documents | Nuggets, usable for context recall | CC BY-SA 4.0 | Confirmatory | | BioASQ mini (`enelpol/rag-mini-bioasq`, test split) | 707 | 40,181 PubMed passages | Passages | Yes | CC BY 2.5 | Confirmatory | | Legal RAG Bench (`isaacus/legal-rag-bench`) | 100 | 4,876 passages (Victorian Criminal Charge Book) | One passage per question | Yes, written by experts | CC BY-NC-SA 4.0 | Exploratory (low power) | @@ -40,7 +42,7 @@ Total: 1,160 questions. Notes: - The HF id `PatronusAI/financebench` was checked on 2026-11-12 (150 rows, `evidence_page_num` per evidence item). The plan already used it. -- FreshStack: the earlier plan counted 223 queries. That number held 20 continuity queries from Experiment 9, which are not part of the public benchmark. A1 uses the 203 FreshStack queries only. The 15 continuity documents and the overlay runbook served only the continuity queries, so they leave the corpus (operator decision, 2026-11-13). The corpus is the public LangChain slice: 10,009 manifest rows, about 9,994 unique files after the 2 repeated upstream ids are counted once. The harness drops its overlay references (`datasets35.py` `_FS_OVERLAY`, `_FS_OVERLAY_ID` and the `continuity/` glob), and a new corpus tree hash is recorded at the freeze. +- FreshStack: the earlier plan counted 223 queries. That number held 20 continuity queries from Experiment 9, which are not part of the public benchmark. A1 uses the 203 FreshStack queries only. The 15 continuity documents and the overlay runbook served only the continuity queries, so they leave the corpus (operator decision, 2026-11-13). (Superseded by A4, D0.15: all 9 FreshStack indexes use the full public corpus of 49,514 documents at a pinned revision; the subset below is retired.) The corpus was the public LangChain slice: 10,009 manifest rows, about 9,994 unique files after the 2 repeated upstream ids are counted once. The harness drops its overlay references (`datasets35.py` `_FS_OVERLAY`, `_FS_OVERLAY_ID` and the `continuity/` glob), and a new corpus tree hash is recorded at the freeze. - Legal RAG Bench: `qa` config (100 rows: `id`, `question`, `answer`, `relevant_passage_id`) and `corpus` config (4,876 rows: `id`, `title`, `text`, `footnotes`), both split `test`. Checked on 2026-11-12. With 100 questions and one gold passage each, the minimum detectable effect is about 5 nDCG points. Its results are labelled exploratory in every table. CC BY-NC-SA: never released under CC BY, derived files under CC BY-NC-SA. - LegalBench-RAG and LLeQA are out. - FinanceBench answers also get a number-match check: the numbers in the reference answer must appear in the generated answer after unit normalisation (D0.6). @@ -71,7 +73,7 @@ Decision (operator, 2026-11-13): BioASQ and Legal passages are written as `.md`, ### D0.4 Hypotheses (pre-registered) -Unit: the question. A factor's marginal score for a question is its mean nDCG@10 over every cell of the other factors at that level. For example, the 4B marginal averages the 24 setups with the 4B embedder. Each contrast is a paired permutation test (10,000 shuffles, seed 20261101) on per-question marginal scores, with a paired 95% bootstrap interval. Holm correction runs across the 11 primary contrasts within one dataset. A contrast counts when the Holm-adjusted p is below 0.05, the interval excludes zero, and the sign matches the stated direction. +(Superseded by A4, D0.15: rows, tests and Holm families are in `preregistration.md` sections 7 to 9. This paragraph and the table are A1 history.) Unit: the question. A factor's marginal score for a question is its mean nDCG@10 over every cell of the other factors at that level. For example, the 4B marginal averages the 24 setups with the 4B embedder. Each contrast is a paired permutation test (10,000 shuffles, seed 20261101) on per-question marginal scores, with a paired 95% bootstrap interval. Holm correction runs across the 11 primary contrasts within one dataset. A contrast counts when the Holm-adjusted p is below 0.05, the interval excludes zero, and the sign matches the stated direction. | Id | Statement | Contrast(s) | |---|---|---| @@ -83,9 +85,9 @@ Unit: the question. A factor's marginal score for a question is its mean nDCG@10 | H6 (superseded by D0.12 and D0.13: confirmatory on FinanceBench and FreshStack only; the short-set clause is withdrawn) | Chunk size changes search quality on long-document sets (FinanceBench, FreshStack), not on short ones (BioASQ median passage about 364 tokens, Legal passages at most 512) | 1,024 vs 512; 2,048 vs 1,024. Holds when at least one contrast counts on each long set, and on each short set both paired intervals lie inside plus or minus the minimum detectable effect | | H7 | Bigger embedders gain more from bigger chunks (embedder x chunk size interaction) | Per question: (8B at 2,048 minus 8B at 512) vs (0.6B at 2,048 minus 0.6B at 512), paired test of the two differences | -BioASQ and Legal act as a built-in control for H6. Legal results are exploratory (D0.2). Minimum detectable effect, stated before the run with `min_detectable_effect`: about 4 points at 150 and 203 questions, about 2 at 707, about 5 at 100. +BioASQ and Legal act as a built-in control for H6. Legal results are exploratory (D0.2). Minimum detectable effect, stated before the run with `min_detectable_effect`: about 4 points at 150 and 203 questions, about 2 at 707, about 5 at 100. (Superseded by A4, D0.15: the MDE table comes from the pilot, `preregistration.md` section 10.) -Code: `paired_permutation`, `paired_bootstrap`, `holm`, `wilson` and `min_detectable_effect` from the existing statistics code (`experiments/_lib/stats.py`, `stats_d6.py`). No new statistics code. The marginal averaging is data preparation, not a new statistic. +Code: `paired_permutation`, `paired_bootstrap`, `holm`, `wilson` and `min_detectable_effect` from the existing statistics code (`experiments/_lib/stats.py`, `stats_d6.py`). No new statistics code. The marginal averaging is data preparation, not a new statistic. (Superseded by A4, D0.15: the readiness change adds tested statistics code, `stats_cluster.py`.) ### D0.5 Answer stage @@ -97,17 +99,17 @@ Code: `paired_permutation`, `paired_bootstrap`, `holm`, `wilson` and `min_detect The rule runs on saved search scores on the Mac. Its inputs and result go into `protocol.md` as a dated note before the replicate pods start. -**Answer setups.** At the chosen embedder and reranker size: chunk size (3) x reranker on or off x hybrid on or off = 12 setups. Each runs 3 times on 1,160 questions: 41,760 answers. +**Answer setups.** At the chosen embedder and reranker size: chunk size (3) x reranker on or off x hybrid on or off = 12 setups. Each runs 3 times on 1,160 questions: 41,760 answers. (Amended by A2 and A3: 15 setups.) -Fixed: answer model `deepseek/deepseek-v4.1-flash-1017` (OpenRouter, one pinned host, temperature 0, seed 20261101); Ragas 0.4.3 judge `openai/gpt-oss-120b`; claim checkers GLM-5.3 (main, Z.AI Coding Plan endpoint as in `arms.py`) and Clef-Flash (second, local). Cross-check judge `openai/gpt-6.1-terra` on 50 hand-checked items as before. +Fixed: answer model `deepseek/deepseek-v4.1-flash-1017` (OpenRouter, one pinned host, temperature 0, seed 20261101); Ragas 0.4.3 judge `openai/gpt-oss-120b`; claim checkers GLM-5.3 (main, Z.AI Coding Plan endpoint as in `arms.py`) and Clef-Flash (second, local). No cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate included it (A4, D0.15). **Refusal rule (kept, no longer tied to H-CC).** `openai/gpt-oss-120b` sorts each claim as fact, refusal or mixed. Primary faithfulness uses factual claims (fact and mixed). Refusals are reported apart. The classifier gate stays: agreement of at least 0.90 and refusal recall of at least 0.90 on 100 operator-labelled claims, seed 20261112. The GLM-5.3 against Clef-Flash kappa report stays. -**Answer-stage hypothesis H8.** Chunk size changes faithfulness and correctness. Expected direction: faithfulness falls as chunks grow. Wang et al. (EMNLP 2024) report faithfulness 97.6 at 512, 94.3 at 1,024 and 80.4 at 2,048 tokens. Contrasts per dataset: 1,024 vs 512 and 2,048 vs 1,024, marginal over reranker and hybrid, with the 3 runs averaged per question first. Outcomes: GLM-5.3 faithfulness on factual claims; Ragas answer correctness on all four datasets; FinanceBench number match. Holm across the H8 contrasts within one dataset. +**Answer-stage hypothesis H8.** (Superseded by A4, D0.15: H8 splits into H8a and H8b, Legal RAG Bench is exploratory, and the Holm families are fixed in `preregistration.md` section 8.) Chunk size changes faithfulness and correctness. Expected direction: faithfulness falls as chunks grow. Wang et al. (EMNLP 2024) report faithfulness 97.6 at 512, 94.3 at 1,024 and 80.4 at 2,048 tokens. Contrasts per dataset: 1,024 vs 512 and 2,048 vs 1,024, marginal over reranker and hybrid, with the 3 runs averaged per question first. Outcomes: GLM-5.3 faithfulness on factual claims; Ragas answer correctness on all four datasets; FinanceBench number match. Holm across the H8 contrasts within one dataset. **Answer correctness on all four datasets (operator decision, 2026-11-13).** The Ragas stage adds a fifth metric: `AnswerCorrectness` from `ragas.metrics.collections`, Ragas 0.4.3 (the version the harness pins), called as `ascore(user_input, response, reference)` with each dataset's human reference answer. It uses the frozen Ragas judge `openai/gpt-oss-120b` and the local 4B embedder for its similarity part, with the library's default weights (factual 0.75, semantic similarity 0.25), recorded in the protocol. References: FinanceBench `answer`, FreshStack reference answer, BioASQ `answer`, Legal RAG Bench `answer`. The FinanceBench number match stays as a second, deterministic check. -**Correctness validation before the metric counts.** The operator hand-labels a stratified, seeded sample of 100 answers (seed 20261113): 25 per dataset, spread evenly over the three chunk sizes, blind to the Ragas score, as correct or not correct against the reference. The Ragas score is made binary at a threshold fixed before labelling (0.5). The report gives Cohen's kappa between judge and operator, per dataset and pooled. Rule fixed before labelling: with pooled kappa of at least 0.60, answer correctness is a reported H8 outcome; below 0.60, it is reported as exploratory only and H8 rests on faithfulness and the FinanceBench number match. Labelling uses answers from the pre-freeze test pod session, not from the scored run. +**Correctness validation before the metric counts.** (Superseded by A2 M4 and A4, D0.15: 200 main-run answers, 50 per dataset, and a gate per core dataset on the kappa point estimate and lower bound.) The operator hand-labels a stratified, seeded sample of 100 answers (seed 20261113): 25 per dataset, spread evenly over the three chunk sizes, blind to the Ragas score, as correct or not correct against the reference. The Ragas score is made binary at a threshold fixed before labelling (0.5). The report gives Cohen's kappa between judge and operator, per dataset and pooled. Rule fixed before labelling: with pooled kappa of at least 0.60, answer correctness is a reported H8 outcome; below 0.60, it is reported as exploratory only and H8 rests on faithfulness and the FinanceBench number match. Labelling uses answers from the pre-freeze test pod session, not from the scored run. **Answer context budget: one round (operator decision, 2026-11-13).** At 2,048 tokens and `top_k` 10, the evidence is about 20,000 Qwen3 tokens, at most about 92,000 characters (10 chunks of up to about 9,200 characters). OMRS packs evidence into at most `answer.max_rounds` rounds of `answer.context_window` characters, minus 2,048 reserved for the reply and 1,536 for the template (`src/omrs/core/answer/synthesis.py:169-189`). A1 sets `answer.context_window` to **131,072 characters** and `answer.max_rounds` to **1** in every answer setup. The round budget is then 127,488 characters, enough for every setup's evidence in one completion call, with no refine round. The prompt is then about 35,000 tokens at most. A pre-freeze task reads the context length of `deepseek/deepseek-v4.1-flash-1017` from the OpenRouter models API on the pinned host and records it; the freeze is blocked unless it is at least 65,536 tokens (prompt plus the 2,048-token reply, with margin). A test fails if any answer setup's largest evidence set needs more than one round. The harness still records per answer the rounds, the evidence characters and any `[evidence truncated]` marker; truncation is data, never silent. @@ -132,14 +134,14 @@ A tested function extracts the numbers from the reference answer and from the ge ### D0.9 Budget and stop rule -The $250 cap is gone. The operator has about £2,000 to £3,000. The estimate (method in the readiness change, design D23, on an RTX 4090 at $0.89 per hour) is about **$650** (final figures after A2c, D0.13): about $354 for about 382 pod hours and the network volume, and about $298 for hosted calls (answers about $155 incl. the token-matched and closed-book runs, Ragas about $122 with answer correctness, claim classifier about $11; GLM-5.3 on the Z.AI Coding Plan has no per-token cost). **Ceiling: $975 (1.5 x the estimate).** The largest unknown is reranker speed (llama.cpp issue #30422): at 3 times slower than assumed the total is about $870; at 6 times slower it passes the ceiling and the stop rule triggers before the freeze. +(Superseded by A3 and A4, D0.15: estimate about $730, ceiling $975, results-blind drop order.) The $250 cap is gone. The operator has about £2,000 to £3,000. The estimate (method in the readiness change, design D23, on an RTX 4090 at $0.89 per hour) is about **$650** (final figures after A2c, D0.13): about $354 for about 382 pod hours and the network volume, and about $298 for hosted calls (answers about $155 incl. the token-matched and closed-book runs, Ragas about $122 with answer correctness, claim classifier about $11; GLM-5.3 on the Z.AI Coding Plan has no per-token cost). **Ceiling: $975 (1.5 x the estimate).** The largest unknown is reranker speed (llama.cpp issue #30422): at 3 times slower than assumed the total is about $870; at 6 times slower it passes the ceiling and the stop rule triggers before the freeze. ### D0.9a GPU choice and parallel index builds (operator instruction, 2026-11-12) - **GPU choice.** Before the freeze, the same fixed job of about 10 minutes runs on an RTX 4090 (24 GB), an L40S (48 GB) and an H100 (80 GB): embed 2,000 fixed FinanceBench and FreshStack chunks at 1,024 and 2,048 Qwen3 tokens with Qwen3-Embedding-8B Q8_0, and rerank a fixed set of query-passage pairs with Qwen3-Reranker-8B Q8_0. Each run records chunks per second, pairs per second, peak VRAM, whether the 8B embedder and the 8B reranker load together without swapping, the llama.cpp build, the driver and the image digest. The live price and stock are re-checked before each pod, and the operator gives the go. - **Decision rule, written into `protocol.md` before the first measurement.** Among the GPUs that hold the 8B embedder and the 8B reranker together without swapping, choose the lowest projected total GPU cost for the main build and search work (9 index builds per dataset plus 72 search setups). When two GPUs are within 10% on cost, choose the faster one. The protocol records the measurements, prices and choice. - **One GPU type.** Every main-run pod (index build pods, search and replicate pods) uses that one GPU type, and every stage runs on it. GPU types are never mixed. The earlier 4090 pre-freeze runs were preparation, not results. -- **Parallel builds.** The 9 indexes are independent. They build on 9 parallel pods of the chosen GPU type, each into its own store on the RunPod network volume; each pod seals its index and runs its 8 search setups. On a 4090 this takes about 18 h of wall time instead of about 133 h on one pod, for about the same pod hours. The three replicate pods start after the selection rule. End to end, about 4 days of pod wall time. Spending is checked at three points: before the freeze (test pod extrapolation), on the build pod after the search stage (before the replicate pods), and while the replicate pods run (pod billing plus provider spend). When the projected or actual total reaches the ceiling, the run stops before the next stage and the operator decides. The OpenRouter key gets a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. +- **Parallel builds.** The 9 indexes are independent. They build on 9 parallel pods of the chosen GPU type, each into its own store on the RunPod network volume; each pod seals its index and runs its 8 search setups. On a 4090 this takes about 18 h of wall time instead of about 133 h on one pod, for about the same pod hours. The three replicate pods start after the selection rule. End to end, about 4 days of pod wall time. Spending is checked at three points: before the freeze (test pod extrapolation), on the build pod after the search stage (before the replicate pods), and while the replicate pods run (pod billing plus provider spend). When the projected total reaches the ceiling, the results-blind drop order of `preregistration.md` section 17 applies (A4, D0.15); no operator choice decides what is dropped. The OpenRouter key gets a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. ### D0.10 Fact checks (2026-11-12) @@ -170,19 +172,19 @@ An independent methods review ran on 2026-11-13. The operator approved every fix 3. Keep the first occurrence of each unit, in rank order. 4. Cut at 10 units. nDCG@10, Recall@10, MRR@10 and hit@page are computed on these 10 units. A list with fewer than 10 distinct units is scored as it is (missing ranks count as misses). -FinanceBench page credit: a chunk counts for its **first page** (the lowest page label it carries). Reason: the rule is fixed per chunk and does not depend on the gold label. The current code gives a spanning chunk the gold page when any page matches (`scoring35.py:49-50`), which favours big chunks again. The token-share rule needs per-page token offsets that the reader does not store. Every chunk's other pages are recorded, and a sensitivity table reports hit@page with "any page" credit. +FinanceBench page credit: (for `evidence_recall_20k`, superseded by A4, D0.15: any-page credit is primary there and first-page credit is a sensitivity; nDCG@10 keeps first-page credit) a chunk counts for its **first page** (the lowest page label it carries). Reason: the rule is fixed per chunk and does not depend on the gold label. The current code gives a spanning chunk the gold page when any page matches (`scoring35.py:49-50`), which favours big chunks again. The token-share rule needs per-page token offsets that the reader does not store. Every chunk's other pages are recorded, and a sensitivity table reports hit@page with "any page" credit. **Token-matched control.** Equal evidence text: 512 tokens x 40 chunks, 1,024 x 20 and 2,048 x 10 (about 20,480 tokens each). Search stage: a secondary metric, evidence recall at 20,480 tokens (the share of gold units present in the first N chunks, N = 20,480 / chunk size), from the same depth-100 lists, so no extra retrieval. Answer stage: two extra setups (D0.12 M5). -**B2. H6 confirmatory on long-document sets only (blocker).** H6 is now: chunk size changes unit-level nDCG@10 on FinanceBench and FreshStack. It holds when at least one chunk-size contrast counts on both datasets. BioASQ results on chunk size are a reported sanity check, not evidence. Legal RAG Bench is exploratory throughout and is removed from every confirmatory hypothesis. +**B2. H6 confirmatory on long-document sets only (blocker).** (Rows, endpoints and the holds rule: `preregistration.md` section 7, A4, D0.15.) H6 is now: chunk size changes unit-level nDCG@10 on FinanceBench and FreshStack. It holds when at least one chunk-size contrast counts on both datasets. BioASQ results on chunk size are a reported sanity check, not evidence. Legal RAG Bench is exploratory throughout and is removed from every confirmatory hypothesis. -**M1. Minimum detectable effect from data.** Before the freeze, compute the standard deviation of per-question paired differences per dataset and per contrast, from Experiment 22 (FreshStack, per-query nDCG@10) and the pre-freeze smoke and test-pod data (all four datasets). Compute the MDE with `min_detectable_effect` at the Holm-adjusted alpha of the worst rank (0.05 / the number of contrasts per dataset), power 0.80. Separate power checks: the H7 interaction (difference of differences) and any equivalence claim (TOST). After B2 no equivalence claim remains in a confirmatory hypothesis, so TOST power is computed only if one is added. The numbers and their sources go into `protocol.md` before the freeze. The earlier MDE figures (about 4, 2 and 5 points) were guesses and are withdrawn. +**M1. Minimum detectable effect from data.** (Superseded by A4, D0.15: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the MDE table; SESOI 0.02 nDCG@10 waits for the operator.) Before the freeze, compute the standard deviation of per-question paired differences per dataset and per contrast, from Experiment 22 (FreshStack, per-query nDCG@10) and the pre-freeze smoke and test-pod data (all four datasets). Compute the MDE with `min_detectable_effect` at the Holm-adjusted alpha of the worst rank (0.05 / the number of contrasts per dataset), power 0.80. Separate power checks: the H7 interaction (difference of differences) and any equivalence claim (TOST). After B2 no equivalence claim remains in a confirmatory hypothesis, so TOST power is computed only if one is added. The numbers and their sources go into `protocol.md` before the freeze. The earlier MDE figures (about 4, 2 and 5 points) were guesses and are withdrawn. -**M2. One mixed model per dataset, then planned contrasts.** Per dataset, one linear mixed model: `ndcg10 ~ C(embedder) * C(chunk) * C(reranker) * C(hybrid)` with a random intercept per question, fitted with `statsmodels` `MixedLM` (REML), the version pinned in `uv.lock` (a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13)). The planned contrasts of H1 to H7 are estimated from the model as follow-ups. Holm runs across every H1 to H8 contrast of one dataset. The paired permutation test and paired bootstrap stay as a robustness check. FinanceBench bootstraps resample whole documents (`paired_bootstrap(groups=doc_name)`, `experiments/_lib/stats_d6.py:140-160`). Cross-dataset rule: a hypothesis holds overall when it is supported on at least 2 of the 3 confirmatory datasets (FinanceBench, FreshStack, BioASQ); H6 needs both long-document sets. Legal is exploratory and never counts. +**M2. One mixed model per dataset, then planned contrasts.** (Superseded by A2c and A4, D0.15: no mixed model; per-question contrast scores with GEE or the FinanceBench document-level sign-flip test; Holm families fixed at 19, 19 and 8 after the second round.) Per dataset, one linear mixed model: `ndcg10 ~ C(embedder) * C(chunk) * C(reranker) * C(hybrid)` with a random intercept per question, fitted with `statsmodels` `MixedLM` (REML), the version pinned in `uv.lock` (a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13)). The planned contrasts of H1 to H7 are estimated from the model as follow-ups. Holm runs across every H1 to H8 contrast of one dataset. The paired permutation test and paired bootstrap stay as a robustness check. FinanceBench bootstraps resample whole documents (`paired_bootstrap(groups=doc_name)`, `experiments/_lib/stats_d6.py:140-160`). Cross-dataset rule: a hypothesis holds overall when it is supported on at least 2 of the 3 confirmatory datasets (FinanceBench, FreshStack, BioASQ); H6 needs both long-document sets. Legal is exploratory and never counts. -**M3. Split-half selection.** Each dataset's questions are split into halves A and B with seed 20261115 (stratified by dataset). The embedder and reranker size for the answer stage are chosen on half A only, with the D0.5 rule. All answer-stage headline results are reported on half B. The answer stage still runs on all questions; half A results are secondary. Search-stage hypotheses use all questions. +**M3. Split-half selection.** (Amended by A4, D0.15: FinanceBench is split by document; the others by question.) Each dataset's questions are split into halves A and B with seed 20261115 (stratified by dataset). The embedder and reranker size for the answer stage are chosen on half A only, with the D0.5 rule. All answer-stage headline results are reported on half B. The answer stage still runs on all questions; half A results are secondary. Search-stage hypotheses use all questions. -**M4. Human validation.** Two labellers, recorded as L1 (the operator) and L2 (the second labeller). No names in public records. +**M4. Human validation.** (Superseded by A4, D0.15: L1 labels 750 items, L2 225; gates and samples in `preregistration.md` section 11.) Two labellers, recorded as L1 (the operator) and L2 (the second labeller). No names in public records. | Set | Size (L1) | Drawn from | Stratified by | L2 share | |---|---:|---|---|---:| @@ -195,13 +197,13 @@ Reports: Cohen's kappa of GLM-5.3 and of Clef-Flash against L1 on faithfulness, **M5. Answer stage: chunk size is confounded with context length.** Chunk-size effects in the answer stage are reported as "chunk size / context length". Two token-matched answer setups join the 12: 512 x 40 and 1,024 x 20 chunks, at the chosen reranker size, reranker on and hybrid on (fixed now, not chosen from data). They are compared with `c2048` x 10 at the same reranker and hybrid levels (amended by A3, D0.14: the comparison uses a separate `c2048-t20k` setup). Total: 14 answer setups (15 after A3). GLM-5.3 against Clef-Flash agreement is reported per chunk size. -**M6. Contamination check.** One closed-book run (no retrieval, same answer model and settings, one run) on all 1,160 questions, scored with `AnswerCorrectness` and the FinanceBench number match. The report splits RAG answer correctness by retrieval hit versus miss (gold unit in the evidence or not), beside the closed-book correctness. +**M6. Contamination check.** (Amended by A4, D0.15: a second closed-book run uses the RAG prompt with empty evidence; rows 14, 15, 18 and 19 are re-run on questions both closed-book runs answer wrongly.) One closed-book run (no retrieval, same answer model and settings, one run) on all 1,160 questions, scored with `AnswerCorrectness` and the FinanceBench number match. The report splits RAG answer correctness by retrieval hit versus miss (gold unit in the evidence or not), beside the closed-book correctness. **M7. Provenance of hosted calls.** Every hosted call (answers, Ragas, GLM-5.3, classifier) records the model id and provider that the reply returns, beside the requested ones. A difference is reported. Raw verdicts (GLM-5.3, Clef-Flash, classifier, Ragas) are released where the dataset licence allows. The local Clef-Flash rerun is named as the path a reader can reproduce without a hosted model. **M8. Reranker pins and a rank check.** Every reranker GGUF is pinned in the `plan.json` runtime block with its SHA-256 before the freeze. On the test pod, each size is checked against the model card's transformers recipe on 200 seeded query-passage pairs (seed 20261117): Spearman rank correlation of the scores at least 0.95. A file below the threshold is not pinned. -**M9. Public pre-registration.** At the freeze commit (G7), the operator deposits `protocol.md` and `plan.json` on OSF as a public pre-registration. The deposit carries no personal data beyond the operator name already allowed in records. +**M9. Public pre-registration.** (Amended by A4, D0.15: `preregistration.md` is the main file of the OSF deposit.) At the freeze commit (G7), the operator deposits `protocol.md` and `plan.json` on OSF as a public pre-registration. The deposit carries no personal data beyond the operator name already allowed in records. **Answer model (decided by the operator, 2026-11-13).** The answer model stays `deepseek/deepseek-v4.1-flash-1017` on the OpenRouter API. Reason: the open-weights DeepSeek V4.1 Flash has 552B parameters, about 508 GB at 8-bit, which no single pod can host; smaller open models answer worse; the API costs about $127 for all answers. A single answer model is a stated limitation. Oracle retrieval was not added at this point; the operator approved an oracle run later the same day (see the oracle paragraph and D0.13). @@ -268,7 +270,7 @@ Experiment 35 becomes that experiment. - Hit rule: a retrieved chunk is a hit when its source is the gold `evidence_doc_name` and its page is the gold page. `evidence_page_num` is zero-indexed. One function converts it to the page numbering that OMRS stores, with a unit test on a known page. A chunk that spans pages is a hit when any of its pages is the gold page. A result row's pages are its `page_labels` (a merged row lists every page it spans, from `add-merged-result-page-labels`), with `page_label` as the fallback for a row without them. A question with several evidence entries is a hit at K when any gold page is hit within the top K. - Licence CC BY-NC 4.0. Keep the PDFs and derived text out of git and out of any CC BY release. -### FreshStack, LangChain slice +### FreshStack, LangChain slice (history; superseded by A4, D0.15: full public corpus) - The slice exported by `prepare_freshstack.py` with seed 20260530: about 10,024 parent documents, all 223 queries, the qrels of Experiments 9a and 22. Indexing is the main cost; a smaller query set only widens the intervals. - Build a new index for each arm. Experiment 22 does not count as a baseline. It used default chunking (32,631 chunks), `qwen/qwen3-embedding-4b` through OpenRouter, hybrid RRF k=60, rerank off and top_k 50. Report it beside the result as a reference and name the settings that differ. Experiment 9a used one vector per parent document, so its numbers are not comparable. @@ -323,7 +325,7 @@ Notes: - Ragas 0.4.3: faithfulness, answer relevancy, context precision, context recall. - Main Ragas judge (operator, 2026-11-08): `openai/gpt-oss-120b` through OpenRouter, a hosted reference (ADR-103), reasoning `low`, temperature 0, fixed seed, one pinned upstream host that honours `seed`. Different family from the answer model. Evidence: Judge's Verdict (arXiv 2510.09738), kappa 0.756, human-like tier. -- Cross-check judge: `openai/gpt-6.1-terra` through OpenRouter on the 50 hand-checked items. It takes a fixed seed but no temperature setting. It is the same vendor family as the main judge, so it checks run-to-run and model-size effects, not family bias. The hand check is the independent evidence. Report agreement between the main judge, the cross-check judge and the hand check. +- (Removed by A4, D0.15; no cross-check judge runs.) Cross-check judge: `openai/gpt-6.1-terra` through OpenRouter on the 50 hand-checked items. It takes a fixed seed but no temperature setting. It is the same vendor family as the main judge, so it checks run-to-run and model-size effects, not family bias. The hand check is the independent evidence. Report agreement between the main judge, the cross-check judge and the hand check. - Hand-check agreement measure (fixed before the check): the operator scores each of the 50 items blind to the judge, per metric, as supported or not supported. Agreement is Cohen's kappa between the operator and each judge, with a 95% bootstrap interval over items, after the judge score is cut at 0.5. Fit for use: kappa 0.6 or higher (substantial). Below that, the report keeps the Ragas numbers and states the shortfall. ### Faithfulness claim check (OMRS answer verification) @@ -356,7 +358,7 @@ Notes: - The 3 runs measure the stages that can change: the generated answer, the Ragas judge and latency. The rebuild check covers index building. Report the run-to-run standard deviation for each. - Tests compare arms question by question. Average the 3 runs for each question first. -## D6. Statistics (basic set) +## D6. Statistics (basic set; history, superseded by A2c and A4, D0.15: see `preregistration.md` section 9) For each dataset and arm: @@ -452,11 +454,11 @@ Resolved on 2026-10-27 by the publication proposal: datasets and sizes, the base **Pre-freeze amendment A2c (2026-11-13): second methods review.** It wins over D0.12 where they differ. - **N1. Rerank pool stays 100.** With `top_k` 100 for depth-100 scoring, OMRS would size the rerank pool as max(`rerank_max_fetch`, `top_k` x `rerank_fetch_multiplier`) = max(100, 300) = 300 (`src/omrs/core/retrieval/policy.py:63-67`, `settings.py:61-62`). The harness passes `fetch_k=100` to `Engine.search` (it reaches the existing `fetch_k_override`, `pipeline.py:279-284`), so every reranker setup scores a pool of exactly 100 and returns all 100 in order. Dense and hybrid setups also fetch 100. -- **N2. Equal evidence by real tokens.** The equal-evidence metric `evidence_recall_20k` and the equal-text answer setups no longer cut by chunk count. They take chunks in rank order and add each chunk's real Qwen3 token count to a running total; a chunk is included when the total before it is below 20,480; the chunk that crosses the limit is included whole (so the evidence is at most one chunk over 20,480), and no later chunk is. Short documents and short last chunks therefore no longer under-fill the budget. The setup names become `c512-t20k` and `c1024-t20k`; the `c2048` x 10 comparison is cut by the same rule. -- **N3. H6 endpoint on FinanceBench.** First-page credit penalises big chunks. H6 on FinanceBench is tested on `evidence_recall_20k` (gold pages within the 20,480-token evidence), its confirmatory endpoint for H6. nDCG@10 with first-page credit and with any-page credit are sensitivity results. H6 counts on FinanceBench only if the 20k measure supports it, and the report gives whether the two page rules agree in sign per contrast. FreshStack keeps unit-level nDCG@10 for H6. -- **N4. p-values from cluster-robust errors.** The mixed model with a random intercept per question stays for estimates and description. Each planned contrast's p-value comes from a GEE with the same mean model (`statsmodels` `GEE`, Gaussian, exchangeable working correlation) and cluster-robust standard errors clustered by question; on FinanceBench clustered by document. Holm runs across the contrasts of each dataset. The paired permutation test, or on FinanceBench the document-grouped bootstrap, is the pre-registered sensitivity check. +- **N2. Equal evidence by real tokens.** The equal-evidence metric `evidence_recall_20k` and the equal-text answer setups no longer cut by chunk count. They take chunks in rank order and add each chunk's real Qwen3 token count to a running total; a chunk is included when the total before it is below 20,480; the chunk that crosses the limit is truncated at the limit (20,480 is a hard limit, and the truncation is recorded per question), and no later chunk is included. Short documents and short last chunks therefore no longer under-fill the budget. The setup names become `c512-t20k` and `c1024-t20k`; the `c2048` x 10 comparison is cut by the same rule. +- **N3. H6 endpoint on FinanceBench.** First-page credit penalises big chunks. H6 on FinanceBench is tested on `evidence_recall_20k` (gold pages within the 20,480-token evidence), its confirmatory endpoint for H6 (A4, D0.15: with any-page credit; first-page credit is a sensitivity). nDCG@10 with first-page credit and with any-page credit are sensitivity results. H6 counts on FinanceBench only if the 20k measure supports it, and the report gives whether the two page rules agree in sign per contrast. FreshStack keeps unit-level nDCG@10 for H6. +- **N4. p-values from cluster-robust errors.** (Superseded by A4, D0.15: no mixed model; the analysis is `preregistration.md` section 9.) The mixed model with a random intercept per question stays for estimates and description. Each planned contrast's p-value comes from a GEE with the same mean model (`statsmodels` `GEE`, Gaussian, exchangeable working correlation) and cluster-robust standard errors clustered by question; on FinanceBench clustered by document. Holm runs across the contrasts of each dataset. The paired permutation test, or on FinanceBench the document-grouped bootstrap, is the pre-registered sensitivity check. - **N5. H8 on all questions.** The selection rule averages over chunk size, so choosing on half A barely biases the chunk-size contrasts. H8 is tested on all questions. Descriptive headline answer numbers stay on half B. -- **Minor.** Only nDCG@10 is primary (the tolerance line names the others as secondary). FreshStack's reference answer is `answer_text` of `freshstack/queries-oct-2024`, config `langchain` (the accepted Stack Overflow answer). Oracle evidence is capped at 20,480 Qwen3 tokens for every dataset (FinanceBench, BioASQ and FreshStack; a Legal passage is at most 512), with truncation recorded. The one-round test covers the oracle and closed-book stages too. Floor (closed-book) and ceiling (oracle) are single runs, while main setups average 3 runs; the report says so. The report adds a per-question cross-table on retrieval misses: closed-book correct or not against RAG correct or not. Oracle retrieval is now part of the plan (an oracle run, approved 2026-11-13). +- **Minor.** Only nDCG@10 is primary (the tolerance line names the others as secondary; A4, D0.15: nDCG@10 stays the primary metric of the study; `evidence_recall_20k` is the confirmatory endpoint for H6 and H7 on FinanceBench, so `plan.json` flags it `primary` too). FreshStack's reference answer is `answer_text` of `freshstack/queries-oct-2024`, config `langchain` (the accepted Stack Overflow answer). Oracle evidence is capped at 20,480 Qwen3 tokens for every dataset (FinanceBench, BioASQ and FreshStack; a Legal passage is at most 512), with truncation recorded. The one-round test covers the oracle and closed-book stages too. Floor (closed-book) and ceiling (oracle) are single runs, while main setups average 3 runs; the report says so. The report adds a per-question cross-table on retrieval misses: closed-book correct or not against RAG correct or not. Oracle retrieval is now part of the plan (an oracle run, approved 2026-11-13). - **Cost.** N1 to N5 add no pod hours or calls (the pool stays 100). Final estimate: about **$650** (pods and network volume about $354 for about 382 pod hours, hosted calls about $298), ceiling **$975** (1.5 x). ### D0.14 Pre-freeze amendment A3: selection rule and hand-off to Part 2 (2026-11-13) @@ -468,5 +470,25 @@ Decision (operator, 2026-11-13). The master plan is `docs/proposals/first-public - **Hand-off.** Part 2 runs on half B only, once per system. Experiment 35 does not run Part 2, does not run RAGFlow, and does not draw labels from RAGFlow answers. Part 2 has its own protocol (C1, C2, fairness rules, labels). - **Token-matched control (PR 209 review, 2026-11-13).** The factorial cell `c2048` x 10 keeps its 10-chunk evidence. A separate 15th answer setup, `c2048-t20k`, cuts evidence at 20,480 real Qwen3 tokens at 2,048-token chunks, reranker on and hybrid on, beside `c512-t20k` and `c1024-t20k`. Each chunk size has a token-matched cell, and the token-matched comparison of H8 uses the three `-t20k` setups. Registered before the freeze. It amends D0.13 N2, which cut the `c2048` comparison by the same rule. Cost: the answer stage grows by about 1/14 (about 7%). Counted from the readiness design D35 table (two token-matched setups: +36 pod hours, +$47 hosted), one more setup adds about $24 of hosted calls (answers, five Ragas metrics, judges and classifier) and about 18 pod hours (about $16 at $0.89 per hour, about 6 hours per replicate pod), so about $40 in all. The estimate moves from about $650 to about $690. The ceiling stays $975 (1.5 x the old estimate, 1.4 x the new one). At 3 times slower reranking the total is about $910, still under it. - **FinanceBench unit in Part 2 (2026-11-13).** Part 2 scores C1 at page level for every FinanceBench question and both systems, the unit used here (first page of a chunk). Pre-freeze gate and document-level fallback are in the master plan, Step 3 rule 3, and protocol section 0.12 item 4. -- **Counts.** The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each. The 3 runs and the 400 L1 labels stay as in D0.12. +- **Counts.** The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each. The 3 runs stay as in D0.12. (Labels superseded by A4, D0.15: 750 L1 and 225 L2.) - **Framework comparison.** Out of Experiment 35 scope. Part 2 holds it. + +### D0.15 Pre-freeze amendment A4: publication review fixes (2026-11-14) + +Decision (operator, 2026-11-14). A publication-grade methods review gave 30 findings, F1 to F30. The operator approved every recommended fix. Detail: `protocol.md` section 0.13 and `plan.json` amendment A4. + +- **Governing document.** `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` holds the live design of Experiment 35. It wins over this change, `protocol.md` and `plan.json` wherever they differ. It is the main file of the OSF deposit. The proposal stays the master plan for scope, order of work and Part 2. For Part 2 and E1, `preregistration.md` section 18 wins over the proposal on the points it fixes; the frozen Part 2 protocol governs the rest after its freeze. +- **History rule.** D0 to D0.14 and D1 to D10 stay as history. Text that A4 changes carries the marker "(superseded by A4, D0.15)" or "(amended by A4, D0.15)". Unmarked text that disagrees with `preregistration.md` is an error to fix, not a rule. +- **What changes in this change (finding ids from the review):** + - F1: all 9 FreshStack indexes use the full public LangChain corpus (49,514 documents, pinned revision). The 10,000-document subset is retired. alpha-nDCG@10 is secondary if the nugget labels allow it. + - F4, F7, F10: no mixed model. Per-question contrast scores. On FreshStack, BioASQ and Legal the decision test is defined by its closed form (mean; Mancl and DeRouen bias-reduced variance; t on n minus 1), with `statsmodels` GEE as the implementation checked against it (`preregistration.md` section 9 point 3); and a document-level sign-flip test on FinanceBench. Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; second round C3, pending open item O6). A gated row stays in its family with p = 1. No equivalence claim, no TOST, no McNemar test as a main test. + - F5, F22, F23: H8 splits into H8a (crossed setups) and H8b (token-matched setups). Both are confirmatory. Every hypothesis has scope S, numbered rows and a holds rule. H7 is confirmatory on FinanceBench and FreshStack. After the second round (C3, C4), H6, H7, H8a and H8b hold only on FinanceBench and FreshStack, and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs. Legal RAG Bench is exploratory in every hypothesis. + - F3, F8, F19: L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 225. Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. + - F12: `evidence_recall_20k` gives any-page credit and is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11). nDCG@10 stays the primary metric of the study. First-page credit is a sensitivity. + - F13: a second closed-book run uses the RAG prompt with empty evidence. + - F14, F21: FinanceBench is split by document. Order: score half A, write `selection.json`, time-stamp the configuration hash on OSF, then score half B. The RQ1 retrieval headline is the A3 cell on half B. "Sealed" applies to indexes only. + - F15: the token cut truncates the crossing chunk at 20,480 tokens (D0.13 N2 now says so). The `openai/gpt-6.1-terra` cross-check is removed. `plan.json` `frozen` stays empty until the freeze; the pins table (`preregistration.md` section 16) fills it. + - F17: estimate about $730 plus the pilot (projected from P2), ceiling $975. At the ceiling a results-blind drop order applies: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop (second round C6; E1 has its own ceiling). No operator choice. + - F20: index cost and configuration hash are defined in `preregistration.md` section 14.1. + - Second round, C1 to C24 (2026-11-14, `protocol.md` section 0.13): the rulings above plus gate intervals by percentile bootstrap, the half-A rule for samples drawn after `halves.json` exists, the seeds table and open item O6. +- **Out of scope here.** Harness code, scorers and statistics code belong to change `exp35-main-run-readiness`. Part 2 and E1 rules belong to the proposal and change `rag-benchmark-readiness-flow`. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md index cd6ed1c3..b1e3cb8c 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md @@ -1,6 +1,6 @@ # Experiment 35: Reproducible Local RAG Benchmark of OMRS -> Status: DRAFT. Purpose `publication`, size Confirmatory. Commit `protocol.md` and `plan.json` before the first measured run. +> Status: DRAFT. Purpose `publication`, size Confirmatory. Commit `protocol.md` and `plan.json` before the first measured run. From amendment A4 (2026-11-14), `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs the live design; where this change differs, it wins. ## Pre-freeze amendment A1 (2026-11-12) @@ -9,15 +9,26 @@ The operator and the agent agreed a full redesign before the freeze. It wins ove - **Datasets (English only, 1,160 questions):** FinanceBench (`PatronusAI/financebench`, 150), FreshStack LangChain (203 public queries, not 223; the 20 Experiment 9 continuity queries leave), BioASQ mini (707) and Legal RAG Bench (`isaacus/legal-rag-bench`, 100 expert questions, one gold passage each, 4,876 passages, CC BY-NC-SA 4.0; exploratory because of low power). LegalBench-RAG and LLeQA are out. FinanceBench answers also get a number-match check. - **Search stage, full factorial on all four datasets:** embedder (Qwen3-Embedding 0.6B, 4B, 8B) x chunk size (512, 1,024, 2,048 Qwen3 tokens, overlap 15%: 77, 154, 307, the same for Markdown and plain text) x reranker (off, Qwen3-Reranker 0.6B, 4B, 8B) x hybrid (off, on with BM25 pinned). That is 9 indexes per dataset and 72 search setups. Setups are named by their settings, never by a profile name. - **Dropped:** the six one-factor arms, the `chunking-off` arm, MiniLM (it truncates at 512 tokens), Granite Guardian, hypothesis H-CC, the $250 cap. Isnad moved to v4. -- **Hypotheses H1 to H7** (search) and **H8** (answer stage, chunk size against faithfulness and correctness), each with a paired test and Holm correction. +- **Hypotheses H1 to H7** (search) and **H8** (answer stage, chunk size against faithfulness and correctness), each with a paired test and Holm correction. (Superseded by A4: H8 splits into H8a and H8b; the tests and Holm families are in `preregistration.md` sections 7 to 9.) - **Answer stage:** one embedder and one reranker size, chosen by a rule fixed before the run; then chunk size x reranker on or off x hybrid on or off = 12 crossed setups (plus 3 token-matched setups from A2 and A3, 15 in all) x 3 runs x 1,160 questions. Answer model, Ragas judge and claim checkers GLM-5.3 (main) and Clef-Flash (second) as before. The refusal rule stays. -- **Budget:** re-estimated at about $650, with a stop rule at a ceiling of $975 (1.5 x the estimate). +- **Budget:** re-estimated at about $650, with a stop rule at a ceiling of $975 (1.5 x the estimate). (Superseded by A4: about $730, ceiling $975, results-blind drop order.) - **Hardware:** one GPU type for every main-run pod, chosen before the freeze from a 10-minute timing job on RTX 4090, L40S and H100 by a rule written before measuring. The 9 indexes build on parallel pods of that type. - **New OMRS code:** a llama.cpp reranker backend (one new file and one `register()` line) in change `exp35-main-run-readiness`. ## Pre-freeze amendment A2 (2026-11-13) -An independent methods review ran; the operator approved every fix (design D0.12): unit-level scoring after collapsing the top 100 chunks; token-matched controls; H6 confirmatory on FinanceBench and FreshStack only, Legal exploratory throughout; one mixed model per dataset with a 2-of-3 cross-dataset rule; MDE from data; split-half selection; two labellers; a closed-book run; returned-model provenance; reranker rank checks; OSF pre-registration. Budget about $650, ceiling $975. +An independent methods review ran; the operator approved every fix (design D0.12): unit-level scoring after collapsing the top 100 chunks; token-matched controls; H6 confirmatory on FinanceBench and FreshStack only, Legal exploratory throughout; one mixed model per dataset (dropped by A2c and A4) with a 2-of-3 cross-dataset rule; MDE from data; split-half selection; two labellers; a closed-book run; returned-model provenance; reranker rank checks; OSF pre-registration. Budget about $650, ceiling $975. + +## Pre-freeze amendment A4 (2026-11-14) + +A publication-grade methods review gave 30 findings, F1 to F30. The operator approved every recommended fix (design D0.15, `protocol.md` section 0.13, `plan.json` amendment A4). The live design now sits in one file, `preregistration.md`, which is the main file of the OSF deposit. Main changes: + +- FreshStack indexes use the full public LangChain corpus (49,514 documents). The 10,000-document subset is retired. +- H8 splits into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory. Every hypothesis is scoped, with numbered rows and a holds rule. Legal RAG Bench is exploratory everywhere. +- No mixed model. Per-question contrasts with GEE, and a document-level sign-flip test on FinanceBench. Holm families fixed at 19, 19 and 8 (second round C3; BioASQ rows 9 to 19 are sanity checks, pending open item O6). +- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items and L2 labels 225. +- `evidence_recall_20k` gives any-page credit. A second closed-book run uses the RAG prompt. FinanceBench is split by document. +- The `openai/gpt-6.1-terra` cross-check is removed. Budget about $730 plus the pilot, ceiling $975, with a results-blind drop order that no longer holds E1 (E1 has its own ceiling). ## Why @@ -52,7 +63,7 @@ The pipeline sweep (`openspec/changes/pipeline-sweep-llamacpp/`, optimisation, S | Dataset | Domain | Questions | Corpus | Gold labels | Licence | |---|---|---|---|---|---| | FinanceBench (open sample) | Finance, real PDFs | 150 | 10-K and similar filings | Gold document and page | CC BY-NC 4.0 | -| FreshStack, LangChain slice | Software | 223 | 10,024 documents | Gold documents | CC BY-SA 4.0 | +| FreshStack, LangChain slice (history; A4 uses the full corpus of 49,514 documents and 203 queries) | Software | 223 | 10,024 documents | Gold documents | CC BY-SA 4.0 | | BioASQ (`enelpol/rag-mini-bioasq`, test split) | Biomedicine | 707 | 40.2K PubMed abstracts | Gold passages and reference answers | CC BY 2.5 | | NFCorpus (fallback for BioASQ) | Biomedicine | (check) | (check) | Graded relevance | Check the academic-use terms first | diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/codebase-rag-evaluation/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/codebase-rag-evaluation/spec.md index 72627766..2ff37502 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/codebase-rag-evaluation/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/codebase-rag-evaluation/spec.md @@ -2,9 +2,9 @@ ## ADDED Requirements -### Requirement: Software retrieval is measured on the FreshStack LangChain slice +### Requirement: Software retrieval is measured on the full FreshStack LangChain corpus -Experiment 35 SHALL measure software retrieval on the 203 public FreshStack queries of the frozen LangChain slice (`prepare_freshstack.py`, seed 20260530). The 20 continuity queries from Experiment 9 SHALL NOT be scored (pre-freeze amendment A1, 2026-11-12). It SHALL build its own 9 indexes and SHALL run the same 72 search setups as the document benchmark. FreshStack nuggets MAY serve as the reference for Ragas context recall. +Experiment 35 SHALL measure software retrieval on the 203 public FreshStack queries (`freshstack/queries-oct-2024`, config `langchain`) against the full public LangChain corpus (`freshstack/corpus-oct-2024`, 49,514 documents) at a pinned revision (amendment A4, 2026-11-14). It SHALL build its own 9 indexes and run the same 72 search setups as the document benchmark. The 20 Experiment 9 continuity queries SHALL NOT be scored. #### Scenario: An existing index is reused - **WHEN** a run scores against an index built by an earlier experiment @@ -12,7 +12,11 @@ Experiment 35 SHALL measure software retrieval on the 203 public FreshStack quer #### Scenario: The query set is reduced - **WHEN** a run scores fewer than the 203 frozen queries without a plan amendment -- **THEN** it SHALL NOT count as the measured result +- **THEN** it SHALL NOT count as a measured result + +#### Scenario: The corpus is a subset +- **WHEN** a FreshStack index holds fewer documents than the pinned full corpus, such as the retired 10,000-document subset of Experiments 9a and 22 +- **THEN** its scores SHALL NOT count as Experiment 35 results #### Scenario: Continuity documents in the corpus - **WHEN** the FreshStack corpus is loaded @@ -22,6 +26,11 @@ Experiment 35 SHALL measure software retrieval on the 203 public FreshStack quer - **WHEN** a FreshStack run includes one of the 20 Experiment 9 continuity queries - **THEN** that query SHALL NOT enter any Experiment 35 figure +#### Scenario: Nugget labels +- **WHEN** the FreshStack nugget labels allow it +- **THEN** alpha-nDCG@10 SHALL be reported as a secondary metric with the benchmark's own scorer, else as not available +- **AND** the nuggets MAY serve as the reference for Ragas context recall + ### Requirement: Retrieved chunks count once per parent document Each retrieved chunk SHALL be mapped to its parent document, and only the first rank of each parent SHALL count. The mapping SHALL reuse the code of Experiment 22's `run_eval.py`. @@ -32,12 +41,12 @@ Each retrieved chunk SHALL be mapped to its parent document, and only the first ### Requirement: An earlier FreshStack run is reported only as a reference -Experiment 22 MAY be reported beside the result as a reference. The report SHALL label it a reference and SHALL list the settings that differ. Experiment 9a SHALL NOT be compared, because it used one vector per parent document. +Experiment 22 MAY be reported beside the result as a reference. The report SHALL label it a reference, SHALL state that it used the retired 10,000-document subset, and SHALL list the settings that differ. Experiment 9a SHALL NOT be compared, because it used one vector per parent document. No FreshStack result SHALL be compared with the public leaderboard unless the same corpus and metric are used. #### Scenario: An earlier run is reported as a reference - **WHEN** the report shows Experiment 22 beside the Experiment 35 FreshStack result -- **THEN** it SHALL label Experiment 22 as a reference, not the baseline -- **AND** it SHALL list the settings that differ, such as top_k, chunking and embedding model +- **THEN** it SHALL label Experiment 22 a reference, not a baseline +- **AND** it SHALL name the corpus difference and list the settings that differ, such as top_k, chunking and the embedding model ### Requirement: Code answer benchmarks without retrieval labels are out of scope diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md index b136d345..ecea4ae7 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md @@ -55,41 +55,65 @@ For each chunk-size level, the harness SHALL set `chunking.chunk_size` and `chun - **WHEN** the process environment or `.env` sets `CHUNKING__CHUNK_SIZE` or `CHUNKING__MARKDOWN_CHUNK_SIZE` - **THEN** the stage SHALL stop with a setup error before ingest and name the setting -### Requirement: Search hypotheses are pre-registered with a paired test and Holm correction +### Requirement: Hypotheses are pre-registered with numbered rows and fixed Holm families -The protocol SHALL register H1 to H8 before the run. Per dataset, a mixed model (`MixedLM`, random intercept per question) SHALL give estimates, and a GEE with cluster-robust errors SHALL give each contrast's p-value, Holm-corrected within the dataset. A contrast SHALL count when the adjusted p is below 0.05 and its sign matches. A hypothesis SHALL hold overall on at least 2 of FinanceBench, FreshStack and BioASQ; Legal SHALL stay exploratory. +The protocol SHALL register H1 to H7, H8a and H8b before the run, each within scope S, with the rows, endpoints, null and holds rule of `preregistration.md` sections 7 and 8. Every null SHALL be a zero effect. H8a (12 crossed setups) and H8b (3 token-matched setups) SHALL both be confirmatory. Legal RAG Bench SHALL stay exploratory in every hypothesis, H8a and H8b correctness included. + +#### Scenario: A row counts +- **WHEN** a directional row has a Holm-adjusted p below 0.05 and an estimate with the predicted sign, or a two-sided row has a Holm-adjusted p below 0.05 +- **THEN** the row SHALL count on that dataset + +#### Scenario: A hypothesis holds +- **WHEN** the rows of H1 to H5 count on at least 2 of 3 core datasets by the hypothesis's rule, or the rows of H6, H7, H8a or H8b count on both FinanceBench and FreshStack by the hypothesis's rule +- **THEN** the hypothesis SHALL hold + +#### Scenario: A two-sided row counts with opposite signs +- **WHEN** row 9 counts on FinanceBench with a positive estimate and on FreshStack with a negative estimate +- **THEN** the report SHALL state the row as a dataset-dependent effect with both directions, and H6 SHALL NOT hold on that row #### Scenario: Direction opposite to the hypothesis -- **WHEN** a contrast is significant but its sign is opposite to the stated direction -- **THEN** the report SHALL say the hypothesis does not hold, and SHALL report the opposite effect as a finding +- **WHEN** a directional row is significant but its sign is opposite to the stated direction +- **THEN** the report SHALL say the row does not count, and SHALL report the opposite effect as a finding -#### Scenario: H6 on short-document sets -- **WHEN** chunk-size contrasts are computed on BioASQ -- **THEN** they SHALL be reported as a sanity check and SHALL NOT count for or against H6, which is tested on FinanceBench (endpoint `evidence_recall_20k`) and FreshStack (nDCG@10) and holds only when both support it +#### Scenario: Chunk-size rows on BioASQ +- **WHEN** rows 9 to 19 are computed on BioASQ +- **THEN** they SHALL be reported as sanity checks with unadjusted 95% intervals and SHALL NOT count for or against H6, H7, H8a or H8b, which are tested on FinanceBench and FreshStack #### Scenario: Legal result - **WHEN** a contrast counts on Legal RAG Bench - **THEN** it SHALL NOT count towards any hypothesis +#### Scenario: H8a holds and H8b does not +- **WHEN** H8a holds for an outcome and H8b does not +- **THEN** the report SHALL claim only that chunk size together with context length changes that outcome + #### Scenario: FinanceBench interval -- **WHEN** a bootstrap interval is computed on FinanceBench -- **THEN** whole documents SHALL be resampled +- **WHEN** a decision p-value or 95% interval is computed on FinanceBench +- **THEN** it SHALL come from the document-level sign-flip permutation test, so questions that share a document move together ### Requirement: Retrieval is scored on distinct units after collapse -Each search setup SHALL save the top 100 chunks, with a rerank pool of exactly 100. The scorer SHALL collapse chunks to units (document, FinanceBench first page, passage) in rank order and cut at 10. Equal evidence SHALL be cut by the running total of real Qwen3 tokens up to 20,480. +Each search setup SHALL save its top 100 chunks, with a rerank pool of exactly 100. The scorer SHALL collapse chunks to units (FreshStack document, FinanceBench page, BioASQ or Legal passage) in rank order, keep the first occurrence of each unit and cut at 10 units. `evidence_recall_20k` SHALL take chunks in rank order up to 20,480 real Qwen3 tokens and truncate the chunk that crosses the limit. #### Scenario: Big chunks gain nothing by construction - **WHEN** two setups give the same unit order but one uses 2,048-token chunks - **THEN** both SHALL score the same nDCG@10 -#### Scenario: Spanning page +#### Scenario: Spanning page in nDCG@10 - **WHEN** a FinanceBench chunk spans pages 41 and 42 and the gold page is 42 -- **THEN** the chunk SHALL count for page 41, and the "any page" credit SHALL appear only in a sensitivity column +- **THEN** for nDCG@10, Recall@10, MRR@10 and hit@page the chunk SHALL count for page 41, and any-page credit SHALL appear only in the sensitivity column + +#### Scenario: Spanning page in evidence_recall_20k +- **WHEN** a whole chunk inside the 20,480-token evidence spans pages 41 and 42 and the gold page is 42 +- **THEN** `evidence_recall_20k` SHALL count gold page 42 as covered (any-page credit), and first-page credit SHALL appear only in the sensitivity column + +#### Scenario: Truncated crossing chunk +- **WHEN** the chunk that crosses 20,480 tokens spans pages 7 and 8 and is truncated +- **THEN** it SHALL cover only page 7, because the reader stores no page offsets inside a chunk -### Requirement: The answer stage uses a selection rule fixed before the run +### Requirement: The answer stage uses the selection rule fixed before the run -On half A (seed 20261115), the harness SHALL choose the smallest embedder, then reranker size, within 0.01 pooled nDCG@10 of the best (ties to the smaller). The pooled score SHALL be the mean over the three core datasets (FinanceBench, FreshStack, BioASQ); Legal RAG Bench SHALL take no part. The answer stage SHALL run 15 setups (12 crossed, 3 token-matched; A3), 3 runs each, plus closed-book and oracle runs. H8 SHALL use all questions; descriptive headlines half B. +On half A only, the harness SHALL choose the smallest embedder, then the smallest reranker size, within 0.01 of the best pooled nDCG@10 over FinanceBench, FreshStack and BioASQ; Legal RAG Bench SHALL take no part. The answer stage SHALL run 15 setups (12 crossed, 3 token-matched), 3 runs each, plus two closed-book runs and one oracle run. H8a and H8b SHALL use all questions. #### Scenario: Two sizes within the margin - **WHEN** the 4B embedder pools 0.612 and the 8B embedder pools 0.619 @@ -99,21 +123,45 @@ On half A (seed 20261115), the harness SHALL choose the smallest embedder, then - **WHEN** the harness pools the per-dataset nDCG@10 means for a size - **THEN** it SHALL average the FinanceBench, FreshStack and BioASQ means and SHALL NOT include the Legal RAG Bench mean -#### Scenario: Selection recorded before answers -- **WHEN** a replicate pod starts an answer stage -- **THEN** `protocol.md` SHALL already hold a dated note with the pooled scores and the chosen sizes +### Requirement: Halves are fixed before any score -### Requirement: Half A also fixes the OMRS setup for Part 2 +Each dataset SHALL be split into half A and half B with seed 20261115. FinanceBench SHALL be split by document, so no document sits in both halves; FreshStack, BioASQ and Legal RAG Bench SHALL be split by question. Every selection step, and every pre-freeze sample drawn after `halves.json` exists, SHALL read half A only. Samples drawn earlier (smoke runs, test pod, classifier-gate claims) SHALL be listed with their question IDs and, once `halves.json` exists, their half. -At the chosen embedder and reranker size, the harness SHALL pick chunk size, reranker on or off and hybrid on or off by the pooled half-A mean nDCG@10 over the three core datasets. Cells within 0.01 of the best SHALL tie, and ties SHALL resolve in this order: reranker off, hybrid off, lowest index cost, smaller chunk size. The configuration hash of the chosen cell SHALL go to `protocol.md` before half B opens for Part 2. +#### Scenario: FinanceBench document in both halves +- **WHEN** `halves.json` puts two questions of one FinanceBench document in different halves +- **THEN** the split SHALL fail its check before any score is computed + +#### Scenario: A pre-freeze sample reaches half B +- **WHEN** the MDE pilot or a RAGFlow pre-freeze check, drawn after `halves.json` exists, draws a half-B question +- **THEN** that sample SHALL NOT be used + +#### Scenario: A sample drawn before the halves +- **WHEN** a smoke-run, test-pod or classifier-gate sample was drawn before `halves.json` existed +- **THEN** it SHALL be listed with its question IDs and their halves, and SHALL NOT be scored as a result + +### Requirement: Half A also fixes the OMRS setup for the RQ1 headline and Part 2 + +At the chosen embedder and reranker size, the harness SHALL pick chunk size, reranker on or off and hybrid on or off by pooled half-A mean nDCG@10. Cells within 0.01 of the best SHALL tie, resolved in order: reranker off, hybrid off, lowest index cost, smaller chunk size. The order SHALL be: score half A, write `selection.json`, time-stamp the configuration hash on OSF, then score half B. #### Scenario: Tie goes to the cheaper cell -- **WHEN** the best cell has reranker on and a cell with reranker off scores 0.008 lower -- **THEN** the rule SHALL choose the cell with reranker off +- **WHEN** the best cell has the reranker on and a cell with the reranker off scores 0.008 lower +- **THEN** the rule SHALL choose the cell with the reranker off + +#### Scenario: Index cost decides a tie +- **WHEN** tied cells are compared on index cost +- **THEN** index cost SHALL be the stored vector values (chunks times embedding dimension) summed over the three core datasets, known from the sealed indexes before any score -#### Scenario: Hash recorded before half B opens -- **WHEN** Part 2 is about to read half B -- **THEN** `protocol.md` SHALL already hold a dated note with the cell scores, the chosen cell and its configuration hash +#### Scenario: Configuration hash is written +- **WHEN** `selection.json` is written +- **THEN** it SHALL hold the pooled scores, both chosen sizes, the chosen cell and the configuration hash: the SHA-256 of the canonical JSON of the cell's resolved settings plus the `halves.json` SHA-256 + +#### Scenario: Half-B score before the time stamp +- **WHEN** a step asks for a half-B score +- **THEN** `selection.json` SHALL already exist, and the OSF time stamp of its hash SHALL already exist + +#### Scenario: RQ1 headline +- **WHEN** the report gives the RQ1 retrieval headline +- **THEN** it SHALL be the chosen cell's nDCG@10 on half B with its 95% interval, and the 72-setup table on all questions SHALL be labelled descriptive ### Requirement: Every answer is one synthesis round @@ -127,13 +175,18 @@ Every answer setup SHALL use `answer.context_window` 131,072 characters and `ans - **WHEN** OMRS cuts evidence to fit the round budget - **THEN** the answer row SHALL record the truncation and the characters dropped, and the report SHALL count truncated answers per setup -### Requirement: Answer correctness is judged against reference answers and validated by hand +### Requirement: Answer correctness is judged against reference answers and gated per dataset + +The Ragas stage SHALL score every answer with Ragas 0.4.3 `AnswerCorrectness` against the reference answer, and the judge SHALL never see the retrieved contexts. L1 SHALL label 50 half-B questions per core dataset (seed 20261113), each with one answer from run 1 of an answer setup the seed picks uniformly at random, and 50 Legal RAG Bench answers drawn the same way and reported without a gate. -The Ragas stage SHALL score every answer with Ragas 0.4.3 `AnswerCorrectness` against the reference answer. L1 SHALL label 100 main-run answers (25 per dataset) and 200 claims; L2 SHALL label 30% of each set. The report SHALL give kappa with intervals per dataset and checker, and L1 against L2. Below pooled kappa 0.60, correctness SHALL be exploratory. +#### Scenario: Gate passes on a dataset +- **WHEN** Cohen's kappa between the score cut at 0.5 and the human label has a point estimate of at least 0.60 and a lower 95% bound of at least 0.40, with "can't tell" items scored both ways, and the AUC of the continuous score has a lower 95% bound above 0.5 +- **THEN** rows 14, 15, 18 and 19 SHALL be testable on that dataset -#### Scenario: Low agreement -- **WHEN** the pooled kappa between the binarised Ragas score and the operator labels is 0.52 -- **THEN** answer correctness SHALL be labelled exploratory, and H8 SHALL rest on faithfulness and the FinanceBench number match +#### Scenario: Low agreement on one dataset +- **WHEN** the correctness gate fails on FreshStack +- **THEN** rows 14, 15, 18 and 19 SHALL be reported as not testable on FreshStack, staying in its Holm family with p = 1 +- **AND** the other datasets SHALL keep their rows ### Requirement: FinanceBench answers get a number-match check @@ -149,10 +202,10 @@ A tested, deterministic function SHALL compare the numbers of each FinanceBench ### Requirement: Document retrieval metrics remain primary -nDCG@10 on distinct units SHALL be the only primary metric. Recall@10, MRR@10 and hit@page (FinanceBench) SHALL be secondary retrieval metrics. Scoring SHALL count documents, pages or passages, never chunks. Judge-based scores SHALL NOT replace these metrics. +nDCG@10 on distinct units SHALL be the primary metric of the study. `evidence_recall_20k` SHALL be the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11). Recall@10, MRR@10, hit@page (FinanceBench) and alpha-nDCG@10 (FreshStack, when its nugget labels allow) SHALL be secondary retrieval metrics. Scoring SHALL count documents, pages or passages, never chunks. Judge-based scores SHALL NOT replace these metrics. #### Scenario: Ragas answer scores improve while retrieval degrades -- **WHEN** a secondary Ragas score improves but a primary metric worsens +- **WHEN** a secondary Ragas score improves but the primary metric worsens - **THEN** the report SHALL present both outcomes - **AND** it SHALL NOT hide the retrieval regression behind the secondary score @@ -202,10 +255,35 @@ Every reranker setup SHALL pass `fetch_k` 100 so the rerank pool is 100 chunks w - **WHEN** a reranker search asks for 100 results without `fetch_k` - **THEN** the pool would be 300, and the harness SHALL refuse that call shape -### Requirement: Closed-book misses are cross-tabulated +### Requirement: Closed-book runs bound contamination -On questions where retrieval missed every gold unit, the report SHALL cross-tabulate closed-book correct or not against RAG correct or not, per dataset. +Two closed-book runs SHALL answer every question once without evidence: one with an own-knowledge prompt, and one with the main answer prompt and an empty evidence block. Both SHALL be descriptive single runs. #### Scenario: Correct without retrieval - **WHEN** a RAG answer is correct although its evidence holds no gold unit -- **THEN** the cross-table SHALL show whether the closed-book answer was also correct +- **THEN** the per-dataset cross-table SHALL show whether the closed-book answers were also correct + +#### Scenario: Contamination subset +- **WHEN** the H8a and H8b correctness rows 14, 15, 18 and 19 are reported +- **THEN** the report SHALL also give them, marked as a registered sensitivity analysis, on the questions both closed-book runs answer wrongly (Ragas correctness below 0.5) + +### Requirement: Missing outcomes keep every question in the denominator + +Every question SHALL stay in its dataset's denominator, and every exclusion SHALL be counted per setup and row, by the rules of `preregistration.md` section 12. + +#### Scenario: Short search list +- **WHEN** a search list holds fewer than 10 units +- **THEN** the missing ranks SHALL score as not relevant + +#### Scenario: Answer call fails after resend +- **WHEN** an answer is still missing when the run closes +- **THEN** it SHALL score 0 for correctness and number match and stay in the denominator +- **AND** the report SHALL count it + +#### Scenario: Answer with no factual claim +- **WHEN** an answer holds no factual claim +- **THEN** faithfulness SHALL be undefined for that run, and the two registered bounds (scored 1, scored 0) SHALL be reported + +#### Scenario: Row still unjudged +- **WHEN** a Ragas or checker row is still unjudged when the run closes +- **THEN** it SHALL leave that row's complete cases and be counted, and a sensitivity analysis SHALL score it 0 diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index 47e969a3..d0800f21 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -4,12 +4,17 @@ ### Requirement: The benchmark is a Confirmatory publication experiment -Experiment 35 SHALL have purpose `publication` and size Confirmatory. Its `protocol.md` and `plan.json` SHALL be committed before the first measured run. Later changes SHALL be recorded only as plan amendments. +Experiment 35 SHALL have purpose `publication` and size Confirmatory. `preregistration.md`, `protocol.md` and `plan.json` SHALL be committed before the first measured run. From amendment A4 (2026-11-14), `preregistration.md` SHALL govern the live design: where another file of the experiment or this change differs from it, `preregistration.md` SHALL win. After the freeze, every change SHALL be a dated, numbered deviation, added to the `plan.json` `amendments` list and uploaded to OSF. #### Scenario: A measured run starts before the plan is committed - **WHEN** the run record shows a plan file that differs from the committed `plan.json` - **THEN** the run SHALL NOT count as a measured result +#### Scenario: Two files disagree on a design fact +- **WHEN** `design.md` or `plan.json` states a design fact that differs from `preregistration.md` +- **THEN** the `preregistration.md` value SHALL apply +- **AND** the other file SHALL be corrected before the freeze + ### Requirement: Local models are pinned and checked before scoring `plan.json` SHALL hold an ADR-103 `runtime` block. Every local model SHALL run through llama.cpp with a public GGUF file, pinned by repository, file, SHA-256, build and server arguments, with a tolerance per primary metric. Hosted models SHALL appear only as named reference arms. @@ -36,25 +41,64 @@ Each index SHALL be built once, and each search setup SHALL run once (run 1) for - **WHEN** two answer setups are compared - **THEN** the 3 runs SHALL be averaged per question before questions are paired -### Requirement: Spending stops at a stated ceiling +### Requirement: Spending stops at a stated ceiling with a results-blind drop order -The protocol SHALL state a cost estimate and a ceiling of 1.5 times the estimate, for pods and hosted calls together. The projected total SHALL be checked before the freeze, on the build pod after the search stage, and while the replicate pods run. When the projected or actual total reaches the ceiling, the run SHALL stop before the next stage, and the operator SHALL decide. +The protocol SHALL state a cost estimate and a ceiling, pods and hosted calls together (about $730 and $975, `preregistration.md` section 17). Projections SHALL use billing and measured rates only, never scores, before the freeze, after the search stage and while the replicate pods run. At the ceiling, work SHALL be dropped in a results-blind order fixed before the run, and no confirmatory item SHALL be dropped by choice. #### Scenario: Projection above the ceiling -- **WHEN** the build pod's projection, with the measured index and search hours, exceeds the ceiling -- **THEN** the replicate pods SHALL NOT start until the operator decides +- **WHEN** a projection, from measured index and search hours, reaches the ceiling +- **THEN** the run SHALL drop the next step in this order and compute the projection again: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt +- **AND** E1 SHALL NOT be in this order, because it has its own ceiling +- **AND** the run record SHALL name the projection that triggered each step + +#### Scenario: Drop order exhausted +- **WHEN** the projection still reaches the ceiling after step (3) +- **THEN** the run SHALL stop +- **AND** no operator choice SHALL select what is dropped + +#### Scenario: Run finishes after drop steps +- **WHEN** every confirmatory item is done after one or more drop steps +- **THEN** the run SHALL be reported as "complete with registered drops", and each drop SHALL be a numbered deviation +- **AND** a run SHALL be reported as incomplete only when a confirmatory item is missing + +#### Scenario: Closed-book RAG-prompt run dropped +- **WHEN** step (3) drops the closed-book run with the RAG prompt +- **THEN** the contamination sensitivity SHALL use the own-knowledge closed-book run alone, and the report SHALL say so + +### Requirement: Statistics follow the pre-registered analysis -### Requirement: Statistics follow the frozen basic set +The analysis SHALL follow `preregistration.md` section 9 and SHALL fit no mixed model. Each contrast row SHALL use per-question contrast scores (cells weighted equally, answer scores averaged over 3 runs first), and the estimate SHALL be their plain mean. The report SHALL give unadjusted 95% intervals beside each decision, marked against the SESOI and the minimum detectable effect. -For each dataset and setup the report SHALL give the mean with a 95% interval. For each registered contrast it SHALL give the paired difference with a 95% interval, a paired permutation test for scores, McNemar's exact test for hit or miss, Holm correction across the primary contrasts of one dataset, and wins, losses and ties. The minimum detectable effect SHALL be stated before the run. +#### Scenario: Test on FreshStack, BioASQ or Legal RAG Bench +- **WHEN** a row's p-value and interval are computed on FreshStack, BioASQ or Legal RAG Bench +- **THEN** they SHALL come from the closed form of pre-registration section 9 point 3: the mean of the contrast scores, the Mancl and DeRouen bias-reduced variance (sum of squared residuals times (n / (n minus 1)) squared, divided by n squared) and a t reference on n minus 1 degrees of freedom +- **AND** the `statsmodels` GEE implementation (intercept only, independence working correlation, question clusters, `cov_type="bias_reduced"`) SHALL be checked against that closed form by a known-answer test, and the closed form SHALL be used if they differ -#### Scenario: No difference is found -- **WHEN** the Holm-adjusted p is 0.05 or more, or the paired interval includes zero -- **THEN** the report SHALL state that no difference larger than the minimum detectable effect was found +#### Scenario: Test on FinanceBench +- **WHEN** a row's p-value and interval are computed on FinanceBench +- **THEN** they SHALL come from a document-level sign-flip permutation test with 100,000 draws and seed 20261101 + +#### Scenario: No difference found +- **WHEN** the Holm-adjusted p is 0.05 or more +- **THEN** the report SHALL state that no difference was detected at this sample size +- **AND** the report SHALL NOT claim equivalence, and no TOST SHALL run + +#### Scenario: Decision and sensitivity check disagree +- **WHEN** a row counts under the decision test but not under its sensitivity check (paired permutation by question on FreshStack and BioASQ, GEE with document clusters on FinanceBench), or the other way +- **THEN** the report SHALL list the row in one disagreement table, and the decision test SHALL decide #### Scenario: New statistics code is written - **WHEN** the analysis needs a statistic -- **THEN** it SHALL use the functions in `experiments/_lib/stats.py` or the s-experiment reference `stats.py` +- **THEN** it SHALL use the functions in `experiments/_lib/stats.py` or the s-experiment reference `stats.py`, or a tested function added for this analysis + +### Requirement: Holm families are fixed before the run + +Holm correction (alpha 0.05) SHALL run within families of fixed size: FinanceBench 19 rows, FreshStack 19 rows and BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks, pending open item O6), 46 in all. Legal RAG Bench SHALL have no family. No gate decided on main-run labels SHALL change a family size. + +#### Scenario: A gate fails +- **WHEN** the faithfulness gate of GLM-5.3 fails +- **THEN** rows 12, 13, 16 and 17 SHALL stay in every Holm family with p = 1 and be reported as not testable +- **AND** the other rows SHALL keep the same number of comparisons ### Requirement: The benchmark emits structured stage traces @@ -87,22 +131,42 @@ Ragas 0.4.3 MAY score faithfulness, answer relevancy, context precision and cont - **THEN** primary retrieval metrics SHALL remain runnable - **AND** Ragas metrics SHALL be reported as not measured -### Requirement: Judge scores are checked by hand +### Requirement: Faithfulness checkers pass a registered gate -About 50 Ragas-judged items SHALL be checked by hand, with the agreement measure defined in the protocol before the check. The report SHALL state the agreement. The judge SHALL run through llama.cpp when possible. +Each faithfulness checker SHALL pass when Cohen's kappa against the human label has a point estimate of at least 0.60 and a lower 95% bound of at least 0.40, on 450 claims (150 per chunk size) from answers to all questions, pooled over the core datasets. Each gate interval SHALL be a percentile bootstrap (10,000 resamples by question, own seed). Gate decisions SHALL be recorded, with their time, before any H8 contrast result is computed. There SHALL be no cross-check judge. #### Scenario: The judge configuration changes - **WHEN** the judge model, its settings or the Ragas version changes - **THEN** the resulting scores SHALL belong to a new evaluation revision +#### Scenario: Checker specificity differs by chunk size +- **WHEN** the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction +- **THEN** H8a and H8b faithfulness rows 12, 13, 16 and 17 SHALL be exploratory and reported as not testable + +#### Scenario: The second checker fails its gate +- **WHEN** Clef-Flash fails its faithfulness gate +- **THEN** its results SHALL be reported as not validated and SHALL change no confirmatory row + +### Requirement: Human labels follow the registered counts + +L1 SHALL label 750 items (claim kinds 100, faithfulness 450 from all questions, correctness 200 from half B), and L2 SHALL label a 30% subset of each set (225 items). The claim classifier SHALL reach agreement of at least 0.90 and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. + +#### Scenario: A label file is released +- **WHEN** human labels are released +- **THEN** they SHALL carry the roles L1 and L2 only, and no labeller name SHALL appear + ### Requirement: Outputs respect dataset licences -`output/` SHALL be split into shareable and restricted parts. Releases SHALL hold scores, question IDs and rebuild scripts, not corpus text. FinanceBench-derived files SHALL NOT be released under CC BY. Each dataset's derived files SHALL go in their own Zenodo archive with their own licence. +`output/` SHALL be split into shareable and restricted parts. Releases SHALL follow the release table of `preregistration.md` section 19, each artefact under its dataset's licence, and SHALL NOT hold corpus text, PDFs or derived text. FinanceBench-derived files SHALL NOT be released under CC BY. Each dataset's derived files SHALL go in their own Zenodo archive with their own licence. #### Scenario: Corpus text reaches a shareable folder - **WHEN** a file under the shareable part of `output/` contains corpus text - **THEN** the release check SHALL fail before any upload +#### Scenario: A reader judges the answers again +- **WHEN** a reader wants to judge the released answers with a newer model +- **THEN** the re-judge kit SHALL rebuild the evidence text from the pinned dataset revisions and the released IDs + ### Requirement: Public releases carry an example set and no personal identifiers The release SHALL include a small shareable example input set with expected output and a reading guide. Before the repository or any release goes public, names, login names and home-folder paths SHALL be removed and the upload searched for them. @@ -121,8 +185,12 @@ Every hosted call SHALL record the model id and provider that the reply returns, ### Requirement: The protocol is publicly pre-registered -The `protocol.md` and `plan.json` of the freeze commit SHALL be deposited on OSF. A later commit, made before the first measured run, SHALL record the OSF link and the freeze commit hash in `protocol.md`; the freeze commit cannot hold its own hash. +`preregistration.md` (main file), `protocol.md`, `plan.json`, the proposal and the Part 2 protocol of the freeze commit SHALL be deposited on OSF. A later commit, made before the first measured run, SHALL record the OSF link and the freeze commit hash in the `plan.json` `frozen` list with the other pins of `preregistration.md` section 16; the freeze commit cannot hold its own hash. The A3 configuration hash SHALL be time-stamped on the registered OSF project before any half-B score is computed. #### Scenario: Run before the deposit -- **WHEN** the first measured run starts and `protocol.md` holds no OSF link +- **WHEN** the first measured run starts and `plan.json` `frozen` holds no OSF link - **THEN** the freeze SHALL count as incomplete + +#### Scenario: Half-B score before the time stamp +- **WHEN** a half-B score is computed before the A3 configuration hash is time-stamped on OSF +- **THEN** the run SHALL record a deviation, and the RQ1 headline SHALL state it diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index 7d656da1..1b3d53f2 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -50,27 +50,27 @@ The full-factorial redesign (design D0) replaces the six arms. Tasks 5.1 to 5.4 - [x] 4A.1 Write amendment A1 into this change (proposal, design D0, specs), `protocol.md` (section 0) and `plan.json` (`amendments`, hypotheses, metrics, runtime). Verify: `openspec validate experiment-35-full-document-rag-benchmark --strict` passes. Done 2026-11-12. - [x] 4A.2 Resolve the A1 open questions (design D0.11) with the operator. Done 2026-11-13: all five proposed answers chosen (corpus without continuity files, `.md` passages, Ragas `AnswerCorrectness` plus number match, one answer round, one search run per setup); only the 4B and 8B reranker repositories wait for the test pod. Verify: `protocol.md` section 0 records each answer with the date. - [ ] 4A.3 Pin the three Qwen3-Reranker GGUF files in the `plan.json` `runtime` block (repository, file, SHA-256, build 11941, server arguments) after the test pod check (readiness task 9.2). Verify: `load_runtime_block` accepts the block and `omrs experiments check` reports no error. -- [ ] 4A.4 Prepare Legal RAG Bench and the 203-query FreshStack set: record the HF revision, the SHA-256 of each file and the licence. Verify: protocol section 4.1 holds one line per file, and the questions load with every gold passage present. -- [ ] 4A.5 Replace task 1.11 (budget): the estimate and the $975 ceiling of design D0.9. Verify: `protocol.md` states the estimate, the ceiling, the three checks and the date. -- [ ] 4A.4a Validate answer correctness (design D0.5): the operator labels 100 seeded answers (seed 20261113, 25 per dataset, even over chunk sizes) from the test pod session, blind to the Ragas score. Verify: `protocol.md` records the threshold (0.5) and the kappa rule (pooled kappa of at least 0.60) in a commit before labelling, then the kappa per dataset and pooled, and whether correctness counts for H8. +- [ ] 4A.4 Prepare Legal RAG Bench and the 203-query FreshStack set: record the HF revision, the SHA-256 of each file and the licence. Verify: protocol section 4.1 holds one line per file, and the questions load with every gold passage present. (Amended by A4: FreshStack uses the full 49,514-document corpus; see 4D.2.) +- [ ] 4A.5 Replace task 1.11 (budget): the estimate and the $975 ceiling of design D0.9. Verify: `protocol.md` states the estimate, the ceiling, the three checks and the date. (Amended by A4: estimate about $730, ceiling $975, results-blind drop order; see `preregistration.md` section 17.) +- [ ] 4A.4a Validate answer correctness (design D0.5): the operator labels 100 seeded answers (seed 20261113, 25 per dataset, even over chunk sizes) from the test pod session, blind to the Ragas score. Verify: `protocol.md` records the threshold (0.5) and the kappa rule (pooled kappa of at least 0.60) in a commit before labelling, then the kappa per dataset and pooled, and whether correctness counts for H8. (Superseded by A4: 200 answers, 50 per core dataset, half B; gate per core dataset on kappa point estimate at least 0.60 and lower bound at least 0.40, plus AUC; see `preregistration.md` section 11.2 and RDY task 14.6.) - [ ] 4A.4b Record the context length of `deepseek/deepseek-v4.1-flash-1017` on the pinned OpenRouter host. Verify: `protocol.md` states it with the date, and it is at least 65,536 tokens; otherwise the freeze stops and the operator decides. - [ ] 4A.5a Choose the GPU type by the rule in `protocol.md` section 0.8a, from the timing job on the RTX 4090, L40S and H100 (readiness task 9.3). Verify: `protocol.md` holds the rule in a commit dated before the first measurement, then the measurements, prices and choice. -- [ ] 4A.6 Build the 9 indexes on 9 parallel pods of the chosen GPU type, each sealing its index and running its 8 search setups (72 setups, run 1), copy the scores to the Mac, and apply the selection rule (design D0.5). Verify: `protocol.md` holds a dated note with the pooled scores and the chosen embedder and reranker sizes before any replicate pod starts. +- [ ] 4A.6 Build the 9 indexes on 9 parallel pods of the chosen GPU type, each sealing its index and running its 8 search setups (72 setups, run 1), copy the scores to the Mac, and apply the selection rule (design D0.5). Verify: `protocol.md` holds a dated note with the pooled scores and the chosen embedder and reranker sizes before any replicate pod starts. (Amended by A4: score half A only at this step; half B is scored after the OSF time stamp in 4C.2.) - [ ] 4A.7 Run the 15 answer setups (12 plus 3 token-matched, amendments A2 and A3) x 3 runs on the three replicate pods. Verify: the join passes and G1 holds or its mismatches are recorded. -- [ ] 4A.8 Compute H1 to H7 on the search scores and H8 on the answer scores (design D0.4, D0.5). Label every Legal RAG Bench figure exploratory. Verify: each number names its `output/` file. +- [ ] 4A.8 Compute H1 to H7 on the search scores and H8 on the answer scores (design D0.4, D0.5). Label every Legal RAG Bench figure exploratory. Verify: each number names its `output/` file. (Amended by A4: H8 splits into H8a and H8b; analysis and Holm families per `preregistration.md` sections 7 to 9.) ## 4B. Pre-freeze amendment A2: methods review (2026-11-13) -Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4B wins: 4A.4a now draws its 100 answers from the main run (half B), not the test pod. +Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4B wins: 4A.4a now draws its 100 answers from the main run (half B), not the test pod. (A4: 200 answers, see 4D.6.) - [x] 4B.1 Write amendment A2 into this change (design D0.12, specs), `protocol.md` (section 0.10) and `plan.json` (amendment A2, metrics, hypotheses). Verify: both `openspec validate --strict` runs and `omrs experiments check` pass. Done 2026-11-13. -- [ ] 4B.2 Compute the minimum detectable effects from data (M1, readiness task 15.11) and write the table with its sources into `protocol.md` section 0.10. Verify: the table is in a commit before the freeze commit. +- [ ] 4B.2 Compute the minimum detectable effects from data (M1, readiness task 15.11) and write the table with its sources into `protocol.md` section 0.10. Verify: the table is in a commit before the freeze commit. (Amended by A4: the table lives in `preregistration.md` section 10; see 4D.5.) - [ ] 4B.3 Pin every reranker GGUF with its SHA-256 (M8) only after its Spearman check (at least 0.95 on 200 seeded pairs) passes on the test pod. Verify: `plan.json` runtime holds three reranker entries, `load_runtime_block` accepts them, and `protocol.md` lists each correlation. -- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. -- [ ] 4B.5 Run the closed-book run (M6) on replicate pod 1 and report correctness by retrieval hit or miss. Verify: the report table names its `output/` file. +- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 225; see 4D.6.) +- [ ] 4B.5 Run the closed-book run (M6) on replicate pod 1 and report correctness by retrieval hit or miss. Verify: the report table names its `output/` file. (Amended by A4: two closed-book runs; see 4D.7.) - [ ] 4B.5a Run the oracle run (design D0.12, readiness tasks 15.12 and 15.13) on replicate pod 1 and report it beside closed-book as ceiling and floor, descriptive only. Verify: the table names its `output/` file and states the no-distractor caveat. -- [ ] 4B.6 Deposit `protocol.md` and `plan.json` on OSF at the freeze commit (M9; the operator does the deposit). Verify: a commit after the freeze, before the first measured run, adds the OSF link and the freeze commit hash to `protocol.md`, and the deposit holds no personal data beyond the operator name. -- [ ] 4B.7 Write the limitations into `report.md`: one answer model, English only, the web-search reranker instruction applied to finance and legal text, 3 runs that measure hosted-API nondeterminism, chunk size confounded with context length in the answer stage. Verify: the report's limitations section names each. +- [ ] 4B.6 Deposit `protocol.md` and `plan.json` on OSF at the freeze commit (M9; the operator does the deposit). Verify: a commit after the freeze, before the first measured run, adds the OSF link and the freeze commit hash to `protocol.md`, and the deposit holds no personal data beyond the operator name. (Amended by A4: `preregistration.md` is the main file of the deposit; the OSF link and freeze commit go into `plan.json` `frozen`.) +- [ ] 4B.7 Write the limitations into `report.md`: one answer model, English only, the web-search reranker instruction applied to finance and legal text, 3 runs that measure hosted-API nondeterminism, chunk size confounded with context length in the answer stage. Verify: the report's limitations section names each. (Amended by A4: add scope S and the limitations of `preregistration.md` section 22.) - [x] 4B.8 Write amendment A2c (second methods review, design D0.13) into this change, `protocol.md` section 0.11 and `plan.json`. Verify: both strict validations and `omrs experiments check` pass. Done 2026-11-13. @@ -79,9 +79,24 @@ Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4 Design D0.14. Protocol section 0.12. - [x] 4C.1 Write amendment A3 into this change (design D0.14, spec), `protocol.md` (section 0.12, with a pointer in section 0.5) and `plan.json` (amendment A3). Verify: both `openspec validate --strict` runs and `omrs experiments check` pass. Done 2026-11-13. -- [ ] 4C.2 After the search stage is scored, apply the A3 rule on half A. Write a dated note into `protocol.md` section 0.12 with the inputs, the chosen cell and its configuration hash. Verify: the note exists before anyone opens half B for Part 2. +- [ ] 4C.2 After the search stage is scored, apply the A3 rule on half A. Write a dated note into `protocol.md` section 0.12 with the inputs, the chosen cell and its configuration hash. Verify: the note exists before anyone opens half B for Part 2. (Amended by A4: write `selection.json`, then time-stamp the configuration hash on OSF before any half-B score is computed.) - [ ] 4C.3 Draw correctness labels from RAGFlow answers. This task does not belong here. Part 2 owns it (master flow group 7). Verify: no Experiment 35 label count changes. +## 4D. Pre-freeze amendment A4: publication review fixes (2026-11-14) + +Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbers refer to its section 23. Harness and statistics code for these tasks lives in change `exp35-main-run-readiness`. + +- [ ] 4D.1 Write amendment A4 into this change (proposal, design D0.15, specs), `plan.json` (amendment A4, hypotheses, decision rule, metrics, gates, references) and `protocol.md` section 0.13. Verify: `openspec validate experiment-35-full-document-rag-benchmark --strict` and `omrs experiments check` pass. +- [ ] 4D.2 Download the full FreshStack LangChain corpus at a pinned revision after the operator approves (P1). Record the revision, row count (49,514) and tree hash. Verify: protocol section 4.1 and the pins table hold the three values. +- [ ] 4D.3 Time one full-corpus FreshStack index build and update the estimate (P2). Verify: the estimate in `preregistration.md` section 17 names the measured hours. +- [ ] 4D.4 Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's scorer (P3), and whether each dataset's qrels list judged non-relevant units (P4). Verify: the scorer is pinned or recorded as "not available", and each dataset has one line. +- [ ] 4D.5 Fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the operator's approval, run the retrieval-only pilot on 40 seeded half-A questions per core dataset, and fill the MDE table (P5, P6); if the operator declines, fill it from assumed SDs of 0.15 to 0.25, labelled as assumed. Restate the checker 2-point tolerance against the table (P7). Verify: the MDE table has one row per dataset and row group, with its SD source. +- [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 225. Verify: the guidelines are committed before the first label, and the gate results are recorded with their time before any H8a or H8b contrast result. +- [ ] 4D.7 Run the second closed-book run (RAG prompt, empty evidence) beside the first, and the length-controlled check on 300 claims. Verify: each table names its `output/` file and is marked descriptive or sensitivity. +- [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if O3 is yes); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. +- [ ] 4D.9 Get the operator's answers to O1 to O6 (OSF time-stamp mechanism, SESOI value, optional blind audit, L2 identity, ethics query, BioASQ rows 12 to 19 as sanity checks). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. +- [ ] 4D.10 Check before the freeze that no live text in this change, `protocol.md` section 0 or `plan.json` contradicts `preregistration.md`. Verify: every hit for "MixedLM", "included whole", "gpt-6.1-terra", "operator decides" and "equivalence" is marked superseded or removed. + ## 5. Run - [ ] 5.1 Run the six arms (or fewer, where 1.9 marks a cell not applicable) on FinanceBench, 3 runs each. Verify: one run record per run, each with a passed live-server check. @@ -93,8 +108,8 @@ Design D0.14. Protocol section 0.12. ## 6. Analyse and report -- [ ] 6.1 Compute the design D6 statistics per dataset and arm, averaging 3 runs per question first. Verify: every number in the result tables comes from a named `output/` file. -- [ ] 6.1a Test the secondary hypothesis H-CC (protocol 4.4): per-question unsupported-rate gap, Granite minus GLM-5.3, with a question-level bootstrap per dataset. Exclude the 15 smoke questions. Then the operator labels 100 disagreement claims blind, shuffled, from both directions (25 per direction from each of FreshStack and FinanceBench), at most one claim per question. A question with no claims leaves the bootstrap, and a dataset with fewer than 30 questions left is inconclusive (protocol 4.4). Verify: the gap intervals and the hand-label share come from named `output/` files, and no smoke question ID appears in the input. +- [ ] 6.1 Compute the statistics of `preregistration.md` section 9 per dataset and contrast row, averaging 3 runs per question first (design D6 is history). Verify: every number in the result tables comes from a named `output/` file. +- [ ] 6.1a (History: H-CC and Granite Guardian were dropped on 2026-11-12, design D0; this task does not run and is not ticked.) Test the secondary hypothesis H-CC (protocol 4.4): per-question unsupported-rate gap, Granite minus GLM-5.3, with a question-level bootstrap per dataset. Exclude the 15 smoke questions. Then the operator labels 100 disagreement claims blind, shuffled, from both directions (25 per direction from each of FreshStack and FinanceBench), at most one claim per question. A question with no claims leaves the bootstrap, and a dataset with fewer than 30 questions left is inconclusive (protocol 4.4). Verify: the gap intervals and the hand-label share come from named `output/` files, and no smoke question ID appears in the input. - [ ] 6.2 Write `report.md` with the Answer box, one table per dataset in the proposal layout, the determinism result, run-to-run SD, hand-check agreement and limitations. Report Experiment 22 as a reference only. Verify: `omrs experiments check` passes and every number names its source file. - [ ] 6.3 Update `status` and `finding` in `experiment.json`, then rebuild the index with `omrs experiments index --write`. Verify: the index row shows the new status. diff --git a/openspec/changes/rag-benchmark-readiness-flow/design.md b/openspec/changes/rag-benchmark-readiness-flow/design.md index 5d413ae5..2529b6aa 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/design.md +++ b/openspec/changes/rag-benchmark-readiness-flow/design.md @@ -61,7 +61,7 @@ The cuts change and the router change each change the index identity. The operat | G6 | 5.3 | RunPod budget and start of the sweep | | G7a | 6.2 | Pre-freeze pod session: quoted price, go | | G7b | 6.8 | Merge the Exp 35 harness PR | -| G7 | 6.11 and 7.3 | Freeze Experiment 35, approve the run budget, and deposit Exp 35, A3 and the Part 2 protocol on OSF together | +| G7 | 6.11 and 7.3 | Freeze Experiment 35, approve the run budget, and deposit the pre-registration (main file), the proposal, `protocol.md`, `plan.json` and the Part 2 protocol on OSF together | ### D4. Rules every group follows @@ -89,9 +89,13 @@ D. Section 2 harness, PR (G7b) | E. Arm smoke run on the run machine; pod stopped after it | +E2. A4 pre-freeze items: FreshStack full corpus pin and timing, + | halves and MDE pilot on half A, P1 to P16, open items O1 to O6 + | (with Part 2 steps 7.1, 7.2 and 7.2a) F. Freeze (G7) | -G. Section 5 runs (3 per arm), section 6 report, section 7 release +G. Search stage; score half A; selection.json; OSF time-stamp of the + hash; score half B; replicate pods; section 6 report; section 7 release ``` Reasons: @@ -100,6 +104,15 @@ Reasons: - The arm smoke run goes after the harness. Every arm needs the section 2 code. Archived task 10.6 and Exp 35 task 4.1 are the same check, so one run does both. - The smoke run uses the run machine. Corpus and indexes stay in place for section 5. A stopped pod bills only its disk while the operator decides at G7. - `pre-exp35-harness-and-answer-fixes` was archived with group 10 open. This group owns those tasks now. The detail stays in that change's design D13. +- Step E2 (added 2026-11-14, amendment A4) holds the pre-freeze items of `preregistration.md` section 23. The pilot and every pre-freeze sample drawn after the halves exist come from half A, so the halves are written first; earlier samples are listed with their halves (P12). Step G follows the order of work of pre-registration section 14.2: no half-B score exists before the configuration hash is time-stamped on OSF. + +### D6. Amendment A4: the pre-registration governs; deviation policy (2026-11-14) + +A publication-grade methods review (2026-11-13) gave 30 findings, F1 to F30. The user approved every recommended fix on 2026-11-14. The file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35 and is the main file of the OSF deposit. The proposal stays the master plan for scope, order of work, Part 2 and E1. For Part 2 and E1, pre-registration section 18 wins over the proposal on the points it fixes; the proposal governs every other fact until the Part 2 protocol is frozen, and then the Part 2 protocol governs. The step 7.2a cost decision, recorded before any half-B score, fixes the Part 2 estimand: "selected OMRS against selected RAGFlow" if the matched search runs, else "selected OMRS against default RAGFlow". `exp35-main-run-readiness` design D38 and its task group 16 build the design; group 6 (steps 6.10a to 6.10d and 6.12 to 6.14) and group 7 (steps 7.1 to 7.6, with 7.2a added) of this change order it. + +- **Deviation policy.** After the freeze, every change is a dated, numbered deviation. The report lists each with its reason, its time (before or after seeing the affected data) and its effect on each claim. Each goes into `plan.json` `amendments` and is uploaded to the OSF project. A run on another llama.cpp build needs a deviation first. Part 2 states the same policy in its protocol. +- **Budget.** Experiment 35: about $730 (estimate), ceiling $975, with the results-blind drop order of pre-registration section 17: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt; then stop. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings at step 7.2a; E1 is not in the Experiment 35 drop order. +- **Reporting.** The Experiment 35 report holds the items of pre-registration section 21 (decisions with "not testable" marks, unjudged share, sensitivity tables, compute table with every pre-freeze pod, returned model IDs, deviation log). The Part 2 report holds its own release table and maintainer correspondence log. ## Risks diff --git a/openspec/changes/rag-benchmark-readiness-flow/proposal.md b/openspec/changes/rag-benchmark-readiness-flow/proposal.md index 61df26b3..927f355d 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/proposal.md +++ b/openspec/changes/rag-benchmark-readiness-flow/proposal.md @@ -15,6 +15,7 @@ Without one recorded order, two branches can touch the same specs at once, the l - A task order only (tasks.md groups 0 to 8) with gates. No code and no capability deltas. - The order (operator decisions 2026-11-06): cuts, then router, then one live switch (no re-index: the live collections are test data, operator 2026-11-07), then the pipeline sweep on RunPod, then Experiment 35. The page routing panel follows the router and can run beside the sweep, because it touches no runtime and no result. +- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the user's open items O1 to O6) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). ## Capabilities @@ -30,5 +31,5 @@ None. - No code. Each group names the change or experiment whose own tasks carry the detail. - Live state: group 4 changes the live app (`~/.omrs`, `~/.config/omrs/omrs.jsonc`, the Llama app) and re-indexes every collection once. It needs the operator's word (gate G5). -- Cost: group 5 runs on RunPod and needs the operator's word on budget (gate G6). Group 6 spends money at the pre-freeze pod and GLM-5.3 run (gate G7a, about 2 to 4 hours of pod plus about $11) and at the smoke and section 5 runs (gate G7). Group 7 adds the Part 2 RAGFlow run on half B and the descriptive E1 run; E1 is dropped first when the budget is short. +- Cost: group 5 runs on RunPod and needs the operator's word on budget (gate G6). Group 6 spends money at the pre-freeze pod and GLM-5.3 run (gate G7a, about 2 to 4 hours of pod plus about $11) and at the smoke and section 5 runs (gate G7). Group 6 adds the FreshStack full-corpus builds and the MDE pilot (estimate about $730 in all, ceiling $975, with a results-blind drop order). Group 7 adds the Part 2 RAGFlow run on half B and the descriptive E1 run, each with its own estimate and ceiling; E1 is dropped first when the budget is short. - Parked, not in this flow: `paddleocr-vl-llamacpp-backend`, `tripwire-retirement-and-provider-symmetry` (watch its overlap with the router's key retirements), issue #161 old-print follow-up, an Indra-on-Latin follow-up, issues #173 and #174. diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 8d620f03..36155086 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -1,6 +1,6 @@ # Tasks -The order is the operator's (2026-11-06, design D1). Do not reorder groups. Exception: steps 7.1 and 7.2 run before task 6.11 (the freeze), as group 7 states. Stop at every gate (design D3). Each group names the change whose own `tasks.md` carries the detail. Rules: design D4. +The order is the operator's (2026-11-06, design D1). Do not reorder groups. Exception: steps 7.1, 7.2 and 7.2a run before task 6.11 (the freeze), as group 7 states. Stop at every gate (design D3). Each group names the change whose own `tasks.md` carries the detail. Rules: design D4. ## 0. Preflight @@ -81,26 +81,54 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime - [ ] 6.10 One smoke run that does archived 10.6 and Exp 35 task 4.1 together: a seeded sample of the factorial setups (Exp 35 amendment A1), 5 seeded questions per dataset, twice, through the claim checkers GLM-5.3 (main) and Clef-Flash (second), and `chat`. Granite Guardian is dropped. Run it on the machine that task 1.5 names. On a pod, quote the price first, and stop the pod after the smoke run (only its disk bills) until G7. Verify: zero errors, a verdict per strategy for every claim, complete call records, an identical second run, and the budget of Exp 35 task 4A.5 (ceiling $975) not exceeded on extrapolation. History: the first smoke run, under the old six-arm design, was done 2026-11-10/11 on pod walsi2c5gfb8xc: all 6 arms, 3 datasets, 5 questions, 2 runs, all three checkers; retrieval identical in 18 of 18 pairs. Report: DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/prefreeze-report-2026-11-11.md. Pod deleted. Open: the extrapolation check needs the budget of Exp 35 task 4A.5. The FreshStack-only smoke rerun is done (2026-11-11, pod 2elh5ji2og6d74, deleted): Clef-Flash passes the guard (ADR-119 A7), and its two harness errors are fixed in PR #207. Tick this task only when the factorial smoke run above passes every Verify check, and after 4A.5 and the zero-error recheck of FreshStack questions 77536364 and 77762811 (Exp 35 task 4.1 (c)). The old six-arm smoke run does not count for the factorial setups (PR 209 review, 2026-11-13). +### E2. Amendment A4 pre-freeze items (steps 6.10a to 6.10d added 2026-11-14; design D6) + +`experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35 ("PR n" means its section n). "RDY 16.N" means task 16.N in `exp35-main-run-readiness/tasks.md`. Steps 6.10a to 6.10d run after step 6.10 and before the freeze. + +- [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. +- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval, run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: every pilot question is in half A, and the MDE table is filled. +- [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. +- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28). O1 the OSF mechanism that time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 the university ethics query and its answer, before the first label; O6 BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack) or the alternative (16 BioASQ rows, 2-of-3 for H8). Record each answer; never guess one. Verify: PR 23 shows each item decided. + ### F. Freeze -- [ ] 6.11 GATE G7: the operator approves the freeze and the run budget. Exp 35 tasks 4.2 and 4.3: commit `protocol.md` and `plan.json`, then remove the DRAFT status. The OSF deposit (Exp 35 task 4B.6) is shared with group 7 step 7.3: one deposit holds Exp 35, A3 and the Part 2 protocol. Do not deposit before step 7.1 is done. +- [ ] 6.11 GATE G7: the user approves the freeze and the run budget (estimate about $730, ceiling $975, PR 17). Steps 6.10a to 6.10d and 7.1 to 7.2a are done. Exp 35 tasks 4.2 and 4.3: commit `preregistration.md`, `protocol.md` and `plan.json` (`frozen` filled), then remove the DRAFT status. The OSF deposit (Exp 35 task 4B.6) is shared with group 7 step 7.3. It holds `preregistration.md` as the main file, with the proposal, `protocol.md` (history), `plan.json` and the Part 2 protocol. Do not deposit before step 7.1 is done. After the freeze every change is a deviation (design D6). ### G. Run and report -- [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text): 72 search setups per dataset, run once each, then the 15 answer setups (12 crossed plus 3 token-matched, A3), 3 runs each. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first arm, and delete the pod when its runs and its copy are done. -- [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics, `report.md`, `experiment.json` status. -- [ ] 6.14 Exp 35 section 7 (tasks 7.1 to 7.5): split, copy to `DATA/omrs/`, Zenodo, identifier search. Merge and archive Exp 35 on the operator's word. +- [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (mechanism O1); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. +- [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm families fixed at 19, 19 and 8 (O6), gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. +- [ ] 6.14 Exp 35 section 7 (tasks 7.1 to 7.5): the release table of PR 19 (answers, evidence IDs, ranked lists, human labels and verdicts per licence, never corpus text) and the re-judge kit, copy to `DATA/omrs/`, Zenodo (one archive per dataset licence), identifier search. Merge and archive Exp 35 on the user's word. ## 7. Part 2 (RAGFlow on half B) and E1 -Master plan: `docs/proposals/first-publication-benchmark.md`, "Order and gates", Step 3 and Step 4. "A3" means Exp 35 protocol section 0.12. Part 2 needs its own experiment number and protocol. Steps 7.1 and 7.2 run before the freeze (6.11). Steps 7.4 to 7.6 run after Exp 35 starts. - -- [ ] 7.1 Reserve the Part 2 experiment number with `omrs experiments new`. Write its protocol: hypotheses C1 and C2 (Holm per dataset across C1 and C2, then 2 of 3 core datasets; Legal exploratory; C2 exploratory if no correctness measure passes kappa 0.60), the fairness rules of the proposal Step 3, the defaults row, the human labels drawn from both systems, and a pointer to A3. Verify: `omrs experiments check` and `openspec validate --all --strict` pass. -- [ ] 7.2 Prepare RAGFlow before the freeze. Pin the release and the container digest. Connect it to the llama.cpp models through OpenAI-compatible endpoints. Time one index per dataset and record the GPU cost. Check that each FinanceBench chunk carries a page number on a seeded FinanceBench sample (the page-level gate of the Part 2 relevance unit: if RAGFlow returns no page for a chunk, the Part 2 primary unit falls back to the document for every question and both systems; the page-level result becomes a labelled sensitivity analysis). Repeat RAGFlow retrieval on a small seeded sample and report whether the ranked lists match. Send the configuration to the RAGFlow maintainers and log every reply. Verify: `protocol.md` of Part 2 holds each result. A blocker is recorded and reported, and no other model is substituted. -- [ ] 7.3 GATE G7 (shared with 6.11): OSF deposit at the Exp 35 freeze. One deposit holds the Exp 35 protocol and `plan.json`, amendment A3 and the Part 2 protocol (C1, C2). The operator does the deposit. Verify: the OSF link and the commit hash are in both protocols. -- [ ] 7.4 After the Exp 35 search stage is scored, apply A3 on half A: the embedder and reranker size from the three core datasets, then chunk size, reranker on or off and hybrid on or off. Freeze the chosen cell. Record its configuration hash on OSF before half B opens. Verify: the dated note in Exp 35 protocol section 0.12 exists before any half-B score for Part 2. -- [ ] 7.5 Run Part 2 on half B, once per system: the A3 cell and RAGFlow on the same shared models. RAGFlow evidence goes through the common OMRS answer harness (same prompt, answer model, 20,480-token budget). Run the defaults row beside it. Quote the pod price first and delete the pod when the copy is done. Verify: one table per dataset, separate from the Exp 35 tables, with paired confidence intervals. -- [ ] 7.6 E1: estimate the cost and set a ceiling for the full 511,962-document corpus. Pin the benchmark commit, the corpus revision and the leaderboard revision. Confirm the GPT-5.4 judging and submission procedure with Onyx. Run E1 only if the budget allows; E1 is the first item to drop. The answer model is GPT-5.4, as in the benchmark's own baselines, and E1 stays descriptive. Verify: the cost note and the ceiling exist before any E1 run. +Master plan: `docs/proposals/first-publication-benchmark.md`, "Order and gates", Step 3 and Step 4, and its list "Part 2 protocol must hold before freeze". "A3" means the selection rule of Exp 35 pre-registration section 14.1 (amendment A3, `protocol.md` section 0.12, is its history). "PR n" means pre-registration section n; section 18 fixes what Part 2 inherits. Part 2 needs its own experiment number and protocol. Steps 7.1, 7.2 and 7.2a run before the freeze (6.11). Steps 7.4 to 7.6 run after Exp 35 starts. Step 7.2a is added (2026-11-14) between 7.2 and 7.3. + +- [ ] 7.1 Reserve the Part 2 experiment number with `omrs experiments new` (`experiment.json` purpose `publication`, size Confirmatory). Write its protocol. It must hold, before the freeze: + 1. Estimand (PR 18), fixed by the step 7.2a decision before any half-B score: "selected OMRS against selected RAGFlow" if the matched search runs (RAGFlow hash time-stamped on OSF with the OMRS hash; default RAGFlow then a secondary, descriptive row), else "selected OMRS against default RAGFlow"; shared models fixed. + 2. RAGFlow pin: release tag, container digest of every service (RAGFlow, the search engine, the database, object store and cache), DeepDoc model versions and the API version. + 3. Settings table with three columns (shared and fixed, default kept, set for measurement), covering chunk method and size, tokenizer, parser and layout model, every LLM-based parsing feature (off or stated), embedder, query instruction, reranker and its instruction, keyword and vector weights, similarity threshold, knn depth, page size and depth. Shared and fixed: the chosen embedder; the reranker when the A3 cell has it on (same Qwen3-Reranker file and size, through RAGFlow's rerank setting); the answer model, prompt and 20,480-token budget; checkers and judges; the corpus snapshot (full FreshStack corpus); question IDs and halves. Set for measurement: similarity cut-off 0, retrieval depth 100. Every other default is read from the pinned release, not from main-branch documentation (P16). + 4. Serving: RAGFlow calls the same llama.cpp router, build and GGUF hashes as OMRS, with the environment record and `verify_llamacpp_runtime` before the first call; run and environment record rules as Part 1. + 5. Evidence mapping (chunk to unit ID per dataset; FinanceBench page unit with the page gate and the document fallback of PR 18) and the common answer harness input format (evidence item, Qwen3 token count, truncation record). + 6. Pre-freeze RAGFlow samples (page-number gate, repeatability check) from half A only, with seeds; the repeatability rule and what a mismatch means for C1. + 7. C1 and C2: one Holm family of 2 per core dataset, the 2-of-3 rule with the same sign on at least 2 core datasets (opposite signs reported as a dataset-dependent effect; the hypothesis does not hold), Legal exploratory. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`, with number match secondary. Tests as PR 9: GEE on question clusters for FreshStack and BioASQ, the document-level sign-flip test on FinanceBench; no McNemar test. C2 needs its own correctness gate (the PR 11.2 rule on labels from both systems); a failed gate leaves C2 in its family as "not testable". This replaces the earlier rule "C2 exploratory below kappa 0.60". + 8. Missing outcomes (PR 18): every half-B question stays in both systems' denominators; a RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2; a short list scores missing ranks as not relevant; a document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects; an unmappable chunk counts as not relevant and is counted; a sensitivity analysis uses the questions where both systems indexed every gold document. + 9. Labels: L2 is the primary Part 2 labeller; system identity is hidden; both systems' answers are shuffled into one list; written guidelines (P8); counts and seeds. + 10. Tuning budget: the exact matched RAGFlow cell list (settings varied and their levels), fixed and recorded before the step 7.2 cost measurement; the matched RAGFlow search decision of step 7.2a; the search-cost figure with its estimator and a RAGFlow curve only if the matched search runs. + 11. Defaults row: one RAGFlow run (pinned-release defaults, the shared parts and the two measurement settings, PR 18) and its secondary role; when the matched search does not run, it is the main RAGFlow run and the table shows it once. + 12. Own estimate, ceiling and drop order for Part 2 (P16). + 13. Conflict of interest: the authors built OMRS; safeguards as PR 11.3, the maintainer correspondence log and a rerun kit. + 14. Release table for RAGFlow configurations, outputs and maintainer replies (PR 19). + 15. The A3 configuration hash definition (PR 14.1) and its OSF time-stamp before any half-B score (O1). + 16. A deviation policy as PR 20. + + Verify: the protocol holds items 1 to 16; `omrs experiments check` and `openspec validate --all --strict` pass. +- [ ] 7.2 Prepare RAGFlow before the freeze. Pin the release and the container digests (item 2). Read every RAGFlow default from the pinned release and fill the settings table (item 3). Connect RAGFlow to the llama.cpp models through OpenAI-compatible endpoints. Time one index per dataset (corpus only) and record the GPU cost, so the projected pod cost of a matched RAGFlow search on half A is known. Check that each FinanceBench chunk carries a page number on a seeded half-A FinanceBench sample (the page-level gate of the Part 2 relevance unit: if RAGFlow returns no page for a chunk, the Part 2 primary unit falls back to the document for every question and both systems; the page-level result becomes a labelled sensitivity analysis). Repeat RAGFlow retrieval on a small seeded half-A sample and report whether the ranked lists match. Send the configuration to the RAGFlow maintainers and log every reply. Quote the pod price first and delete the pod when done. Verify: the Part 2 protocol holds each result, and every sampled question is in half A. A blocker is recorded and reported, and no other model is substituted. +- [ ] 7.2a Matched RAGFlow search decision (added 2026-11-14; F2, P16). From the step 7.2 timing, project the pod cost of a retrieval-only RAGFlow search on half A (the same selection rule, over the cell list fixed in item 10 before step 7.2). If it is at most $50, the search runs on half A before any half-B score exists; otherwise it does not run. This recorded decision fixes the C1 and C2 estimand (PR 18), so it is never chosen after seeing results: if the search runs, "selected OMRS against selected RAGFlow" (the RAGFlow cell chosen on half A by the same rule, its configuration hash time-stamped on OSF with the OMRS hash before any half-B score; default RAGFlow becomes a secondary, descriptive row); if not, "selected OMRS against default RAGFlow". Write the Part 2 estimate and ceiling, and the E1 estimate and ceiling (P16). Verify: the decision, the estimand, the projected cost and the date are in the Part 2 protocol before the freeze and before any half-B score. +- [ ] 7.3 GATE G7 (shared with 6.11): OSF deposit at the Exp 35 freeze. One deposit holds `preregistration.md` as the main file, the proposal, `protocol.md` (history), `plan.json` and the Part 2 protocol (C1, C2). The user does the deposit. Verify: the OSF link and the commit hash are in the pre-registration and the Part 2 protocol. +- [ ] 7.4 After the Exp 35 search stage, score half A only and apply A3 (PR 14.1): the embedder and reranker size from the three core datasets, then chunk size, reranker on or off and hybrid on or off. Write `selection.json` with the configuration hash (PR 14.1 definition). Time-stamp the hash on OSF (O1) before any half-B score, for Exp 35 or Part 2. If the matched RAGFlow search runs (step 7.2a), it runs on half A in this step too, and the selected RAGFlow configuration hash is time-stamped with the OMRS hash. Verify: the OSF time-stamp is older than the first half-B score. +- [ ] 7.5 Run Part 2 on half B, once per system: the A3 cell and RAGFlow (the cell selected in step 7.4 if the matched search ran, else default RAGFlow) on the same shared models and settings table. RAGFlow evidence goes through the common OMRS answer harness (same prompt, answer model, 20,480-token budget). Run the defaults row beside it. L2 labels with system identity hidden. Apply the Part 2 missing-outcome rules and drop order. Quote the pod price first and delete the pod when the copy is done. Verify: one table per dataset, separate from the Exp 35 tables, with C1 and C2 decided by the PR 9 method and Holm over the family of 2; the release table items exist. +- [ ] 7.6 E1: use the E1 estimate and ceiling of step 7.2a for the full 511,962-document corpus. Pin the benchmark commit, the corpus revision and the leaderboard revision. Confirm the GPT-5.4 judging and submission procedure with Onyx. Run E1 only if the budget allows; E1 has its own ceiling and is dropped first when the project budget is short (proposal; it is not in the PR 17 drop order). The answer model is GPT-5.4, as in the benchmark's own baselines, and E1 stays descriptive. Verify: the cost note and the ceiling exist before any E1 run. ## 8. Close From 21c661ee689cd750bf1cf0d35617cfe7a972927f Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 17:39:36 +0000 Subject: [PATCH 2/9] docs(exp35): address PR 210 review round 1 Fix the twelve CodeRabbit threads on PR 210. The pre-registration names the primary measure as faithfulness of cited factual claims, adds the uncited-segment share, a registered sensitivity and open item O7, makes undefined gate statistics fail closed, limits the early stop to preflight, G1 to G3 and the ceiling, states one 2-of-3 same-sign rule for C1 and C2, and sets Part 2 C2 missing-outcome rules. OpenSpec tasks and specs mirror these and fix the half-B hash guard, the 52,200-answer budget count, the pilot-declined path, the closed-book drop fallback, the pre-freeze task split and the O6 family size. --- docs/proposals/first-publication-benchmark.md | 10 +++---- .../plan.json | 9 ++++-- .../preregistration.md | 30 +++++++++++++------ .../protocol.md | 15 +++++++++- .../exp35-main-run-readiness/design.md | 6 ++-- .../exp35-main-run-readiness/proposal.md | 4 +-- .../specs/benchmark-answer-scoring/spec.md | 26 ++++++++++++++++ .../specs/benchmark-harness/spec.md | 7 ++++- .../changes/exp35-main-run-readiness/tasks.md | 22 +++++++------- .../specs/evaluation-observability/spec.md | 4 +++ .../tasks.md | 16 +++++----- .../rag-benchmark-readiness-flow/design.md | 2 +- .../rag-benchmark-readiness-flow/proposal.md | 2 +- .../rag-benchmark-readiness-flow/tasks.md | 12 ++++---- 14 files changed, 116 insertions(+), 49 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index 2fe4b77e..d6348452 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -55,7 +55,7 @@ The claims stay testable. OMRS reaches stated scores under stated conditions. Na ## Order and gates -1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The open items O1 to O6 and the pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The Part 2 protocol exists before this point (it is not written yet). +1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The open items O1 to O7 and the pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The Part 2 protocol exists before this point (it is not written yet). 2. Deposit on OSF in one deposit: the pre-registration as main file, this proposal, the protocol history, `plan.json` and the Part 2 protocol (C1, C2). Record the link and the commit hash. 3. Build and seal the 9 indexes per dataset. The rebuild gate G2 passes before any search writes output. Then run the 72 search setups on all questions (pre-registration section 14.2). 4. Score half A only. Apply the A3 rule. Write `selection.json`. @@ -106,7 +106,7 @@ The primary metric is nDCG@10 at unit level, with the trec_eval formulas written The answer stage uses the embedder and reranker size chosen on half A (pre-registration section 14.1). It has 15 setups, each run 3 times on all 1,160 questions: 12 crossed setups (chunk size x reranker on or off x hybrid on or off, 10 chunks of evidence each) and 3 token-matched setups (`c512-t20k`, `c1024-t20k`, `c2048-t20k`, reranker and hybrid on, evidence cut at 20,480 Qwen3 tokens). Two closed-book runs (own-knowledge prompt, and the RAG prompt with an empty evidence block) and an oracle run (gold evidence given directly) frame the results (pre-registration section 13). -- Faithfulness: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. +- Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; open item O7 decides before the freeze whether it becomes primary). - Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. - Human check: two labellers score samples blind (L1 750 labels, L2 225). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. @@ -192,7 +192,7 @@ Half B only. No selection step in Part 1 or Part 2 reads half B. Every pre-freez 7. **Human labels.** Part 2 adds its own correctness labels and human-agreement sample, drawn from the answers of both systems, so judge agreement covers RAGFlow answers. L2 is the primary Part 2 labeller, so the OMRS authors' labels are the check, not the gate. System identity is hidden: both systems' answers are shuffled into one list. Labellers follow a written Part 2 correctness guideline, with worked examples from outside every sample, fixed before the first label. They run in Part 2. The Experiment 35 label counts do not change. 8. **Report.** Map every retrieved chunk to the dataset's unit ID. Give one table per dataset, separate from the Experiment 35 tables. 9. **Matched RAGFlow search (conditional).** A retrieval-only RAGFlow search on half A (the same selection rule, 12 to 24 cells). Its exact cell list (the RAGFlow settings varied and their levels) is fixed and recorded before the cost measurement at master flow step 7.2. It runs only if its projected pod cost, measured at master flow step 7.2, is at most $50. The decision and its cost figure are recorded at master flow step 7.2a, before any half-B score exists. The decision fixes the estimand (see "Estimand" above). If the search runs, the selected RAGFlow cell enters C1 and C2, its configuration hash is time-stamped on OSF with the OMRS hash before any half-B score exists, default RAGFlow becomes a secondary, descriptive row, and the search-cost figure shows a RAGFlow curve. -10. **Missing outcomes.** Every half-B question stays in both systems' denominators. Retries follow the content-blind resend rule of Part 1 only. A RAGFlow timeout or error that remains scores 0 for C1 and gets an answer from empty evidence for C2. A list shorter than 10 units scores the missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. A chunk that does not map to a dataset unit counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. +10. **Missing outcomes.** Every half-B question stays in both systems' denominators. Retries follow the content-blind resend rule of Part 1 only. A RAGFlow timeout or error that remains scores 0 for C1 and gets an answer from empty evidence for C2. A list shorter than 10 units scores the missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. A chunk that does not map to a dataset unit counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. The pre-registration section 12 rules apply to both systems alike: an answer still missing after the resend rule, from either system, scores 0 for correctness. A question enters the C2 contrast only when both systems' Ragas rows are judged (complete pairs). Every exclusion is counted per system, and a sensitivity analysis scores unjudged rows 0 and keeps every question (pre-registration section 18). 11. **Conflict of interest.** The authors built OMRS. The paper states this. The safeguards are: the pre-registration and its deviation log; selection on half A only; Holm families fixed before the run; blind labels with L2 as primary labeller; the published RAGFlow settings table and maintainer correspondence; and release of every output, RAGFlow outputs included. ### Hypotheses C1 and C2 @@ -202,7 +202,7 @@ Half B only. No selection step in Part 1 or Part 2 reads half B. Every pre-freez - **C1:** selected OMRS and RAGFlow differ in retrieval nDCG@10. Null: zero difference. Two-sided, per core dataset. - **C2:** selected OMRS and RAGFlow differ in answer correctness, measured by continuous Ragas `AnswerCorrectness`. Null: zero difference. Two-sided, per core dataset. FinanceBench number match is secondary. -C1 and C2 form their own Holm family, separate from H1 to H8b: Holm runs per dataset across C1 and C2 (a family of 2 per core dataset). The cross-dataset rule is the 2-of-3 rule of Part 1 with the same-sign rule: a hypothesis holds when it counts with the same sign on at least 2 of the 3 core datasets (FinanceBench, FreshStack, BioASQ). Opposite signs are reported as a dataset-dependent effect, and the hypothesis does not hold. The report states the direction. Each test pairs the two systems question by question, with the analysis of pre-registration section 9: +C1 and C2 form their own Holm family, separate from H1 to H8b: Holm runs per dataset across C1 and C2 (a family of 2 per core dataset). The cross-dataset rule is the 2-of-3 rule of Part 1 with the same-sign rule: a hypothesis holds when it counts with the same sign on at least 2 of the 3 core datasets (FinanceBench, FreshStack, BioASQ). A count with the opposite sign on the third dataset is reported as a dataset-dependent effect and does not stop the hypothesis holding. The report states the direction. Each test pairs the two systems question by question, with the analysis of pre-registration section 9: - FreshStack and BioASQ: GEE on question clusters with bias-reduced covariance (Mancl and DeRouen 2001) and a t reference distribution. - FinanceBench: questions share documents, so the decision p-value comes from a document-level sign-flip permutation test, and the interval inverts the same test. The randomisation test is the recommended test for comparing two IR systems (Smucker, Allan and Carterette 2007); here it flips signs per document, not per question. @@ -358,7 +358,7 @@ The Experiment 35 limitations are in pre-registration section 22. The paper repe 4. Ask the Napier library to confirm the article type and the deal. 5. Done: Experiment 35 amendment A3 (protocol section 0.12) holds the selection rule for Part 2. 6. Done: amendment A4 (protocol section 0.13) and the pre-registration, after the methods review of 13 November 2026. -7. Close the open items O1 to O6 and the pre-freeze tasks P1 to P16 of pre-registration section 23. +7. Close the open items O1 to O7 and the pre-freeze tasks P1 to P16 of pre-registration section 23. 8. Reserve an experiment number for Part 2. Write its protocol, covering every item of "What the Part 2 protocol must hold before its freeze". 9. Before the freeze, pin the RAGFlow release and container digests and connect it to the llama.cpp models. Read the RAGFlow defaults from the pinned release. Time one index per dataset (corpus only). Measure the projected pod cost of the matched search at master flow step 7.2 and record the decision at step 7.2a; it fixes the Part 2 estimand. Check page provenance on a seeded half-A FinanceBench sample (the rule 3 gate: if RAGFlow returns no page for a chunk, the primary unit falls back to the document for both systems) and retrieval repeatability on half A. Send the settings table to the RAGFlow maintainers. 10. Write the Part 2 estimate and ceiling, and the E1 estimate and ceiling (pre-freeze task P16; no figure exists yet). diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 0dae682f..92ac0ea6 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -59,7 +59,7 @@ "null": "each contrast is zero" } ], - "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks, rows 12 to 19 pending open item O6); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs; opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", + "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks, rows 12 to 19 pending open item O6); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis needs the same sign: under the both-datasets rule (H6, H8a, H8b) the row counts with the same sign on FinanceBench and FreshStack, and opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold; under a 2-of-3 rule (Part 2 C1 and C2; H8a and H8b under the O6 alternative) the row counts with the same sign on at least 2 of the 3 core datasets, and an opposite-sign count on the third is reported as a dataset-dependent effect and does not stop the hypothesis holding. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", "stats": "expert", "seed": 20261101, "baseline": "none: full factorial of embedder x chunk size x reranker x hybrid (72 search setups per dataset); e4b-c1024-roff-hoff is the reference cell of the G2 rebuild check only", @@ -90,6 +90,11 @@ "binary": false, "primary": false }, + { + "name": "uncited_segment_share", + "binary": false, + "primary": false + }, { "name": "financebench_number_match", "binary": true, @@ -375,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b by open item O6), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF, then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O1 to O6 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF.", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b by open item O6), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF, then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O1 to O7 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (open item O7 decides whether it becomes primary); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file.", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index b5cadf63..22e7b3af 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -77,13 +77,14 @@ For question q with gold set G_q (size R_q) and unit list L_q of up to 10 units, **Answer metrics.** Per answer, then averaged over the 3 runs per question, then over questions. -- **Faithfulness** (`faithfulness_glm_factual`): share of the answer's factual claims (classifier kinds fact and mixed) that GLM-5.3 calls supported. Refusal claims are reported apart. Claims and citations come from the answer itself: the answer model cites evidence by ordinal (for example `[1]`), the saved answer is split into claims, only prose segments with at least one valid ordinal go to the checker, and the checker sees one claim with the text of the evidence its ordinals cite (OpenSpec specs `grounded-answer-synthesis` and `benchmark-answer-scoring`). +- **Faithfulness of cited factual claims** (`faithfulness_glm_factual`): share of the answer's cited factual claims (classifier kinds fact and mixed) that GLM-5.3 calls supported. Refusal claims are reported apart. Claims and citations come from the answer itself: the answer model cites evidence by ordinal (for example `[1]`), the saved answer is split into claims, only prose segments with at least one valid ordinal go to the checker, and the checker sees one claim with the text of the evidence its ordinals cite (OpenSpec specs `grounded-answer-synthesis` and `benchmark-answer-scoring`). A prose segment with no valid ordinal (an uncited segment) is not a claim and is not checked, because it names no evidence to check against. +- **Uncited segments.** Co-outcome, reported per setup and dataset: `uncited_segment_share`, the share of the answer's prose segments that carry no valid ordinal. Registered sensitivity analysis: the classifier also sorts the uncited segments (it sees only the question and the segment, section 11.1); uncited segments it calls fact or mixed count as unsupported, and faithfulness is computed again over cited and uncited factual segments together. Open item O7 decides before the freeze whether this version becomes the primary measure. - **Correctness** (`ragas_answer_correctness`): Ragas 0.4.3 (Es et al. 2023) `AnswerCorrectness`, default weights (factual 0.75, similarity 0.25), judge `openai/gpt-oss-120b`, local 4B embedder. The judge sees the question, the answer and the reference answer, never the contexts. FreshStack's reference is `answer_text` (the accepted Stack Overflow answer). - **Secondary**: Clef-Flash faithfulness, FinanceBench number match (`protocol.md` section 0.6 rule), refusal rate, the four other Ragas metrics, closed-book and oracle correctness. ## 7. Hypotheses -Each null is a zero effect. Every test is two-sided. A directional contrast counts when its Holm-adjusted p is below 0.05 and its estimate has the predicted sign (this is a one-sided test at 0.025). A two-sided contrast counts when its Holm-adjusted p is below 0.05. A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs. Rows that count with opposite signs are reported as a dataset-dependent effect, and the hypothesis does not hold. The report always states the direction. Marginal contrasts weight every cell of the other factors equally (for example, the embedder effect averages over 24 cells, 18 of them reranker-on). Row numbers refer to section 8. +Each null is a zero effect. Every test is two-sided. A directional contrast counts when its Holm-adjusted p is below 0.05 and its estimate has the predicted sign (this is a one-sided test at 0.025). A two-sided contrast counts when its Holm-adjusted p is below 0.05. A two-sided hypothesis needs the same sign. Under a both-datasets rule (H6, H8a, H8b), the row must count with the same sign on FinanceBench and on FreshStack; if it counts with opposite signs, it is reported as a dataset-dependent effect, and the hypothesis does not hold. Under a 2-of-3 rule (C1 and C2 in section 18; H8a and H8b under the O6 alternative), the row must count with the same sign on at least 2 of the 3 core datasets. A count with the opposite sign on the third dataset is reported as a dataset-dependent effect and does not stop the hypothesis holding. The report always states the direction. Marginal contrasts weight every cell of the other factors equally (for example, the embedder effect averages over 24 cells, 18 of them reranker-on). Row numbers refer to section 8. - **H1 (embedder).** Within scope S, a bigger embedder ranks gold units higher. Rows 1 (4B minus 0.6B) and 2 (8B minus 4B), nDCG@10, directional. Null: each contrast is zero. Holds: each step holds when its row counts on at least 2 of the 3 core datasets. H1 holds in full when both steps hold, in part when one does. - **H2 (reranker).** Within scope S, the reranker raises nDCG@10. Row 3 (mean of the three reranker sizes minus off), directional. Null: zero. Holds when row 3 counts on at least 2 of 3 core datasets. @@ -162,7 +163,7 @@ A non-significant result means no difference was detected at this sample size. T - **Main claim checker**: GLM-5.3 (Z.AI Coding Plan endpoint, `chat`, reasoning `low`, temperature 0, seed 20261109). **Second checker**: Clef-Flash (local GGUF). If Clef-Flash fails its faithfulness gate, its results are reported as not validated and it changes no confirmatory row. The runner never switches checker itself; a failed call stops the stage and `--resume` retries it. If GLM-5.3 is withdrawn before the run ends, that is a deviation (section 20). - **Checker tolerance.** A local checker may replace the main one only under the rule of `protocol.md` section 4.4. Its 2-point allowance is restated against the H8a and H8b faithfulness MDE once the MDE table is filled: the bar stays only if it is at most half that MDE (P7). The withdrawn "about 4 points" figure is not used. -- **Claim classifier**: `openai/gpt-oss-120b` sorts claims into fact, refusal or mixed. Pre-freeze gate: agreement at least 0.90 and refusal recall at least 0.90 on 100 L1-labelled smoke claims (seed 20261112). +- **Claim classifier**: `openai/gpt-oss-120b` sorts claims into fact, refusal or mixed. It also sorts the uncited segments for the sensitivity analysis of section 6, with the same prompt and settings. Pre-freeze gate: agreement at least 0.90 and refusal recall at least 0.90 on 100 L1-labelled smoke claims (seed 20261112). A failed classifier gate blocks the freeze, so it cannot stop a measured run. - **Ragas judge**: `openai/gpt-oss-120b` (reasoning `low`, temperature 0, one pinned upstream host). Its seed, and the classifier's seed, are pins still to record (P10). There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. ### 11.2 Gates @@ -171,6 +172,15 @@ Gate decisions are computed and recorded, with a time, before any H8 contrast re **Interval method for every gate.** Every gate interval (kappa, the AUC lower bound, the specificity difference) is a percentile bootstrap: 10,000 resamples, resampling by question, so all claims or answers of one question move together. Each interval has its own seed (section 16). +**Undefined statistics fail closed.** A statistic is undefined when its sample lacks a class it needs: kappa when both label sets hold one and the same class (chance agreement 1); AUC when the human labels hold one class; the specificity difference when a chunk-size level holds no human-supported claim. The rule: + +1. Undefined on the full labelled sample: the gate fails. For the differential-error rule, rows 12, 13, 16 and 17 become not testable. +2. A resample is never drawn again. Each undefined resample is counted, and the report gives the count. +3. More than 5% of resamples undefined (over 500 of 10,000): the gate fails as in point 1. +4. Otherwise an undefined kappa resample takes 0 and an undefined AUC resample takes 0.5 (the chance values). Both sit below their pass thresholds, so they can never help a gate pass. An undefined specificity-difference resample has no chance value, so it is left out of the interval; point 3 caps this at 5%. + +Why: a chance value makes an undefined resample count against a pass, and point 3 stops a sample with too few items of one class from passing on its defined resamples alone. A bootstrap stratified by label would keep both classes in every resample, but the resampling unit is the question, and label and chunk size are not question properties. + - **Faithfulness gate, per checker.** Cohen's kappa (Sim and Wright 2005) of the checker against the human label: point estimate at least 0.60 and lower 95% bound at least 0.40. Sample: 450 claims from the crossed setups, 150 per chunk size, 50 per core dataset within each chunk size, at most one claim per answer, seed 20261116. The claims are drawn from answers to all questions (both halves), because H8a and H8b are tested on all questions. The correctness sample below stays on half B. The gate is pooled over the three core datasets because 150 claims per dataset are too few for a per-dataset gate; kappa per checker and per core dataset is reported with its interval as descriptive and decides nothing. "Can't tell" counts as not supported (the binary benchmark scale); kappa is also reported with "can't tell" items left out. If GLM-5.3 fails, rows 12, 13, 16 and 17 are not testable on every dataset. - **Differential-error rule.** "Unsupported" is the positive class. Specificity is the share of human-supported claims the checker calls supported. The rule uses GLM-5.3, the main checker; Clef-Flash specificity is reported only. If the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, H8a and H8b faithfulness (rows 12, 13, 16, 17) is exploratory and not testable. Sensitivity per chunk size is reported. - **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. The Ragas score is cut at 0.5 (fixed before labelling). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). @@ -204,7 +214,7 @@ All are single runs on all 1,160 questions, scored like the main runs, descripti - **Closed-book, own-knowledge prompt (floor).** The answer model with no evidence and a fixed prompt that asks for an answer from its own knowledge. - **Closed-book, RAG prompt.** The main answer prompt with an empty evidence block, so it differs from the RAG setups only in the evidence. A registered sensitivity analysis. - **Oracle (ceiling).** Gold units given directly as evidence (Legal gold passage, FinanceBench gold page texts, BioASQ gold passages, FreshStack gold documents), capped at 20,480 Qwen3 tokens with truncation recorded. It has no distractors, so it is an upper bound, not a fair comparison. -- **Contamination sensitivity.** Rows 14, 15, 18 and 19 are re-run on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5). The report gives closed-book correctness per dataset beside every correctness table, and a per-question cross-table on retrieval misses (closed-book correct or not against RAG correct or not). +- **Contamination sensitivity.** Rows 14, 15, 18 and 19 are re-run on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5). If the drop order of section 17 drops the RAG-prompt run, the subset uses the own-knowledge run alone, and the report says so. The report gives closed-book correctness per dataset beside every correctness table, and a per-question cross-table on retrieval misses (closed-book correct or not against RAG correct or not). ## 14. Selection rule, order of work and RQ1 headline @@ -312,7 +322,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti - **Ceiling**: **$975**, pods and hosted calls together (1.5 x the A2 estimate, about 1.34 x this one). At 3 times slower reranking than assumed, the total is about $950 (estimate), still under the ceiling with little margin. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. - **Checks**: before the freeze (test pod extrapolation), on the build pods after the search stage, and at each stage end while the replicate pods run. Each projection uses billing and measured rates only, never scores. - **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. -- **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early when the preflight fails, a gate fails, or a projection reaches the ceiling after the last drop step. +- **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early only when the preflight fails, an execution-integrity gate (G1 to G3, section 15) fails, or a projection reaches the ceiling after the last drop step. A failed outcome-validation gate (faithfulness or correctness, section 11.2) does not stop the run: its rows stay in their families as "not testable" (p = 1), and the rest of the run continues. The classifier gate is decided before the freeze (section 11.1). - **Part 2 and E1** each get their own estimate and ceiling before their runs (P16). ## 18. What Experiment 35 fixes for Part 2 and E1 @@ -330,8 +340,8 @@ Experiment 35 fixes the points below. On these points this section wins over the - **Matched RAGFlow search** on half A (same selection rule, 12 to 24 cells, retrieval only) guards against a weak-baseline comparison (Armstrong et al. 2009; Lin 2019; Yang et al. 2019). Its exact cell list (the RAGFlow settings varied and their levels) is fixed and recorded before the cost measurement at FLOW step 7.2. It runs only if its projected pod cost, measured at FLOW step 7.2, is at most $50. The decision is recorded at FLOW step 7.2a, before any half-B score exists, and it fixes the estimand above. - **Pre-freeze RAGFlow samples** (page-number gate, seed 20261130; repeatability check, seed 20261131) come from half A only. - **FinanceBench unit**: page for every question and both systems; pre-freeze gate that RAGFlow returns page numbers; fallback to document for every question and both systems if it fails; the other unit as labelled sensitivity (`protocol.md` section 0.12 item 4). -- **C1 and C2**: one Holm family of 2 per core dataset, Legal exploratory. Each holds when it counts with the same sign on at least 2 core datasets; opposite signs are reported as a dataset-dependent effect, and it does not hold. The report states the direction. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`; number match is secondary. Both use the analysis of section 9 (GEE on question clusters for FreshStack and BioASQ, document-level sign-flip on FinanceBench). No McNemar test where documents cluster (Obuchowski 1998). C2 needs its own Part 2 correctness gate (the section 11.2 rule, labels from both systems); a failed gate leaves C2 in the family as not testable. -- **Part 2 missing outcomes**: every half-B question stays in both systems' denominators. A RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2. A short list scores missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. An unmappable chunk counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. +- **C1 and C2**: one Holm family of 2 per core dataset, Legal exploratory. Each holds when it counts with the same sign on at least 2 of the 3 core datasets (the 2-of-3 rule of section 7). A count with the opposite sign on the third dataset is reported as a dataset-dependent effect and does not stop the hypothesis holding. The report states the direction. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`; number match is secondary. Both use the analysis of section 9 (GEE on question clusters for FreshStack and BioASQ, document-level sign-flip on FinanceBench). No McNemar test where documents cluster (Obuchowski 1998). C2 needs its own Part 2 correctness gate (the section 11.2 rule, labels from both systems); a failed gate leaves C2 in the family as not testable. +- **Part 2 missing outcomes**: every half-B question stays in both systems' denominators. A RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2. A short list scores missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. An unmappable chunk counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. The section 12 rules apply to both systems alike. An answer still missing after the resend rule, from either system, scores 0 for correctness. A question enters the C2 contrast only when both systems' Ragas rows are judged (complete pairs); every exclusion is counted per system, and a sensitivity analysis scores unjudged rows 0 and keeps every question. - **Labels**: L2 is the primary Part 2 labeller; system identity is hidden; both systems' answers are shuffled into one list (seed 20261132). The Part 2 correctness-gate sample and its bootstraps use seeds 20261133 to 20261135. - **Budget**: Part 2 gets its own estimate and ceiling (P16). @@ -361,7 +371,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - Decisions per row and per hypothesis, estimates with unadjusted 95% intervals, SESOI and MDE lines, and the gated rows marked "not testable". - **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed. - Realised chunk-length distribution per chunk-size level and dataset. -- Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit. +- Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit; faithfulness with uncited factual segments as unsupported, beside the uncited-segment share. - **Compute table**: GPU type, pod hours per stage including every pre-freeze pod (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, an energy estimate (GPU hours times board power; Strubell et al. 2019; Lacoste et al. 2019), `resent_calls`, error rows per stage and role, and every `out_of_credit` stop. - Returned model ID and provider per hosted call, with the count of differences from the requested values. - AI assistance: the plan was drafted with AI agents; the paper says so. @@ -374,6 +384,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - BioASQ is split by question, so some gold passages are shared across halves. - Chunk size is a cap, not the realised length; BioASQ and Legal chunks barely change with it. - H8 faithfulness is checker-measured, and the checker sees longer passages at bigger chunks; the differential-error rule and the length-controlled check address this only in part. +- The primary faithfulness measure checks cited factual claims only. An answer can add uncited facts without losing faithfulness; the uncited-segment share and the sensitivity analysis of section 6 show how much this happens. - H8b is conditional on reranker on and hybrid on at the chosen sizes. - The answer model may have seen benchmark text in training; the closed-book runs bound this only in part. - One index build per cell. @@ -389,6 +400,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - O4. L2's name, role and relation to the project. - O5. The ethics query to the university and its answer. - O6. BioASQ rows 12 to 19 as sanity checks (BioASQ family 8, families 19 / 19 / 8, 46 contrasts), with H8a and H8b holding only on FinanceBench and FreshStack. This changes the earlier 2-of-3 rule for H8. The alternative: keep rows 12 to 19 in the BioASQ family (16 rows, 54 contrasts, alpha 0.05/16 for the BioASQ MDE) and keep 2-of-3 for H8a and H8b. The user confirms one before the freeze. +- O7. Whether the sensitivity version of faithfulness (uncited factual segments count as unsupported, section 6) becomes the primary measure. Until the user decides, the primary measure stays faithfulness of cited factual claims. **Pre-freeze tasks:** @@ -400,7 +412,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - P6. Name the smoke and test-pod answer files that feed the H8 MDE rows. - P7. Restate the checker 2-point tolerance against the MDE table. - P8. Write the label guidelines for each set, with worked examples outside every sample. -- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16), and the length-controlled sample size (300). Seed 20261120 (audit) is used only if O3 is yes; confirm it then. +- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16), and the length-controlled sample size (300). The pilot seed is confirmed before the pilot runs (P5). Seed 20261120 (audit) is used only if O3 is yes; confirm it then. - P10. Record the tokenizer revision, answer-model host, reasoning mode, output cap and context length, and the judge, classifier and GLM-5.3 settings (the Ragas judge and classifier seeds included) in `plan.json`. - P11. Record the query instruction, the reranker instruction text and the hybrid settings in `plan.json`. - P12. Write `halves.json`; record its SHA-256; list every smoke and pre-freeze question ID with its half. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index 511cf204..887bb39a 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -56,7 +56,7 @@ Per-question marginal nDCG@10 (mean over every cell of the other factors at that ### 0.5 Answer stage - **Selection rule (amended A3, section 0.12: the pool is the three core datasets, Legal excluded; the sentence below is kept for history).** Pooled score = plain mean of the four per-dataset mean nDCG@10 values. Embedder: averaged over all other factors. Reranker size: at the chosen embedder, reranker-on setups, averaged over chunk size and hybrid. Choose the smallest size within 0.01 of the best (a gap of exactly 0.01 counts as within); equal scores go to the smaller size. The inputs and the choice go into a dated note here before the replicate pods start. -- **Setups.** Chunk size (3) x reranker on or off x hybrid on or off = 12 setups x 3 runs x 1,160 questions = 41,760 answers. +- **Setups.** Chunk size (3) x reranker on or off x hybrid on or off = 12 setups x 3 runs x 1,160 questions = 41,760 answers. (A2 and A3 add 3 token-matched setups: 15 setups, 52,200 answers.) - **Fixed.** Answer model `deepseek/deepseek-v4.1-flash-1017` (OpenRouter, one pinned host, temperature 0, seed 20261101). Ragas 0.4.3, judge `openai/gpt-oss-120b`. Claim checkers GLM-5.3 (main, Z.AI Coding Plan endpoint) and Clef-Flash (second, local). Granite Guardian is dropped, and so is H-CC (section 4.4 keeps its history). - **Refusal rule.** `openai/gpt-oss-120b` sorts claims as fact, refusal or mixed. Primary faithfulness uses fact and mixed claims; refusals are reported apart. Gate: agreement of at least 0.90 and refusal recall of at least 0.90 on 100 operator-labelled claims, seed 20261112. GLM-5.3 against Clef-Flash kappa is reported. - **Answer correctness (decision 2026-11-13).** The Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3, `ragas.metrics.collections`, default weights: factual 0.75, similarity 0.25; judge `openai/gpt-oss-120b`, local 4B embedder) against each dataset's reference answer, on all four datasets. Validation before it counts: the operator labels 100 seeded answers (seed 20261113; 25 per dataset, even over chunk sizes; from the test pod session), blind to the score, correct or not. Ragas scores are binarised at 0.5. Pooled Cohen's kappa of at least 0.60: correctness is an H8 outcome. Below 0.60: exploratory only. The FinanceBench number match stays. @@ -204,6 +204,19 @@ Written before the freeze. No measured run has started. A publication-grade meth C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 markers, RQ3 wording, OSF deposit list). +**CodeRabbit round 1 (PR 210, 2026-11-14).** Twelve review threads, all fixed before the freeze: + +| Change | Pre-registration section | +| --- | --- | +| Primary measure named "faithfulness of cited factual claims", definition unchanged; uncited-segment share as co-outcome; sensitivity with uncited factual segments as unsupported; open item O7; limitation added | 6, 11.1, 21, 22, 23 | +| Undefined gate statistics fail closed (one-class sample fails; undefined resample takes kappa 0 or AUC 0.5, or is left out for the specificity difference; over 5% undefined fails) | 11.2 | +| Early stop only on preflight, G1 to G3 or the ceiling after the last drop step; a failed outcome gate leaves its rows not testable | 17 | +| One same-sign rule: both-datasets rules need both; 2-of-3 rules (C1, C2) need 2 of 3, and an opposite third is reported | 7, 18 | +| Part 2 C2: section 12 rules for both systems, complete pairs, unjudged-as-0 sensitivity | 18 | +| Contamination subset falls back to the own-knowledge run when the RAG-prompt run is dropped | 13 | +| Pilot seed confirmed before the pilot | 23 (P9) | +| OpenSpec only: half-B guard checks the hash of the current selection file; budget check uses 52,200 answers (15 setups x 3 runs x 1,160); pilot-declined path; labelling and closed-book run tasks moved out of the pre-freeze section; Holm family size follows the recorded O6 answer | none | + Cost: estimate about $730 (was about $690) plus the pilot, projected from P2 before the freeze; ceiling $975 unchanged (pre-registration section 17). ## 1. Question diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index 3ba73aae..7dad1065 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -326,7 +326,7 @@ One function `cohen_kappa(a, b)` goes into `experiments/_lib/stats_d6.py`, test- ### D22. Replicate pod time, Ragas included -**Revision (redesign, 2026-11-12).** The figures below the revision assumed six arms and 19,440 answers. The redesign has 12 answer setups x 3 runs x 1,160 questions = 41,760 answers, 13,920 per replicate pod. Claims per answer stay at about 4.47 (smoke data), so about 186,700 claims in all and 62,200 per pod. Only GLM-5.3 and Clef-Flash judge. Times are for an RTX 4090; the GPU choice (D24 revision) may change the GPU-bound rows. +**Revision (redesign, 2026-11-12).** The figures below the revision assumed six arms and 19,440 answers. The redesign has 12 answer setups x 3 runs x 1,160 questions = 41,760 answers, 13,920 per replicate pod. (A2 and A3 add 3 token-matched setups, priced as separate rows in D35 and below: 15 setups, 52,200 answers, 17,400 per replicate pod.) Claims per answer stay at about 4.47 (smoke data), so about 186,700 claims in all and 62,200 per pod. Only GLM-5.3 and Clef-Flash judge. Times are for an RTX 4090; the GPU choice (D24 revision) may change the GPU-bound rows. | Stage (per replicate pod) | Basis | Hours | | --- | --- | ---: | @@ -631,7 +631,7 @@ Written before the freeze; no measured run has started. A publication-grade meth - **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. - **Pins, release, determinism, existing data (F16, F18, F20, F25; PR 5, 15, 16, 19).** One pins table (P10 to P14). A release table per artefact and licence, and a re-judge kit. One build per cell is assumed; an optional `e8b-c2048` rebuild check on FinanceBench runs only when the cost projection after the search stage is at or below the estimate, and is reported, not gated. The existing-data table lists every earlier look; P12 lists each smoke and pre-freeze question ID with its half. -Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O6 are group 16. +Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O7 are group 16. ## Risks / Trade-offs @@ -663,7 +663,7 @@ Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack c 1. Build and test the code (tasks 1 to 8) on the feature branch. The default behaviour stays serial-equivalent: concurrency 1 gives today's files. 2. Run the classifier gate (tasks 8.1 to 8.7). The classifier calls run on a 4090 pod; labelling and scoring run on the Mac. 3. Resolve and record the image digest (task 9.1). -4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the user's decisions on O1 to O6. +4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the user's decisions on O1 to O7. 5. Write the procedure, the measured values, the GPU choice, the G2 stop rule and the budget with its ceiling into `protocol.md` (group 10). 6. At G7, freeze `preregistration.md` (the governing file, D38), `protocol.md` and `plan.json` with the measured values, after every group 16 task is done. 7. Index pods (D32, 9 in parallel on the chosen GPU type): each builds one index for all four datasets and seals it. The `e4b-c1024` rebuild check (D20) runs after that seal, and each pod then runs its 8 search setups once `rebuild.json` passes. A failed check stops here: find the cause, fix, build again. Then the FreshStack recheck on the real indexes with zero errors in every answer setup; on the Mac, half-A scoring, the selection rule (D31) and `selection.json`; the OSF time-stamp of the configuration hash; then half-B scoring (D38); and the budget re-check (D23). When the projection reaches $975, apply the drop order of D38. diff --git a/openspec/changes/exp35-main-run-readiness/proposal.md b/openspec/changes/exp35-main-run-readiness/proposal.md index 8051320f..11b7ff78 100644 --- a/openspec/changes/exp35-main-run-readiness/proposal.md +++ b/openspec/changes/exp35-main-run-readiness/proposal.md @@ -6,7 +6,7 @@ On 2026-11-12 the operator agreed a full-factorial redesign of Experiment 35 (pr - Four datasets (FinanceBench 150, FreshStack 203, BioASQ 707, Legal RAG Bench 100; 1,160 questions). - Search stage: embedder (Qwen3-Embedding 0.6B, 4B, 8B) x chunk size (512, 1,024, 2,048 tokens, 15% overlap) x reranker (off, Qwen3-Reranker 0.6B, 4B, 8B) x hybrid (off, BM25 on). 9 indexes and 72 search setups per dataset, run once each. Setups are named by their settings. -- Answer stage: 12 setups (chunk size x reranker on or off x hybrid on or off) at the embedder and reranker sizes chosen by a pre-registered rule, 3 runs each: 41,760 answers. +- Answer stage: 12 setups (chunk size x reranker on or off x hybrid on or off) at the embedder and reranker sizes chosen by a pre-registered rule, 3 runs each: 41,760 answers. (A2 and A3 add 3 token-matched setups: 15 setups x 3 runs x 1,160 questions = 52,200 answers.) - Claim checkers GLM-5.3 (main) and Clef-Flash (second). Granite Guardian and hypothesis H-CC are dropped. The refusal rule and its classifier gate stay. - The $250 cap is replaced by an estimate (about $650 then, about $730 after A4) and a stop rule at a $975 ceiling. @@ -29,7 +29,7 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set on the full public corpus (A4), and a FinanceBench number-match check. - Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against human labels by Cohen's kappa (design D33; A4 makes it a gate per core dataset on 50 labels each, D38); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget then: about $650, ceiling $975. - Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; `statsmodels` as an experiment-only dependency (approved); two labellers with a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. (The A2 mixed model, the 400 L1 items and the $650 estimate are replaced by A4 below.) -- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3: BioASQ rows 9 to 19 are sanity checks, open item O6) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O1 to O6 are tasks. +- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3: BioASQ rows 9 to 19 are sanity checks, open item O6) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O1 to O7 are tasks. - Answer context budget (design D30): a test proves that the largest evidence set of every answer setup synthesises without the earlier context overflow, and each answer row records its rounds, evidence characters and truncation. - Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8b with the existing statistics functions. - The harness reads the `rate_limited` record instead of parsing error text, and stops at once on `out_of_credit`. Every hybrid-on setup pins `retrieval.hybrid_sparse_backend: bm25`, as the protocol defines it. diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md index d248afb4..0271e5aa 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md @@ -149,6 +149,15 @@ Primary faithfulness, per answer setup and per checker (GLM-5.3 and Clef-Flash), - **WHEN** an answer setup has refusal claims - **THEN** its primary faithfulness MUST leave them out +### Requirement: Uncited segments are reported and tested in a sensitivity analysis + +Primary faithfulness SHALL cover cited factual claims only. The report SHALL give, per setup and dataset, the share of prose segments with no valid ordinal. The classifier SHALL also sort uncited segments, with the question and segment only. A registered sensitivity analysis SHALL count uncited fact and mixed segments as unsupported (pre-registration section 6; open item O7). + +#### Scenario: One cited and one uncited fact + +- **WHEN** an answer holds one cited fact the checker calls supported and one uncited segment the classifier calls fact +- **THEN** primary faithfulness MUST be 1.0, the sensitivity MUST be 0.5, and the uncited share MUST be 0.5 + ### Requirement: Refusals are reported separately A refusal report step SHALL read a joined main run and SHALL refuse to start without a passing join report. It SHALL give the refusal rate per answer setup and dataset as a primary descriptive outcome. For refusal claims only, it SHALL give each checker's unsupported rate and the Cohen's kappa of Clef-Flash against GLM-5.3. For factual claims, it SHALL give the Cohen's kappa of Clef-Flash against GLM-5.3 per dataset. @@ -190,6 +199,17 @@ A sampling step SHALL draw 50 half-B questions per dataset with seed 20261113, e - **WHEN** the agreement step computes any gate interval (kappa, AUC or specificity difference) - **THEN** it MUST use a percentile bootstrap with 10,000 resamples by question and the seed listed for that interval in pre-registration section 16 +#### Scenario: Statistic undefined on the full sample + +- **WHEN** the labelled sample of a dataset holds only one human class, so the AUC or kappa is undefined +- **THEN** the gate MUST fail on that dataset + +#### Scenario: Undefined resamples + +- **WHEN** a resample holds only one class +- **THEN** the step MUST NOT draw it again, MUST give it kappa 0 or AUC 0.5, and MUST count it +- **AND** the gate MUST fail when more than 500 of the 10,000 resamples are undefined + #### Scenario: Point estimate passes but lower bound fails - **WHEN** kappa on FinanceBench is 0.65 with a lower 95% bound of 0.35 @@ -243,6 +263,12 @@ The agreement step SHALL give each checker's specificity (the share of human-sup - **WHEN** only the Clef-Flash specificity difference excludes zero - **THEN** it MUST be reported only and MUST NOT change any row +#### Scenario: Specificity difference undefined + +- **WHEN** a chunk-size level of the full sample, or of more than 500 of the 10,000 resamples, holds no human-supported claim +- **THEN** rows 12, 13, 16 and 17 MUST be "not testable" +- **AND** a resample with no human-supported claim at a level MUST NOT be drawn again, MUST be left out of the interval and MUST be counted + #### Scenario: Gate decided before H8a and H8b results - **WHEN** the analysis starts an H8a or H8b faithfulness row and no gate record with a time exists diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md index b11d2c4f..55022efa 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md @@ -171,13 +171,18 @@ The selection step SHALL write the cell scores, the chosen cell and the configur ### Requirement: Half B is scored only after the hash is time-stamped -The harness SHALL follow pre-registration section 14.2: score half A only; apply the selection rule and write the selection file; time-stamp the configuration hash with a file upload to the registered OSF project; then score half B and start the replicate pods. A scoring step SHALL refuse any half-B score while the selection file or the time-stamp record is missing. A replicate pod's first stage SHALL refuse to start under the same condition. +The harness SHALL follow pre-registration section 14.2: score half A; write the selection file; time-stamp its configuration hash by an OSF upload; then score half B and start the replicate pods. A scoring step SHALL refuse any half-B score unless the selection file exists and a time-stamp record holds its current configuration hash. A replicate pod's first stage SHALL refuse to start under the same condition. #### Scenario: Half-B score before the time-stamp - **WHEN** the scoring step is asked for half-B scores and no time-stamp record exists - **THEN** it MUST stop before it computes any half-B score and MUST name the missing record +#### Scenario: Time-stamp of an earlier selection + +- **WHEN** the scoring step is asked for half-B scores and the time-stamp record holds a hash that differs from the configuration hash in the selection file +- **THEN** it MUST stop before it computes any half-B score and MUST name both hashes + #### Scenario: Half-A scoring - **WHEN** the scoring step is asked for half-A scores and no selection file exists diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index f21cfac0..5c7ca674 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -320,7 +320,7 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d 2a. Reranker files (design D24 revision, item 1): for each Qwen3-Reranker GGUF candidate, a relevant pair scores near 1 and an irrelevant pair near 0, and the ranking of 20 query-passage sets matches the model card's transformers recipe. Record the result per file. 2b. Reranker throughput at 0.6B, 4B and 8B on 100-candidate pools at 512-, 1,024- and 2,048-token passages; the 8B embedder and the 8B reranker loaded together (peak VRAM, any swap); index build time per chunk size on a fixed slice of each corpus for each embedder. 3. Calibration: answers at 16, `chat` at 4, Ragas at 16, `claimkinds` at 16. Record per role the 429 and 402 counts, the waits, the stops and the calls per minute. Halve a value that hit a rate-limit stop and repeat that role once. Check that each answer's evidence chunk IDs equal the same run's search IDs. Measure wall seconds per answer per stage and the reported `cost_usd` per answer. - 4. The first budget check (design D23 revision): measured build hours per chunk size times 9 indexes, plus the search stage from the measured reranker throughput, plus three replicate pods at the measured hours, all times the pod price, plus the hosted cost per answer times 41,760, plus the full-corpus FreshStack builds from task 16.2, the two closed-book runs, the oracle run, the length-controlled check and the pilot. When the projection reaches the $975 ceiling, apply the results-blind drop order of design D38 (pre-registration section 17) in order and record each step with the projection that triggered it; no confirmatory item is dropped by choice. + 4. The first budget check (design D23 revision): measured build hours per chunk size times 9 indexes, plus the search stage from the measured reranker throughput, plus three replicate pods at the measured hours, all times the pod price, plus the hosted cost per answer times 52,200 (15 answer setups x 3 runs x 1,160 questions), plus the full-corpus FreshStack builds from task 16.2, the two closed-book runs, the oracle run, the length-controlled check and the pilot. When the projection reaches the $975 ceiling, apply the results-blind drop order of design D38 (pre-registration section 17) in order and record each step with the projection that triggered it; no confirmatory item is dropped by choice. 5. Copy the session outputs to the Mac, then delete the pod. Verify: steps 1 and 2 pass with zero errors; a dated calibration note in `protocol.md` gives the measured values, the counts, the hours per replicate pod and the extrapolated total; design.md D22 holds the measured rates; the `arms.py` defaults equal the measured values; the pod no longer exists in the RunPod pod list. A failure in step 1 or 2 is fixed in code and the session runs again. @@ -367,14 +367,12 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d Verify: the operator reads and accepts the text. - [ ] 10.4 Write the concurrency options, the rate-limit exit code 5 and the stop-and-resume rule into `protocol.md` sections 5 and 7. The values are the measured ones from task 9.2. Verify: the values match the `arms.py` constants and the calibration note. -## 11. Integration checks +## 11. Integration checks (run last, after groups 12 to 16 and the code tasks of group 16; its pre-freeze tasks close before G7) - [ ] 11.1 Update the clean-base tripwire count in `tests/test_clean_base_tripwire.py`. Diff the collected test IDs against the last pin, not the per-file counts. Verify: `uv run --no-sync pytest tests/test_clean_base_tripwire.py -q` passes. - [ ] 11.2 Run `./scripts/local_ci.sh` (after the operator agrees to a full run). Verify: the final line reports success, and no step prints `SKIPPED`. - [ ] 11.3 Run `uv run --no-sync omrs experiments check` and `openspec validate exp35-main-run-readiness --strict`. Verify: both pass. -Group 11 runs last, after groups 12 to 16 below (the code tasks of group 16; its pre-freeze tasks close before G7). - ## 12. App addition C: llama.cpp reranker backend (redesign, 2026-11-12; design D27) Architecture invariant 11: one new file and one `register()` line, no branch on strategy names. Each test is proved to fail on the current code first. @@ -427,7 +425,7 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 14.2 Write failing tests that each answer row records `completion_calls`, the evidence characters, and the truncated passages and characters dropped, computed with OMRS's own `_fit_and_measure` on the same rows and settings. Prove they fail, then implement in `answers35.py`. Verify: the tests pass, and a row with cut evidence is still saved. - [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score, both chosen sizes, the A3 cell and the configuration hash. Then (A4, pre-registration section 14.1) tests that: index cost is the number of chunks times the embedding dimension, summed over the three core datasets, read from the sealed indexes with no score; the configuration hash is the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings with the SHA-256 of `halves.json` as one field, and the same inputs give the same hash twice; the any-page sensitivity repeats step 3 with any-page nDCG@10 on FinanceBench, records whether the cell changes, and never changes the registered cell. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. - [ ] 14.5 Ragas `AnswerCorrectness` (design D33). Write failing tests with a stub LLM and stub embeddings: the `ragas` stage scores five metrics, `AnswerCorrectness` gets the question, the answer and the dataset's reference answer (never the contexts), uses the frozen judge and the local 4B embedder, and records the weights; an answer without a reference records "not judged" and sends no call; resume skips saved rows. Prove they fail, then implement in `ragas35.py` with Ragas 0.4.3. Verify: the tests pass, and the version check still refuses another Ragas version. -- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B, each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). +- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B, each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). - [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome (O6); a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). ## 15. Methods-review fixes (pre-freeze amendment A2, 2026-11-13; design D35) @@ -446,7 +444,7 @@ Each test is proved to fail on the current code before the fix. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.2 Depth 100. Write failing tests: every search stage saves 100 chunks per question; at depth 100 every reranker setup reranks a pool of exactly 100 (the harness passes `fetch_k=100`; without it `policy.py:63-67` gives 300, which the test must show failing first); a reranker-on search returns the whole reranked pool (100) in order; the answer stage takes its 10 (or 40 or 20) chunks from the head of that list. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.3 Token-matched answer setups (M5, revised by A2c N2). Write failing tests: the answer setup table has 15 setups (A3): the 12 crossed setups on 10 chunks each, plus `c512-t20k`, `c1024-t20k` and `c2048-t20k` at the chosen sizes with reranker on and hybrid on; the evidence of the three `-t20k` setups is cut by real Qwen3 tokens with the rule of task 15.14, while the `c2048` x 10 factorial cell keeps its 10-chunk evidence (a test with short fixture chunks proves `c2048` x 10 still sends 10 chunks and `c2048-t20k` sends more); the cost line of D35 (about +$40, ceiling $975) is checked in the test pod budget sum; with one answer round (task 14.1) the largest 40-chunk set still fits. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.4 Closed-book stages (M6; A4, pre-registration section 13). Write failing tests: `--stage closedbook` runs two prompts, each once on all 1,160 questions: the own-knowledge prompt (floor) and the main RAG answer prompt with an empty evidence block; neither sends a search call; both use the frozen answer settings, write one row per question for all four datasets, run `AnswerCorrectness` and the number match, and resume without repeat calls. Then report tests: closed-book correctness per dataset beside every correctness table; the per-question cross-table on retrieval misses; and the registered contamination sensitivity, which re-runs rows 14, 15, 18 and 19 on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5). Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.4 Closed-book stages (M6; A4, pre-registration section 13). Write failing tests: `--stage closedbook` runs two prompts, each once on all 1,160 questions: the own-knowledge prompt (floor) and the main RAG answer prompt with an empty evidence block; neither sends a search call; both use the frozen answer settings, write one row per question for all four datasets, run `AnswerCorrectness` and the number match, and resume without repeat calls. Then report tests: closed-book correctness per dataset beside every correctness table; the per-question cross-table on retrieval misses; and the registered contamination sensitivity, which re-runs rows 14, 15, 18 and 19 on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5); when the drop order (PR 17) has dropped the RAG-prompt run, the subset uses the own-knowledge run alone and the table is labelled "one closed-book run". Prove they fail, then implement. Verify: the tests pass, the fallback case included. - [ ] 15.5 Split halves (M3; A4, pre-registration section 4). Write failing tests: `split35.py` with seed 20261115 gives the same halves twice; FinanceBench is split by document (documents shuffled with the seed, each going with all its questions to the half that has fewer questions so far, ties to A), so no document sits in both halves; FreshStack, BioASQ and Legal are split by question (shuffled, first half rounded up is half A); `halves.json` lists every question ID with its half; `select35.py` reads half A only (a planted half-B effect does not change the choice). Prove they fail, then implement. Verify: the tests pass. - [ ] 15.6 Clustered analysis (A4 replaces M2 and A2c N4; pre-registration section 9). No MixedLM. Write failing tests for `experiments/_lib/stats_cluster.py` and `factorial35.py`: - the per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells (equal cell weights), with answer scores averaged over the 3 runs first; the estimate is the plain mean of d_qk; on FreshStack, BioASQ and Legal the decision test is the closed form of PR 9 point 3: variance = (sum of squared residuals) x (n / (n minus 1)) squared / n squared, and the p-value and 95% interval use t on n minus 1 degrees of freedom, computed by our code whatever reference the library reports by default (`statsmodels` GEE results use a normal reference unless told otherwise); the implementation is `statsmodels` `GEE` of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`); a known-answer test on a fixed fixture checks that the GEE estimate and variance equal the closed form and that the closed form equals the paired t-test variance times n / (n minus 1); if the library does not match, the code uses the closed form and the report states the difference; @@ -467,17 +465,17 @@ Each test is proved to fail on the current code before the fix. ## 16. Publication-review fixes (pre-freeze amendment A4, 2026-11-14; design D38) -`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 are code. Tasks 16.23 to 16.28 are the user's decisions O1 to O6; record each answer, never guess it. Every task closes before G7. +`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 and 16.29 are code. Tasks 16.23 to 16.28 and 16.30 are the user's decisions O1 to O7; record each answer, never guess it. Every task closes before G7. - [ ] 16.1 P1. FreshStack full corpus. Ask the user to approve the download of `freshstack/corpus-oct-2024`, config `langchain`, at a pinned revision. Then download it into `~/Development/DATA/omrs/` and record the revision, the row count (49,514 expected) and the corpus tree hash in `plan.json` and in the pins table (PR 16). Verify: the loader of task 13.5 accepts the corpus, and its check refuses a changed row count. - [ ] 16.2 P2. Time one full-corpus FreshStack index build on the chosen GPU type (part of task 9.2 or 9.3 when possible). Update the estimate in PR 17 from the measured time before relying on it. Verify: the compute table and the estimate name the measured hours; the "30 to 40 pod hours" reviewer figure is no longer the basis. - [ ] 16.3 P3. Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's own scorer. If yes, pin the scorer (version or commit) and write failing tests that a FreshStack secondary column holds it, then implement. If no, record "not available" in PR 6. Verify: PR 6 and `plan.json` say which. - [ ] 16.4 P4. Check, per dataset, whether the qrels list judged non-relevant units. Record the answer. Then write failing report tests: the unjudged share at 10 per setup and dataset; condensed-list nDCG@10 (unjudged units removed) as a sensitivity analysis for H1 to H5 where non-relevant judgements exist; and the statement "neither can be computed" where the qrels hold positives only. Prove they fail, then implement. Verify: the tests pass and PR 21 matches the recorded answer. -- [ ] 16.5 P5. MDE pilot. Fix the pilot cell list before the pilot runs: at most 4 indexes per core dataset, the H7 corner cells, built with the pinned settings and not reused. Project the pilot cost from the P2 timing, add it to the PR 17 estimate and get the user's approval; if declined, apply the PR 10 fallback (assumed SDs 0.15 to 0.25). Draw 40 questions per core dataset from half A with seed 20261118 (after task 16.12). Run retrieval only on the full-corpus indexes, with no hosted call, after a price quote and the user's go. Run `mde35.py` (task 15.11) and fill the MDE table of PR 10. Verify: every pilot question ID is in half A, the cell list has a commit dated before the pilot, the table is filled, and the pilot pod hours are in the compute table. +- [ ] 16.5 P5. MDE pilot. Fix the pilot cell list before the pilot runs: at most 4 indexes per core dataset, the H7 corner cells, built with the pinned settings and not reused. Project the pilot cost from the P2 timing, add it to the PR 17 estimate and ask the user. If approved: confirm seed 20261118 (task 16.9), draw 40 questions per core dataset from half A with it (after task 16.12), run retrieval only on the full-corpus indexes, with no hosted call, after a price quote and the user's go, run `mde35.py` (task 15.11) and fill the MDE table of PR 10. Verify: every pilot question ID is in half A, the cell list has a commit dated before the pilot, the table is filled, and the pilot pod hours are in the compute table. If declined: fill the table from the PR 10 fallback (assumed SDs 0.15 to 0.25, labelled assumed). Verify: the declined answer and its date are recorded, every row shows the assumed range and names PR 10 as its source, and no pilot pod hours appear. - [ ] 16.6 P6. Name the smoke and test-pod answer files that feed the H8a and H8b MDE rows, with their SHA-256. Verify: PR 10 names them and marks those rows rough. - [ ] 16.7 P7. Restate the checker 2-point tolerance against the H8a and H8b faithfulness MDE: the bar stays only if it is at most half that MDE. Verify: PR 11.1 and `protocol.md` section 4.4 give the restated rule; the "about 4 points" figure is not used. - [ ] 16.8 P8. Write one label guideline per set (claim kinds, faithfulness, correctness, Part 2 correctness), with worked examples drawn from outside every sample. Faithfulness follows the LLM-AggreFact task definition. Fix each guideline before its first label. Verify: the guidelines are committed before any label, and no worked example is in a sample (a test of `labels35.py` checks the IDs). -- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Seed 20261120 (audit) is used only if O3 (task 16.25) is yes; confirm it then. Verify: PR 16 and `plan.json` hold the confirmed values. +- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Confirm the pilot seed before the pilot runs (task 16.5). Seed 20261120 (audit) is used only if O3 (task 16.25) is yes; confirm it then. Verify: PR 16 and `plan.json` hold the confirmed values. - [ ] 16.10 P10. Record in `plan.json`: the tokenizer revision; the answer-model host, reasoning mode, output cap and context length (task 14.1a); the Ragas judge and classifier host and settings; the GLM-5.3 settings. Verify: each "Missing: P10" cell of PR 16 points to a `plan.json` key. - [ ] 16.11 P11. Record in `plan.json`: the query instruction (empty string), the reranker instruction text, and the hybrid settings (RRF constant, per-side fusion depth, BM25 tokeniser and parameters). Write a failing test that the run record of a search stage holds the same values, then implement. Verify: the test passes. - [ ] 16.12 P12. Run `split35.py` (task 15.5), write `halves.json`, record its SHA-256 in `plan.json`, and list every smoke and pre-freeze question ID with its half in the existing-data table (PR 5). Verify: the hash in `plan.json` equals the file, and every listed ID has a half. @@ -485,8 +483,8 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.14 P14. Set the `evidence_recall_20k` tolerance on another back end, with its reason, in `plan.json` `runtime.tolerance`. Verify: PR 15 names it. - [ ] 16.15 P15. Check that no file still names `openai/gpt-6.1-terra` as a cross-check judge (A4 removed it from `plan.json`). This change never planned it. The `experiment-35-full-document-rag-benchmark` design marks its cross-check text as removed by A4. Verify: a search for `gpt-6.1-terra` finds it only in history text. - [ ] 16.16 P16. Part 2 and E1 inputs (readiness flow group 7): own estimates and ceilings for Part 2 and for E1; RAGFlow defaults read from the pinned release, not main-branch documentation; the three-column settings table; the matched RAGFlow cell list fixed before FLOW step 7.2; the cost measured at FLOW step 7.2 and the decision at FLOW step 7.2a (at most $50 projected pod cost), recorded before any half-B score exists. Verify: the Part 2 protocol holds each item and FLOW steps 7.1, 7.2 and 7.2a are done. -- [ ] 16.17 Half-B guard (PR 14.2). Write failing tests: the scorer refuses any half-B score while `selection.json` or the OSF time-stamp record (upload time and file hash, task 16.23) is missing, and names both; the replicate pods' first stage refuses to start under the same condition; half-A scoring needs neither. Prove they fail, then implement. Verify: the tests pass. -- [ ] 16.18 Faithfulness gate and differential-error rule (PR 11.2). Write failing tests for `labels35.py`: per checker, kappa against the human label on the 450-claim sample (drawn from answers to all questions, both halves), with "can't tell" as not supported (also reported with those items left out), passes at a point estimate of at least 0.60 and a lower bound of at least 0.40 (percentile bootstrap by question, seed 20261125); a failed Clef-Flash gate marks its results not validated and changes no row; specificity per chunk size, and when the 95% interval of the GLM-5.3 2,048 minus 512 difference (seed 20261126) excludes zero in either direction, rows 12, 13, 16 and 17 become "not testable" (p = 1, task 15.7); sensitivity per chunk size is reported; the gate file records its time and the analysis refuses to compute an H8a or H8b row before it exists. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.17 Half-B guard (PR 14.2). Write failing tests: the scorer refuses any half-B score while `selection.json` or the OSF time-stamp record (upload time and file hash, task 16.23) is missing, and names both; it also refuses when no time-stamp record holds the configuration hash of the current `selection.json` (a record of an earlier selection), and names both hashes; the replicate pods' first stage refuses to start under the same condition; half-A scoring needs neither. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.18 Faithfulness gate and differential-error rule (PR 11.2). Write failing tests for `labels35.py`: per checker, kappa against the human label on the 450-claim sample (drawn from answers to all questions, both halves), with "can't tell" as not supported (also reported with those items left out), passes at a point estimate of at least 0.60 and a lower bound of at least 0.40 (percentile bootstrap by question, seed 20261125); a failed Clef-Flash gate marks its results not validated and changes no row; specificity per chunk size, and when the 95% interval of the GLM-5.3 2,048 minus 512 difference (seed 20261126) excludes zero in either direction, rows 12, 13, 16 and 17 become "not testable" (p = 1, task 15.7); sensitivity per chunk size is reported; undefined statistics fail closed (PR 11.2: an undefined difference resample is left out and counted, and over 5% undefined makes rows 12, 13, 16 and 17 not testable); the gate file records its time and the analysis refuses to compute an H8a or H8b row before it exists. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.19 Length-controlled check (PR 11.2). Write failing tests for a `lengthcheck` stage: 300 claims from `c2048` crossed answers with seed 20261119; each cited chunk split into consecutive 512-token windows, the window that shares the most distinct lower-case claim words kept (ties to the earlier window); GLM-5.3 and Clef-Flash judge the claim again; the report gives the verdict change rate with a Wilson interval; no gate. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.20 Report items (PR 21). Write failing report tests: the realised chunk-length distribution per chunk-size level and dataset; one sensitivity table set (clustered method against sensitivity check, FinanceBench page rules, missing-outcome bounds, closed-book subset, length-controlled check, A3 cell under any-page credit); the compute table (GPU type, pod hours per stage with every pre-freeze pod, hosted tokens and cost per role, energy estimate, `resent_calls`, error rows per stage and role, every `out_of_credit` stop); the returned model ID and provider per hosted call with the count of differences; every row with SESOI and MDE lines and "not testable" marks. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.21 Release kit (PR 19). Write failing tests for a re-judge script that rebuilds evidence text from the pinned dataset revisions and the released unit and chunk IDs, on a fixture; and for an export that writes ranked lists in TREC run format, never corpus text. Prove they fail, then implement. Verify: the tests pass, and the release table of PR 19 names each artefact and its licence. @@ -497,3 +495,5 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.26 O4 (user decides). L2's name, role and relation to the project. Record it in the run record only, never in public files. Verify: no public file holds the name. - [ ] 16.27 O5 (user decides). The ethics query to the university and its answer, before the first label. Verify: the answer is recorded before any label. - [ ] 16.28 O6 (user decides). BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack), or the alternative: rows 12 to 19 in the BioASQ family (16 rows) with 2-of-3 for H8. Record the answer in PR 7, 8 and 23. Verify: recorded before G7. +- [ ] 16.29 Uncited segments (PR 6). Write failing tests: the join counts per answer the prose segments with no valid ordinal (the segments `split_claims` drops) and gives `uncited_segment_share` per setup and dataset; `claimkinds` also sorts uncited segments with the frozen classifier (question and segment only); the sensitivity counts uncited fact and mixed segments as unsupported; a fixture answer with one supported cited fact and one uncited fact scores 1.0 primary and 0.5 sensitivity. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.30 O7 (user decides). Whether the uncited-segment sensitivity of PR 6 becomes the primary faithfulness measure. Record the answer in PR 6 and 23. Verify: recorded before G7. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index d0800f21..de9145ca 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -143,6 +143,10 @@ Each faithfulness checker SHALL pass when Cohen's kappa against the human label - **WHEN** the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction - **THEN** H8a and H8b faithfulness rows 12, 13, 16 and 17 SHALL be exploratory and reported as not testable +#### Scenario: A gate statistic is undefined +- **WHEN** kappa, the AUC or the specificity difference is undefined on the full labelled sample, or on more than 500 of the 10,000 resamples +- **THEN** the gate SHALL fail closed by the rule of `preregistration.md` section 11.2, and its rows SHALL be reported as not testable + #### Scenario: The second checker fails its gate - **WHEN** Clef-Flash fails its faithfulness gate - **THEN** its results SHALL be reported as not validated and SHALL change no confirmatory row diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index 1b3d53f2..46057455 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -1,6 +1,6 @@ # Tasks: Experiment 35 Reproducible Local RAG Benchmark -> DRAFT. Do not start a measured run until section 4 is complete and committed. Section 1 starts only after `pipeline-sweep-llamacpp` passes its exit rule. +> DRAFT. Do not start a measured run until the pre-freeze tasks of section 4 are complete and committed (tasks 4A.4a, 4A.6 to 4A.8, 4B.4, 4B.5, 4B.5a, 4B.7 and 4C.2 run during or after the measured run and stay beside their amendment). Section 1 starts only after `pipeline-sweep-llamacpp` passes its exit rule. ## 1. Prepare the protocol inputs @@ -61,13 +61,13 @@ The full-factorial redesign (design D0) replaces the six arms. Tasks 5.1 to 5.4 ## 4B. Pre-freeze amendment A2: methods review (2026-11-13) -Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4B wins: 4A.4a now draws its 100 answers from the main run (half B), not the test pod. (A4: 200 answers, see 4D.6.) +Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4B wins: 4A.4a now draws its 100 answers from the main run (half B), not the test pod. (A4: 200 answers, see 5.7.) - [x] 4B.1 Write amendment A2 into this change (design D0.12, specs), `protocol.md` (section 0.10) and `plan.json` (amendment A2, metrics, hypotheses). Verify: both `openspec validate --strict` runs and `omrs experiments check` pass. Done 2026-11-13. - [ ] 4B.2 Compute the minimum detectable effects from data (M1, readiness task 15.11) and write the table with its sources into `protocol.md` section 0.10. Verify: the table is in a commit before the freeze commit. (Amended by A4: the table lives in `preregistration.md` section 10; see 4D.5.) - [ ] 4B.3 Pin every reranker GGUF with its SHA-256 (M8) only after its Spearman check (at least 0.95 on 200 seeded pairs) passes on the test pod. Verify: `plan.json` runtime holds three reranker entries, `load_runtime_block` accepts them, and `protocol.md` lists each correlation. -- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 225; see 4D.6.) -- [ ] 4B.5 Run the closed-book run (M6) on replicate pod 1 and report correctness by retrieval hit or miss. Verify: the report table names its `output/` file. (Amended by A4: two closed-book runs; see 4D.7.) +- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 225; see 5.7.) +- [ ] 4B.5 Run the closed-book run (M6) on replicate pod 1 and report correctness by retrieval hit or miss. Verify: the report table names its `output/` file. (Amended by A4: two closed-book runs; see 5.8.) - [ ] 4B.5a Run the oracle run (design D0.12, readiness tasks 15.12 and 15.13) on replicate pod 1 and report it beside closed-book as ceiling and floor, descriptive only. Verify: the table names its `output/` file and states the no-distractor caveat. - [ ] 4B.6 Deposit `protocol.md` and `plan.json` on OSF at the freeze commit (M9; the operator does the deposit). Verify: a commit after the freeze, before the first measured run, adds the OSF link and the freeze commit hash to `protocol.md`, and the deposit holds no personal data beyond the operator name. (Amended by A4: `preregistration.md` is the main file of the deposit; the OSF link and freeze commit go into `plan.json` `frozen`.) - [ ] 4B.7 Write the limitations into `report.md`: one answer model, English only, the web-search reranker instruction applied to finance and legal text, 3 runs that measure hosted-API nondeterminism, chunk size confounded with context length in the answer stage. Verify: the report's limitations section names each. (Amended by A4: add scope S and the limitations of `preregistration.md` section 22.) @@ -91,10 +91,10 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 4D.3 Time one full-corpus FreshStack index build and update the estimate (P2). Verify: the estimate in `preregistration.md` section 17 names the measured hours. - [ ] 4D.4 Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's scorer (P3), and whether each dataset's qrels list judged non-relevant units (P4). Verify: the scorer is pinned or recorded as "not available", and each dataset has one line. - [ ] 4D.5 Fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the operator's approval, run the retrieval-only pilot on 40 seeded half-A questions per core dataset, and fill the MDE table (P5, P6); if the operator declines, fill it from assumed SDs of 0.15 to 0.25, labelled as assumed. Restate the checker 2-point tolerance against the table (P7). Verify: the MDE table has one row per dataset and row group, with its SD source. -- [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 225. Verify: the guidelines are committed before the first label, and the gate results are recorded with their time before any H8a or H8b contrast result. -- [ ] 4D.7 Run the second closed-book run (RAG prompt, empty evidence) beside the first, and the length-controlled check on 300 claims. Verify: each table names its `output/` file and is marked descriptive or sensitivity. +- [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Verify: the guidelines are committed before the freeze. (The labelling itself is task 5.7.) +- [ ] 4D.7 Fix the second closed-book run (RAG prompt, empty evidence) and the length-controlled check (300 claims) in `plan.json` and the pre-registration. Verify: both are listed before the freeze. (The runs are task 5.8.) - [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if O3 is yes); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. -- [ ] 4D.9 Get the operator's answers to O1 to O6 (OSF time-stamp mechanism, SESOI value, optional blind audit, L2 identity, ethics query, BioASQ rows 12 to 19 as sanity checks). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. +- [ ] 4D.9 Get the operator's answers to O1 to O7 (OSF time-stamp mechanism, SESOI value, optional blind audit, L2 identity, ethics query, BioASQ rows 12 to 19 as sanity checks, uncited-segment faithfulness as primary). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. - [ ] 4D.10 Check before the freeze that no live text in this change, `protocol.md` section 0 or `plan.json` contradicts `preregistration.md`. Verify: every hit for "MixedLM", "included whole", "gpt-6.1-terra", "operator decides" and "equivalence" is marked superseded or removed. ## 5. Run @@ -105,6 +105,8 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 5.4 Run the determinism check for every arm, and the rebuild check (4B baseline index rebuilt once per dataset in a fresh process). Verify: the output states match or mismatch per arm and dataset, and for each rebuild. - [ ] 5.5 Run Ragas on each arm where the dataset supports each metric. Verify: unsupported metrics show "not measured". - [ ] 5.6 Check about 50 judged items by hand. Verify: the agreement is computed with the measure frozen in 1.10. +- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 225, with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. +- [ ] 5.8 (A4, moved from 4D.7.) Run the second closed-book run (RAG prompt, empty evidence) beside the first, and the length-controlled check on 300 claims. Verify: each table names its `output/` file and is marked descriptive or sensitivity. ## 6. Analyse and report diff --git a/openspec/changes/rag-benchmark-readiness-flow/design.md b/openspec/changes/rag-benchmark-readiness-flow/design.md index 2529b6aa..73589593 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/design.md +++ b/openspec/changes/rag-benchmark-readiness-flow/design.md @@ -90,7 +90,7 @@ D. Section 2 harness, PR (G7b) E. Arm smoke run on the run machine; pod stopped after it | E2. A4 pre-freeze items: FreshStack full corpus pin and timing, - | halves and MDE pilot on half A, P1 to P16, open items O1 to O6 + | halves and MDE pilot on half A, P1 to P16, open items O1 to O7 | (with Part 2 steps 7.1, 7.2 and 7.2a) F. Freeze (G7) | diff --git a/openspec/changes/rag-benchmark-readiness-flow/proposal.md b/openspec/changes/rag-benchmark-readiness-flow/proposal.md index 927f355d..6dfb05bc 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/proposal.md +++ b/openspec/changes/rag-benchmark-readiness-flow/proposal.md @@ -15,7 +15,7 @@ Without one recorded order, two branches can touch the same specs at once, the l - A task order only (tasks.md groups 0 to 8) with gates. No code and no capability deltas. - The order (operator decisions 2026-11-06): cuts, then router, then one live switch (no re-index: the live collections are test data, operator 2026-11-07), then the pipeline sweep on RunPod, then Experiment 35. The page routing panel follows the router and can run beside the sweep, because it touches no runtime and no result. -- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the user's open items O1 to O6) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). +- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the user's open items O1 to O7) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). ## Capabilities diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 36155086..6146fdeb 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -86,9 +86,9 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35 ("PR n" means its section n). "RDY 16.N" means task 16.N in `exp35-main-run-readiness/tasks.md`. Steps 6.10a to 6.10d run after step 6.10 and before the freeze. - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. -- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval, run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: every pilot question is in half A, and the MDE table is filled. -- [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28). O1 the OSF mechanism that time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 the university ethics query and its answer, before the first label; O6 BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack) or the alternative (16 BioASQ rows, 2-of-3 for H8). Record each answer; never guess one. Verify: PR 23 shows each item decided. +- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. +- [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. +- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 the OSF mechanism that time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 the university ethics query and its answer, before the first label; O6 BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack) or the alternative (16 BioASQ rows, 2-of-3 for H8); O7 whether faithfulness with uncited factual segments as unsupported becomes primary (RDY 16.30). Record each answer; never guess one. Verify: PR 23 shows each item decided. ### F. Freeze @@ -97,7 +97,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime ### G. Run and report - [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (mechanism O1); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. -- [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm families fixed at 19, 19 and 8 (O6), gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. +- [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm family sizes as recorded for O6 in PR 8 at the freeze: 19, 19 and 8, or 19, 19 and 16 under the O6 alternative; gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. - [ ] 6.14 Exp 35 section 7 (tasks 7.1 to 7.5): the release table of PR 19 (answers, evidence IDs, ranked lists, human labels and verdicts per licence, never corpus text) and the re-judge kit, copy to `DATA/omrs/`, Zenodo (one archive per dataset licence), identifier search. Merge and archive Exp 35 on the user's word. ## 7. Part 2 (RAGFlow on half B) and E1 @@ -111,8 +111,8 @@ Master plan: `docs/proposals/first-publication-benchmark.md`, "Order and gates", 4. Serving: RAGFlow calls the same llama.cpp router, build and GGUF hashes as OMRS, with the environment record and `verify_llamacpp_runtime` before the first call; run and environment record rules as Part 1. 5. Evidence mapping (chunk to unit ID per dataset; FinanceBench page unit with the page gate and the document fallback of PR 18) and the common answer harness input format (evidence item, Qwen3 token count, truncation record). 6. Pre-freeze RAGFlow samples (page-number gate, repeatability check) from half A only, with seeds; the repeatability rule and what a mismatch means for C1. - 7. C1 and C2: one Holm family of 2 per core dataset, the 2-of-3 rule with the same sign on at least 2 core datasets (opposite signs reported as a dataset-dependent effect; the hypothesis does not hold), Legal exploratory. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`, with number match secondary. Tests as PR 9: GEE on question clusters for FreshStack and BioASQ, the document-level sign-flip test on FinanceBench; no McNemar test. C2 needs its own correctness gate (the PR 11.2 rule on labels from both systems); a failed gate leaves C2 in its family as "not testable". This replaces the earlier rule "C2 exploratory below kappa 0.60". - 8. Missing outcomes (PR 18): every half-B question stays in both systems' denominators; a RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2; a short list scores missing ranks as not relevant; a document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects; an unmappable chunk counts as not relevant and is counted; a sensitivity analysis uses the questions where both systems indexed every gold document. + 7. C1 and C2: one Holm family of 2 per core dataset, the 2-of-3 rule with the same sign on at least 2 of the 3 core datasets (a count with the opposite sign on the third is reported as a dataset-dependent effect and does not stop the hypothesis holding; PR 7 and 18), Legal exploratory. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`, with number match secondary. Tests as PR 9: GEE on question clusters for FreshStack and BioASQ, the document-level sign-flip test on FinanceBench; no McNemar test. C2 needs its own correctness gate (the PR 11.2 rule on labels from both systems); a failed gate leaves C2 in its family as "not testable". This replaces the earlier rule "C2 exploratory below kappa 0.60". + 8. Missing outcomes (PR 18): every half-B question stays in both systems' denominators; a RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2; a short list scores missing ranks as not relevant; a document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects; an unmappable chunk counts as not relevant and is counted; a sensitivity analysis uses the questions where both systems indexed every gold document. The PR 12 rules apply to both systems alike: an answer still missing after the resend rule scores 0 for correctness; C2 uses complete pairs (both systems' Ragas rows judged), every exclusion is counted, and a sensitivity analysis scores unjudged rows 0. 9. Labels: L2 is the primary Part 2 labeller; system identity is hidden; both systems' answers are shuffled into one list; written guidelines (P8); counts and seeds. 10. Tuning budget: the exact matched RAGFlow cell list (settings varied and their levels), fixed and recorded before the step 7.2 cost measurement; the matched RAGFlow search decision of step 7.2a; the search-cost figure with its estimator and a RAGFlow curve only if the matched search runs. 11. Defaults row: one RAGFlow run (pinned-release defaults, the shared parts and the two measurement settings, PR 18) and its secondary role; when the matched search does not run, it is the main RAGFlow run and the table shows it once. From c4948b2f08bb6b8237bd939871e44ae1fdb9cfe1 Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 18:11:25 +0000 Subject: [PATCH 3/9] build(exp35): add stats dependency group and record the O6 decision Add statsmodels 0.15.0 in a new stats dependency group (included by dev, absent from the base install). The closed-form clustered variance matched the statsmodels GEE to 1e-10 on a seeded fixture. Teach the dependency floor test to skip PEP 735 include-group entries. Record the O6 decision of 2026-11-14: BioASQ rows 12 to 19 are sanity checks, families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack. Remove the open-item conditionals from the pre-registration, plan, protocol, proposals, tasks and flow. --- docs/proposals/first-publication-benchmark.md | 4 +- .../plan.json | 8 +- .../preregistration.md | 14 +-- .../protocol.md | 6 +- .../exp35-main-run-readiness/design.md | 4 +- .../exp35-main-run-readiness/proposal.md | 2 +- .../changes/exp35-main-run-readiness/tasks.md | 8 +- .../design.md | 4 +- .../proposal.md | 2 +- .../specs/evaluation-observability/spec.md | 2 +- .../rag-benchmark-readiness-flow/tasks.md | 4 +- pyproject.toml | 5 + tests/test_dependency_floors.py | 2 + uv.lock | 115 ++++++++++++++++++ 14 files changed, 151 insertions(+), 29 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index d6348452..efc42300 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -122,7 +122,7 @@ Full wording, contrast rows and "holds" rules: pre-registration sections 7 and 8 - **H5:** a bigger reranker ranks better. - **H6:** chunk size changes retrieval (the same row, same sign, on FinanceBench and FreshStack). - **H7:** a bigger embedder gains more from bigger chunks (FinanceBench and FreshStack; expected to be underpowered). -- **H8a:** chunk size changes faithfulness and correctness in the crossed setups, where context length grows with chunk size (the same row, same sign, on FinanceBench and FreshStack; open item O6). +- **H8a:** chunk size changes faithfulness and correctness in the crossed setups, where context length grows with chunk size (the same row, same sign, on FinanceBench and FreshStack). - **H8b:** chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. The paper says "chunk size itself changes X" only when H8b holds for X. ### Methods in short @@ -131,7 +131,7 @@ Detail for each item is in the pre-registration section named. - Search runs once per setup on a sealed index, and each index is built once. A rebuild check and a repeat check confirm determinism (section 15). Answer setups repeat to measure drift in the hosted API. - Analysis: no mixed model. One clustered method gives every estimate, interval and p-value: GEE with bias-reduced covariance on question clusters for FreshStack, BioASQ and Legal, and a document-level sign-flip permutation test on FinanceBench (section 9). -- Holm families are fixed now: 19 contrasts on FinanceBench, 19 on FreshStack, 8 on BioASQ (46 in all; BioASQ rows 9 to 19 are sanity checks, open item O6); Legal is exploratory. No gate changes a family size (section 8). H1 to H5 hold on at least 2 of 3 core datasets. H6, H7, H8a and H8b hold only on FinanceBench and FreshStack. A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs (section 7). +- Holm families are fixed now: 19 contrasts on FinanceBench, 19 on FreshStack, 8 on BioASQ (46 in all; BioASQ rows 9 to 19 are sanity checks); Legal is exploratory. No gate changes a family size (section 8). H1 to H5 hold on at least 2 of 3 core datasets. H6, H7, H8a and H8b hold only on FinanceBench and FreshStack. A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs (section 7). - Missing outcomes: every question stays in its denominator, with a rule per outcome (section 12). - Power: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the minimum detectable effect table before the freeze. It builds at most the four H7 corner indexes per core dataset and does not reuse them. Its cost is projected from the P2 timing and approved before the pod starts; if the user declines, the table is filled from assumed SDs and labelled so. The smallest effect of interest is 0.02 nDCG@10 (open item O2). No equivalence claim is made (section 10). - Every main-run pod uses one RunPod GPU type, chosen by a timing job before the freeze. The Mac scores the results. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 92ac0ea6..f6ec634f 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -49,7 +49,7 @@ { "id": "H8a", "role": "answer-stage", - "statement": "within scope S (preregistration.md section 2), chunk size changes GLM-5.3 faithfulness on factual claims and Ragas AnswerCorrectness in the 12 crossed setups, where context length grows with chunk size: rows 12 to 13 (faithfulness) and 14 to 15 (correctness), 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided, all questions, 3 runs averaged per question; per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack (open item O6); BioASQ rows are sanity checks; faithfulness is expected to fall as chunks grow; Legal RAG Bench is exploratory; FinanceBench number match is secondary; confirmatory", + "statement": "within scope S (preregistration.md section 2), chunk size changes GLM-5.3 faithfulness on factual claims and Ragas AnswerCorrectness in the 12 crossed setups, where context length grows with chunk size: rows 12 to 13 (faithfulness) and 14 to 15 (correctness), 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided, all questions, 3 runs averaged per question; per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack; BioASQ rows are sanity checks; faithfulness is expected to fall as chunks grow; Legal RAG Bench is exploratory; FinanceBench number match is secondary; confirmatory", "null": "each contrast is zero" }, { @@ -59,7 +59,7 @@ "null": "each contrast is zero" } ], - "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks, rows 12 to 19 pending open item O6); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis needs the same sign: under the both-datasets rule (H6, H8a, H8b) the row counts with the same sign on FinanceBench and FreshStack, and opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold; under a 2-of-3 rule (Part 2 C1 and C2; H8a and H8b under the O6 alternative) the row counts with the same sign on at least 2 of the 3 core datasets, and an opposite-sign count on the third is reported as a dataset-dependent effect and does not stop the hypothesis holding. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", + "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE (version 0.15.0, `stats` dependency group, recorded in uv.lock) of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. The closed form was checked against statsmodels 0.15.0 on 2026-11-14 and matched to 1e-10. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis needs the same sign: under the both-datasets rule (H6, H8a, H8b) the row counts with the same sign on FinanceBench and FreshStack, and opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold; under a 2-of-3 rule (Part 2 C1 and C2) the row counts with the same sign on at least 2 of the 3 core datasets, and an opposite-sign count on the third is reported as a dataset-dependent effect and does not stop the hypothesis holding. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", "stats": "expert", "seed": 20261101, "baseline": "none: full factorial of embedder x chunk size x reranker x hybrid (72 search setups per dataset); e4b-c1024-roff-hoff is the reference cell of the G2 rebuild check only", @@ -359,7 +359,7 @@ "id": "A2b", "date": "2026-11-13", "before_first_run": true, - "summary": "Oracle run approved: one run on all 1,160 questions with gold units as evidence (Legal gold passage, FinanceBench gold page texts, BioASQ gold passages, FreshStack gold documents capped at 20,480 tokens with truncation recorded), same answer prompt and settings, scored like the main runs; reported with closed-book as ceiling and floor, descriptive only. Answer model stays deepseek/deepseek-v4.1-flash-1017 on the API; statsmodels in the stats dependency group. Budget estimate $650, ceiling $975.", + "summary": "Oracle run approved: one run on all 1,160 questions with gold units as evidence (Legal gold passage, FinanceBench gold page texts, BioASQ gold passages, FreshStack gold documents capped at 20,480 tokens with truncation recorded), same answer prompt and settings, scored like the main runs; reported with closed-book as ceiling and floor, descriptive only. Answer model stays deepseek/deepseek-v4.1-flash-1017 on the API; statsmodels 0.15.0 in the stats dependency group. Budget estimate $650, ceiling $975.", "where": "protocol.md section 0.10; change design D0.12 and readiness D36" }, { @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b by open item O6), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF, then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O1 to O7 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (open item O7 decides whether it becomes primary); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file.", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF, then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O1 to O7 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (open item O7 decides whether it becomes primary); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file.", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index 22e7b3af..a76b31a1 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -84,7 +84,7 @@ For question q with gold set G_q (size R_q) and unit list L_q of up to 10 units, ## 7. Hypotheses -Each null is a zero effect. Every test is two-sided. A directional contrast counts when its Holm-adjusted p is below 0.05 and its estimate has the predicted sign (this is a one-sided test at 0.025). A two-sided contrast counts when its Holm-adjusted p is below 0.05. A two-sided hypothesis needs the same sign. Under a both-datasets rule (H6, H8a, H8b), the row must count with the same sign on FinanceBench and on FreshStack; if it counts with opposite signs, it is reported as a dataset-dependent effect, and the hypothesis does not hold. Under a 2-of-3 rule (C1 and C2 in section 18; H8a and H8b under the O6 alternative), the row must count with the same sign on at least 2 of the 3 core datasets. A count with the opposite sign on the third dataset is reported as a dataset-dependent effect and does not stop the hypothesis holding. The report always states the direction. Marginal contrasts weight every cell of the other factors equally (for example, the embedder effect averages over 24 cells, 18 of them reranker-on). Row numbers refer to section 8. +Each null is a zero effect. Every test is two-sided. A directional contrast counts when its Holm-adjusted p is below 0.05 and its estimate has the predicted sign (this is a one-sided test at 0.025). A two-sided contrast counts when its Holm-adjusted p is below 0.05. A two-sided hypothesis needs the same sign. Under a both-datasets rule (H6, H8a, H8b), the row must count with the same sign on FinanceBench and on FreshStack; if it counts with opposite signs, it is reported as a dataset-dependent effect, and the hypothesis does not hold. Under a 2-of-3 rule (C1 and C2 in section 18), the row must count with the same sign on at least 2 of the 3 core datasets. A count with the opposite sign on the third dataset is reported as a dataset-dependent effect and does not stop the hypothesis holding. The report always states the direction. Marginal contrasts weight every cell of the other factors equally (for example, the embedder effect averages over 24 cells, 18 of them reranker-on). Row numbers refer to section 8. - **H1 (embedder).** Within scope S, a bigger embedder ranks gold units higher. Rows 1 (4B minus 0.6B) and 2 (8B minus 4B), nDCG@10, directional. Null: each contrast is zero. Holds: each step holds when its row counts on at least 2 of the 3 core datasets. H1 holds in full when both steps hold, in part when one does. - **H2 (reranker).** Within scope S, the reranker raises nDCG@10. Row 3 (mean of the three reranker sizes minus off), directional. Null: zero. Holds when row 3 counts on at least 2 of 3 core datasets. @@ -93,7 +93,7 @@ Each null is a zero effect. Every test is two-sided. A directional contrast coun - **H5 (reranker size).** Within scope S, a bigger reranker ranks better (reranker-on setups). Rows 7 (4B minus 0.6B) and 8 (8B minus 4B), directional. Holds as H1. - **H6 (chunk size, search).** Within scope S, chunk size changes retrieval. Rows 9 (1,024 minus 512) and 10 (2,048 minus 1,024), two-sided. Endpoint: `evidence_recall_20k` on FinanceBench, nDCG@10 on FreshStack. Null: each contrast is zero. Holds when the same row (9 or 10) counts with the same sign on FinanceBench and on FreshStack. BioASQ is a sanity check (most passages are shorter than 512 tokens, so chunk size barely changes its chunks); Legal is exploratory. FinanceBench sensitivity: nDCG@10 with first-page and with any-page credit; the report gives sign agreement per row. - **H7 (embedder by chunk size).** Within scope S, a bigger embedder gains more from bigger chunks. Row 11: (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, with the H6 endpoints. Null: zero. Holds when row 11 counts on both FinanceBench and FreshStack. BioASQ is a sanity check, Legal exploratory. Expected to be underpowered (section 10). -- **H8a (chunk size with context length, answers).** Within scope S, chunk size changes faithfulness and correctness in the 12 crossed setups, where context length grows with chunk size (long contexts can hide evidence: Liu et al. 2024). Rows 12 to 13 (faithfulness) and 14 to 15 (correctness): 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided. Expected direction for faithfulness: falls as chunks grow (Wang et al. 2024); the report gives the sign. Null: each contrast is zero. Per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack. Legal is exploratory. BioASQ chunks barely change with chunk size, so BioASQ rows 12 to 19 are sanity checks, like rows 9 to 11 (section 8). This rule depends on open item O6. +- **H8a (chunk size with context length, answers).** Within scope S, chunk size changes faithfulness and correctness in the 12 crossed setups, where context length grows with chunk size (long contexts can hide evidence: Liu et al. 2024). Rows 12 to 13 (faithfulness) and 14 to 15 (correctness): 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided. Expected direction for faithfulness: falls as chunks grow (Wang et al. 2024); the report gives the sign. Null: each contrast is zero. Per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack. Legal is exploratory. BioASQ chunks barely change with chunk size, so BioASQ rows 12 to 19 are sanity checks, like rows 9 to 11 (section 8). - **H8b (chunk size at equal tokens, answers).** Within scope S, chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. Rows 16 to 17 (faithfulness) and 18 to 19 (correctness): `c1024-t20k` minus `c512-t20k` and `c2048-t20k` minus `c1024-t20k`, two-sided, at reranker on and hybrid on with the chosen sizes (so H8b is conditional on that cell). Holds as H8a (per outcome, the same row counts with the same sign on FinanceBench and on FreshStack). **Reading fixed now:** "chunk size itself changes X" is claimed only when H8b holds for outcome X. When H8a holds and H8b does not, the claim is "chunk size together with context length changes X". **Error rates of the composite rule.** Holm within each dataset plus "at least 2 of 3 datasets" (H1 to H5) or "FinanceBench and FreshStack" (H6 to H8b) is conservative. Under the null on three independent datasets, a false "holds" for one directional contrast has probability about 3 x 0.025 squared, about 0.002, before Holm. Under the FinanceBench-and-FreshStack rule it is about 0.025 squared (about 0.0006) for a directional row, and 2 x 0.025 squared (about 0.001) for a two-sided row under the same-sign rule. Power falls the other way: at a per-dataset power of 0.6, the 2-of-3 rule has about 0.65; at 0.5, about 0.50. @@ -126,7 +126,7 @@ Contrasts are per question, marginal over every factor not named. Search rows us "Reranker on" in rows 5 and 6 is the mean over the three reranker sizes. BioASQ uses nDCG@10 for rows 1 to 8. -**Family sizes, fixed now.** FinanceBench: 19 (rows 1 to 19). FreshStack: 19 (rows 1 to 19). BioASQ: 8 (rows 1 to 8). Legal RAG Bench: no family; all 19 rows are exploratory. In all, 46 contrasts sit in Holm families. BioASQ rows 9 to 19 are sanity checks, reported with unadjusted 95% intervals and no decision. The reason is the same for all of them: most BioASQ passages are shorter than 512 tokens, so chunk size barely changes BioASQ chunks. Rows 12 to 19 as sanity checks depend on open item O6. +**Family sizes, fixed now.** FinanceBench: 19 (rows 1 to 19). FreshStack: 19 (rows 1 to 19). BioASQ: 8 (rows 1 to 8). Legal RAG Bench: no family; all 19 rows are exploratory. In all, 46 contrasts sit in Holm families. BioASQ rows 9 to 19 are sanity checks, reported with unadjusted 95% intervals and no decision. The reason is the same for all of them: most BioASQ passages are shorter than 512 tokens, so chunk size barely changes BioASQ chunks. **Gated rows stay in the family.** No gate changes a family size. A row whose measure fails its gate (section 11), or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as "not testable". With p = 1 it takes the last Holm rank, so the other rows keep the same number of comparisons. @@ -136,7 +136,7 @@ One clustered method gives every estimate, interval and p-value. There is no mix 1. **Per-question contrast score.** For row k and question q, d_qk is the mean of question q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights). Answer scores are first averaged over the 3 runs. 2. **Estimate.** The plain mean of d_qk over the questions that enter the row (section 12). -3. **FreshStack, BioASQ and Legal.** The decision test is defined by its closed form. Let n be the number of questions that enter the row and r_q = d_qk minus the estimate. Estimate: the mean of d_qk. Variance: (sum of r_q squared) x (n / (n minus 1)) squared / n squared. Reference distribution: t on n minus 1 degrees of freedom, for the p-value and the 95% interval. This is the Mancl and DeRouen (2001) bias-reduced sandwich for an intercept-only model with one row per cluster. It equals the paired t-test variance times n / (n minus 1), so it needs no convergence fallback. With complete data the estimate equals the cell-means contrast of the full-factorial model. **Implementation:** `statsmodels` GEE (Liang and Zeger 1986) of d_qk on an intercept (Gaussian family, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) gives the estimate and the variance. The p-value and the interval are always computed from the t reference above, whatever reference the library reports by default. A known-answer test (RDY task 15.6) checks that the library estimate and variance equal the closed form. If they do not, the closed form is used and the difference is reported. +3. **FreshStack, BioASQ and Legal.** The decision test is defined by its closed form. Let n be the number of questions that enter the row and r_q = d_qk minus the estimate. Estimate: the mean of d_qk. Variance: (sum of r_q squared) x (n / (n minus 1)) squared / n squared. Reference distribution: t on n minus 1 degrees of freedom, for the p-value and the 95% interval. This is the Mancl and DeRouen (2001) bias-reduced sandwich for an intercept-only model with one row per cluster. It equals the paired t-test variance times n / (n minus 1), so it needs no convergence fallback. With complete data the estimate equals the cell-means contrast of the full-factorial model. **Implementation:** `statsmodels` GEE (Liang and Zeger 1986) of d_qk on an intercept (Gaussian family, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) gives the estimate and the variance. The p-value and the interval are always computed from the t reference above, whatever reference the library reports by default. A known-answer test (RDY task 15.6) checks that the library estimate and variance equal the closed form. If they do not, the closed form is used and the difference is reported. The closed form was checked against `statsmodels` 0.15.0 on 2026-11-14 on a seeded fixture of 30 questions: the GEE estimate and variance matched it to 1e-10. The library reports a normal reference for the p-value (`use_t=False`), so the code takes the t reference itself. The build-time test of RDY task 15.6 stays. 4. **FinanceBench (questions share documents).** The decision p-value comes from a document-level sign-flip permutation test. The statistic is the question-weighted mean of d_qk. Under the null, the sign of each document's sum of d_qk is flipped at random: 100,000 draws, seed 20261101, p = (1 + number of draws with an absolute statistic at least as large) / 100,001. The 95% interval inverts the same test (the set of shifts not rejected at 0.05, with the same draws). The test assumes each document's sum is symmetric about zero under the null. The interval also assumes a constant additive shift for every question (a location-shift model); the point estimate and the p-value do not need it. 5. **Decision.** Holm (alpha 0.05) within each family (Carterette 2012) on the decision p-values, then the sign rule of section 7, then the 2-of-3 or both-datasets rule of each hypothesis (with the same-sign rule of section 7 for two-sided rows). 6. **Sensitivity checks, registered, never decisive.** FreshStack and BioASQ: paired permutation test by question (10,000 shuffles, seed 20261101; Smucker, Allan and Carterette 2007; Urbano, Lima and Hanjalic 2019). FinanceBench: the GEE of point 3 with document clusters. Every row where the decision and the sensitivity check disagree on "counts" is listed in one table. @@ -272,7 +272,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti | Answer model host, reasoning mode, output cap, context length (at least 65,536 tokens) | To record | Missing: P10 | | Ragas judge and classifier | `openai/gpt-oss-120b`, reasoning `low`, temperature 0, fixed seed, one host | `plan.json` `runtime.references` (model only); host and settings P10 | | GLM-5.3 | Z.AI, `chat`, reasoning `low`, temperature 0, seed 20261109 | `protocol.md` section 4.4; `plan.json` model only (P10) | -| Ragas, statsmodels | Ragas 0.4.3; statsmodels in the `stats` dependency group | Ragas: `uv.lock`. statsmodels: Missing: RDY task 15.6 adds a `stats` dependency group and pins the version | +| Ragas, statsmodels | Ragas 0.4.3; statsmodels 0.15.0 in the `stats` dependency group | `uv.lock` (statsmodels added 2026-11-14) | | Query instruction | Empty string | Missing: P11 | | Reranker instruction | llama-server default Qwen3-Reranker instruction, text recorded | Missing: P11 | | Hybrid settings | RRF constant, per-side fusion depth, BM25 tokeniser and parameters | Missing: P11 | @@ -321,7 +321,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti - **Estimate**: about **$730** on an RTX 4090 at $0.89 per hour (estimate). It is the A3 estimate of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 extra pod hours, about $30 to $40, not timed: P2), the second closed-book run (about $3 hosted and about 1 pod hour, by analogy with the first) and the pilot. The pilot cost is not yet estimated: it is projected from the P2 timing, added here before the freeze and approved by the user before the pod starts (section 10). Search cost barely changes with corpus size, because the rerank pool stays 100. - **Ceiling**: **$975**, pods and hosted calls together (1.5 x the A2 estimate, about 1.34 x this one). At 3 times slower reranking than assumed, the total is about $950 (estimate), still under the ceiling with little margin. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. - **Checks**: before the freeze (test pod extrapolation), on the build pods after the search stage, and at each stage end while the replicate pods run. Each projection uses billing and measured rates only, never scores. -- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. +- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. - **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early only when the preflight fails, an execution-integrity gate (G1 to G3, section 15) fails, or a projection reaches the ceiling after the last drop step. A failed outcome-validation gate (faithfulness or correctness, section 11.2) does not stop the run: its rows stay in their families as "not testable" (p = 1), and the rest of the run continues. The classifier gate is decided before the freeze (section 11.1). - **Part 2 and E1** each get their own estimate and ceiling before their runs (P16). @@ -399,7 +399,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - O3. Whether the optional blind relevance audit for H3 and C1 runs. - O4. L2's name, role and relation to the project. - O5. The ethics query to the university and its answer. -- O6. BioASQ rows 12 to 19 as sanity checks (BioASQ family 8, families 19 / 19 / 8, 46 contrasts), with H8a and H8b holding only on FinanceBench and FreshStack. This changes the earlier 2-of-3 rule for H8. The alternative: keep rows 12 to 19 in the BioASQ family (16 rows, 54 contrasts, alpha 0.05/16 for the BioASQ MDE) and keep 2-of-3 for H8a and H8b. The user confirms one before the freeze. +- O6. Decided by the owner on 2026-11-14: BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8, 46 contrasts); H8a and H8b hold on FinanceBench and FreshStack only. - O7. Whether the sensitivity version of faithfulness (uncited factual segments count as unsupported, section 6) becomes the primary measure. Until the user decides, the primary measure stays faithfulness of cited factual claims. **Pre-freeze tasks:** diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index 887bb39a..1b830513 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -184,12 +184,12 @@ Written before the freeze. No measured run has started. A publication-grade meth | --- | --- | --- | | C1 | OSF category "registration before analysis of the outcome data"; pilot scores on half A are main-run outcome data for the pilot cells | 1 | | C2 | Pilot: at most the 4 H7 corner indexes per core dataset, not reused; stand-in SD for adjacent levels; cost projected from P2 and approved before the pod; registered fallback without a pilot | 10, 17, 23 (P5) | -| C3 | BioASQ rows 12 to 19 become sanity checks; families 19 / 19 / 8, 46 contrasts; BioASQ MDE alpha 0.05/8; open item O6 | 7, 8, 10, 23 | +| C3 | BioASQ rows 12 to 19 become sanity checks; families 19 / 19 / 8, 46 contrasts; BioASQ MDE alpha 0.05/8; open item O6, decided by the owner on 2026-11-14 | 7, 8, 10, 23 | | C4 | Two-sided hypotheses hold only when the same row counts with the same sign on every dataset the rule needs (H6, H8a, H8b, C1, C2) | 7, 18 | | C5 | One gate interval method: percentile bootstrap, 10,000 resamples by question, own seed; differential-error rule on GLM-5.3, either direction | 11.2 | | C6 | Drop order without E1 (E1 has its own ceiling); contamination check falls back to the own-knowledge run; "complete with registered drops" | 13, 17 | | C7 | Half-A rule covers samples drawn after `halves.json` exists; earlier samples listed in section 5 | 4 | -| C8 | `statsmodels` pin marked missing: RDY task 15.6 adds the `stats` group | 16 | +| C8 | `statsmodels` 0.15.0 added to the `stats` dependency group and `uv.lock` on 2026-11-14; the closed form was checked against its GEE (match to 1e-10) | 16 | | C12 | Search-cost figure: RAGFlow curve only if its matched search runs; "adapted from Dodge et al. 2019", ascending sort | 14.3 | | C13 | Sign-flip interval assumes a constant additive shift | 9 | | C14 | Seeds table, one row per seed; new seeds 20261121 to 20261135 wait for P9 | 16, 23 | @@ -215,7 +215,7 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | Part 2 C2: section 12 rules for both systems, complete pairs, unjudged-as-0 sensitivity | 18 | | Contamination subset falls back to the own-knowledge run when the RAG-prompt run is dropped | 13 | | Pilot seed confirmed before the pilot | 23 (P9) | -| OpenSpec only: half-B guard checks the hash of the current selection file; budget check uses 52,200 answers (15 setups x 3 runs x 1,160); pilot-declined path; labelling and closed-book run tasks moved out of the pre-freeze section; Holm family size follows the recorded O6 answer | none | +| OpenSpec only: half-B guard checks the hash of the current selection file; budget check uses 52,200 answers (15 setups x 3 runs x 1,160); pilot-declined path; labelling and closed-book run tasks moved out of the pre-freeze section; Holm family sizes are 19 / 19 / 8 (O6, decided by the owner on 2026-11-14) | none | Cost: estimate about $730 (was about $690) plus the pilot, projected from P2 before the freeze; ceiling $975 unchanged (pre-registration section 17). diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index 7dad1065..7fed6986 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -575,7 +575,7 @@ The operator approved every fix of an independent methods review (Exp 35 change - **Setups (M5).** Three token-matched answer setups, `c512-t20k`, `c1024-t20k` and (added by A3, 2026-11-13) `c2048-t20k`, at the chosen embedder and reranker size, reranker on and hybrid on; the setup name gains `-t20k` when evidence is cut by the token budget instead of 10 chunks. 15 answer setups in all (14 before A3). - **Closed-book run (M6).** A `closedbook` stage: one run, all 1,160 questions, the answer model with no evidence (a fixed prompt that asks for an answer from the model's own knowledge), then `AnswerCorrectness` and the number match. It runs on replicate pod 1. **A4 (D38):** a second closed-book run uses the main RAG answer prompt with an empty evidence block, and the contamination sensitivity re-runs rows 14, 15, 18 and 19 on the questions both closed-book runs answer wrongly. - **Split halves (M3).** `split35.py` writes `halves.json` per dataset (seed 20261115). `select35.py` reads half A only. The report reads half B for answer-stage headlines. **A4 (D38):** FinanceBench is split by document, so no document sits in both halves; the other datasets are split by question. -- **Clustered analysis (M2, A2c N4; replaced by A4, D38).** No MixedLM. For FreshStack, BioASQ and Legal the decision test is defined by its closed form (pre-registration section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster; the paired t-test variance times n / (n minus 1)); t reference on n minus 1 degrees of freedom. `experiments/_lib/stats_cluster.py` implements it with `statsmodels` `GEE` on per-question contrast scores (intercept only, Gaussian, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) and computes the p-value and interval from the t reference itself, because GEE results report a normal reference by default (`use_t=False` in the statsmodels source). The known-answer test of task 15.6 checks the GEE estimate and variance against the closed form; on a mismatch the closed form is used and the difference reported. The same module runs the document-level sign-flip permutation test (100,000 draws, seed 20261101) for FinanceBench; one method gives every estimate, interval and p-value (D38). It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock`. The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. +- **Clustered analysis (M2, A2c N4; replaced by A4, D38).** No MixedLM. For FreshStack, BioASQ and Legal the decision test is defined by its closed form (pre-registration section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster; the paired t-test variance times n / (n minus 1)); t reference on n minus 1 degrees of freedom. `experiments/_lib/stats_cluster.py` implements it with `statsmodels` `GEE` on per-question contrast scores (intercept only, Gaussian, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) and computes the p-value and interval from the t reference itself, because GEE results report a normal reference by default (`use_t=False` in the statsmodels source). The known-answer test of task 15.6 checks the GEE estimate and variance against the closed form; on a mismatch the closed form is used and the difference reported. The same module runs the document-level sign-flip permutation test (100,000 draws, seed 20261101) for FinanceBench; one method gives every estimate, interval and p-value (D38). It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock` (`statsmodels` 0.15.0, added 2026-11-14). The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. - **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750, L2 225. - **Provenance (M7).** Each hosted call row records `model_returned` and `provider_returned` from the reply (OpenRouter `model` and `provider`; Z.AI `model`), beside the requested values. The join counts differences per role. - **Reranker check (M8).** `rerank_check35.py` scores 200 seeded pairs (seed 20261117) per size with llama-server and with the transformers recipe, and reports Spearman correlation; below 0.95 the file is not pinned. @@ -620,7 +620,7 @@ Written before the freeze; no measured run has started. A publication-grade meth - **Deviation policy (F15, F29; PR 20).** After the freeze, every change is a dated, numbered deviation. The report gives its reason, its time (before or after seeing the affected data) and its effect on each claim. Each goes into `plan.json` `amendments` and is uploaded to the OSF project. - **FreshStack full corpus (F1; PR 4).** All 9 FreshStack indexes are built on the full public LangChain corpus (49,514 documents) at a pinned revision (P1). The download needs the user's approval at build time. The 10,000-document subset of Experiments 9a and 22 is retired for Experiment 35, and the continuity queries, continuity documents and overlay runbook are not used. No result is compared with the FreshStack leaderboard unless the corpus and the metric are the same. alpha-nDCG@10 is a secondary FreshStack metric if the nugget labels allow it (P3). One full-corpus build is timed before the estimate is relied on (P2). -- **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3, O6: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. +- **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. - **Analysis (F7, F10; PR 9).** No MixedLM. The per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells; the estimate is its plain mean. FreshStack, BioASQ and Legal: the closed-form test of PR 9 point 3 (mean of d_qk; Mancl and DeRouen bias-reduced variance, sum of squared residuals x (n / (n minus 1)) squared / n squared; t on n minus 1 degrees of freedom), implemented with `statsmodels` GEE on an intercept (independence working correlation, question clusters, `cov_type="bias_reduced"`) and checked against the closed form by the known-answer test of task 15.6. FinanceBench: document-level sign-flip permutation test, 100,000 draws, seed 20261101, interval by inverting the test. Registered sensitivity checks, never decisive: a paired permutation test by question (10,000 shuffles) on FreshStack and BioASQ, and the GEE with document clusters on FinanceBench; disagreements go in one table. No McNemar test and no equivalence test. The type I error simulation test now covers this method (task 15.6). - **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). - **Missing outcomes (F9; PR 12).** Every question stays in its dataset's denominator, and every exclusion is counted per setup and row. A short list scores missing ranks as not relevant. A failed answer call follows the resend rule (D3); an answer still missing at run close scores 0 for correctness and number match and has no factual claim. Faithfulness uses complete cases per row, with two sensitivity analyses (no-claim answers scored 1, and 0). A row still unjudged at run close is counted and scored 0 in a sensitivity analysis. diff --git a/openspec/changes/exp35-main-run-readiness/proposal.md b/openspec/changes/exp35-main-run-readiness/proposal.md index 11b7ff78..04ca8b43 100644 --- a/openspec/changes/exp35-main-run-readiness/proposal.md +++ b/openspec/changes/exp35-main-run-readiness/proposal.md @@ -29,7 +29,7 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set on the full public corpus (A4), and a FinanceBench number-match check. - Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against human labels by Cohen's kappa (design D33; A4 makes it a gate per core dataset on 50 labels each, D38); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget then: about $650, ceiling $975. - Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; `statsmodels` as an experiment-only dependency (approved); two labellers with a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. (The A2 mixed model, the 400 L1 items and the $650 estimate are replaced by A4 below.) -- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3: BioASQ rows 9 to 19 are sanity checks, open item O6) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O1 to O7 are tasks. +- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O1 to O7 are tasks. - Answer context budget (design D30): a test proves that the largest evidence set of every answer setup synthesises without the earlier context overflow, and each answer row records its rounds, evidence characters and truncation. - Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8b with the existing statistics functions. - The harness reads the `rate_limited` record instead of parsing error text, and stops at once on `out_of_credit`. Every hybrid-on setup pins `retrieval.hybrid_sparse_backend: bm25`, as the protocol defines it. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index 5c7ca674..b708d6a0 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -426,7 +426,7 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score, both chosen sizes, the A3 cell and the configuration hash. Then (A4, pre-registration section 14.1) tests that: index cost is the number of chunks times the embedding dimension, summed over the three core datasets, read from the sealed indexes with no score; the configuration hash is the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings with the SHA-256 of `halves.json` as one field, and the same inputs give the same hash twice; the any-page sensitivity repeats step 3 with any-page nDCG@10 on FinanceBench, records whether the cell changes, and never changes the registered cell. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. - [ ] 14.5 Ragas `AnswerCorrectness` (design D33). Write failing tests with a stub LLM and stub embeddings: the `ragas` stage scores five metrics, `AnswerCorrectness` gets the question, the answer and the dataset's reference answer (never the contexts), uses the frozen judge and the local 4B embedder, and records the weights; an answer without a reference records "not judged" and sends no call; resume skips saved rows. Prove they fail, then implement in `ragas35.py` with Ragas 0.4.3. Verify: the tests pass, and the version check still refuses another Ragas version. - [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B, each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). -- [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome (O6); a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). +- [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome; a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). ## 15. Methods-review fixes (pre-freeze amendment A2, 2026-11-13; design D35) @@ -451,8 +451,8 @@ Each test is proved to fail on the current code before the fix. - FinanceBench uses the document-level sign-flip permutation test: the statistic is the question-weighted mean d_qk, each document's sum is flipped at random, 100,000 draws with seed 20261101, p = (1 + draws at least as large in absolute value) / 100,001, and the 95% interval inverts the same test on the same draws; - registered sensitivity checks, never decisive: a paired permutation test by question on FreshStack and BioASQ (10,000 shuffles, seed 20261101, `paired_permutation`), and the GEE above with document clusters on FinanceBench; every row where decision and sensitivity disagree on "counts" goes into one table; - type I error simulation, now for this method: with no planted effect and unequal error variance across cells, the rejection rate at 0.05 stays near 0.05 over 200 simulations for the GEE (FreshStack-sized data) and for the sign-flip test (84 documents, about 1.8 questions each); a fixture with complete data gives the same estimate as the cell-means contrast of the full-factorial model. - - Add `statsmodels` in a new `stats` dependency group (decided by the operator, 2026-11-13; not yet in `pyproject.toml` or `uv.lock`), with `{include-group = "stats"}` in the `dev` group; a test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. -- [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8; O6); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. + - The `stats` dependency group exists (decided by the operator, 2026-11-13; added on 2026-11-14 with `statsmodels` 0.15.0 in `pyproject.toml` and `uv.lock`, and `{include-group = "stats"}` in the `dev` group). The wrapper and its tests are not built yet, so this box stays open. A test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. +- [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the L2 subset is 30% of each set with seeds 20261122, 20261123 and 20261124 (225 in all); no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.9 Hosted-call provenance (M7). Write failing tests: each answer, Ragas, GLM-5.3 and classifier row records `model_returned` and `provider_returned` from the stubbed reply; the join counts rows whose returned model differs from the requested one per role. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.10 Reranker rank check (M8). Write failing tests for `rerank_check35.py`: 200 pairs drawn with seed 20261117 are the same twice; Spearman below 0.95 marks the file "not pinned". Prove they fail, then implement. Run it on the test pod for each size (task 9.2 step 2a). Verify: the tests pass and `protocol.md` lists the correlation per file. @@ -494,6 +494,6 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.25 O3 (user decides). Whether the optional blind relevance audit for H3 and C1 runs (about 300 to 400 L1 labels, seed 20261120, descriptive). Record the answer. If yes, add its sample and page to `labels35.py` with tests first. Verify: the answer is recorded before G7. - [ ] 16.26 O4 (user decides). L2's name, role and relation to the project. Record it in the run record only, never in public files. Verify: no public file holds the name. - [ ] 16.27 O5 (user decides). The ethics query to the university and its answer, before the first label. Verify: the answer is recorded before any label. -- [ ] 16.28 O6 (user decides). BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack), or the alternative: rows 12 to 19 in the BioASQ family (16 rows) with 2-of-3 for H8. Record the answer in PR 7, 8 and 23. Verify: recorded before G7. +- [ ] 16.28 O6 (decided by the owner on 2026-11-14): BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack). The answer is recorded in PR 7, 8 and 23. Verify: recorded before G7. - [ ] 16.29 Uncited segments (PR 6). Write failing tests: the join counts per answer the prose segments with no valid ordinal (the segments `split_claims` drops) and gives `uncited_segment_share` per setup and dataset; `claimkinds` also sorts uncited segments with the frozen classifier (question and segment only); the sensitivity counts uncited fact and mixed segments as unsupported; a fixture answer with one supported cited fact and one uncited fact scores 1.0 primary and 0.5 sensitivity. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.30 O7 (user decides). Whether the uncited-segment sensitivity of PR 6 becomes the primary faithfulness measure. Record the answer in PR 6 and 23. Verify: recorded before G7. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md index 47bce1d0..a4efbe38 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md @@ -481,7 +481,7 @@ Decision (operator, 2026-11-14). A publication-grade methods review gave 30 find - **History rule.** D0 to D0.14 and D1 to D10 stay as history. Text that A4 changes carries the marker "(superseded by A4, D0.15)" or "(amended by A4, D0.15)". Unmarked text that disagrees with `preregistration.md` is an error to fix, not a rule. - **What changes in this change (finding ids from the review):** - F1: all 9 FreshStack indexes use the full public LangChain corpus (49,514 documents, pinned revision). The 10,000-document subset is retired. alpha-nDCG@10 is secondary if the nugget labels allow it. - - F4, F7, F10: no mixed model. Per-question contrast scores. On FreshStack, BioASQ and Legal the decision test is defined by its closed form (mean; Mancl and DeRouen bias-reduced variance; t on n minus 1), with `statsmodels` GEE as the implementation checked against it (`preregistration.md` section 9 point 3); and a document-level sign-flip test on FinanceBench. Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; second round C3, pending open item O6). A gated row stays in its family with p = 1. No equivalence claim, no TOST, no McNemar test as a main test. + - F4, F7, F10: no mixed model. Per-question contrast scores. On FreshStack, BioASQ and Legal the decision test is defined by its closed form (mean; Mancl and DeRouen bias-reduced variance; t on n minus 1), with `statsmodels` GEE as the implementation checked against it (`preregistration.md` section 9 point 3); and a document-level sign-flip test on FinanceBench. Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; second round C3). A gated row stays in its family with p = 1. No equivalence claim, no TOST, no McNemar test as a main test. - F5, F22, F23: H8 splits into H8a (crossed setups) and H8b (token-matched setups). Both are confirmatory. Every hypothesis has scope S, numbered rows and a holds rule. H7 is confirmatory on FinanceBench and FreshStack. After the second round (C3, C4), H6, H7, H8a and H8b hold only on FinanceBench and FreshStack, and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs. Legal RAG Bench is exploratory in every hypothesis. - F3, F8, F19: L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 225. Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. - F12: `evidence_recall_20k` gives any-page credit and is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11). nDCG@10 stays the primary metric of the study. First-page credit is a sensitivity. @@ -490,5 +490,5 @@ Decision (operator, 2026-11-14). A publication-grade methods review gave 30 find - F15: the token cut truncates the crossing chunk at 20,480 tokens (D0.13 N2 now says so). The `openai/gpt-6.1-terra` cross-check is removed. `plan.json` `frozen` stays empty until the freeze; the pins table (`preregistration.md` section 16) fills it. - F17: estimate about $730 plus the pilot (projected from P2), ceiling $975. At the ceiling a results-blind drop order applies: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop (second round C6; E1 has its own ceiling). No operator choice. - F20: index cost and configuration hash are defined in `preregistration.md` section 14.1. - - Second round, C1 to C24 (2026-11-14, `protocol.md` section 0.13): the rulings above plus gate intervals by percentile bootstrap, the half-A rule for samples drawn after `halves.json` exists, the seeds table and open item O6. + - Second round, C1 to C24 (2026-11-14, `protocol.md` section 0.13): the rulings above plus gate intervals by percentile bootstrap, the half-A rule for samples drawn after `halves.json` exists, the seeds table and open item O6 (decided by the owner on 2026-11-14). - **Out of scope here.** Harness code, scorers and statistics code belong to change `exp35-main-run-readiness`. Part 2 and E1 rules belong to the proposal and change `rag-benchmark-readiness-flow`. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md index b1e3cb8c..8d2dc082 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md @@ -25,7 +25,7 @@ A publication-grade methods review gave 30 findings, F1 to F30. The operator app - FreshStack indexes use the full public LangChain corpus (49,514 documents). The 10,000-document subset is retired. - H8 splits into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory. Every hypothesis is scoped, with numbered rows and a holds rule. Legal RAG Bench is exploratory everywhere. -- No mixed model. Per-question contrasts with GEE, and a document-level sign-flip test on FinanceBench. Holm families fixed at 19, 19 and 8 (second round C3; BioASQ rows 9 to 19 are sanity checks, pending open item O6). +- No mixed model. Per-question contrasts with GEE, and a document-level sign-flip test on FinanceBench. Holm families fixed at 19, 19 and 8 (second round C3; BioASQ rows 9 to 19 are sanity checks). - Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items and L2 labels 225. - `evidence_recall_20k` gives any-page credit. A second closed-book run uses the RAG prompt. FinanceBench is split by document. - The `openai/gpt-6.1-terra` cross-check is removed. Budget about $730 plus the pilot, ceiling $975, with a results-blind drop order that no longer holds E1 (E1 has its own ceiling). diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index de9145ca..ee42002f 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -93,7 +93,7 @@ The analysis SHALL follow `preregistration.md` section 9 and SHALL fit no mixed ### Requirement: Holm families are fixed before the run -Holm correction (alpha 0.05) SHALL run within families of fixed size: FinanceBench 19 rows, FreshStack 19 rows and BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks, pending open item O6), 46 in all. Legal RAG Bench SHALL have no family. No gate decided on main-run labels SHALL change a family size. +Holm correction (alpha 0.05) SHALL run within families of fixed size: FinanceBench 19 rows, FreshStack 19 rows and BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks), 46 in all. Legal RAG Bench SHALL have no family. No gate decided on main-run labels SHALL change a family size. #### Scenario: A gate fails - **WHEN** the faithfulness gate of GLM-5.3 fails diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 6146fdeb..ad24422d 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -88,7 +88,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. - [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. - [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 the OSF mechanism that time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 the university ethics query and its answer, before the first label; O6 BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack) or the alternative (16 BioASQ rows, 2-of-3 for H8); O7 whether faithfulness with uncited factual segments as unsupported becomes primary (RDY 16.30). Record each answer; never guess one. Verify: PR 23 shows each item decided. +- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 the OSF mechanism that time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 the university ethics query and its answer, before the first label; O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 whether faithfulness with uncited factual segments as unsupported becomes primary (RDY 16.30). Record each answer; never guess one. Verify: PR 23 shows each item decided. ### F. Freeze @@ -97,7 +97,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime ### G. Run and report - [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (mechanism O1); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. -- [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm family sizes as recorded for O6 in PR 8 at the freeze: 19, 19 and 8, or 19, 19 and 16 under the O6 alternative; gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. +- [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm family sizes 19, 19 and 8 as fixed in PR 8; gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. - [ ] 6.14 Exp 35 section 7 (tasks 7.1 to 7.5): the release table of PR 19 (answers, evidence IDs, ranked lists, human labels and verdicts per licence, never corpus text) and the re-judge kit, copy to `DATA/omrs/`, Zenodo (one archive per dataset licence), identifier search. Merge and archive Exp 35 on the user's word. ## 7. Part 2 (RAGFlow on half B) and E1 diff --git a/pyproject.toml b/pyproject.toml index 81b45f6c..556a6de6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -759,7 +759,12 @@ select = ["E", "F", "I", "UP", "B", "S"] # `uv sync --no-group dev --group ragas`. It conflicts with `dev` so uv resolves it in # its own fork; Ragas needs older rich and jiter, and the base keeps its own versions. ragas = ["ragas==0.4.3", "langchain-community==0.3.31"] +# Experiment 35 clustered analysis (statsmodels GEE, readiness task 15.6). It is +# experiment-only and never in the app install; `uv.lock` records the exact version. +# `dev` includes it so the fast suite runs the analysis tests. +stats = ["statsmodels>=0.15.0"] dev = [ + {include-group = "stats"}, "import-linter>=2.13", "ipywidgets>=8.1.8", "jupytext>=1.19.4", diff --git a/tests/test_dependency_floors.py b/tests/test_dependency_floors.py index 9028d6a3..b9f789eb 100644 --- a/tests/test_dependency_floors.py +++ b/tests/test_dependency_floors.py @@ -182,6 +182,8 @@ def _declared_floors() -> list[tuple[str, str, tuple[int, ...]]]: for group, reqs in pyproject.get("dependency-groups", {}).items(): for req in reqs: + if not isinstance(req, str): + continue # PEP 735 `{include-group = "..."}` table, no floor to read parsed = _extract_floor(req) if parsed: name, version = parsed diff --git a/uv.lock b/uv.lock index 2111d106..6966d151 100644 --- a/uv.lock +++ b/uv.lock @@ -1370,6 +1370,26 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" }, ] +[[package]] +name = "formulaic" +version = "1.2.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "interface-meta" }, + { name = "narwhals" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "pandas" }, + { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "scipy", version = "1.18.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "typing-extensions" }, + { name = "wrapt" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f1/87/e7b9b39d218793f159d7425e264c0ac196da07f1decc333001502fac02fb/formulaic-1.2.2.tar.gz", hash = "sha256:c99e8f11ff7d327eaecaf63855ca69b7fa0da100ad6c0041ef80912fbac667e6", size = 657522, upload-time = "2026-06-02T18:24:42.632Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fc/75/0576b03f7889ad25b5385b4f6c69e0543713425a6f193b056c9b0a2e65ce/formulaic-1.2.2-py3-none-any.whl", hash = "sha256:0f84ff49e3fc9dc0e68ab08a0a9427874021aa6c558e66b44dc634a35739b09b", size = 118939, upload-time = "2026-06-02T18:24:41.344Z" }, +] + [[package]] name = "frozenlist" version = "1.8.0" @@ -2095,6 +2115,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/42/22/429e950b50569cb56eb324cc62992c0a458121e5d5a43acec295a127ade7/instructor-1.17.0-py3-none-any.whl", hash = "sha256:f6b1832d178e899ee0a6a3f247e221f19580fa525c815211c02447691d761217", size = 269394, upload-time = "2026-09-09T02:25:54.888Z" }, ] +[[package]] +name = "interface-meta" +version = "2.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/28/5b/202a48a1ccdef721a95357fbe263c54ef87c18c4972df1ab5c22fe0f33d5/interface_meta-2.0.1.tar.gz", hash = "sha256:902bd9a95a12f195f15753a1080075d4eca7a2cb934fac7ac03c9e362b50796a", size = 15281, upload-time = "2026-04-30T22:35:36.882Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a1/d3/20a61a248feb1249cd81f83ce731f0bdf5170ed96a08a298f7081cda9e90/interface_meta-2.0.1-py3-none-any.whl", hash = "sha256:f38016bef9a4429b6d0792d809be7b65e9781820c674bf7f463999086b6e6323", size = 15330, upload-time = "2026-04-30T22:35:35.201Z" }, +] + [[package]] name = "ipython" version = "9.16.1" @@ -4223,11 +4252,15 @@ dev = [ { name = "pytest-cov" }, { name = "pytest-xdist" }, { name = "ruff" }, + { name = "statsmodels" }, ] ragas = [ { name = "langchain-community" }, { name = "ragas" }, ] +stats = [ + { name = "statsmodels" }, +] [package.metadata] requires-dist = [ @@ -4291,11 +4324,13 @@ dev = [ { name = "pytest-cov", specifier = ">=7.1.0" }, { name = "pytest-xdist", specifier = ">=3.8.0" }, { name = "ruff", specifier = ">=0.16.0" }, + { name = "statsmodels", specifier = ">=0.15.0" }, ] ragas = [ { name = "langchain-community", specifier = "==0.3.31" }, { name = "ragas", specifier = "==0.4.3" }, ] +stats = [{ name = "statsmodels", specifier = ">=0.15.0" }] [[package]] name = "onnxruntime" @@ -4705,6 +4740,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a2/e8/6d75ffd9784bce2e93d1ae4415649427e39a53bb172d4672b2b59c6f0a7b/pathable-0.6.0-py3-none-any.whl", hash = "sha256:82c4ca6c98c502ad12e0d4e9779b6210afee93c38990988c8c5d1b49bdcdf566", size = 18983, upload-time = "2026-05-19T18:15:10.728Z" }, ] +[[package]] +name = "patsy" +version = "1.0.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "packaging" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/08/e3/950e513affdfe1c1864397b4f40d6ed06ec1b54f608e462c27f151dd72df/patsy-1.0.3.tar.gz", hash = "sha256:79ebf4c93ff4d296e58a9d5be2b2ee31bd49d737cf11d70ffbd8a44b2de42e65", size = 399992, upload-time = "2026-08-29T21:35:32.258Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/70/cd3cf5cff538323076a879edef02cce6f13f7d75e03d12f211b0a0dbd178/patsy-1.0.3-py2.py3-none-any.whl", hash = "sha256:d3dbebe8fd5f46e29912d030b63c6268647b59bf788a99e2af28a30234cf357c", size = 233318, upload-time = "2026-08-29T21:35:30.913Z" }, +] + [[package]] name = "pdf-inspector" version = "1.17.0" @@ -6551,6 +6600,72 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c8/cb/6a6a47d5b464bd08695d254f3da6e7986cc70c9fa5d778eda57538edfe56/starlette-1.6.0-py3-none-any.whl", hash = "sha256:a86dd39d14bb45f85a3d18525215a9ef0cfd1f192ac793220e72598c90335f0c", size = 75969, upload-time = "2026-08-08T18:27:56.196Z" }, ] +[[package]] +name = "statsmodels" +version = "0.15.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "formulaic" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "packaging" }, + { name = "pandas" }, + { name = "patsy" }, + { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "scipy", version = "1.18.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a8/1f/a3cbf6ed7cf6286afec694bfef561f3e51483197716717fb8476f1f0c50e/statsmodels-0.15.0.tar.gz", hash = "sha256:5d257fe58d0772bc46a557880ca78e2a8e07fec7bfd9d11074aef8e33e1aecbc", size = 15736853, upload-time = "2026-08-27T10:45:21.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f5/2e/bba0e676ff1df16be77fc70f652c0261d8ca3a7467dda5ecf39afa7bded0/statsmodels-0.15.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:103a83723f998897fbb5d5d0ff728cbe3755a90d9b1f22406bf58f305faff2e7", size = 11714869, upload-time = "2026-08-27T10:34:52.281Z" }, + { url = "https://files.pythonhosted.org/packages/58/c8/08a71f7ad3f0506967d0ab0cdc16a6dec0fe94d0b317f5567836010a10f6/statsmodels-0.15.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0ca5d27711f171ee478d547d5a8c5424d1820a1b73c2a3d6bf63f71ad4525f71", size = 11567437, upload-time = "2026-08-27T10:35:08.711Z" }, + { url = "https://files.pythonhosted.org/packages/51/bd/e9292d873ed61f6d02864dd6a936e53c4129e10eee39117d7cd2d0da845c/statsmodels-0.15.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7ef37437f740620a8da03437a29b1fc21bd9a70eeb231cd0d223cc4d766adb78", size = 11827696, upload-time = "2026-08-27T14:30:23.194Z" }, + { url = "https://files.pythonhosted.org/packages/fa/6f/235505a633776a2c1188e583e5a507a548b4fbd21abc14e199f1a84e8ee2/statsmodels-0.15.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b67886b66d9c7ca118526accedb5c6de7ffd74c015dc0492ea0b1690192b65da", size = 12087914, upload-time = "2026-08-27T14:30:57.648Z" }, + { url = "https://files.pythonhosted.org/packages/04/d1/d64a5a3eef6cd7b1cc71645b3f625c16e024113e7ffcd1b4ec3da71605ec/statsmodels-0.15.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:dd184e911a1a8c52e5c1c27b43ccd10c24faec08053271ca1a35276cae0a3ada", size = 12109911, upload-time = "2026-08-27T14:31:25.914Z" }, + { url = "https://files.pythonhosted.org/packages/b8/be/77c4b5d3f61eab5676981d20f705acee8222ad31de8aca42a331dcd4cfdb/statsmodels-0.15.0-cp311-cp311-win_amd64.whl", hash = "sha256:5e1214312d6362cebe0b5c53b4f478d9072559709000369dfc528211cb993e8a", size = 11361029, upload-time = "2026-08-27T10:35:24.869Z" }, + { url = "https://files.pythonhosted.org/packages/1f/17/14121b56377f2881ce94c0d0a01466585468923340d94841337d324f6be6/statsmodels-0.15.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8e8deb1e3d4ab89fec2a52815f1ec924920c4e0cdb307421a4e47cfed87c03ff", size = 11654329, upload-time = "2026-08-27T15:09:42.207Z" }, + { url = "https://files.pythonhosted.org/packages/ef/f5/6b550b5c936dd9ec5c859477698b4fcbccc826e9a4a4a52847fa51673357/statsmodels-0.15.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c6e20788141df134b83ffb43b8cccf135a876188f84124835c86f011f138adee", size = 11527231, upload-time = "2026-08-27T10:35:42.669Z" }, + { url = "https://files.pythonhosted.org/packages/f0/a5/0e7e87e0dedc68bfeecc24bea92876023db09bac1c0fe50584a7d0819fb5/statsmodels-0.15.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4f90dc4989bd82837c80059fdce132976d4e5a80100119d16e1b00537eb55867", size = 11747428, upload-time = "2026-08-27T14:32:28.987Z" }, + { url = "https://files.pythonhosted.org/packages/41/c7/aeadca282baf51d7b9c135674bac79ded91f4db79c69c188575d22234342/statsmodels-0.15.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c6f2bd48915cc9b3b95c3c7f6ac363b6f8aaa38172962ba5f7f6885f0dfe57f2", size = 12042643, upload-time = "2026-08-27T14:33:06.022Z" }, + { url = "https://files.pythonhosted.org/packages/9c/44/7d3eb3836b3c054280baca7ec91bd12858825de437c6319c1212c2531d94/statsmodels-0.15.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:733887a09ff878a2e29504eda684488b26bd134ba3539ac58d2136a58914c65a", size = 12079497, upload-time = "2026-08-27T14:33:47.147Z" }, + { url = "https://files.pythonhosted.org/packages/03/e3/d863228849da191d37c54bdd72634fbbcfadd12e910e29e4bdca1b952bb6/statsmodels-0.15.0-cp312-cp312-win_amd64.whl", hash = "sha256:ba89eb3ccb1c9c1e400d5081157305495b09350b3a65f2d66d3424ad1d3c4209", size = 11322106, upload-time = "2026-08-27T10:35:58.135Z" }, + { url = "https://files.pythonhosted.org/packages/87/17/fee8cfc100c37b1c553515dbad59607a55085a5b9beac2b008e4c03b6388/statsmodels-0.15.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:09fe34ac80e147cd76cb702ba672559ee3c5dd3ba15c25a232922b46e5ffe70e", size = 11696848, upload-time = "2026-08-27T10:36:13.782Z" }, + { url = "https://files.pythonhosted.org/packages/9e/60/a885751cd3aa796ef94b60f6d3de0c9ace4190feb3e55414ee728d2a0abc/statsmodels-0.15.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0ffe08752f420f12a0b03e8f7d8f3a45417615309b1f37f10561f9a53f044737", size = 11556170, upload-time = "2026-08-27T10:36:29.848Z" }, + { url = "https://files.pythonhosted.org/packages/0f/f4/482701d2dc7540897b13ead827cf8499b11c7325753b8cc4ff1a82c87a18/statsmodels-0.15.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a7ab98634d47048c6b7c168b3f8f71369e256d7314bc386168526f34c7699d7", size = 11752604, upload-time = "2026-08-27T10:47:50.809Z" }, + { url = "https://files.pythonhosted.org/packages/cc/88/5c8819f232448d52514b76a64cf285bc608cff9179f3eeacf58750612323/statsmodels-0.15.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bcaa1e4499697c4e023e5fa6fb04459a34e4f4a449c45ae441992e933bde24db", size = 12055386, upload-time = "2026-08-27T10:49:13.7Z" }, + { url = "https://files.pythonhosted.org/packages/19/a8/298e9b5c993cad3b58913d61b7e7cb0e2fba12e39d3114cd8484f922a5d1/statsmodels-0.15.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:88dbe345ecf317862481b3c86305f42133082e5b366a3a8cf240a6caac404daf", size = 12088356, upload-time = "2026-08-27T10:48:10.645Z" }, + { url = "https://files.pythonhosted.org/packages/28/f5/386c6cfda27c47df81643dc9466b3e28dc2a82152e524131977275a88e0c/statsmodels-0.15.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:ac9bce461d0c8f529f8ed331a469eaa5dd7c3213c06b62623a2e5438a6383093", size = 9809418, upload-time = "2026-08-27T10:36:50.646Z" }, + { url = "https://files.pythonhosted.org/packages/15/b4/a8a4d89918599309bebfc09a2f793a910279f3900671bc959ac8c810edd1/statsmodels-0.15.0-cp313-cp313-win_amd64.whl", hash = "sha256:081adf7e5f2da63f63cbd90491447de5d4c4c2921f19814f235bcf3e736edd3c", size = 11312877, upload-time = "2026-08-27T10:37:08.325Z" }, + { url = "https://files.pythonhosted.org/packages/7a/79/66b57afdcbf9a8b21b1a8e4414ddb006e819b2a81243cd74c12f3301eca3/statsmodels-0.15.0-cp313-cp313-win_arm64.whl", hash = "sha256:3e5f870037ca154a4d177aeac68704a784b24cf2c45d86b046fc28e37a2a5584", size = 10812282, upload-time = "2026-08-27T10:37:26.708Z" }, + { url = "https://files.pythonhosted.org/packages/86/51/daf737efb6002905bd006d296f9ad031ed3861b640075bde3b734706b49d/statsmodels-0.15.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:04e65ee05475aeceb9cb954cf96c0f9361b9cdfd497b8aae47afbab0c7c786dd", size = 11698848, upload-time = "2026-08-27T10:37:46.312Z" }, + { url = "https://files.pythonhosted.org/packages/22/b1/ca42c91e427db43917d4ab5cc6991e7e50b574bcfdf6c4c8eb9590af71d3/statsmodels-0.15.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1f1443cc1ff7683f3d7a9054e6dccd92040fbd4324616a0b6dbb8af1ce7889fb", size = 11571229, upload-time = "2026-08-27T10:38:04.723Z" }, + { url = "https://files.pythonhosted.org/packages/5b/04/d803e8d014d4643d9c0a5eec78343073fbfff1c1018f776dd416c07fc076/statsmodels-0.15.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a45026596c71afa18543ec2af40b895609c181fc3649433c1cded5ce23ec5cd4", size = 11801588, upload-time = "2026-08-27T15:00:31.603Z" }, + { url = "https://files.pythonhosted.org/packages/fd/44/40d7678858065e9ec8efd9f37f04227b22a66632ba194e4aa43fda023f6a/statsmodels-0.15.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:edd803f4606061af8c7a83e2326ea294cbfa802a908dfbda4a88347301fb6820", size = 12062254, upload-time = "2026-08-27T15:00:49.803Z" }, + { url = "https://files.pythonhosted.org/packages/51/69/16c73c3c09f4b8d85f125d7d3ae0f7092d4736a83d24c0159e4fb21632ec/statsmodels-0.15.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:eb7fc1147cd8d17132190db7a8e63cd76a06145dec502816d8f3c20619923f2b", size = 12101259, upload-time = "2026-08-27T15:01:08.755Z" }, + { url = "https://files.pythonhosted.org/packages/23/4c/1138aad8e3c27ebf941d732788488cce7c28d76404a7b7fc1186faad2890/statsmodels-0.15.0-cp314-cp314-win_amd64.whl", hash = "sha256:c3c1138b4d0e5b0c2387b17dafd2a0fd137c67bd869f70d4ee56feecc918cb0d", size = 11348219, upload-time = "2026-08-27T15:11:12.924Z" }, + { url = "https://files.pythonhosted.org/packages/3a/bd/53546d8c9ab013d9a0513706d2a09a03aff12034e6f898987e8ce0f38257/statsmodels-0.15.0-cp314-cp314-win_arm64.whl", hash = "sha256:74c0b63448f03020cc27a8ee698fbcf2ba91e61a9f9eee7978c0c4ed33a9b17f", size = 10866645, upload-time = "2026-08-27T10:38:20.886Z" }, + { url = "https://files.pythonhosted.org/packages/38/a1/0d38328643d4704c8de7867518c3f52ec145ff62537b2ca1e14cdeecf51b/statsmodels-0.15.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:6197b4cca69fd7bc23ab9430abf1c479764cfe74767467f51964a633dfade7a5", size = 11870287, upload-time = "2026-08-27T10:38:38.271Z" }, + { url = "https://files.pythonhosted.org/packages/28/55/fac7e03895f24a0b4883ff9fc69f9421a5df1b60e7edd7e68d49f4327593/statsmodels-0.15.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5538a734dd28efd450bb38a5d5fc94d13ee709bc497e3e4c0a24b3803e3eac39", size = 11755467, upload-time = "2026-08-27T10:38:54.683Z" }, + { url = "https://files.pythonhosted.org/packages/4c/38/5321b9f6065d1c5861e87853720603d56ae81795a83d3040410ca2684305/statsmodels-0.15.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dde0657d18e322eedb2540a84c0582e97afd0f78c370b45eebb0d4ac228a7a83", size = 11636274, upload-time = "2026-08-27T15:03:41.86Z" }, + { url = "https://files.pythonhosted.org/packages/28/56/c75e984c492509d9ab2ba5f2c0e4eab7177cd2cec424cef9dafb42f0cc70/statsmodels-0.15.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90e75e726be2c9fc1d1d4c43337503a35864bdf5b4693773a08c6ee92d04a153", size = 11865818, upload-time = "2026-08-27T15:04:01.503Z" }, + { url = "https://files.pythonhosted.org/packages/61/72/9e03975701c808767c19f394f670941a962204a4f6a00eeaf56b560348e8/statsmodels-0.15.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:65e64afc2655a486ea44a5ffe4e2b05ee3abbcb10ea61d1c02d3e9e519338d9b", size = 11907800, upload-time = "2026-08-27T15:05:49.786Z" }, + { url = "https://files.pythonhosted.org/packages/4f/a7/c467019a8bf1ab6a07ca88004f0a1d4ea2dae89642aa804e49437b256243/statsmodels-0.15.0-cp314-cp314t-win_amd64.whl", hash = "sha256:edee066ac9b171d95c3de925225331a21de1deb5b6452f252bc853e44566e72b", size = 11559083, upload-time = "2026-08-27T15:06:05.872Z" }, + { url = "https://files.pythonhosted.org/packages/54/10/dc78352367cd5bb298340b57c390c944b66236f6d026cc238bc8264c7633/statsmodels-0.15.0-cp314-cp314t-win_arm64.whl", hash = "sha256:40b2737456e75d96866943e33017a1dffc9166025ee33b7eb373fbe8c7a85e09", size = 11029613, upload-time = "2026-08-27T10:39:10.043Z" }, + { url = "https://files.pythonhosted.org/packages/de/cb/a1f5ad730cb66c7d911ebb45413880333d837f1cffff990a4c6d2c541b95/statsmodels-0.15.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:d48ba61db0d13d3033330e6bd1f864dfa9b48c0b952b3b0f7570206b35d552e5", size = 11702125, upload-time = "2026-08-30T16:04:07.635Z" }, + { url = "https://files.pythonhosted.org/packages/ce/db/fb95f242adce112c1507e2062ad6b7b00aea338925f2a03fbe47973c4d72/statsmodels-0.15.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:24b9917a09bbe0227047e512a40652c4b9117c2cfbf19c4a5b9971a8944a8606", size = 11571622, upload-time = "2026-08-30T16:04:29.558Z" }, + { url = "https://files.pythonhosted.org/packages/b5/3f/f4cc7e5f4d6c8f46551aee63e59d9b964881070f552400ebb76667b39ad9/statsmodels-0.15.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:333d0c70f269ff98b647f157d2ae7fbcf0617e6f9a5b1817fc6609766502292c", size = 11780713, upload-time = "2026-08-30T07:40:06.332Z" }, + { url = "https://files.pythonhosted.org/packages/e9/20/fa315a013f70efa6674adfaca59365ea7562371914c408551fc3c399792f/statsmodels-0.15.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:80b946d7c922f46f045f25e50daf13439f79c05e403ba50081cad509892c8b26", size = 12070117, upload-time = "2026-08-30T07:40:27.04Z" }, + { url = "https://files.pythonhosted.org/packages/91/43/a63f7880050c2c24056fa8d663b7c8a1816b25864660abbc73afff3f3558/statsmodels-0.15.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cf897abce5616e2763ab7d7a7fa914677df87a649971f877b83a358c98f6e291", size = 12105153, upload-time = "2026-08-30T07:40:45.814Z" }, + { url = "https://files.pythonhosted.org/packages/d4/d4/523ddb13862c228407297754486b6c9e8777e5114f4532b667b47728c544/statsmodels-0.15.0-cp315-cp315-win_amd64.whl", hash = "sha256:511a4f9cadbf91a690bfa863bb3e25dce583d8140046886fdf35fbfabfbce401", size = 11352844, upload-time = "2026-08-30T16:04:48.14Z" }, + { url = "https://files.pythonhosted.org/packages/53/59/f3058214e61b15582ef17d761c9333def46960ee521313409b2705791029/statsmodels-0.15.0-cp315-cp315-win_arm64.whl", hash = "sha256:0cb6228713fe47046b606daee63ed4473a7c246fa97cf0a6d6f20d3b0c96c051", size = 10866685, upload-time = "2026-08-30T16:05:03.971Z" }, + { url = "https://files.pythonhosted.org/packages/4a/2d/43089b37f4bbbf7373929cd198de6cbdb0b5a4580a6e5e32bf9fe74fccab/statsmodels-0.15.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:27e9467fae83ff327c9d16ae5ea7e771bcdd66a682f49aa324eeefea5584f2d5", size = 11858149, upload-time = "2026-08-30T16:05:20.585Z" }, + { url = "https://files.pythonhosted.org/packages/7d/38/ca88421b81666cdcfc3d0eb12c714f80f2be327669ececbb5930b68a71c0/statsmodels-0.15.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:ecf84918dcd410ab34d3d85d34e0b16ebb10319c9fc77dbf9a1f1a49324c59f8", size = 11739931, upload-time = "2026-08-30T16:05:37.286Z" }, + { url = "https://files.pythonhosted.org/packages/ab/64/98b9e2b1fbfdc4e2392bff349718d8dc796403006a3106ec375c59a5d3f8/statsmodels-0.15.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5891ad077b36d6401e44967739df3d2cdd380a0e6e9e46dadec7e3d3a828af55", size = 11657996, upload-time = "2026-08-30T07:41:05.608Z" }, + { url = "https://files.pythonhosted.org/packages/34/6d/d8c9e73013ee854031218e8a044ba679535268bc61cf59466f19ac8d980c/statsmodels-0.15.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:af80f9658c48792947fc754a026ffbebca61a8b483253781973f604a67cf1f8c", size = 11893402, upload-time = "2026-08-30T07:41:27.553Z" }, + { url = "https://files.pythonhosted.org/packages/cc/f5/b9b47f3c688a8af1b199af96e098b8f20897a60b45187c57ce6fcde77a5f/statsmodels-0.15.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:583eb2a46f7c7a5c34a26715c8b6d92199bfb4f04623db5fff9902a3c7c832d3", size = 11934786, upload-time = "2026-08-30T07:41:46.809Z" }, + { url = "https://files.pythonhosted.org/packages/c4/08/608871f18bc18c6cba0b484e81fd25a023ef3ca6541f6793ec839fa49e1f/statsmodels-0.15.0-cp315-cp315t-win_amd64.whl", hash = "sha256:38038823fbe86f11e09433b4a0f4af5f102327c3137a1604b002ce3cf722a31f", size = 11549124, upload-time = "2026-08-30T16:05:53.179Z" }, + { url = "https://files.pythonhosted.org/packages/e3/eb/90a506dbee5cde669e2ef28565ff6b7a0791ab415eab2da82f92d033cccc/statsmodels-0.15.0-cp315-cp315t-win_arm64.whl", hash = "sha256:90df413d7e09474f7e7cbf8807440c1bf76298cdd9daabf9141f76d7627c9151", size = 11021454, upload-time = "2026-08-30T16:06:08.124Z" }, +] + [[package]] name = "striprtf" version = "0.0.26" From 3bf3d876f8d68e41972bd4581819b50a64f04295 Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 18:15:34 +0000 Subject: [PATCH 4/9] docs(exp35): record owner decisions O1, O5 and O7 --- docs/proposals/first-publication-benchmark.md | 10 +++++----- .../35-full-rag-benchmark-2026-11-01/plan.json | 2 +- .../preregistration.md | 12 ++++++------ .../35-full-rag-benchmark-2026-11-01/protocol.md | 14 +++++++++++--- .../changes/exp35-main-run-readiness/design.md | 4 ++-- .../changes/exp35-main-run-readiness/proposal.md | 2 +- .../specs/benchmark-answer-scoring/spec.md | 2 +- openspec/changes/exp35-main-run-readiness/tasks.md | 10 +++++----- .../tasks.md | 2 +- .../changes/rag-benchmark-readiness-flow/design.md | 3 ++- .../rag-benchmark-readiness-flow/proposal.md | 2 +- .../changes/rag-benchmark-readiness-flow/tasks.md | 4 ++-- 12 files changed, 38 insertions(+), 29 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index efc42300..37ca5e08 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -55,11 +55,11 @@ The claims stay testable. OMRS reaches stated scores under stated conditions. Na ## Order and gates -1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The open items O1 to O7 and the pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The Part 2 protocol exists before this point (it is not written yet). +1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The open items O2, O3 and O4 and the pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The owner decided O1, O5, O6 and O7 on 2026-11-14. The Part 2 protocol exists before this point (it is not written yet). 2. Deposit on OSF in one deposit: the pre-registration as main file, this proposal, the protocol history, `plan.json` and the Part 2 protocol (C1, C2). Record the link and the commit hash. 3. Build and seal the 9 indexes per dataset. The rebuild gate G2 passes before any search writes output. Then run the 72 search setups on all questions (pre-registration section 14.2). 4. Score half A only. Apply the A3 rule. Write `selection.json`. -5. Time-stamp the A3 configuration hash by a file upload to the registered OSF project (open item O1: the user confirms the mechanism). +5. Time-stamp the A3 configuration hash by a file upload to the registered OSF project (O1, decided by the owner on 2026-11-14). 6. Only then score half B and start the answer stage. "Sealed" means the indexes only: half-B ranked lists exist on disk before step 5, but no half-B score is computed before it. 7. Run Part 2 on half B, once per system. 8. Run E1 if the budget allows. E1 is the first item to drop when the budget is short. @@ -106,7 +106,7 @@ The primary metric is nDCG@10 at unit level, with the trec_eval formulas written The answer stage uses the embedder and reranker size chosen on half A (pre-registration section 14.1). It has 15 setups, each run 3 times on all 1,160 questions: 12 crossed setups (chunk size x reranker on or off x hybrid on or off, 10 chunks of evidence each) and 3 token-matched setups (`c512-t20k`, `c1024-t20k`, `c2048-t20k`, reranker and hybrid on, evidence cut at 20,480 Qwen3 tokens). Two closed-book runs (own-knowledge prompt, and the RAG prompt with an empty evidence block) and an oracle run (gold evidence given directly) frame the results (pre-registration section 13). -- Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; open item O7 decides before the freeze whether it becomes primary). +- Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; O7, decided by the owner on 2026-11-14: the primary measure stays cited factual claims, because OMRS answers are built to cite their evidence). - Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. - Human check: two labellers score samples blind (L1 750 labels, L2 225). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. @@ -171,7 +171,7 @@ The main comparison uses one OMRS setup chosen on half A of the Experiment 35 sp - Choose on the three core datasets pooled. Legal RAG Bench takes no part in selection. - Setups within 0.01 of the best tie. Ties go to the cheaper setup in this order: reranker off, hybrid off, lowest index cost, then smaller chunk size. Index cost is the number of stored vector values (chunks times embedding dimension) summed over the three core datasets, known before any score. -- The configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the SHA-256 of `halves.json`. A file upload to the registered OSF project time-stamps it before any half-B score is computed (open item O1). +- The configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the SHA-256 of `halves.json`. A file upload to the registered OSF project time-stamps it before any half-B score is computed (O1, decided by the owner on 2026-11-14). A **defaults row** sits beside the main comparison: OMRS shipped defaults against default RAGFlow. Both use the embedder chosen on half A and the same answer model. OMRS keeps its own shipped retrieval settings. Default RAGFlow is one RAGFlow run: its defaults read from the pinned release, plus the shared parts (the embedder, and the reranker when the A3 cell has it on) and the two measurement settings (cut-off 0, depth 100) of rule 1. When the matched search does not run, the RAGFlow side of the main row and of the defaults row is this same run, and the table shows its scores once. The row names the tuning gap, which is the part of any OMRS lead that comes from choosing a setup on half A. The defaults row is secondary. It gets paired confidence intervals and no confirmatory claim. @@ -358,7 +358,7 @@ The Experiment 35 limitations are in pre-registration section 22. The paper repe 4. Ask the Napier library to confirm the article type and the deal. 5. Done: Experiment 35 amendment A3 (protocol section 0.12) holds the selection rule for Part 2. 6. Done: amendment A4 (protocol section 0.13) and the pre-registration, after the methods review of 13 November 2026. -7. Close the open items O1 to O7 and the pre-freeze tasks P1 to P16 of pre-registration section 23. +7. Close the open items O2, O3 and O4 and the pre-freeze tasks P1 to P16 of pre-registration section 23. 8. Reserve an experiment number for Part 2. Write its protocol, covering every item of "What the Part 2 protocol must hold before its freeze". 9. Before the freeze, pin the RAGFlow release and container digests and connect it to the llama.cpp models. Read the RAGFlow defaults from the pinned release. Time one index per dataset (corpus only). Measure the projected pod cost of the matched search at master flow step 7.2 and record the decision at step 7.2a; it fixes the Part 2 estimand. Check page provenance on a seeded half-A FinanceBench sample (the rule 3 gate: if RAGFlow returns no page for a chunk, the primary unit falls back to the document for both systems) and retrieval repeatability on half A. Send the settings table to the RAGFlow maintainers. 10. Write the Part 2 estimate and ceiling, and the E1 estimate and ceiling (pre-freeze task P16; no figure exists yet). diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index f6ec634f..1810d142 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF, then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O1 to O7 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (open item O7 decides whether it becomes primary); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file.", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O2, O3 and O4 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence.", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index a76b31a1..e43833c7 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -78,7 +78,7 @@ For question q with gold set G_q (size R_q) and unit list L_q of up to 10 units, **Answer metrics.** Per answer, then averaged over the 3 runs per question, then over questions. - **Faithfulness of cited factual claims** (`faithfulness_glm_factual`): share of the answer's cited factual claims (classifier kinds fact and mixed) that GLM-5.3 calls supported. Refusal claims are reported apart. Claims and citations come from the answer itself: the answer model cites evidence by ordinal (for example `[1]`), the saved answer is split into claims, only prose segments with at least one valid ordinal go to the checker, and the checker sees one claim with the text of the evidence its ordinals cite (OpenSpec specs `grounded-answer-synthesis` and `benchmark-answer-scoring`). A prose segment with no valid ordinal (an uncited segment) is not a claim and is not checked, because it names no evidence to check against. -- **Uncited segments.** Co-outcome, reported per setup and dataset: `uncited_segment_share`, the share of the answer's prose segments that carry no valid ordinal. Registered sensitivity analysis: the classifier also sorts the uncited segments (it sees only the question and the segment, section 11.1); uncited segments it calls fact or mixed count as unsupported, and faithfulness is computed again over cited and uncited factual segments together. Open item O7 decides before the freeze whether this version becomes the primary measure. +- **Uncited segments.** Co-outcome, reported per setup and dataset: `uncited_segment_share`, the share of the answer's prose segments that carry no valid ordinal. Registered sensitivity analysis: the classifier also sorts the uncited segments (it sees only the question and the segment, section 11.1); uncited segments it calls fact or mixed count as unsupported, and faithfulness is computed again over cited and uncited factual segments together. Decided by the owner on 2026-11-14 (O7): the primary measure stays faithfulness of cited factual claims, and this version stays a sensitivity analysis only. Reason: OMRS answers are built to cite their evidence, so cited claims are the product's claims. - **Correctness** (`ragas_answer_correctness`): Ragas 0.4.3 (Es et al. 2023) `AnswerCorrectness`, default weights (factual 0.75, similarity 0.25), judge `openai/gpt-oss-120b`, local 4B embedder. The judge sees the question, the answer and the reference answer, never the contexts. FreshStack's reference is `answer_text` (the accepted Stack Overflow answer). - **Secondary**: Clef-Flash faithfulness, FinanceBench number match (`protocol.md` section 0.6 rule), refusal rate, the four other Ragas metrics, closed-book and oracle correctness. @@ -192,7 +192,7 @@ Why: a chance value makes an undefined resample count against a pass, and point - **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: 225. - **Guidelines.** One written guideline per set (claim kinds, faithfulness, correctness, and Part 2 correctness), with worked examples drawn from outside every sample, fixed before the first label and released (P8). Faithfulness follows the LLM-AggreFact task definition (MiniCheck): a claim is supported if and only if the cited passages support it, judged as the checker sees it, on meaning not grammar. - **Blinding.** Labellers never see scores, verdicts, setup names or chunk sizes. Cited passage length can still hint at chunk size; the report says so. A training round on practice items comes first and is not evidence. -- **Ethics.** An ethics query goes to the university before the first label (OPEN item O5); the answer is recorded. +- **Ethics.** The study has no human participants. It uses public datasets, and the labellers are members of the research team, not study subjects, so no ethics review is sought (O5, decided by the owner on 2026-11-14). This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation (section 20). - **Conflict of interest.** The authors built OMRS. Safeguards: this pre-registration and its deviation log; selection on half A only; Holm families fixed now; blind labels with L2 as primary labeller in Part 2; published RAGFlow settings and maintainer correspondence; release of every output (section 19). The paper states the conflict. - **Optional blind relevance audit** for H3 and C1 (about 300 to 400 L1 labels on unjudged top-3 units, seed 20261120): runs only if the user approves (OPEN item O3). It would be descriptive. @@ -235,7 +235,7 @@ Pooled score = plain mean of the per-dataset mean nDCG@10 over FinanceBench, Fre 1. Build and seal the 9 indexes; G2 passes before any search writes into the shared output folder. 2. Run the 72 search setups on all questions. 3. Score half A only. Apply 14.1. Write `selection.json`. -4. Time-stamp the configuration hash with a file upload to the registered OSF project (OPEN item O1: the user confirms the mechanism). +4. Time-stamp the configuration hash with a file upload to the registered OSF project (O1, decided by the owner on 2026-11-14). 5. Only then score half B, and start the replicate pods (answer stage). The word "sealed" applies to indexes only. Half B is held out from every selection step; its search lists exist on disk before step 4, but no half-B score is computed before it. @@ -394,13 +394,13 @@ After the freeze, every change is a dated, numbered deviation. The report lists **OPEN (the user decides):** -- O1. The OSF mechanism that time-stamps the A3 configuration hash (proposed: a file upload to the registered project). +- O1. Decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 configuration hash. - O2. The SESOI value, 0.02 nDCG@10. - O3. Whether the optional blind relevance audit for H3 and C1 runs. - O4. L2's name, role and relation to the project. -- O5. The ethics query to the university and its answer. +- O5. Decided by the owner on 2026-11-14: no ethics query (section 11.3). - O6. Decided by the owner on 2026-11-14: BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8, 46 contrasts); H8a and H8b hold on FinanceBench and FreshStack only. -- O7. Whether the sensitivity version of faithfulness (uncited factual segments count as unsupported, section 6) becomes the primary measure. Until the user decides, the primary measure stays faithfulness of cited factual claims. +- O7. Decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims; the uncited-segment version is a sensitivity analysis only (section 6). **Pre-freeze tasks:** diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index 1b830513..e732c98c 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -165,7 +165,7 @@ Written before the freeze. No measured run has started. A publication-grade meth | F16 | Existing data declared | 5 | | F17 | Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop (E1 left the order, C6 below); "complete with registered drops" defined; "the operator decides" removed; Part 2 budget a pre-freeze task | 17 | | F18 | Release table per artefact and licence, re-judge kit | 19 | -| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 and ethics open | 11.3 | +| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 open; no ethics query (O5, decided 2026-11-14) | 11.3 | | F20 | Pins table; index cost and configuration hash defined | 14.1, 16 | | F21 | FinanceBench split by document; pre-freeze samples from half A; order: score half A, hash, score half B | 4, 14.2 | | F22 | "Holds" defined per hypothesis; H7 confirmatory on FinanceBench and FreshStack, as H6; H4 named as simple effects | 7 | @@ -175,7 +175,7 @@ Written before the freeze. No measured run has started. A publication-grade meth | F26 | Checker 2-point tolerance restated against the MDE table | 11.1, 23 (P7) | | F27 | Expected-maximum estimator for the search-cost figure | 14.3 | | F28 | Compute table in the report, pre-freeze pods included | 21 | -| F29 | Deviation policy; OSF file upload time-stamps the A3 hash (open) | 20, 14.2 | +| F29 | Deviation policy; OSF file upload time-stamps the A3 hash (O1, decided 2026-11-14) | 20, 14.2 | | F30 | Reference fixes | 24 | **Second round (C1 to C24).** A post-fix consistency review (2026-11-14) gave 24 findings, C1 to C24. The orchestrator ruled on each the same day. @@ -208,7 +208,7 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | Change | Pre-registration section | | --- | --- | -| Primary measure named "faithfulness of cited factual claims", definition unchanged; uncited-segment share as co-outcome; sensitivity with uncited factual segments as unsupported; open item O7; limitation added | 6, 11.1, 21, 22, 23 | +| Primary measure named "faithfulness of cited factual claims", definition unchanged; uncited-segment share as co-outcome; sensitivity with uncited factual segments as unsupported; O7 decided by the owner on 2026-11-14 (primary stays cited claims); limitation added | 6, 11.1, 21, 22, 23 | | Undefined gate statistics fail closed (one-class sample fails; undefined resample takes kappa 0 or AUC 0.5, or is left out for the specificity difference; over 5% undefined fails) | 11.2 | | Early stop only on preflight, G1 to G3 or the ceiling after the last drop step; a failed outcome gate leaves its rows not testable | 17 | | One same-sign rule: both-datasets rules need both; 2-of-3 rules (C1, C2) need 2 of 3, and an opposite third is reported | 7, 18 | @@ -217,6 +217,14 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | Pilot seed confirmed before the pilot | 23 (P9) | | OpenSpec only: half-B guard checks the hash of the current selection file; budget check uses 52,200 answers (15 setups x 3 runs x 1,160); pilot-declined path; labelling and closed-book run tasks moved out of the pre-freeze section; Holm family sizes are 19 / 19 / 8 (O6, decided by the owner on 2026-11-14) | none | +**Owner decisions (2026-11-14).** Three open items are closed before the freeze. O2, O3 and O4 stay open. + +| Item | Decision | Pre-registration section | +| --- | --- | --- | +| O1 | A file upload to the registered OSF project time-stamps the A3 configuration hash | 14.2, 23 | +| O5 | No ethics query: the study has no human participants, and the labellers are members of the research team. The question reopens as a deviation if a paid external annotator is used | 11.3, 23 | +| O7 | The primary faithfulness measure stays faithfulness of cited factual claims, because OMRS answers are built to cite their evidence. `uncited_segment_share` stays a co-outcome. The uncited-segment version stays a sensitivity analysis only | 6, 23 | + Cost: estimate about $730 (was about $690) plus the pilot, projected from P2 before the freeze; ceiling $975 unchanged (pre-registration section 17). ## 1. Question diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index 7fed6986..d434715e 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -631,7 +631,7 @@ Written before the freeze; no measured run has started. A publication-grade meth - **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. - **Pins, release, determinism, existing data (F16, F18, F20, F25; PR 5, 15, 16, 19).** One pins table (P10 to P14). A release table per artefact and licence, and a re-judge kit. One build per cell is assumed; an optional `e8b-c2048` rebuild check on FinanceBench runs only when the cost projection after the search stage is at or below the estimate, and is reported, not gated. The existing-data table lists every earlier look; P12 lists each smoke and pre-freeze question ID with its half. -Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O7 are group 16. +Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O7 (O1, O5, O6 and O7 decided by the owner on 2026-11-14; O2, O3 and O4 open) are group 16. ## Risks / Trade-offs @@ -663,7 +663,7 @@ Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack c 1. Build and test the code (tasks 1 to 8) on the feature branch. The default behaviour stays serial-equivalent: concurrency 1 gives today's files. 2. Run the classifier gate (tasks 8.1 to 8.7). The classifier calls run on a 4090 pod; labelling and scoring run on the Mac. 3. Resolve and record the image digest (task 9.1). -4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the user's decisions on O1 to O7. +4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the user's decisions on O2, O3 and O4 (O1, O5, O6 and O7 are decided). 5. Write the procedure, the measured values, the GPU choice, the G2 stop rule and the budget with its ceiling into `protocol.md` (group 10). 6. At G7, freeze `preregistration.md` (the governing file, D38), `protocol.md` and `plan.json` with the measured values, after every group 16 task is done. 7. Index pods (D32, 9 in parallel on the chosen GPU type): each builds one index for all four datasets and seals it. The `e4b-c1024` rebuild check (D20) runs after that seal, and each pod then runs its 8 search setups once `rebuild.json` passes. A failed check stops here: find the cause, fix, build again. Then the FreshStack recheck on the real indexes with zero errors in every answer setup; on the Mac, half-A scoring, the selection rule (D31) and `selection.json`; the OSF time-stamp of the configuration hash; then half-B scoring (D38); and the budget re-check (D23). When the projection reaches $975, apply the drop order of D38. diff --git a/openspec/changes/exp35-main-run-readiness/proposal.md b/openspec/changes/exp35-main-run-readiness/proposal.md index 04ca8b43..cf391d62 100644 --- a/openspec/changes/exp35-main-run-readiness/proposal.md +++ b/openspec/changes/exp35-main-run-readiness/proposal.md @@ -29,7 +29,7 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set on the full public corpus (A4), and a FinanceBench number-match check. - Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against human labels by Cohen's kappa (design D33; A4 makes it a gate per core dataset on 50 labels each, D38); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget then: about $650, ceiling $975. - Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; `statsmodels` as an experiment-only dependency (approved); two labellers with a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. (The A2 mixed model, the 400 L1 items and the $650 estimate are replaced by A4 below.) -- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O1 to O7 are tasks. +- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O2, O3 and O4 (O1, O5, O6 and O7 are decided by the owner on 2026-11-14) are tasks. - Answer context budget (design D30): a test proves that the largest evidence set of every answer setup synthesises without the earlier context overflow, and each answer row records its rounds, evidence characters and truncation. - Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8b with the existing statistics functions. - The harness reads the `rate_limited` record instead of parsing error text, and stops at once on `out_of_credit`. Every hybrid-on setup pins `retrieval.hybrid_sparse_backend: bm25`, as the protocol defines it. diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md index 0271e5aa..7ffddac7 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md @@ -151,7 +151,7 @@ Primary faithfulness, per answer setup and per checker (GLM-5.3 and Clef-Flash), ### Requirement: Uncited segments are reported and tested in a sensitivity analysis -Primary faithfulness SHALL cover cited factual claims only. The report SHALL give, per setup and dataset, the share of prose segments with no valid ordinal. The classifier SHALL also sort uncited segments, with the question and segment only. A registered sensitivity analysis SHALL count uncited fact and mixed segments as unsupported (pre-registration section 6; open item O7). +Primary faithfulness SHALL cover cited factual claims only. The report SHALL give, per setup and dataset, the share of prose segments with no valid ordinal. The classifier SHALL also sort uncited segments, with the question and segment only. A registered sensitivity analysis SHALL count uncited fact and mixed segments as unsupported (pre-registration section 6; O7, decided by the owner on 2026-11-14: sensitivity only). #### Scenario: One cited and one uncited fact diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index b708d6a0..14f64986 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -345,7 +345,7 @@ The classifier is the fixed-prompt LLM `openai/gpt-oss-120b`, pinned (operator d - the G2 stop rule in section 6: if a rebuild check fails, the replicate pods do not start; find the cause, fix it, build the indexes again, then seal and check again; - the zero-error recheck of FreshStack `77536364` and `77762811` in every answer-setup shape (`--qids`, `--index-from`, its own output folder); the pass rule: every `answer-summary.json` has `failed` 0, and every judge summary has `errors` 0; - the search-stage time of the recheck on the volume, recorded as a check on read speed; - - the order of work of pre-registration section 14.2 (A4): on the Mac, score half A only of the 72 search setups, run the selection rule (`select35.py`) and write `selection.json`; time-stamp the configuration hash with a file upload to the registered OSF project (mechanism: open item O1, task 16.23); only then score half B and start the replicate pods; the scorer refuses a half-B score while the time-stamp record is missing (task 16.17); + - the order of work of pre-registration section 14.2 (A4): on the Mac, score half A only of the 72 search setups, run the selection rule (`select35.py`) and write `selection.json`; time-stamp the configuration hash with a file upload to the registered OSF project (O1, decided by the owner on 2026-11-14; task 16.23); only then score half B and start the replicate pods; the scorer refuses a half-B score while the time-stamp record is missing (task 16.17); - the budget re-check after the search stage, with the real pod cost so far and the measured rates; when the projection reaches the ceiling, apply the results-blind drop order of design D38 and record each step; no confirmatory item is dropped by choice; - the three replicate pods with `--run 1`, `2` and `3`, `--local-dir` on the container disk, and their own checkouts; each pod runs answer (with its own retrieval), the two judges, `ragas` and `claimkinds` for the 15 answer setups on its own run, one stage at a time, with the budget read at each stage end; - the expected pod hours from design D22 and D32, as measured by tasks 9.2 and 9.3; @@ -465,7 +465,7 @@ Each test is proved to fail on the current code before the fix. ## 16. Publication-review fixes (pre-freeze amendment A4, 2026-11-14; design D38) -`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 and 16.29 are code. Tasks 16.23 to 16.28 and 16.30 are the user's decisions O1 to O7; record each answer, never guess it. Every task closes before G7. +`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 and 16.29 are code. Tasks 16.23 to 16.28 and 16.30 are the user's decisions O1 to O7. The owner decided O1, O5, O6 and O7 on 2026-11-14; O2, O3 and O4 stay open. Record each answer, never guess it. Every task closes before G7. - [ ] 16.1 P1. FreshStack full corpus. Ask the user to approve the download of `freshstack/corpus-oct-2024`, config `langchain`, at a pinned revision. Then download it into `~/Development/DATA/omrs/` and record the revision, the row count (49,514 expected) and the corpus tree hash in `plan.json` and in the pins table (PR 16). Verify: the loader of task 13.5 accepts the corpus, and its check refuses a changed row count. - [ ] 16.2 P2. Time one full-corpus FreshStack index build on the chosen GPU type (part of task 9.2 or 9.3 when possible). Update the estimate in PR 17 from the measured time before relying on it. Verify: the compute table and the estimate name the measured hours; the "30 to 40 pod hours" reviewer figure is no longer the basis. @@ -489,11 +489,11 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.20 Report items (PR 21). Write failing report tests: the realised chunk-length distribution per chunk-size level and dataset; one sensitivity table set (clustered method against sensitivity check, FinanceBench page rules, missing-outcome bounds, closed-book subset, length-controlled check, A3 cell under any-page credit); the compute table (GPU type, pod hours per stage with every pre-freeze pod, hosted tokens and cost per role, energy estimate, `resent_calls`, error rows per stage and role, every `out_of_credit` stop); the returned model ID and provider per hosted call with the count of differences; every row with SESOI and MDE lines and "not testable" marks. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.21 Release kit (PR 19). Write failing tests for a re-judge script that rebuilds evidence text from the pinned dataset revisions and the released unit and chunk IDs, on a fixture; and for an export that writes ranked lists in TREC run format, never corpus text. Prove they fail, then implement. Verify: the tests pass, and the release table of PR 19 names each artefact and its licence. - [ ] 16.22 Optional `e8b-c2048` rebuild check (PR 15). Run it on FinanceBench only when the cost projection after the search stage is at or below the estimate (a cost-only decision, recorded). Report the share of identical ranked lists and the nDCG@10 change; it is not a gate. Verify: the decision and, if run, the result are in the run record. -- [ ] 16.23 O1 (user decides). The OSF mechanism that time-stamps the A3 configuration hash (proposed: a file upload to the registered project). Record the answer in PR 14.2 and PR 23. Verify: the answer is recorded before G7. +- [ ] 16.23 O1 (decided by the owner on 2026-11-14): a file upload to the registered OSF project time-stamps the A3 configuration hash. The answer is recorded in PR 14.2 and PR 23. Verify: recorded before G7. - [ ] 16.24 O2 (user decides). The SESOI value (proposed: 0.02 nDCG@10). Record the answer in PR 10. Verify: recorded before the pilot results are read. - [ ] 16.25 O3 (user decides). Whether the optional blind relevance audit for H3 and C1 runs (about 300 to 400 L1 labels, seed 20261120, descriptive). Record the answer. If yes, add its sample and page to `labels35.py` with tests first. Verify: the answer is recorded before G7. - [ ] 16.26 O4 (user decides). L2's name, role and relation to the project. Record it in the run record only, never in public files. Verify: no public file holds the name. -- [ ] 16.27 O5 (user decides). The ethics query to the university and its answer, before the first label. Verify: the answer is recorded before any label. +- [ ] 16.27 O5 (decided by the owner on 2026-11-14): no ethics query. The study has no human participants, and the labellers are members of the research team, not study subjects, so no ethics review is sought. This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation. The answer is recorded in PR 11.3 and PR 23. Verify: recorded before any label. - [ ] 16.28 O6 (decided by the owner on 2026-11-14): BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack). The answer is recorded in PR 7, 8 and 23. Verify: recorded before G7. - [ ] 16.29 Uncited segments (PR 6). Write failing tests: the join counts per answer the prose segments with no valid ordinal (the segments `split_claims` drops) and gives `uncited_segment_share` per setup and dataset; `claimkinds` also sorts uncited segments with the frozen classifier (question and segment only); the sensitivity counts uncited fact and mixed segments as unsupported; a fixture answer with one supported cited fact and one uncited fact scores 1.0 primary and 0.5 sensitivity. Prove they fail, then implement. Verify: the tests pass. -- [ ] 16.30 O7 (user decides). Whether the uncited-segment sensitivity of PR 6 becomes the primary faithfulness measure. Record the answer in PR 6 and 23. Verify: recorded before G7. +- [ ] 16.30 O7 (decided by the owner on 2026-11-14): the primary faithfulness measure stays faithfulness of cited factual claims. Reason: OMRS answers are built to cite their evidence, so cited claims are the product's claims. `uncited_segment_share` stays a co-outcome, and the uncited-segment sensitivity of PR 6 stays a sensitivity analysis only. The answer is recorded in PR 6 and 23. Verify: recorded before G7. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index 46057455..9e9d6029 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -94,7 +94,7 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Verify: the guidelines are committed before the freeze. (The labelling itself is task 5.7.) - [ ] 4D.7 Fix the second closed-book run (RAG prompt, empty evidence) and the length-controlled check (300 claims) in `plan.json` and the pre-registration. Verify: both are listed before the freeze. (The runs are task 5.8.) - [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if O3 is yes); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. -- [ ] 4D.9 Get the operator's answers to O1 to O7 (OSF time-stamp mechanism, SESOI value, optional blind audit, L2 identity, ethics query, BioASQ rows 12 to 19 as sanity checks, uncited-segment faithfulness as primary). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. +- [ ] 4D.9 Get the operator's answers to O2, O3 and O4 (SESOI value, optional blind audit, L2 identity). The owner decided O1 (OSF file upload time-stamps the hash), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only) on 2026-11-14. Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. - [ ] 4D.10 Check before the freeze that no live text in this change, `protocol.md` section 0 or `plan.json` contradicts `preregistration.md`. Verify: every hit for "MixedLM", "included whole", "gpt-6.1-terra", "operator decides" and "equivalence" is marked superseded or removed. ## 5. Run diff --git a/openspec/changes/rag-benchmark-readiness-flow/design.md b/openspec/changes/rag-benchmark-readiness-flow/design.md index 73589593..0b03b415 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/design.md +++ b/openspec/changes/rag-benchmark-readiness-flow/design.md @@ -90,7 +90,8 @@ D. Section 2 harness, PR (G7b) E. Arm smoke run on the run machine; pod stopped after it | E2. A4 pre-freeze items: FreshStack full corpus pin and timing, - | halves and MDE pilot on half A, P1 to P16, open items O1 to O7 + | halves and MDE pilot on half A, P1 to P16, + | open items O2, O3 and O4 (O1, O5, O6, O7 decided) | (with Part 2 steps 7.1, 7.2 and 7.2a) F. Freeze (G7) | diff --git a/openspec/changes/rag-benchmark-readiness-flow/proposal.md b/openspec/changes/rag-benchmark-readiness-flow/proposal.md index 6dfb05bc..672917c6 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/proposal.md +++ b/openspec/changes/rag-benchmark-readiness-flow/proposal.md @@ -15,7 +15,7 @@ Without one recorded order, two branches can touch the same specs at once, the l - A task order only (tasks.md groups 0 to 8) with gates. No code and no capability deltas. - The order (operator decisions 2026-11-06): cuts, then router, then one live switch (no re-index: the live collections are test data, operator 2026-11-07), then the pipeline sweep on RunPod, then Experiment 35. The page routing panel follows the router and can run beside the sweep, because it touches no runtime and no result. -- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the user's open items O1 to O7) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). +- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the user's open items O2, O3 and O4; O1, O5, O6 and O7 are decided by the owner on 2026-11-14) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). ## Capabilities diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index ad24422d..41e8ec69 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -88,7 +88,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. - [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. - [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 the OSF mechanism that time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 the university ethics query and its answer, before the first label; O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 whether faithfulness with uncited factual segments as unsupported becomes primary (RDY 16.30). Record each answer; never guess one. Verify: PR 23 shows each item decided. +- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. O2, O3 and O4 stay open. Verify: PR 23 shows each item decided. ### F. Freeze @@ -96,7 +96,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime ### G. Run and report -- [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (mechanism O1); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. +- [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (O1, decided 2026-11-14); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. - [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm family sizes 19, 19 and 8 as fixed in PR 8; gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. - [ ] 6.14 Exp 35 section 7 (tasks 7.1 to 7.5): the release table of PR 19 (answers, evidence IDs, ranked lists, human labels and verdicts per licence, never corpus text) and the re-judge kit, copy to `DATA/omrs/`, Zenodo (one archive per dataset licence), identifier search. Merge and archive Exp 35 on the user's word. From 6bac2968770844818d02f3190bf40c21ae8a630d Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 18:48:12 +0000 Subject: [PATCH 5/9] docs(exp35): address PR 210 review round 2 Fix six CodeRabbit findings on PR 210 and reject one. Row 11 (H7) uses coefficients +1, -1, -1, +1 on its corner cells, with its MDE and a known-answer case on that scale. Every question stays in H8b; realised evidence token totals are recorded, and "chunk size itself" also needs a sensitivity on questions with at least 95% of the token budget. Every undefined resample is counted once and left out, apart from the over-5% rule. Drop step 1 removes the Legal correctness labels (150, L1 700, L2 210). The readiness flow asks O2 before any pilot result is read. --- docs/proposals/first-publication-benchmark.md | 6 ++-- .../plan.json | 8 +++--- .../preregistration.md | 28 +++++++++---------- .../protocol.md | 11 ++++++++ .../exp35-main-run-readiness/design.md | 6 ++-- .../specs/benchmark-answer-scoring/spec.md | 15 ++++++++-- .../specs/benchmark-harness/spec.md | 5 ++++ .../changes/exp35-main-run-readiness/tasks.md | 12 ++++---- .../design.md | 2 +- .../specs/document-rag-evaluation/spec.md | 12 ++++++++ .../specs/evaluation-observability/spec.md | 2 +- .../rag-benchmark-readiness-flow/tasks.md | 4 +-- 12 files changed, 74 insertions(+), 37 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index 37ca5e08..218617ee 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -108,7 +108,7 @@ The answer stage uses the embedder and reranker size chosen on half A (pre-regis - Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; O7, decided by the owner on 2026-11-14: the primary measure stays cited factual claims, because OMRS answers are built to cite their evidence). - Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. -- Human check: two labellers score samples blind (L1 750 labels, L2 225). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). +- Human check: two labellers score samples blind (L1 750 labels, L2 225; L1 700 and L2 210 if drop step 1 removes Legal answers). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. ### Hypotheses H1 to H8b @@ -121,9 +121,9 @@ Full wording, contrast rows and "holds" rules: pre-registration sections 7 and 8 - **H4:** reranker plus hybrid beats each alone (simple effects, not an interaction test). - **H5:** a bigger reranker ranks better. - **H6:** chunk size changes retrieval (the same row, same sign, on FinanceBench and FreshStack). -- **H7:** a bigger embedder gains more from bigger chunks (FinanceBench and FreshStack; expected to be underpowered). +- **H7:** a bigger embedder gains more from bigger chunks (FinanceBench and FreshStack; expected to be underpowered). Row 11 is a difference of differences with coefficients +1, -1, -1, +1 on the four corner cells, after averaging over reranker and hybrid. - **H8a:** chunk size changes faithfulness and correctness in the crossed setups, where context length grows with chunk size (the same row, same sign, on FinanceBench and FreshStack). -- **H8b:** chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. The paper says "chunk size itself changes X" only when H8b holds for X. +- **H8b:** chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. The paper says "chunk size itself changes X" only when H8b holds for X and a registered sensitivity agrees: the same row, on the questions where all three `-t20k` setups reach at least 95% of 20,480 tokens, has the same sign on FinanceBench and FreshStack. Every question stays in H8b, and realised evidence token totals are reported per setup. ### Methods in short diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 1810d142..7fdbc911 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -43,7 +43,7 @@ { "id": "H7", "role": "primary", - "statement": "within scope S (preregistration.md section 2), a bigger embedder gains more from bigger chunks: row 11, (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, on the H6 endpoints; holds when row 11 counts on both FinanceBench and FreshStack; BioASQ is a sanity check and Legal RAG Bench is exploratory; expected to be underpowered", + "statement": "within scope S (preregistration.md section 2), a bigger embedder gains more from bigger chunks: row 11, (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, on the H6 endpoints; a difference of differences with coefficients +1, -1, -1, +1 on those four cells; holds when row 11 counts on both FinanceBench and FreshStack; BioASQ is a sanity check and Legal RAG Bench is exploratory; expected to be underpowered", "null": "the difference of differences is zero" }, { @@ -55,11 +55,11 @@ { "id": "H8b", "role": "answer-stage", - "statement": "within scope S (preregistration.md section 2), chunk size changes faithfulness and correctness when evidence is held at 20,480 Qwen3 tokens: rows 16 to 17 (faithfulness) and 18 to 19 (correctness), c1024-t20k minus c512-t20k and c2048-t20k minus c1024-t20k, two-sided, at reranker on and hybrid on with the chosen sizes (conditional on that cell); holds as H8a (same row, same sign, on FinanceBench and on FreshStack); 'chunk size itself changes X' is claimed only when H8b holds for outcome X; confirmatory", + "statement": "within scope S (preregistration.md section 2), chunk size changes faithfulness and correctness when evidence is held at 20,480 Qwen3 tokens: rows 16 to 17 (faithfulness) and 18 to 19 (correctness), c1024-t20k minus c512-t20k and c2048-t20k minus c1024-t20k, two-sided, at reranker on and hybrid on with the chosen sizes (conditional on that cell); holds as H8a (same row, same sign, on FinanceBench and on FreshStack); 'chunk size itself changes X' is claimed only when H8b holds for outcome X and the registered sensitivity on questions where all three -t20k setups reach at least 19,456 tokens (95% of 20,480) gives the same sign on FinanceBench and FreshStack; every question stays in H8b; realised evidence token totals are recorded per question and setup; confirmatory", "null": "each contrast is zero" } ], - "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE (version 0.15.0, `stats` dependency group, recorded in uv.lock) of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. The closed form was checked against statsmodels 0.15.0 on 2026-11-14 and matched to 1e-10. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis needs the same sign: under the both-datasets rule (H6, H8a, H8b) the row counts with the same sign on FinanceBench and FreshStack, and opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold; under a 2-of-3 rule (Part 2 C1 and C2) the row counts with the same sign on at least 2 of the 3 core datasets, and an opposite-sign count on the third is reported as a dataset-dependent effect and does not stop the hypothesis holding. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", + "decision_rule": "preregistration.md sections 7 to 9 govern. No mixed model. For row k and question q, the contrast score d_qk is the mean of q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights; answer scores averaged over 3 runs first); this recipe does not apply to row 11 (H7): each of its four corner cells is first averaged over reranker and hybrid, then d_q11 = (+1) e8b-c2048 (-1) e8b-c512 (-1) e0.6b-c2048 (+1) e0.6b-c512, so scores 0.8, 0.2, 0.3, 0.7 give 1.0, and its estimate, interval and MDE use that scale (known-answer fixture case); the estimate is the plain mean of d_qk over questions. FreshStack, BioASQ and Legal: the decision test is defined by its closed form (preregistration.md section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster); t reference on n minus 1 degrees of freedom for the p-value and interval. Implementation: statsmodels GEE (version 0.15.0, `stats` dependency group, recorded in uv.lock) of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, cov_type bias_reduced); a known-answer test checks it against the closed form, and the closed form is used if they differ. The closed form was checked against statsmodels 0.15.0 on 2026-11-14 and matched to 1e-10. FinanceBench: the decision p-value and 95% interval come from a document-level sign-flip permutation test (100,000 draws, seed 20261101). Holm (alpha 0.05) within families fixed now: FinanceBench 19 rows, FreshStack 19 rows, BioASQ 8 rows (rows 1 to 8; rows 9 to 19 are sanity checks); Legal RAG Bench has no family and is exploratory. A row whose measure fails its gate, or that the differential-error rule makes exploratory, stays in its family with p = 1 and is reported as not testable. A directional row counts when the Holm-adjusted p is below 0.05 and the estimate has the predicted sign; a two-sided row counts when the Holm-adjusted p is below 0.05. Hypotheses hold by the rule in each statement (H1 to H5: 2 of 3 core datasets; H6, H7, H8a and H8b: FinanceBench and FreshStack). A two-sided hypothesis needs the same sign: under the both-datasets rule (H6, H8a, H8b) the row counts with the same sign on FinanceBench and FreshStack, and opposite signs are reported as a dataset-dependent effect and the hypothesis does not hold; under a 2-of-3 rule (Part 2 C1 and C2) the row counts with the same sign on at least 2 of the 3 core datasets, and an opposite-sign count on the third is reported as a dataset-dependent effect and does not stop the hypothesis holding. Fixed search settings: similarity_threshold 0.0, metadata extraction disabled, exact dense search with no approximate vector index, chunk overlap 15% (77, 154 and 307 tokens). Search scoring: top 100 chunks (rerank pool 100, fetch_k 100) collapsed to distinct units and cut at 10; FinanceBench nDCG@10 gives a chunk its first page (any-page credit is a sensitivity); evidence_recall_20k gives any-page credit (first-page credit is a sensitivity). Registered sensitivity checks, never decisive: paired permutation by question (10,000 shuffles, seed 20261101) on FreshStack and BioASQ; GEE with document clusters on FinanceBench. A non-significant result means no difference was detected; there is no equivalence claim and no TOST.", "stats": "expert", "seed": 20261101, "baseline": "none: full factorial of embedder x chunk size x reranker x hybrid (72 search setups per dataset); e4b-c1024-roff-hoff is the reference cell of the G2 rebuild check only", @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O2, O3 and O4 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence.", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O2, O3 and O4 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 210; no confirmatory row changes).", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index e43833c7..9e1b6dec 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -30,7 +30,7 @@ - **Search stage.** Full factorial: 9 indexes (`e-c`) and 72 search setups (`e-c-r-h`) per dataset. Each setup runs once on its sealed index. Each index is built once (one-build-per-cell assumption, section 15). - **Fixed in search.** Every setup saves its top 100 chunks (`top_k` 100, `fetch_k` 100, so the rerank pool is exactly 100 and the reranker returns all 100 in order). `similarity_threshold` 0.0. Metadata extraction `disabled`. All datasets go through the Markdown splitter and count tokens with the pinned Qwen3 tokenizer (revision `5cf2132a`). BioASQ and Legal passages are written as `.md`. Dense search is exact: no approximate vector index is built. - **Answer stage.** 15 answer setups at the embedder and reranker size chosen on half A (section 14.1): 12 crossed setups (chunk size 3 x reranker on or off x hybrid on or off), each with 10 chunks of evidence; and 3 token-matched setups `c512-t20k`, `c1024-t20k`, `c2048-t20k` (reranker on, hybrid on), each with evidence cut at 20,480 real Qwen3 tokens. The `c2048` x 10 crossed cell keeps its 10 chunks. Each answer setup runs 3 times on all 1,160 questions. -- **Token cut.** Chunks are taken in rank order and their Qwen3 token counts are summed. A chunk is included when the total before it is below 20,480. The chunk that crosses the limit is truncated at the limit, and no later chunk is included. Truncation is recorded per question. +- **Token cut.** Chunks are taken in rank order and their Qwen3 token counts are summed. A chunk is included when the total before it is below 20,480. The chunk that crosses the limit is truncated at the limit, and no later chunk is included. Truncation is recorded per question. When the top 100 chunks hold fewer than 20,480 tokens, the setup keeps all 100 and its evidence is shorter. The realised evidence token total is recorded per question and setup (H8b reading in section 7). - **Other runs.** Two closed-book runs, one oracle run (section 13), and a length-controlled checker run (section 11.2). ## 4. Datasets @@ -92,9 +92,9 @@ Each null is a zero effect. Every test is two-sided. A directional contrast coun - **H4 (combining reranker and hybrid: simple effects).** Within scope S, reranker plus hybrid beats each alone. Row 5 (both on minus reranker only) and row 6 (both on minus hybrid only), directional. These are simple effects, not an interaction test; the interaction is reported descriptively with its interval. Null: each contrast is zero. Holds on a dataset when both rows count there; holds overall on at least 2 of 3 core datasets. - **H5 (reranker size).** Within scope S, a bigger reranker ranks better (reranker-on setups). Rows 7 (4B minus 0.6B) and 8 (8B minus 4B), directional. Holds as H1. - **H6 (chunk size, search).** Within scope S, chunk size changes retrieval. Rows 9 (1,024 minus 512) and 10 (2,048 minus 1,024), two-sided. Endpoint: `evidence_recall_20k` on FinanceBench, nDCG@10 on FreshStack. Null: each contrast is zero. Holds when the same row (9 or 10) counts with the same sign on FinanceBench and on FreshStack. BioASQ is a sanity check (most passages are shorter than 512 tokens, so chunk size barely changes its chunks); Legal is exploratory. FinanceBench sensitivity: nDCG@10 with first-page and with any-page credit; the report gives sign agreement per row. -- **H7 (embedder by chunk size).** Within scope S, a bigger embedder gains more from bigger chunks. Row 11: (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, with the H6 endpoints. Null: zero. Holds when row 11 counts on both FinanceBench and FreshStack. BioASQ is a sanity check, Legal exploratory. Expected to be underpowered (section 10). +- **H7 (embedder by chunk size).** Within scope S, a bigger embedder gains more from bigger chunks. Row 11: (8B at 2,048 minus 8B at 512) minus (0.6B at 2,048 minus 0.6B at 512), marginal over reranker and hybrid, directional, with the H6 endpoints. It is a difference of differences with coefficients +1, -1, -1, +1 on these four cells (section 9 point 1). Null: zero. Holds when row 11 counts on both FinanceBench and FreshStack. BioASQ is a sanity check, Legal exploratory. Expected to be underpowered (section 10). - **H8a (chunk size with context length, answers).** Within scope S, chunk size changes faithfulness and correctness in the 12 crossed setups, where context length grows with chunk size (long contexts can hide evidence: Liu et al. 2024). Rows 12 to 13 (faithfulness) and 14 to 15 (correctness): 1,024 minus 512 and 2,048 minus 1,024, marginal over reranker and hybrid, two-sided. Expected direction for faithfulness: falls as chunks grow (Wang et al. 2024); the report gives the sign. Null: each contrast is zero. Per outcome, holds when the same row counts with the same sign on FinanceBench and on FreshStack. Legal is exploratory. BioASQ chunks barely change with chunk size, so BioASQ rows 12 to 19 are sanity checks, like rows 9 to 11 (section 8). -- **H8b (chunk size at equal tokens, answers).** Within scope S, chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. Rows 16 to 17 (faithfulness) and 18 to 19 (correctness): `c1024-t20k` minus `c512-t20k` and `c2048-t20k` minus `c1024-t20k`, two-sided, at reranker on and hybrid on with the chosen sizes (so H8b is conditional on that cell). Holds as H8a (per outcome, the same row counts with the same sign on FinanceBench and on FreshStack). **Reading fixed now:** "chunk size itself changes X" is claimed only when H8b holds for outcome X. When H8a holds and H8b does not, the claim is "chunk size together with context length changes X". +- **H8b (chunk size at equal tokens, answers).** Within scope S, chunk size changes faithfulness and correctness when evidence is held at 20,480 tokens. Rows 16 to 17 (faithfulness) and 18 to 19 (correctness): `c1024-t20k` minus `c512-t20k` and `c2048-t20k` minus `c1024-t20k`, two-sided, at reranker on and hybrid on with the chosen sizes (so H8b is conditional on that cell). Holds as H8a (per outcome, the same row counts with the same sign on FinanceBench and on FreshStack). **Reading fixed now:** "chunk size itself changes X" is claimed only when H8b holds for outcome X and a registered sensitivity analysis agrees. Every question stays in H8b. The sensitivity analysis repeats the H8b row on the questions where all three `-t20k` setups reach at least 95% of 20,480 tokens (19,456), and agrees when that row's estimate has the same sign on FinanceBench and on FreshStack. The report gives the questions it keeps per dataset and the realised token totals per setup. When H8a holds and H8b does not, the claim is "chunk size together with context length changes X". **Error rates of the composite rule.** Holm within each dataset plus "at least 2 of 3 datasets" (H1 to H5) or "FinanceBench and FreshStack" (H6 to H8b) is conservative. Under the null on three independent datasets, a false "holds" for one directional contrast has probability about 3 x 0.025 squared, about 0.002, before Holm. Under the FinanceBench-and-FreshStack rule it is about 0.025 squared (about 0.0006) for a directional row, and 2 x 0.025 squared (about 0.001) for a two-sided row under the same-sign rule. Power falls the other way: at a per-dataset power of 0.6, the 2-of-3 rule has about 0.65; at 0.5, about 0.50. @@ -134,7 +134,7 @@ Contrasts are per question, marginal over every factor not named. Search rows us One clustered method gives every estimate, interval and p-value. There is no mixed model. -1. **Per-question contrast score.** For row k and question q, d_qk is the mean of question q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights). Answer scores are first averaged over the 3 runs. +1. **Per-question contrast score.** For row k and question q, d_qk is the mean of question q's scores over the cells at the first level minus the mean over the cells at the second level (equal cell weights). Answer scores are first averaged over the 3 runs. This recipe does not apply to row 11. For row 11, each of the four H7 corner cells is first averaged over reranker and hybrid, and d_q11 = (+1) e8b-c2048 (-1) e8b-c512 (-1) e0.6b-c2048 (+1) e0.6b-c512. Example: scores 0.8, 0.2, 0.3 and 0.7 for these cells in this order give d = 1.0; the general recipe would give 0.5. The known-answer fixture (RDY task 15.6) holds this case. 2. **Estimate.** The plain mean of d_qk over the questions that enter the row (section 12). 3. **FreshStack, BioASQ and Legal.** The decision test is defined by its closed form. Let n be the number of questions that enter the row and r_q = d_qk minus the estimate. Estimate: the mean of d_qk. Variance: (sum of r_q squared) x (n / (n minus 1)) squared / n squared. Reference distribution: t on n minus 1 degrees of freedom, for the p-value and the 95% interval. This is the Mancl and DeRouen (2001) bias-reduced sandwich for an intercept-only model with one row per cluster. It equals the paired t-test variance times n / (n minus 1), so it needs no convergence fallback. With complete data the estimate equals the cell-means contrast of the full-factorial model. **Implementation:** `statsmodels` GEE (Liang and Zeger 1986) of d_qk on an intercept (Gaussian family, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) gives the estimate and the variance. The p-value and the interval are always computed from the t reference above, whatever reference the library reports by default. A known-answer test (RDY task 15.6) checks that the library estimate and variance equal the closed form. If they do not, the closed form is used and the difference is reported. The closed form was checked against `statsmodels` 0.15.0 on 2026-11-14 on a seeded fixture of 30 questions: the GEE estimate and variance matched it to 1e-10. The library reports a normal reference for the p-value (`use_t=False`), so the code takes the t reference itself. The build-time test of RDY task 15.6 stays. 4. **FinanceBench (questions share documents).** The decision p-value comes from a document-level sign-flip permutation test. The statistic is the question-weighted mean of d_qk. Under the null, the sign of each document's sum of d_qk is flipped at random: 100,000 draws, seed 20261101, p = (1 + number of draws with an absolute statistic at least as large) / 100,001. The 95% interval inverts the same test (the set of shifts not rejected at 0.05, with the same draws). The test assumes each document's sum is symmetric about zero under the null. The interval also assumes a constant additive shift for every question (a location-shift model); the point estimate and the p-value do not need it. @@ -146,16 +146,16 @@ A non-significant result means no difference was detected at this sample size. T ## 10. Power, minimum detectable effect and smallest effect of interest -- **Smallest effect of interest (SESOI).** 0.02 nDCG@10 (OPEN item O2: the user confirms the value; Lakens 2017). Each row's interval is reported against plus and minus the SESOI. No other endpoint has a SESOI; their intervals are reported against the MDE. +- **Smallest effect of interest (SESOI).** 0.02 nDCG@10 (OPEN item O2: the user confirms the value before any pilot result is read; Lakens 2017). Each row's interval is reported against plus and minus the SESOI. No other endpoint has a SESOI; their intervals are reported against the MDE. - **Pilot.** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118), on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. The pilot builds at most 4 indexes per core dataset: the four H7 corner cells (`e0.6b-c512`, `e0.6b-c2048`, `e8b-c512`, `e8b-c2048`). Reranker and hybrid setups run on those indexes as the cell list needs. A contrast between adjacent levels (for example 4B minus 0.6B) uses the SD of the extreme-level contrast (8B minus 0.6B) as a stand-in, and the MDE table says so. Pilot indexes are built with the pinned models and settings and are not reused: the main run builds all 9 indexes after the freeze. The pilot gives the SD of per-question contrast scores per dataset and contrast type, and on FinanceBench the design effect of document clustering. Experiment 22 values are not used. - **Pilot cost.** The pilot cost is projected from the P2 timing, added to the estimate before the freeze and shown in the compute table (section 17). The user approves it before the pod starts. Registered fallback if the user declines: no pilot; the MDE table is filled as a range from assumed SDs of 0.15 to 0.25 for retrieval contrasts, labelled as assumed, and the report says H7 and Part 2 C2 are expected to be underpowered. -- **MDE.** MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha = 0.05 / family size (worst Holm rank: 0.05/19 on FinanceBench and FreshStack, 0.05/8 on BioASQ) and n divided by the design effect on FinanceBench. Answer-stage rows use the SD from named smoke and test-pod answer files (P6) and are labelled rough. +- **MDE.** MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha = 0.05 / family size (worst Holm rank: 0.05/19 on FinanceBench and FreshStack, 0.05/8 on BioASQ) and n divided by the design effect on FinanceBench. Row 11 uses the SD of its own d (section 9 point 1), so its MDE is on the difference-of-differences scale. Answer-stage rows use the SD from named smoke and test-pod answer files (P6) and are labelled rough. | Dataset | Row(s) | SD of d (source) | MDE | Below SESOI? | | --- | --- | --- | --- | --- | | (filled by P5 and P6 before the freeze) | | | | | -- **Known weak spots (reviewer estimates, to be replaced by the table).** H7 compares differences of differences of single cells; at plausible SDs its MDE on FreshStack is about 0.08 to 0.12 nDCG@10, so H7 is expected to be underpowered. Part 2 C2 on half B is expected to detect only large correctness differences. The report states both. +- **Known weak spots (reviewer estimates, to be replaced by the table).** H7 compares differences of differences of single cells; at plausible SDs its MDE on FreshStack is about 0.08 to 0.12 nDCG@10 on that scale (coefficients +1, -1, -1, +1), so H7 is expected to be underpowered. Part 2 C2 on half B is expected to detect only large correctness differences. The report states both. ## 11. Judges, checkers and human labels @@ -175,7 +175,7 @@ Gate decisions are computed and recorded, with a time, before any H8 contrast re **Undefined statistics fail closed.** A statistic is undefined when its sample lacks a class it needs: kappa when both label sets hold one and the same class (chance agreement 1); AUC when the human labels hold one class; the specificity difference when a chunk-size level holds no human-supported claim. The rule: 1. Undefined on the full labelled sample: the gate fails. For the differential-error rule, rows 12, 13, 16 and 17 become not testable. -2. A resample is never drawn again. Each undefined resample is counted, and the report gives the count. +2. Points 2 to 4 apply to every undefined resample, also when the full sample is defined. A resample is never drawn again. Each undefined resample is counted once, whichever chunk-size level or class it lacks, and the report gives the count. 3. More than 5% of resamples undefined (over 500 of 10,000): the gate fails as in point 1. 4. Otherwise an undefined kappa resample takes 0 and an undefined AUC resample takes 0.5 (the chance values). Both sit below their pass thresholds, so they can never help a gate pass. An undefined specificity-difference resample has no chance value, so it is left out of the interval; point 3 caps this at 5%. @@ -183,13 +183,13 @@ Why: a chance value makes an undefined resample count against a pass, and point - **Faithfulness gate, per checker.** Cohen's kappa (Sim and Wright 2005) of the checker against the human label: point estimate at least 0.60 and lower 95% bound at least 0.40. Sample: 450 claims from the crossed setups, 150 per chunk size, 50 per core dataset within each chunk size, at most one claim per answer, seed 20261116. The claims are drawn from answers to all questions (both halves), because H8a and H8b are tested on all questions. The correctness sample below stays on half B. The gate is pooled over the three core datasets because 150 claims per dataset are too few for a per-dataset gate; kappa per checker and per core dataset is reported with its interval as descriptive and decides nothing. "Can't tell" counts as not supported (the binary benchmark scale); kappa is also reported with "can't tell" items left out. If GLM-5.3 fails, rows 12, 13, 16 and 17 are not testable on every dataset. - **Differential-error rule.** "Unsupported" is the positive class. Specificity is the share of human-supported claims the checker calls supported. The rule uses GLM-5.3, the main checker; Clef-Flash specificity is reported only. If the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, H8a and H8b faithfulness (rows 12, 13, 16, 17) is exploratory and not testable. Sensitivity per chunk size is reported. -- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. The Ragas score is cut at 0.5 (fixed before labelling). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). +- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. If drop step 1 (section 17) removes Legal answer generation, the Legal answers are not drawn: the sample is 150 answers, the L1 total is 700, the L2 correctness subset is 45 and the L2 total is 210. Legal has no gate, so no confirmatory row changes. The Ragas score is cut at 0.5 (fixed before labelling). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). - **Length-controlled check (registered sensitivity, no gate).** 300 claims from `c2048` crossed answers (seed 20261119). GLM-5.3 and Clef-Flash judge each claim again against one 512-token window of each cited chunk: the chunk is split into consecutive 512-token windows, and the window that shares the most distinct lower-case claim words is kept (ties go to the earlier window). The report gives the verdict change rate with a Wilson interval. ### 11.3 Human labels - **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller (OPEN item O4: name, role and relation to the project; recorded in the run record, never in public files). L2 labels a 30% subset of each set (seeds 20261122 to 20261124, one per set; section 16). Kappa of L1 against L2 is reported per set with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. -- **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: 225. +- **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: 225. After drop step 1: correctness 150, L1 700, L2 210. - **Guidelines.** One written guideline per set (claim kinds, faithfulness, correctness, and Part 2 correctness), with worked examples drawn from outside every sample, fixed before the first label and released (P8). Faithfulness follows the LLM-AggreFact task definition (MiniCheck): a claim is supported if and only if the cited passages support it, judged as the checker sees it, on meaning not grammar. - **Blinding.** Labellers never see scores, verdicts, setup names or chunk sizes. Cited passage length can still hint at chunk size; the report says so. A training round on practice items comes first and is not evidence. - **Ethics.** The study has no human participants. It uses public datasets, and the labellers are members of the research team, not study subjects, so no ethics review is sought (O5, decided by the owner on 2026-11-14). This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation (section 20). @@ -303,7 +303,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti | 20261121 | Factorial smoke run and test pod: setups and 5 questions per dataset | | 20261122 | L2 subset, claim kinds (30 of 100) | | 20261123 | L2 subset, faithfulness (135 of 450) | -| 20261124 | L2 subset, correctness (60 of 200) | +| 20261124 | L2 subset, correctness (60 of 200; 45 of 150 after drop step 1) | | 20261125 | Faithfulness-gate kappa bootstrap (each checker, pooled and per dataset) | | 20261126 | Differential-error rule bootstrap (GLM-5.3 specificity difference) | | 20261127 | Correctness-gate kappa bootstrap | @@ -321,7 +321,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti - **Estimate**: about **$730** on an RTX 4090 at $0.89 per hour (estimate). It is the A3 estimate of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 extra pod hours, about $30 to $40, not timed: P2), the second closed-book run (about $3 hosted and about 1 pod hour, by analogy with the first) and the pilot. The pilot cost is not yet estimated: it is projected from the P2 timing, added here before the freeze and approved by the user before the pod starts (section 10). Search cost barely changes with corpus size, because the rerank pool stays 100. - **Ceiling**: **$975**, pods and hosted calls together (1.5 x the A2 estimate, about 1.34 x this one). At 3 times slower reranking than assumed, the total is about $950 (estimate), still under the ceiling with little margin. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. - **Checks**: before the freeze (test pod extrapolation), on the build pods after the search stage, and at each stage end while the replicate pods run. Each projection uses billing and measured rates only, never scores. -- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. +- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. Step 1 also removes the 50 Legal correctness labels (section 11.2: correctness sample 150, L1 700, L2 210). E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. - **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early only when the preflight fails, an execution-integrity gate (G1 to G3, section 15) fails, or a projection reaches the ceiling after the last drop step. A failed outcome-validation gate (faithfulness or correctness, section 11.2) does not stop the run: its rows stay in their families as "not testable" (p = 1), and the rest of the run continues. The classifier gate is decided before the freeze (section 11.1). - **Part 2 and E1** each get their own estimate and ceiling before their runs (P16). @@ -371,7 +371,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - Decisions per row and per hypothesis, estimates with unadjusted 95% intervals, SESOI and MDE lines, and the gated rows marked "not testable". - **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed. - Realised chunk-length distribution per chunk-size level and dataset. -- Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit; faithfulness with uncited factual segments as unsupported, beside the uncited-segment share. +- Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit; H8b on questions with at least 95% of the token budget in all three `-t20k` setups, with realised token totals per setup; faithfulness with uncited factual segments as unsupported, beside the uncited-segment share. - **Compute table**: GPU type, pod hours per stage including every pre-freeze pod (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, an energy estimate (GPU hours times board power; Strubell et al. 2019; Lacoste et al. 2019), `resent_calls`, error rows per stage and role, and every `out_of_credit` stop. - Returned model ID and provider per hosted call, with the count of differences from the requested values. - AI assistance: the plan was drafted with AI agents; the paper says so. @@ -395,7 +395,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists **OPEN (the user decides):** - O1. Decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 configuration hash. -- O2. The SESOI value, 0.02 nDCG@10. +- O2. The SESOI value, 0.02 nDCG@10. Answered and dated before any pilot result is read. - O3. Whether the optional blind relevance audit for H3 and C1 runs. - O4. L2's name, role and relation to the project. - O5. Decided by the owner on 2026-11-14: no ethics query (section 11.3). diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index e732c98c..1de138f4 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -217,6 +217,17 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | Pilot seed confirmed before the pilot | 23 (P9) | | OpenSpec only: half-B guard checks the hash of the current selection file; budget check uses 52,200 answers (15 setups x 3 runs x 1,160); pilot-declined path; labelling and closed-book run tasks moved out of the pre-freeze section; Holm family sizes are 19 / 19 / 8 (O6, decided by the owner on 2026-11-14) | none | +**CodeRabbit round 2 (PR 210, 2026-11-14).** Seven findings in six threads on commit 3bf3d876. Six are fixed before the freeze. One is rejected. + +| Change | Pre-registration section | +| --- | --- | +| Row 11 (H7) uses coefficients +1, -1, -1, +1 on the four corner cells after averaging over reranker and hybrid; the general mean-minus-mean recipe does not apply to it; known-answer fixture case (0.8, 0.2, 0.3, 0.7 gives 1.0); MDE on the same scale | 7, 9, 10 | +| Short evidence pools: all 100 chunks are kept when they hold fewer than 20,480 tokens, and the realised token total is recorded per question and setup. Every question stays in H8b. "Chunk size itself" also needs a registered sensitivity on questions where all three `-t20k` setups reach at least 19,456 tokens (95%), with the count kept per dataset | 3, 7, 21 | +| Every undefined resample is never drawn again and is counted once, whichever chunk-size level it lacks, also when the full sample is defined; the over-5% failure rule stays separate | 11.2 | +| Drop step 1 removes the 50 Legal correctness labels: correctness 150, L1 700, L2 210; Legal has no gate, so no confirmatory row changes | 11.2, 16, 17 | +| OpenSpec only: O2 (SESOI) is asked in readiness flow step 6.10b, before any pilot result is read | none | +| Rejected: move O5 before classifier-gate labelling. O5 was decided on 2026-11-14 (no ethics query), so the answer is already recorded before any label | none | + **Owner decisions (2026-11-14).** Three open items are closed before the freeze. O2, O3 and O4 stay open. | Item | Decision | Pre-registration section | diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index d434715e..be64e216 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -620,15 +620,15 @@ Written before the freeze; no measured run has started. A publication-grade meth - **Deviation policy (F15, F29; PR 20).** After the freeze, every change is a dated, numbered deviation. The report gives its reason, its time (before or after seeing the affected data) and its effect on each claim. Each goes into `plan.json` `amendments` and is uploaded to the OSF project. - **FreshStack full corpus (F1; PR 4).** All 9 FreshStack indexes are built on the full public LangChain corpus (49,514 documents) at a pinned revision (P1). The download needs the user's approval at build time. The 10,000-document subset of Experiments 9a and 22 is retired for Experiment 35, and the continuity queries, continuity documents and overlay runbook are not used. No result is compared with the FreshStack leaderboard unless the corpus and the metric are the same. alpha-nDCG@10 is a secondary FreshStack metric if the nugget labels allow it (P3). One full-corpus build is timed before the estimate is relied on (P2). -- **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. +- **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Row 11 (H7) is a difference of differences with coefficients +1, -1, -1, +1 on the four corner cells after averaging over reranker and hybrid; the mean-minus-mean recipe does not apply to it (PR 9 point 1). Every question stays in H8b; "chunk size itself" also needs the registered sensitivity on questions where all three `-t20k` setups reach at least 95% of 20,480 tokens (PR 7), and realised evidence token totals are recorded per question and setup. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. - **Analysis (F7, F10; PR 9).** No MixedLM. The per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells; the estimate is its plain mean. FreshStack, BioASQ and Legal: the closed-form test of PR 9 point 3 (mean of d_qk; Mancl and DeRouen bias-reduced variance, sum of squared residuals x (n / (n minus 1)) squared / n squared; t on n minus 1 degrees of freedom), implemented with `statsmodels` GEE on an intercept (independence working correlation, question clusters, `cov_type="bias_reduced"`) and checked against the closed form by the known-answer test of task 15.6. FinanceBench: document-level sign-flip permutation test, 100,000 draws, seed 20261101, interval by inverting the test. Registered sensitivity checks, never decisive: a paired permutation test by question (10,000 shuffles) on FreshStack and BioASQ, and the GEE with document clusters on FinanceBench; disagreements go in one table. No McNemar test and no equivalence test. The type I error simulation test now covers this method (task 15.6). - **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). - **Missing outcomes (F9; PR 12).** Every question stays in its dataset's denominator, and every exclusion is counted per setup and row. A short list scores missing ranks as not relevant. A failed answer call follows the resend rule (D3); an answer still missing at run close scores 0 for correctness and number match and has no factual claim. Faithfulness uses complete cases per row, with two sensitivity analyses (no-claim answers scored 1, and 0). A row still unjudged at run close is counted and scored 0 in a sensitivity analysis. -- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 225, with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. +- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. Every undefined resample is left out and counted once, whichever level lacks a supported claim; over 500 of 10,000 makes those rows not testable (PR 11.2). A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window of each cited chunk, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 225 (700 and 210 after drop step 1), with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. - **Closed-book runs (F13; PR 13).** Two single runs: own-knowledge prompt (floor) and the RAG prompt with an empty evidence block. Contamination sensitivity: rows 14, 15, 18 and 19 re-run on the questions both runs answer wrongly. - **Halves, order of work and selection (F14, F20, F21, F27; PR 4, 14).** FinanceBench is split by document. Every pre-freeze sample drawn after `halves.json` exists (the pilot, the RAGFlow checks) comes from half A; earlier samples are listed with their IDs and halves (PR 5). Order: build and seal the 9 indexes and pass G2; run the 72 search setups on all questions; score half A only; apply the selection rule and write `selection.json`; time-stamp the configuration hash on OSF (O1); only then score half B and start the replicate pods. The harness enforces it (task 16.17). "Sealed" applies to indexes only. Index cost = chunks times embedding dimension, summed over the three core datasets. Configuration hash = SHA-256 of the canonical JSON of the selected cell's resolved settings, with the `halves.json` SHA-256 as one field. RQ1 retrieval headline = the A3 cell on half B; the 72-setup table is descriptive. The search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, without replacement, ascending sort). - **Pilot and MDE (F6; PR 10).** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118) on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. SESOI 0.02 nDCG@10 (O2). Experiment 22 values are not used. -- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. +- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation, which also removes the 50 Legal correctness labels (correctness 150, L1 700, L2 210; no gate, so no confirmatory row changes); (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. - **Pins, release, determinism, existing data (F16, F18, F20, F25; PR 5, 15, 16, 19).** One pins table (P10 to P14). A release table per artefact and licence, and a re-judge kit. One build per cell is assumed; an optional `e8b-c2048` rebuild check on FinanceBench runs only when the cost projection after the search stage is at or below the estimate, and is reported, not gated. The existing-data table lists every earlier look; P12 lists each smoke and pre-freeze question ID with its half. Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O7 (O1, O5, O6 and O7 decided by the owner on 2026-11-14; O2, O3 and O4 open) are group 16. diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md index 7ffddac7..223d82f5 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-answer-scoring/spec.md @@ -263,11 +263,20 @@ The agreement step SHALL give each checker's specificity (the share of human-sup - **WHEN** only the Clef-Flash specificity difference excludes zero - **THEN** it MUST be reported only and MUST NOT change any row -#### Scenario: Specificity difference undefined +#### Scenario: Specificity difference undefined on the full sample -- **WHEN** a chunk-size level of the full sample, or of more than 500 of the 10,000 resamples, holds no human-supported claim +- **WHEN** the 512 or the 2,048 level of the full labelled sample holds no human-supported claim +- **THEN** rows 12, 13, 16 and 17 MUST be "not testable" + +#### Scenario: Every undefined difference resample is left out and counted + +- **WHEN** a resample holds no human-supported claim at the 512 level, at the 2,048 level or at both, whether or not the full sample is defined +- **THEN** the step MUST NOT draw it again, MUST leave it out of the interval and MUST count it once + +#### Scenario: Too many undefined difference resamples + +- **WHEN** more than 500 of the 10,000 resamples are undefined, each counted once whichever level lacks a human-supported claim - **THEN** rows 12, 13, 16 and 17 MUST be "not testable" -- **AND** a resample with no human-supported claim at a level MUST NOT be drawn again, MUST be left out of the interval and MUST be counted #### Scenario: Gate decided before H8a and H8b results diff --git a/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md b/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md index 55022efa..99c4b1a3 100644 --- a/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md +++ b/openspec/changes/exp35-main-run-readiness/specs/benchmark-harness/spec.md @@ -251,6 +251,11 @@ The answer stage SHALL run 15 setups: the 12 crossed setups on 10 chunks each an - **WHEN** the chosen embedder and reranker retrieve short 2,048-token chunks - **THEN** `c2048` x 10 MUST send exactly 10 chunks, and `c2048-t20k` MUST send chunks until the 20,480-token limit, truncating the chunk that crosses it and recording the truncation +#### Scenario: Short evidence pool + +- **WHEN** the top 100 chunks of a `-t20k` setup hold fewer than 20,480 Qwen3 tokens +- **THEN** the setup MUST keep all 100 chunks and MUST record the realised evidence token total for that question and setup + ### Requirement: FreshStack uses the full public corpus at a pinned revision The FreshStack loader SHALL load the full public `freshstack/corpus-oct-2024` corpus, config `langchain`, at the revision pinned in `plan.json`, and the 203 public queries. It SHALL refuse a corpus whose revision or row count differs from the pin. It SHALL NOT sample a subset, add distractors, load the 20 continuity queries or the continuity documents, or append the overlay runbook. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index 14f64986..7356ce87 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -425,8 +425,8 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 14.2 Write failing tests that each answer row records `completion_calls`, the evidence characters, and the truncated passages and characters dropped, computed with OMRS's own `_fit_and_measure` on the same rows and settings. Prove they fail, then implement in `answers35.py`. Verify: the tests pass, and a row with cut evidence is still saved. - [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score, both chosen sizes, the A3 cell and the configuration hash. Then (A4, pre-registration section 14.1) tests that: index cost is the number of chunks times the embedding dimension, summed over the three core datasets, read from the sealed indexes with no score; the configuration hash is the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings with the SHA-256 of `halves.json` as one field, and the same inputs give the same hash twice; the any-page sensitivity repeats step 3 with any-page nDCG@10 on FinanceBench, records whether the cell changes, and never changes the registered cell. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. - [ ] 14.5 Ragas `AnswerCorrectness` (design D33). Write failing tests with a stub LLM and stub embeddings: the `ragas` stage scores five metrics, `AnswerCorrectness` gets the question, the answer and the dataset's reference answer (never the contexts), uses the frozen judge and the local 4B embedder, and records the weights; an answer without a reference records "not judged" and sends no call; resume skips saved rows. Prove they fail, then implement in `ragas35.py` with Ragas 0.4.3. Verify: the tests pass, and the version check still refuses another Ragas version. -- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B, each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). -- [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome; a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). +- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B (150 with no Legal draw when drop step 1 applies, PR 11.2 and 17), each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). +- [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome; a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups, with the PR 7 sensitivity on questions at 19,456 tokens or more in all three; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). ## 15. Methods-review fixes (pre-freeze amendment A2, 2026-11-13; design D35) @@ -447,19 +447,19 @@ Each test is proved to fail on the current code before the fix. - [ ] 15.4 Closed-book stages (M6; A4, pre-registration section 13). Write failing tests: `--stage closedbook` runs two prompts, each once on all 1,160 questions: the own-knowledge prompt (floor) and the main RAG answer prompt with an empty evidence block; neither sends a search call; both use the frozen answer settings, write one row per question for all four datasets, run `AnswerCorrectness` and the number match, and resume without repeat calls. Then report tests: closed-book correctness per dataset beside every correctness table; the per-question cross-table on retrieval misses; and the registered contamination sensitivity, which re-runs rows 14, 15, 18 and 19 on the questions that both closed-book runs answer wrongly (Ragas correctness below 0.5); when the drop order (PR 17) has dropped the RAG-prompt run, the subset uses the own-knowledge run alone and the table is labelled "one closed-book run". Prove they fail, then implement. Verify: the tests pass, the fallback case included. - [ ] 15.5 Split halves (M3; A4, pre-registration section 4). Write failing tests: `split35.py` with seed 20261115 gives the same halves twice; FinanceBench is split by document (documents shuffled with the seed, each going with all its questions to the half that has fewer questions so far, ties to A), so no document sits in both halves; FreshStack, BioASQ and Legal are split by question (shuffled, first half rounded up is half A); `halves.json` lists every question ID with its half; `select35.py` reads half A only (a planted half-B effect does not change the choice). Prove they fail, then implement. Verify: the tests pass. - [ ] 15.6 Clustered analysis (A4 replaces M2 and A2c N4; pre-registration section 9). No MixedLM. Write failing tests for `experiments/_lib/stats_cluster.py` and `factorial35.py`: - - the per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells (equal cell weights), with answer scores averaged over the 3 runs first; the estimate is the plain mean of d_qk; on FreshStack, BioASQ and Legal the decision test is the closed form of PR 9 point 3: variance = (sum of squared residuals) x (n / (n minus 1)) squared / n squared, and the p-value and 95% interval use t on n minus 1 degrees of freedom, computed by our code whatever reference the library reports by default (`statsmodels` GEE results use a normal reference unless told otherwise); the implementation is `statsmodels` `GEE` of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`); a known-answer test on a fixed fixture checks that the GEE estimate and variance equal the closed form and that the closed form equals the paired t-test variance times n / (n minus 1); if the library does not match, the code uses the closed form and the report states the difference; + - the per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells (equal cell weights), with answer scores averaged over the 3 runs first; row 11 instead uses coefficients +1, -1, -1, +1 on its four corner cells (PR 9 point 1), and a known-answer case (0.8, 0.2, 0.3, 0.7 gives 1.0) checks it; the estimate is the plain mean of d_qk; on FreshStack, BioASQ and Legal the decision test is the closed form of PR 9 point 3: variance = (sum of squared residuals) x (n / (n minus 1)) squared / n squared, and the p-value and 95% interval use t on n minus 1 degrees of freedom, computed by our code whatever reference the library reports by default (`statsmodels` GEE results use a normal reference unless told otherwise); the implementation is `statsmodels` `GEE` of d_qk on an intercept (Gaussian, identity link, independence working correlation, clusters = questions, `cov_type="bias_reduced"`); a known-answer test on a fixed fixture checks that the GEE estimate and variance equal the closed form and that the closed form equals the paired t-test variance times n / (n minus 1); if the library does not match, the code uses the closed form and the report states the difference; - FinanceBench uses the document-level sign-flip permutation test: the statistic is the question-weighted mean d_qk, each document's sum is flipped at random, 100,000 draws with seed 20261101, p = (1 + draws at least as large in absolute value) / 100,001, and the 95% interval inverts the same test on the same draws; - registered sensitivity checks, never decisive: a paired permutation test by question on FreshStack and BioASQ (10,000 shuffles, seed 20261101, `paired_permutation`), and the GEE above with document clusters on FinanceBench; every row where decision and sensitivity disagree on "counts" goes into one table; - type I error simulation, now for this method: with no planted effect and unequal error variance across cells, the rejection rate at 0.05 stays near 0.05 over 200 simulations for the GEE (FreshStack-sized data) and for the sign-flip test (84 documents, about 1.8 questions each); a fixture with complete data gives the same estimate as the cell-means contrast of the full-factorial model. - The `stats` dependency group exists (decided by the operator, 2026-11-13; added on 2026-11-14 with `statsmodels` 0.15.0 in `pyproject.toml` and `uv.lock`, and `{include-group = "stats"}` in the `dev` group). The wrapper and its tests are not built yet, so this box stays open. A test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. - [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the L2 subset is 30% of each set with seeds 20261122, 20261123 and 20261124 (225 in all); no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the L2 subset is 30% of each set with seeds 20261122, 20261123 and 20261124 (225 in all); no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.9 Hosted-call provenance (M7). Write failing tests: each answer, Ragas, GLM-5.3 and classifier row records `model_returned` and `provider_returned` from the stubbed reply; the join counts rows whose returned model differs from the requested one per role. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.10 Reranker rank check (M8). Write failing tests for `rerank_check35.py`: 200 pairs drawn with seed 20261117 are the same twice; Spearman below 0.95 marks the file "not pinned". Prove they fail, then implement. Run it on the test pod for each size (task 9.2 step 2a). Verify: the tests pass and `protocol.md` lists the correlation per file. - [ ] 15.11 MDE from the pilot (M1; A4, pre-registration section 10). Write `mde35.py` on saved files only: per core dataset and contrast type, the SD of the per-question contrast scores from the retrieval-only pilot (task 16.5, P5; an adjacent-level contrast uses the extreme-level SD as a stand-in, marked so), and on FinanceBench the design effect of document clustering; MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha 0.05/19 on FinanceBench and FreshStack and 0.05/8 on BioASQ, and n divided by the design effect on FinanceBench; answer-stage rows use the SD from the smoke and test-pod answer files named in task 16.6 (P6) and are labelled rough; each row is shown against the SESOI (0.02 nDCG@10, open item O2). Experiment 22 values are not used. Write failing tests on a fixture first. Verify: the tests pass and the MDE table of pre-registration section 10 is filled before the freeze. - [ ] 15.12 Oracle contexts (design D36; A2c caps). Write failing tests for `oracle35.py` on fixtures of all four datasets: every oracle context is capped at 20,480 Qwen3 tokens with truncation recorded (FinanceBench and BioASQ too); Legal gives the one gold passage; FinanceBench gives the gold page text(s) from the read PDF pages; BioASQ gives the gold corpus passages by relevant passage id; FreshStack gives the gold documents in qrels order, capped at 20,480 Qwen3 tokens, and records the truncated documents and tokens dropped; a missing gold unit stops with a setup error naming it; no search call is sent. Prove they fail, then implement. - [ ] 15.13 Oracle stage. Write failing tests: `--stage oracle` runs once (run 1), uses the frozen answer prompt and settings with the oracle evidence, and feeds the judge, `ragas` (`AnswerCorrectness`) and number-match stages; the report puts closed-book (floor) and oracle (ceiling) beside the RAG setups, marked descriptive with the no-distractor caveat. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.14 Equal evidence by real tokens (A2c N2; A4 page credit). Write failing tests for `evidence_by_tokens`: chunks are taken in rank order while the running Qwen3 token total before a chunk is below 20,480; the chunk that crosses is truncated at 20,480 tokens, the truncation is recorded per question, and no later chunk is taken; a list of short chunks fills to the budget instead of stopping at a fixed count; `evidence_recall_20k` uses this cut with any-page credit on FinanceBench (a whole chunk covers every gold page it overlaps; the truncated chunk covers only its first page) and reports first-page credit as a sensitivity column; the token counts come from the pinned tokenizer. Prove they fail on the chunk-count rule, then implement. Verify: the tests pass. +- [ ] 15.14 Equal evidence by real tokens (A2c N2; A4 page credit). Write failing tests for `evidence_by_tokens`: chunks are taken in rank order while the running Qwen3 token total before a chunk is below 20,480; the chunk that crosses is truncated at 20,480 tokens, the truncation is recorded per question, and no later chunk is taken; when the top 100 chunks hold fewer than 20,480 tokens all 100 are kept, and the realised token total is recorded per question and setup; a list of short chunks fills to the budget instead of stopping at a fixed count; `evidence_recall_20k` uses this cut with any-page credit on FinanceBench (a whole chunk covers every gold page it overlaps; the truncated chunk covers only its first page) and reports first-page credit as a sensitivity column; the token counts come from the pinned tokenizer. Prove they fail on the chunk-count rule, then implement. Verify: the tests pass. - [ ] 15.15 One round for closed-book and oracle (A2c). Extend the task 14.1 test: the largest oracle context of each dataset and every closed-book prompt synthesise in exactly one completion call. Prove it fails before the caps of task 15.12, then pass it. Verify: the test passes. - [ ] 15.16 Floor, ceiling and the miss cross-table (A2c). Write failing report tests: floor and ceiling rows are marked single-run beside 3-run averages; a per-question cross-table on retrieval misses counts closed-book correct or not against RAG correct or not, per dataset. Prove they fail, then implement. Verify: the tests pass. @@ -484,7 +484,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.15 P15. Check that no file still names `openai/gpt-6.1-terra` as a cross-check judge (A4 removed it from `plan.json`). This change never planned it. The `experiment-35-full-document-rag-benchmark` design marks its cross-check text as removed by A4. Verify: a search for `gpt-6.1-terra` finds it only in history text. - [ ] 16.16 P16. Part 2 and E1 inputs (readiness flow group 7): own estimates and ceilings for Part 2 and for E1; RAGFlow defaults read from the pinned release, not main-branch documentation; the three-column settings table; the matched RAGFlow cell list fixed before FLOW step 7.2; the cost measured at FLOW step 7.2 and the decision at FLOW step 7.2a (at most $50 projected pod cost), recorded before any half-B score exists. Verify: the Part 2 protocol holds each item and FLOW steps 7.1, 7.2 and 7.2a are done. - [ ] 16.17 Half-B guard (PR 14.2). Write failing tests: the scorer refuses any half-B score while `selection.json` or the OSF time-stamp record (upload time and file hash, task 16.23) is missing, and names both; it also refuses when no time-stamp record holds the configuration hash of the current `selection.json` (a record of an earlier selection), and names both hashes; the replicate pods' first stage refuses to start under the same condition; half-A scoring needs neither. Prove they fail, then implement. Verify: the tests pass. -- [ ] 16.18 Faithfulness gate and differential-error rule (PR 11.2). Write failing tests for `labels35.py`: per checker, kappa against the human label on the 450-claim sample (drawn from answers to all questions, both halves), with "can't tell" as not supported (also reported with those items left out), passes at a point estimate of at least 0.60 and a lower bound of at least 0.40 (percentile bootstrap by question, seed 20261125); a failed Clef-Flash gate marks its results not validated and changes no row; specificity per chunk size, and when the 95% interval of the GLM-5.3 2,048 minus 512 difference (seed 20261126) excludes zero in either direction, rows 12, 13, 16 and 17 become "not testable" (p = 1, task 15.7); sensitivity per chunk size is reported; undefined statistics fail closed (PR 11.2: an undefined difference resample is left out and counted, and over 5% undefined makes rows 12, 13, 16 and 17 not testable); the gate file records its time and the analysis refuses to compute an H8a or H8b row before it exists. Prove they fail, then implement. Verify: the tests pass. +- [ ] 16.18 Faithfulness gate and differential-error rule (PR 11.2). Write failing tests for `labels35.py`: per checker, kappa against the human label on the 450-claim sample (drawn from answers to all questions, both halves), with "can't tell" as not supported (also reported with those items left out), passes at a point estimate of at least 0.60 and a lower bound of at least 0.40 (percentile bootstrap by question, seed 20261125); a failed Clef-Flash gate marks its results not validated and changes no row; specificity per chunk size, and when the 95% interval of the GLM-5.3 2,048 minus 512 difference (seed 20261126) excludes zero in either direction, rows 12, 13, 16 and 17 become "not testable" (p = 1, task 15.7); sensitivity per chunk size is reported; undefined statistics fail closed (PR 11.2: every undefined difference resample is left out and counted once, whichever level lacks a supported claim, and over 5% undefined makes rows 12, 13, 16 and 17 not testable); the gate file records its time and the analysis refuses to compute an H8a or H8b row before it exists. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.19 Length-controlled check (PR 11.2). Write failing tests for a `lengthcheck` stage: 300 claims from `c2048` crossed answers with seed 20261119; each cited chunk split into consecutive 512-token windows, the window that shares the most distinct lower-case claim words kept (ties to the earlier window); GLM-5.3 and Clef-Flash judge the claim again; the report gives the verdict change rate with a Wilson interval; no gate. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.20 Report items (PR 21). Write failing report tests: the realised chunk-length distribution per chunk-size level and dataset; one sensitivity table set (clustered method against sensitivity check, FinanceBench page rules, missing-outcome bounds, closed-book subset, length-controlled check, A3 cell under any-page credit); the compute table (GPU type, pod hours per stage with every pre-freeze pod, hosted tokens and cost per role, energy estimate, `resent_calls`, error rows per stage and role, every `out_of_credit` stop); the returned model ID and provider per hosted call with the count of differences; every row with SESOI and MDE lines and "not testable" marks. Prove they fail, then implement. Verify: the tests pass. - [ ] 16.21 Release kit (PR 19). Write failing tests for a re-judge script that rebuilds evidence text from the pinned dataset revisions and the released unit and chunk IDs, on a fixture; and for an export that writes ranked lists in TREC run format, never corpus text. Prove they fail, then implement. Verify: the tests pass, and the release table of PR 19 names each artefact and its licence. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md index a4efbe38..f8da804c 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md @@ -483,7 +483,7 @@ Decision (operator, 2026-11-14). A publication-grade methods review gave 30 find - F1: all 9 FreshStack indexes use the full public LangChain corpus (49,514 documents, pinned revision). The 10,000-document subset is retired. alpha-nDCG@10 is secondary if the nugget labels allow it. - F4, F7, F10: no mixed model. Per-question contrast scores. On FreshStack, BioASQ and Legal the decision test is defined by its closed form (mean; Mancl and DeRouen bias-reduced variance; t on n minus 1), with `statsmodels` GEE as the implementation checked against it (`preregistration.md` section 9 point 3); and a document-level sign-flip test on FinanceBench. Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; second round C3). A gated row stays in its family with p = 1. No equivalence claim, no TOST, no McNemar test as a main test. - F5, F22, F23: H8 splits into H8a (crossed setups) and H8b (token-matched setups). Both are confirmatory. Every hypothesis has scope S, numbered rows and a holds rule. H7 is confirmatory on FinanceBench and FreshStack. After the second round (C3, C4), H6, H7, H8a and H8b hold only on FinanceBench and FreshStack, and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs. Legal RAG Bench is exploratory in every hypothesis. - - F3, F8, F19: L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 225. Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. + - F3, F8, F19 (counts after drop step 1: L1 700, L2 210; second CodeRabbit round): L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 225. Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. - F12: `evidence_recall_20k` gives any-page credit and is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11). nDCG@10 stays the primary metric of the study. First-page credit is a sensitivity. - F13: a second closed-book run uses the RAG prompt with empty evidence. - F14, F21: FinanceBench is split by document. Order: score half A, write `selection.json`, time-stamp the configuration hash on OSF, then score half B. The RQ1 retrieval headline is the A3 cell on half B. "Sealed" applies to indexes only. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md index ecea4ae7..2682f78c 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/document-rag-evaluation/spec.md @@ -87,6 +87,14 @@ The protocol SHALL register H1 to H7, H8a and H8b before the run, each within sc - **WHEN** H8a holds for an outcome and H8b does not - **THEN** the report SHALL claim only that chunk size together with context length changes that outcome +#### Scenario: Chunk size itself +- **WHEN** H8b holds for an outcome +- **THEN** the report SHALL claim that chunk size itself changes it only when the registered sensitivity, on the questions where all three `-t20k` setups reach at least 19,456 tokens, gives the same sign on FinanceBench and FreshStack, and SHALL give the questions kept per dataset + +#### Scenario: H7 contrast scale +- **WHEN** row 11 is computed +- **THEN** each question's score SHALL use coefficients +1, -1, -1, +1 on e8b-c2048, e8b-c512, e0.6b-c2048 and e0.6b-c512 after averaging over reranker and hybrid, not the mean-minus-mean recipe + #### Scenario: FinanceBench interval - **WHEN** a decision p-value or 95% interval is computed on FinanceBench - **THEN** it SHALL come from the document-level sign-flip permutation test, so questions that share a document move together @@ -179,6 +187,10 @@ Every answer setup SHALL use `answer.context_window` 131,072 characters and `ans The Ragas stage SHALL score every answer with Ragas 0.4.3 `AnswerCorrectness` against the reference answer, and the judge SHALL never see the retrieved contexts. L1 SHALL label 50 half-B questions per core dataset (seed 20261113), each with one answer from run 1 of an answer setup the seed picks uniformly at random, and 50 Legal RAG Bench answers drawn the same way and reported without a gate. +#### Scenario: Legal answers dropped +- **WHEN** drop step 1 removes Legal RAG Bench answer generation +- **THEN** no Legal answer SHALL be drawn, the correctness sample SHALL be 150 and no confirmatory row SHALL change + #### Scenario: Gate passes on a dataset - **WHEN** Cohen's kappa between the score cut at 0.5 and the human label has a point estimate of at least 0.60 and a lower 95% bound of at least 0.40, with "can't tell" items scored both ways, and the AUC of the continuous score has a lower 95% bound above 0.5 - **THEN** rows 14, 15, 18 and 19 SHALL be testable on that dataset diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index ee42002f..f260c272 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -153,7 +153,7 @@ Each faithfulness checker SHALL pass when Cohen's kappa against the human label ### Requirement: Human labels follow the registered counts -L1 SHALL label 750 items (claim kinds 100, faithfulness 450 from all questions, correctness 200 from half B), and L2 SHALL label a 30% subset of each set (225 items). The claim classifier SHALL reach agreement of at least 0.90 and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. +L1 SHALL label 750 items (claim kinds 100, faithfulness 450 from all questions, correctness 200 from half B), and L2 SHALL label a 30% subset of each set (225 items); after drop step 1, correctness 150, L1 700 and L2 210. The claim classifier SHALL reach agreement of at least 0.90 and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. #### Scenario: A label file is released - **WHEN** human labels are released diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 41e8ec69..0516ea7a 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -86,9 +86,9 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35 ("PR n" means its section n). "RDY 16.N" means task 16.N in `exp35-main-run-readiness/tasks.md`. Steps 6.10a to 6.10d run after step 6.10 and before the freeze. - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. -- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. +- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.24 (O2): ask the user to confirm or change the SESOI and record the answer and its date in PR 10, before any pilot result is read and before the MDE table is filled (with the pilot or the fallback). Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the O2 answer is dated before any pilot result is read, the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. - [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. O2, O3 and O4 stay open. Verify: PR 23 shows each item decided. +- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed; asked in step 6.10b before any pilot result is read); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. O2, O3 and O4 stay open. Verify: PR 23 shows each item decided. ### F. Freeze From 6ea256c5273f61be1ca46df0b766dba1edd386a8 Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 19:06:41 +0000 Subject: [PATCH 6/9] docs(exp35): record owner decisions O2, O3 and O4 O2: SESOI 0.02 nDCG@10 with a written justification in preregistration section 10. O3: no relevance audit by default; a registered unjudged-share trigger decides. O4: L2 labels every correctness answer and every Part 2 label, and 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1). Seed 20261124 retired. No owner decision is open. --- docs/proposals/first-publication-benchmark.md | 8 ++--- .../plan.json | 2 +- .../preregistration.md | 35 ++++++++++--------- .../protocol.md | 13 ++++--- .../exp35-main-run-readiness/design.md | 16 ++++----- .../exp35-main-run-readiness/proposal.md | 2 +- .../changes/exp35-main-run-readiness/tasks.md | 14 ++++---- .../design.md | 8 ++--- .../proposal.md | 2 +- .../specs/evaluation-observability/spec.md | 2 +- .../tasks.md | 8 ++--- .../rag-benchmark-readiness-flow/design.md | 2 +- .../rag-benchmark-readiness-flow/proposal.md | 2 +- .../rag-benchmark-readiness-flow/tasks.md | 4 +-- 14 files changed, 61 insertions(+), 57 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index 218617ee..666c46a2 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -55,7 +55,7 @@ The claims stay testable. OMRS reaches stated scores under stated conditions. Na ## Order and gates -1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The open items O2, O3 and O4 and the pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The owner decided O1, O5, O6 and O7 on 2026-11-14. The Part 2 protocol exists before this point (it is not written yet). +1. Freeze Part 1. The operator approves the pre-registration, `plan.json` and the run budget (master flow gate G7). The pre-freeze tasks P1 to P16 (pre-registration section 23) close first. The owner decided O1 to O7 on 2026-11-14, so no owner decision is open. The Part 2 protocol exists before this point (it is not written yet). 2. Deposit on OSF in one deposit: the pre-registration as main file, this proposal, the protocol history, `plan.json` and the Part 2 protocol (C1, C2). Record the link and the commit hash. 3. Build and seal the 9 indexes per dataset. The rebuild gate G2 passes before any search writes output. Then run the 72 search setups on all questions (pre-registration section 14.2). 4. Score half A only. Apply the A3 rule. Write `selection.json`. @@ -108,7 +108,7 @@ The answer stage uses the embedder and reranker size chosen on half A (pre-regis - Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; O7, decided by the owner on 2026-11-14: the primary measure stays cited factual claims, because OMRS answers are built to cite their evidence). - Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. -- Human check: two labellers score samples blind (L1 750 labels, L2 225; L1 700 and L2 210 if drop step 1 removes Legal answers). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). +- Human check: two labellers score samples blind (L1 750 labels, L2 365; L1 700 and L2 315 if drop step 1 removes Legal answers). L2 labels every correctness answer and every Part 2 label, and 30% of the faithfulness and claim-kind sets (O4). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. ### Hypotheses H1 to H8b @@ -133,7 +133,7 @@ Detail for each item is in the pre-registration section named. - Analysis: no mixed model. One clustered method gives every estimate, interval and p-value: GEE with bias-reduced covariance on question clusters for FreshStack, BioASQ and Legal, and a document-level sign-flip permutation test on FinanceBench (section 9). - Holm families are fixed now: 19 contrasts on FinanceBench, 19 on FreshStack, 8 on BioASQ (46 in all; BioASQ rows 9 to 19 are sanity checks); Legal is exploratory. No gate changes a family size (section 8). H1 to H5 hold on at least 2 of 3 core datasets. H6, H7, H8a and H8b hold only on FinanceBench and FreshStack. A two-sided hypothesis holds only when the same row counts with the same sign on every dataset its rule needs (section 7). - Missing outcomes: every question stays in its denominator, with a rule per outcome (section 12). -- Power: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the minimum detectable effect table before the freeze. It builds at most the four H7 corner indexes per core dataset and does not reuse them. Its cost is projected from the P2 timing and approved before the pod starts; if the user declines, the table is filled from assumed SDs and labelled so. The smallest effect of interest is 0.02 nDCG@10 (open item O2). No equivalence claim is made (section 10). +- Power: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the minimum detectable effect table before the freeze. It builds at most the four H7 corner indexes per core dataset and does not reuse them. Its cost is projected from the P2 timing and approved before the pod starts; if the user declines, the table is filled from assumed SDs and labelled so. The smallest effect of interest is 0.02 nDCG@10 (O2, decided by the owner on 2026-11-14, with a written justification in pre-registration section 10). No equivalence claim is made (section 10). - Every main-run pod uses one RunPod GPU type, chosen by a timing job before the freeze. The Mac scores the results. - Budget: estimate about $730 (an estimate, not measured), ceiling $975, pods and hosted calls together. The hosted-call key carries a credit limit, so an overrun stops cleanly. At the ceiling a results-blind drop order applies: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop. E1 has its own ceiling and is not in this order. A run that finishes every confirmatory item after drops is "complete with registered drops", each drop a numbered deviation. No confirmatory item is dropped by choice (section 17). - Pre-registration: the pre-registration is the main file of the OSF deposit at the freeze commit, with this proposal, the protocol history, `plan.json` and the Part 2 protocol. @@ -358,7 +358,7 @@ The Experiment 35 limitations are in pre-registration section 22. The paper repe 4. Ask the Napier library to confirm the article type and the deal. 5. Done: Experiment 35 amendment A3 (protocol section 0.12) holds the selection rule for Part 2. 6. Done: amendment A4 (protocol section 0.13) and the pre-registration, after the methods review of 13 November 2026. -7. Close the open items O2, O3 and O4 and the pre-freeze tasks P1 to P16 of pre-registration section 23. +7. Close the pre-freeze tasks P1 to P16 of pre-registration section 23. No owner decision is open (O1 to O7 decided on 2026-11-14). 8. Reserve an experiment number for Part 2. Write its protocol, covering every item of "What the Part 2 protocol must hold before its freeze". 9. Before the freeze, pin the RAGFlow release and container digests and connect it to the llama.cpp models. Read the RAGFlow defaults from the pinned release. Time one index per dataset (corpus only). Measure the projected pod cost of the matched search at master flow step 7.2 and record the decision at step 7.2a; it fixes the Part 2 estimand. Check page provenance on a seeded half-A FinanceBench sample (the rule 3 gate: if RAGFlow returns no page for a chunk, the primary unit falls back to the document for both systems) and retrieval repeatability on half A. Send the settings table to the RAGFlow maintainers. 10. Write the Part 2 estimate and ceiling, and the E1 estimate and ceiling (pre-freeze task P16; no figure exists yet). diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 7fdbc911..702a2f2b 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 225 (30%); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if open item O3 is yes. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 (open items O2, O3 and O4 also close before the freeze) and are not filled here. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 210; no confirmatory row changes).", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 365 (every correctness answer and every Part 2 label, 30% of the faithfulness and claim-kind sets; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2); O4, L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 315 after O4; no confirmatory row changes).", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index 9e1b6dec..6186cf09 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -146,7 +146,8 @@ A non-significant result means no difference was detected at this sample size. T ## 10. Power, minimum detectable effect and smallest effect of interest -- **Smallest effect of interest (SESOI).** 0.02 nDCG@10 (OPEN item O2: the user confirms the value before any pilot result is read; Lakens 2017). Each row's interval is reported against plus and minus the SESOI. No other endpoint has a SESOI; their intervals are reported against the MDE. +- **Smallest effect of interest (SESOI).** 0.02 nDCG@10, decided by the owner on 2026-11-14 and recorded on that date, before any pilot result was read (O2). The concept follows Lakens 2017. It changes no decision: every hypothesis is tested against zero, and each row's interval is read against plus and minus 0.02. No other endpoint has a SESOI; their intervals are reported against the MDE. +- **Why 0.02.** No IR standard fixes this value; it is the authors' justified choice. (1) User terms: for a question with one gold unit, moving it from rank 2 to rank 1 raises nDCG@10 by 1 minus 1/log2(3), about 0.369. A mean gain of 0.02 is about one question in 18 with its gold unit moved from second to first place (0.02 / 0.369 = 0.054); 0.01 is about one in 37, and 0.05 about one in 7. Bringing a gold unit from outside the top 10 to rank 10 adds 1/log2(11), about 0.289, so 0.02 is about one question in 14. (2) Lower bound from the design: selection treats setups within 0.01 as tied (section 14.1) and the tolerance on another back end is 0.005 (section 15), so the SESOI must be above 0.01. (3) Cost: each factor step costs something real. A 4B to 8B model roughly doubles memory, a reranker scores 100 more pairs per query, and hybrid search adds an index. The authors judge that a gain which changes the top result for fewer than about one question in 20 does not pay for such a step for an OMRS user. (4) Published effects (Zhang et al. 2025; MTEB retrieval scores are nDCG@10 x 100): on MTEB(eng, v2) retrieval the 0.6B, 4B and 8B embedders score 61.83, 68.46 and 69.44 (steps about 0.066 and 0.010). On MTEB-R, from a 61.82 first stage, the 0.6B, 4B and 8B rerankers score 65.80, 69.76 and 69.02 (gains about 0.040, 0.079 and 0.072; size steps 0.040 and minus 0.007). These effects span about 0.01 to 0.08 in size, and 0.02 separates the two smallest steps from the rest. (5) Not set by power: the design likely detects only about 0.05 on the smaller datasets, so 0.02 was not chosen to match what can be detected. A row whose interval cannot exclude 0.02 is reported as "cannot rule out a small effect". (6) Sensitivity: the report also reads every interval against 0.01 and 0.05. - **Pilot.** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118), on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. The pilot builds at most 4 indexes per core dataset: the four H7 corner cells (`e0.6b-c512`, `e0.6b-c2048`, `e8b-c512`, `e8b-c2048`). Reranker and hybrid setups run on those indexes as the cell list needs. A contrast between adjacent levels (for example 4B minus 0.6B) uses the SD of the extreme-level contrast (8B minus 0.6B) as a stand-in, and the MDE table says so. Pilot indexes are built with the pinned models and settings and are not reused: the main run builds all 9 indexes after the freeze. The pilot gives the SD of per-question contrast scores per dataset and contrast type, and on FinanceBench the design effect of document clustering. Experiment 22 values are not used. - **Pilot cost.** The pilot cost is projected from the P2 timing, added to the estimate before the freeze and shown in the compute table (section 17). The user approves it before the pod starts. Registered fallback if the user declines: no pilot; the MDE table is filled as a range from assumed SDs of 0.15 to 0.25 for retrieval contrasts, labelled as assumed, and the report says H7 and Part 2 C2 are expected to be underpowered. - **MDE.** MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha = 0.05 / family size (worst Holm rank: 0.05/19 on FinanceBench and FreshStack, 0.05/8 on BioASQ) and n divided by the design effect on FinanceBench. Row 11 uses the SD of its own d (section 9 point 1), so its MDE is on the difference-of-differences scale. Answer-stage rows use the SD from named smoke and test-pod answer files (P6) and are labelled rough. @@ -183,18 +184,18 @@ Why: a chance value makes an undefined resample count against a pass, and point - **Faithfulness gate, per checker.** Cohen's kappa (Sim and Wright 2005) of the checker against the human label: point estimate at least 0.60 and lower 95% bound at least 0.40. Sample: 450 claims from the crossed setups, 150 per chunk size, 50 per core dataset within each chunk size, at most one claim per answer, seed 20261116. The claims are drawn from answers to all questions (both halves), because H8a and H8b are tested on all questions. The correctness sample below stays on half B. The gate is pooled over the three core datasets because 150 claims per dataset are too few for a per-dataset gate; kappa per checker and per core dataset is reported with its interval as descriptive and decides nothing. "Can't tell" counts as not supported (the binary benchmark scale); kappa is also reported with "can't tell" items left out. If GLM-5.3 fails, rows 12, 13, 16 and 17 are not testable on every dataset. - **Differential-error rule.** "Unsupported" is the positive class. Specificity is the share of human-supported claims the checker calls supported. The rule uses GLM-5.3, the main checker; Clef-Flash specificity is reported only. If the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, H8a and H8b faithfulness (rows 12, 13, 16, 17) is exploratory and not testable. Sensitivity per chunk size is reported. -- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. If drop step 1 (section 17) removes Legal answer generation, the Legal answers are not drawn: the sample is 150 answers, the L1 total is 700, the L2 correctness subset is 45 and the L2 total is 210. Legal has no gate, so no confirmatory row changes. The Ragas score is cut at 0.5 (fixed before labelling). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). +- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. If drop step 1 (section 17) removes Legal answer generation, the Legal answers are not drawn: the sample is 150 answers, the L1 total is 700, L2 labels all 150 and the L2 total is 315. Legal has no gate, so no confirmatory row changes. The Ragas score is cut at 0.5 (fixed before labelling). The human label for this gate is the agreed L1 and L2 label (section 11.3). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). - **Length-controlled check (registered sensitivity, no gate).** 300 claims from `c2048` crossed answers (seed 20261119). GLM-5.3 and Clef-Flash judge each claim again against one 512-token window of each cited chunk: the chunk is split into consecutive 512-token windows, and the window that shares the most distinct lower-case claim words is kept (ties go to the earlier window). The report gives the verdict change rate with a Wilson interval. ### 11.3 Human labels -- **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller (OPEN item O4: name, role and relation to the project; recorded in the run record, never in public files). L2 labels a 30% subset of each set (seeds 20261122 to 20261124, one per set; section 16). Kappa of L1 against L2 is reported per set with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. -- **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: 225. After drop step 1: correctness 150, L1 700, L2 210. +- **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller, an author or colleague (O4, decided by the owner on 2026-11-14). L2's name, role and relation to the project go in the run record before the first label (P8), never in public files. L2 labels every correctness answer and every Part 2 label. Reason: each correctness gate is per dataset on 50 answers, and 30% of 50 is too few to estimate human agreement per dataset; in Part 2, L1 is an OMRS author. L2 labels a 30% subset of the faithfulness set (135 of 450, seed 20261123) and of the claim-kind set (30 of 100, seed 20261122). Reason: each has one pooled gate, so a sample estimates agreement. Where both labellers label every item, the reference label for the gate is the agreed label after the disagreement rule. Kappa of L1 against L2 is reported per set, and per core dataset for correctness, with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. +- **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: claim kinds 30, faithfulness 135, correctness 200; 365 in all. After drop step 1: correctness 150, L1 700, L2 315. Part 2 counts are set in the Part 2 protocol (P16), and L2 labels all of them. - **Guidelines.** One written guideline per set (claim kinds, faithfulness, correctness, and Part 2 correctness), with worked examples drawn from outside every sample, fixed before the first label and released (P8). Faithfulness follows the LLM-AggreFact task definition (MiniCheck): a claim is supported if and only if the cited passages support it, judged as the checker sees it, on meaning not grammar. - **Blinding.** Labellers never see scores, verdicts, setup names or chunk sizes. Cited passage length can still hint at chunk size; the report says so. A training round on practice items comes first and is not evidence. - **Ethics.** The study has no human participants. It uses public datasets, and the labellers are members of the research team, not study subjects, so no ethics review is sought (O5, decided by the owner on 2026-11-14). This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation (section 20). - **Conflict of interest.** The authors built OMRS. Safeguards: this pre-registration and its deviation log; selection on half A only; Holm families fixed now; blind labels with L2 as primary labeller in Part 2; published RAGFlow settings and maintainer correspondence; release of every output (section 19). The paper states the conflict. -- **Optional blind relevance audit** for H3 and C1 (about 300 to 400 L1 labels on unjudged top-3 units, seed 20261120): runs only if the user approves (OPEN item O3). It would be descriptive. +- **Blind relevance audit for H3 and C1 (O3, decided by the owner on 2026-11-14).** No audit by default. The audit runs only if this trigger, fixed now, fires: the mean unjudged share in the top 10 units differs by more than 5 percentage points (0.5 units per question) (a) between hybrid on and hybrid off on FreshStack, marginal over the other factors, or (b) between OMRS and RAGFlow on any core dataset in Part 2. The trigger uses the absolute gap, so it is blind to which side scores higher. Trigger (a) is computed on all questions after step 5 of section 14.2, trigger (b) after the Part 2 runs. A unit is unjudged when the qrels hold no judgement for it. Where P4 finds that a dataset's qrels hold positives only, "unjudged" means "not in the gold set". The share is then 1 minus precision@10, so a side that finds more gold units also has a smaller share. Why: an unjudged unit scores as not relevant, so the bias a setup can suffer is bounded by how many more unjudged units it retrieves; with equal shares both sides are treated alike. The 5-point value is a judgement. Rough bound: if a quarter of a 0.5-unit gap were truly relevant, one side would miss about 0.125 relevant units per question. Each such unit is worth 1/log2(rank + 1) over IDCG, on average about 0.45 over IDCG across ranks 1 to 10. So the bias can reach about 0.06 nDCG@10 on a question with one gold unit, about 0.02 with five and about 0.01 with ten. If the trigger fires, L1 labels about 300 to 400 unjudged top-3 units (seed 20261120); the audit is descriptive and changes no decision. The report gives both trigger values. ## 12. Missing outcomes @@ -299,11 +300,11 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti | 20261117 | Reranker rank check (200 pairs per size) | | 20261118 | MDE pilot questions (40 per core dataset, half A) | | 20261119 | Length-controlled check (300 claims) | -| 20261120 | Optional relevance audit (O3) | +| 20261120 | Relevance audit, only if its trigger fires (O3, section 11.3) | | 20261121 | Factorial smoke run and test pod: setups and 5 questions per dataset | | 20261122 | L2 subset, claim kinds (30 of 100) | | 20261123 | L2 subset, faithfulness (135 of 450) | -| 20261124 | L2 subset, correctness (60 of 200; 45 of 150 after drop step 1) | +| 20261124 | Retired, not reused: was the L2 correctness subset; L2 now labels every correctness answer (O4) | | 20261125 | Faithfulness-gate kappa bootstrap (each checker, pooled and per dataset) | | 20261126 | Differential-error rule bootstrap (GLM-5.3 specificity difference) | | 20261127 | Correctness-gate kappa bootstrap | @@ -321,7 +322,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti - **Estimate**: about **$730** on an RTX 4090 at $0.89 per hour (estimate). It is the A3 estimate of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 extra pod hours, about $30 to $40, not timed: P2), the second closed-book run (about $3 hosted and about 1 pod hour, by analogy with the first) and the pilot. The pilot cost is not yet estimated: it is projected from the P2 timing, added here before the freeze and approved by the user before the pod starts (section 10). Search cost barely changes with corpus size, because the rerank pool stays 100. - **Ceiling**: **$975**, pods and hosted calls together (1.5 x the A2 estimate, about 1.34 x this one). At 3 times slower reranking than assumed, the total is about $950 (estimate), still under the ceiling with little margin. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. - **Checks**: before the freeze (test pod extrapolation), on the build pods after the search stage, and at each stage end while the replicate pods run. Each projection uses billing and measured rates only, never scores. -- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. Step 1 also removes the 50 Legal correctness labels (section 11.2: correctness sample 150, L1 700, L2 210). E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. +- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. Step 1 also removes the 50 Legal correctness labels (section 11.2: correctness sample 150, L1 700, L2 315). E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. - **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early only when the preflight fails, an execution-integrity gate (G1 to G3, section 15) fails, or a projection reaches the ceiling after the last drop step. A failed outcome-validation gate (faithfulness or correctness, section 11.2) does not stop the run: its rows stay in their families as "not testable" (p = 1), and the rest of the run continues. The classifier gate is decided before the freeze (section 11.1). - **Part 2 and E1** each get their own estimate and ceiling before their runs (P16). @@ -342,7 +343,7 @@ Experiment 35 fixes the points below. On these points this section wins over the - **FinanceBench unit**: page for every question and both systems; pre-freeze gate that RAGFlow returns page numbers; fallback to document for every question and both systems if it fails; the other unit as labelled sensitivity (`protocol.md` section 0.12 item 4). - **C1 and C2**: one Holm family of 2 per core dataset, Legal exploratory. Each holds when it counts with the same sign on at least 2 of the 3 core datasets (the 2-of-3 rule of section 7). A count with the opposite sign on the third dataset is reported as a dataset-dependent effect and does not stop the hypothesis holding. The report states the direction. C1 is nDCG@10. C2 is continuous `AnswerCorrectness`; number match is secondary. Both use the analysis of section 9 (GEE on question clusters for FreshStack and BioASQ, document-level sign-flip on FinanceBench). No McNemar test where documents cluster (Obuchowski 1998). C2 needs its own Part 2 correctness gate (the section 11.2 rule, labels from both systems); a failed gate leaves C2 in the family as not testable. - **Part 2 missing outcomes**: every half-B question stays in both systems' denominators. A RAGFlow timeout or error after the content-blind retry rule scores 0 for C1 and gets an answer from empty evidence for C2. A short list scores missing ranks as not relevant. A document RAGFlow fails to parse stays in the corpus count and is reported with the questions it affects. An unmappable chunk counts as not relevant and is counted. A sensitivity analysis uses the questions where both systems indexed every gold document. The section 12 rules apply to both systems alike. An answer still missing after the resend rule, from either system, scores 0 for correctness. A question enters the C2 contrast only when both systems' Ragas rows are judged (complete pairs); every exclusion is counted per system, and a sensitivity analysis scores unjudged rows 0 and keeps every question. -- **Labels**: L2 is the primary Part 2 labeller; system identity is hidden; both systems' answers are shuffled into one list (seed 20261132). The Part 2 correctness-gate sample and its bootstraps use seeds 20261133 to 20261135. +- **Labels**: L2 is the primary Part 2 labeller and labels every Part 2 item (O4, section 11.3); system identity is hidden; both systems' answers are shuffled into one list (seed 20261132). The Part 2 correctness-gate sample and its bootstraps use seeds 20261133 to 20261135. - **Budget**: Part 2 gets its own estimate and ceiling (P16). ## 19. Release @@ -369,7 +370,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists ## 21. Reporting - Decisions per row and per hypothesis, estimates with unadjusted 95% intervals, SESOI and MDE lines, and the gated rows marked "not testable". -- **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed. +- **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed as defined; the audit trigger then uses the share not in the gold set (section 11.3). - Realised chunk-length distribution per chunk-size level and dataset. - Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit; H8b on questions with at least 95% of the token budget in all three `-t20k` setups, with realised token totals per setup; faithfulness with uncited factual segments as unsupported, beside the uncited-segment share. - **Compute table**: GPU type, pod hours per stage including every pre-freeze pod (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, an energy estimate (GPU hours times board power; Strubell et al. 2019; Lacoste et al. 2019), `resent_calls`, error rows per stage and role, and every `out_of_credit` stop. @@ -390,14 +391,14 @@ After the freeze, every change is a dated, numbered deviation. The report lists - One index build per cell. - L1 is an OMRS author. -## 23. Open items before the freeze +## 23. Owner decisions and pre-freeze tasks -**OPEN (the user decides):** +**Owner decisions (none open; the pre-freeze tasks P1 to P16 remain):** - O1. Decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 configuration hash. -- O2. The SESOI value, 0.02 nDCG@10. Answered and dated before any pilot result is read. -- O3. Whether the optional blind relevance audit for H3 and C1 runs. -- O4. L2's name, role and relation to the project. +- O2. Decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, with the written justification of section 10, recorded before any pilot result is read. +- O3. Decided by the owner on 2026-11-14: no relevance audit by default; it runs only if the registered unjudged-share trigger fires (section 11.3). +- O4. Decided by the owner on 2026-11-14: L2 is an author or colleague and labels every correctness answer and every Part 2 label, and a 30% subset of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1; section 11.3). The name goes in the run record only (P8). - O5. Decided by the owner on 2026-11-14: no ethics query (section 11.3). - O6. Decided by the owner on 2026-11-14: BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8, 46 contrasts); H8a and H8b hold on FinanceBench and FreshStack only. - O7. Decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims; the uncited-segment version is a sensitivity analysis only (section 6). @@ -411,8 +412,8 @@ After the freeze, every change is a dated, numbered deviation. The report lists - P5. Fix the pilot cell list (at most the four H7 corner indexes per core dataset), project the pilot cost from the P2 timing, get the user's approval, run the pilot, and fill the MDE table (or apply the registered fallback of section 10). - P6. Name the smoke and test-pod answer files that feed the H8 MDE rows. - P7. Restate the checker 2-point tolerance against the MDE table. -- P8. Write the label guidelines for each set, with worked examples outside every sample. -- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16), and the length-controlled sample size (300). The pilot seed is confirmed before the pilot runs (P5). Seed 20261120 (audit) is used only if O3 is yes; confirm it then. +- P8. Write the label guidelines for each set, with worked examples outside every sample. Record L2's name, role and relation to the project in the run record before the first label. +- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16; 20261124 is retired), and the length-controlled sample size (300). The pilot seed is confirmed before the pilot runs (P5). Seed 20261120 (audit) is used only if the audit trigger fires (O3); confirm it then. - P10. Record the tokenizer revision, answer-model host, reasoning mode, output cap and context length, and the judge, classifier and GLM-5.3 settings (the Ragas judge and classifier seeds included) in `plan.json`. - P11. Record the query instruction, the reranker instruction text and the hybrid settings in `plan.json`. - P12. Write `halves.json`; record its SHA-256; list every smoke and pre-freeze question ID with its half. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index 1de138f4..d94eb93f 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -152,12 +152,12 @@ Written before the freeze. No measured run has started. A publication-grade meth | F3 | Faithfulness gate per checker (kappa at least 0.60, lower bound at least 0.40), 450 claims by chunk size from answers to all questions, differential-error rule, length-controlled check | 11.2 | | F4 | Numbered contrast table; Holm families fixed at 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; was 16, changed by C3 below); gated rows stay in with p = 1 | 8 | | F5 | H8 split into H8a (crossed) and H8b (token-matched), both confirmatory | 7, 8 | -| F6 | Retrieval-only pilot on 40 half-A questions per core dataset; SESOI 0.02 nDCG@10 (open); underpowered rows named | 10 | +| F6 | Retrieval-only pilot on 40 half-A questions per core dataset; SESOI 0.02 nDCG@10 (O2, decided by the owner on 2026-11-14); underpowered rows named | 10 | | F7 | MixedLM dropped; one clustered method for estimates, intervals and p-values | 9 | | F8 | Correctness gate per core dataset, 50 items each, point estimate and lower bound; continuous score validated by AUC | 11.2 | | F9 | Missing-outcome rules per outcome, Part 1 and Part 2 | 12, 18 | | F10 | Decision test defined by its closed form (Mancl and DeRouen bias-reduced variance, t on n minus 1), `statsmodels` GEE as the checked implementation; FinanceBench document-level sign-flip; C2 fixed as continuous correctness, no McNemar | 9, 18 | -| F11 | Unjudged share at 10 and condensed-list sensitivity; blind audit optional (open) | 21, 11.3 | +| F11 | Unjudged share at 10 and condensed-list sensitivity; blind audit only if the unjudged-share trigger fires (O3, decided by the owner on 2026-11-14) | 21, 11.3 | | F12 | `evidence_recall_20k` gives any-page credit; first-page credit is sensitivity | 6 | | F13 | Second closed-book run with the RAG prompt; H8a and H8b correctness re-run on closed-book-wrong questions | 13 | | F14 | RQ1 headline is the A3 cell on half B; 72-setup table descriptive | 14.3 | @@ -165,7 +165,7 @@ Written before the freeze. No measured run has started. A publication-grade meth | F16 | Existing data declared | 5 | | F17 | Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop (E1 left the order, C6 below); "complete with registered drops" defined; "the operator decides" removed; Part 2 budget a pre-freeze task | 17 | | F18 | Release table per artefact and licence, re-judge kit | 19 | -| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 open; no ethics query (O5, decided 2026-11-14) | 11.3 | +| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 counts set (O4, decided 2026-11-14); no ethics query (O5, decided 2026-11-14) | 11.3 | | F20 | Pins table; index cost and configuration hash defined | 14.1, 16 | | F21 | FinanceBench split by document; pre-freeze samples from half A; order: score half A, hash, score half B | 4, 14.2 | | F22 | "Holds" defined per hypothesis; H7 confirmatory on FinanceBench and FreshStack, as H6; H4 named as simple effects | 7 | @@ -224,15 +224,18 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | Row 11 (H7) uses coefficients +1, -1, -1, +1 on the four corner cells after averaging over reranker and hybrid; the general mean-minus-mean recipe does not apply to it; known-answer fixture case (0.8, 0.2, 0.3, 0.7 gives 1.0); MDE on the same scale | 7, 9, 10 | | Short evidence pools: all 100 chunks are kept when they hold fewer than 20,480 tokens, and the realised token total is recorded per question and setup. Every question stays in H8b. "Chunk size itself" also needs a registered sensitivity on questions where all three `-t20k` setups reach at least 19,456 tokens (95%), with the count kept per dataset | 3, 7, 21 | | Every undefined resample is never drawn again and is counted once, whichever chunk-size level it lacks, also when the full sample is defined; the over-5% failure rule stays separate | 11.2 | -| Drop step 1 removes the 50 Legal correctness labels: correctness 150, L1 700, L2 210; Legal has no gate, so no confirmatory row changes | 11.2, 16, 17 | +| Drop step 1 removes the 50 Legal correctness labels: correctness 150, L1 700, L2 210 (O4 later set L2 to 315); Legal has no gate, so no confirmatory row changes | 11.2, 16, 17 | | OpenSpec only: O2 (SESOI) is asked in readiness flow step 6.10b, before any pilot result is read | none | | Rejected: move O5 before classifier-gate labelling. O5 was decided on 2026-11-14 (no ethics query), so the answer is already recorded before any label | none | -**Owner decisions (2026-11-14).** Three open items are closed before the freeze. O2, O3 and O4 stay open. +**Owner decisions (2026-11-14).** Six open items are closed before the freeze (O6 is in the table above). No owner decision is open; the pre-freeze tasks P1 to P16 remain. | Item | Decision | Pre-registration section | | --- | --- | --- | | O1 | A file upload to the registered OSF project time-stamps the A3 configuration hash | 14.2, 23 | +| O2 | SESOI 0.02 nDCG@10, with a written justification (user terms, design lower bound, cost, published Qwen3 effect sizes, not set by power, sensitivity at 0.01 and 0.05). Recorded before any pilot result is read | 10, 23 | +| O3 | No blind relevance audit by default. It runs only if the mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2 | 11.3, 23 | +| O4 | L2 (author or colleague) labels every correctness answer and every Part 2 label, and 30% of the faithfulness (135) and claim-kind (30) sets. L2 365, or 315 after drop step 1. Seed 20261124 retired. The name goes in the run record only | 11.3, 16, 23 | | O5 | No ethics query: the study has no human participants, and the labellers are members of the research team. The question reopens as a deviation if a paid external annotator is used | 11.3, 23 | | O7 | The primary faithfulness measure stays faithfulness of cited factual claims, because OMRS answers are built to cite their evidence. `uncited_segment_share` stays a co-outcome. The uncited-segment version stays a sensitivity analysis only | 6, 23 | diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index be64e216..a8610802 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -576,7 +576,7 @@ The operator approved every fix of an independent methods review (Exp 35 change - **Closed-book run (M6).** A `closedbook` stage: one run, all 1,160 questions, the answer model with no evidence (a fixed prompt that asks for an answer from the model's own knowledge), then `AnswerCorrectness` and the number match. It runs on replicate pod 1. **A4 (D38):** a second closed-book run uses the main RAG answer prompt with an empty evidence block, and the contamination sensitivity re-runs rows 14, 15, 18 and 19 on the questions both closed-book runs answer wrongly. - **Split halves (M3).** `split35.py` writes `halves.json` per dataset (seed 20261115). `select35.py` reads half A only. The report reads half B for answer-stage headlines. **A4 (D38):** FinanceBench is split by document, so no document sits in both halves; the other datasets are split by question. - **Clustered analysis (M2, A2c N4; replaced by A4, D38).** No MixedLM. For FreshStack, BioASQ and Legal the decision test is defined by its closed form (pre-registration section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster; the paired t-test variance times n / (n minus 1)); t reference on n minus 1 degrees of freedom. `experiments/_lib/stats_cluster.py` implements it with `statsmodels` `GEE` on per-question contrast scores (intercept only, Gaussian, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) and computes the p-value and interval from the t reference itself, because GEE results report a normal reference by default (`use_t=False` in the statsmodels source). The known-answer test of task 15.6 checks the GEE estimate and variance against the closed form; on a mismatch the closed form is used and the difference reported. The same module runs the document-level sign-flip permutation test (100,000 draws, seed 20261101) for FinanceBench; one method gives every estimate, interval and p-value (D38). It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock` (`statsmodels` 0.15.0, added 2026-11-14). The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. -- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750, L2 225. +- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set; O4 later retired 20261124), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750. O4 (2026-11-14): L2 labels 30% of claim kinds (30) and faithfulness (135) and all 200 correctness answers, so L2 365; the correctness gate uses the agreed label. - **Provenance (M7).** Each hosted call row records `model_returned` and `provider_returned` from the reply (OpenRouter `model` and `provider`; Z.AI `model`), beside the requested values. The join counts differences per role. - **Reranker check (M8).** `rerank_check35.py` scores 200 seeded pairs (seed 20261117) per size with llama-server and with the transformers recipe, and reports Spearman correlation; below 0.95 the file is not pinned. @@ -587,7 +587,7 @@ The operator approved every fix of an independent methods review (Exp 35 change | Depth 100 retrieval with `fetch_k` 100 passed, so the rerank pool stays 100 (without it, 300 and three times the rerank hours); dense and hybrid top 100 cost seconds | about 0 | $0 | | Two token-matched answer setups (6,960 answers at 2,048-level context: answers, five Ragas metrics, classifier; judges and Ragas on the replicate pods) | +36 (about +12 per replicate pod) | +$47 | | Closed-book run (1,160 short answers plus correctness) | +1 | +$3 | -| Human labels (400 L1, 120 L2; A4 raises them to 750 L1 and 225 L2, D38) | 0 | $0 | +| Human labels (400 L1, 120 L2; A4 raises them to 750 L1 and 365 L2, D38 and O4) | 0 | $0 | | **A2 total** | **about 378 pod hours, about $350 with the volume** | **about $290** | | Oracle run (D36) and A2c (no cost) | +4 | +$8 | | **Final total** | **about 382 pod hours, about $354 with the volume** | **about $298** | @@ -622,16 +622,16 @@ Written before the freeze; no measured run has started. A publication-grade meth - **FreshStack full corpus (F1; PR 4).** All 9 FreshStack indexes are built on the full public LangChain corpus (49,514 documents) at a pinned revision (P1). The download needs the user's approval at build time. The 10,000-document subset of Experiments 9a and 22 is retired for Experiment 35, and the continuity queries, continuity documents and overlay runbook are not used. No result is compared with the FreshStack leaderboard unless the corpus and the metric are the same. alpha-nDCG@10 is a secondary FreshStack metric if the nugget labels allow it (P3). One full-corpus build is timed before the estimate is relied on (P2). - **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Row 11 (H7) is a difference of differences with coefficients +1, -1, -1, +1 on the four corner cells after averaging over reranker and hybrid; the mean-minus-mean recipe does not apply to it (PR 9 point 1). Every question stays in H8b; "chunk size itself" also needs the registered sensitivity on questions where all three `-t20k` setups reach at least 95% of 20,480 tokens (PR 7), and realised evidence token totals are recorded per question and setup. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. - **Analysis (F7, F10; PR 9).** No MixedLM. The per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells; the estimate is its plain mean. FreshStack, BioASQ and Legal: the closed-form test of PR 9 point 3 (mean of d_qk; Mancl and DeRouen bias-reduced variance, sum of squared residuals x (n / (n minus 1)) squared / n squared; t on n minus 1 degrees of freedom), implemented with `statsmodels` GEE on an intercept (independence working correlation, question clusters, `cov_type="bias_reduced"`) and checked against the closed form by the known-answer test of task 15.6. FinanceBench: document-level sign-flip permutation test, 100,000 draws, seed 20261101, interval by inverting the test. Registered sensitivity checks, never decisive: a paired permutation test by question (10,000 shuffles) on FreshStack and BioASQ, and the GEE with document clusters on FinanceBench; disagreements go in one table. No McNemar test and no equivalence test. The type I error simulation test now covers this method (task 15.6). -- **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). +- **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). O3 (2026-11-14): the blind relevance audit runs only if the PR 11.3 unjudged-share trigger fires. - **Missing outcomes (F9; PR 12).** Every question stays in its dataset's denominator, and every exclusion is counted per setup and row. A short list scores missing ranks as not relevant. A failed answer call follows the resend rule (D3); an answer still missing at run close scores 0 for correctness and number match and has no factual claim. Faithfulness uses complete cases per row, with two sensitivity analyses (no-claim answers scored 1, and 0). A row still unjudged at run close is counted and scored 0 in a sensitivity analysis. -- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. Every undefined resample is left out and counted once, whichever level lacks a supported claim; over 500 of 10,000 makes those rows not testable (PR 11.2). A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window of each cited chunk, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 225 (700 and 210 after drop step 1), with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. +- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. Every undefined resample is left out and counted once, whichever level lacks a supported claim; over 500 of 10,000 makes those rows not testable (PR 11.2). A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window of each cited chunk, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 365 (700 and 315 after drop step 1; O4: L2 labels every correctness answer and every Part 2 label, 30% of the other two sets), with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. - **Closed-book runs (F13; PR 13).** Two single runs: own-knowledge prompt (floor) and the RAG prompt with an empty evidence block. Contamination sensitivity: rows 14, 15, 18 and 19 re-run on the questions both runs answer wrongly. - **Halves, order of work and selection (F14, F20, F21, F27; PR 4, 14).** FinanceBench is split by document. Every pre-freeze sample drawn after `halves.json` exists (the pilot, the RAGFlow checks) comes from half A; earlier samples are listed with their IDs and halves (PR 5). Order: build and seal the 9 indexes and pass G2; run the 72 search setups on all questions; score half A only; apply the selection rule and write `selection.json`; time-stamp the configuration hash on OSF (O1); only then score half B and start the replicate pods. The harness enforces it (task 16.17). "Sealed" applies to indexes only. Index cost = chunks times embedding dimension, summed over the three core datasets. Configuration hash = SHA-256 of the canonical JSON of the selected cell's resolved settings, with the `halves.json` SHA-256 as one field. RQ1 retrieval headline = the A3 cell on half B; the 72-setup table is descriptive. The search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, without replacement, ascending sort). -- **Pilot and MDE (F6; PR 10).** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118) on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. SESOI 0.02 nDCG@10 (O2). Experiment 22 values are not used. -- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation, which also removes the 50 Legal correctness labels (correctness 150, L1 700, L2 210; no gate, so no confirmatory row changes); (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. +- **Pilot and MDE (F6; PR 10).** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118) on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. SESOI 0.02 nDCG@10 (O2, decided by the owner on 2026-11-14, justified in PR 10). Experiment 22 values are not used. +- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation, which also removes the 50 Legal correctness labels (correctness 150, L1 700, L2 315; no gate, so no confirmatory row changes); (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. - **Pins, release, determinism, existing data (F16, F18, F20, F25; PR 5, 15, 16, 19).** One pins table (P10 to P14). A release table per artefact and licence, and a re-judge kit. One build per cell is assumed; an optional `e8b-c2048` rebuild check on FinanceBench runs only when the cost projection after the search stage is at or below the estimate, and is reported, not gated. The existing-data table lists every earlier look; P12 lists each smoke and pre-freeze question ID with its half. -Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the user's decisions O1 to O7 (O1, O5, O6 and O7 decided by the owner on 2026-11-14; O2, O3 and O4 open) are group 16. +Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open) are group 16. ## Risks / Trade-offs @@ -663,7 +663,7 @@ Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack c 1. Build and test the code (tasks 1 to 8) on the feature branch. The default behaviour stays serial-equivalent: concurrency 1 gives today's files. 2. Run the classifier gate (tasks 8.1 to 8.7). The classifier calls run on a 4090 pod; labelling and scoring run on the Mac. 3. Resolve and record the image digest (task 9.1). -4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the user's decisions on O2, O3 and O4 (O1, O5, O6 and O7 are decided). +4. Pre-freeze test pod session (task 9.2, D24): calibration and the first budget check against the $975 ceiling, reranker file checks and throughput, the VRAM check, build time per chunk size, the FreshStack recheck, and every new stage end to end once. Delete the pod. Then the GPU-choice timing on the L40S and the H100 (task 9.3), each after a price quote and the operator's go. Then the A4 pre-freeze tasks of group 16 (D38): the full FreshStack corpus pin and build timing, the retrieval-only pilot on half A and the MDE table, the pins, the label guidelines, and the record of L2's identity in the run record (O1 to O7 are all decided). 5. Write the procedure, the measured values, the GPU choice, the G2 stop rule and the budget with its ceiling into `protocol.md` (group 10). 6. At G7, freeze `preregistration.md` (the governing file, D38), `protocol.md` and `plan.json` with the measured values, after every group 16 task is done. 7. Index pods (D32, 9 in parallel on the chosen GPU type): each builds one index for all four datasets and seals it. The `e4b-c1024` rebuild check (D20) runs after that seal, and each pod then runs its 8 search setups once `rebuild.json` passes. A failed check stops here: find the cause, fix, build again. Then the FreshStack recheck on the real indexes with zero errors in every answer setup; on the Mac, half-A scoring, the selection rule (D31) and `selection.json`; the OSF time-stamp of the configuration hash; then half-B scoring (D38); and the budget re-check (D23). When the projection reaches $975, apply the drop order of D38. diff --git a/openspec/changes/exp35-main-run-readiness/proposal.md b/openspec/changes/exp35-main-run-readiness/proposal.md index cf391d62..681ff491 100644 --- a/openspec/changes/exp35-main-run-readiness/proposal.md +++ b/openspec/changes/exp35-main-run-readiness/proposal.md @@ -29,7 +29,7 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set on the full public corpus (A4), and a FinanceBench number-match check. - Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against human labels by Cohen's kappa (design D33; A4 makes it a gate per core dataset on 50 labels each, D38); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget then: about $650, ceiling $975. - Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; `statsmodels` as an experiment-only dependency (approved); two labellers with a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. (The A2 mixed model, the 400 L1 items and the $650 estimate are replaced by A4 below.) -- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 225 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the user's open decisions O2, O3 and O4 (O1, O5, O6 and O7 are decided by the owner on 2026-11-14) are tasks. +- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 365 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open) are tasks. - Answer context budget (design D30): a test proves that the largest evidence set of every answer setup synthesises without the earlier context overflow, and each answer row records its rounds, evidence characters and truncation. - Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8b with the existing statistics functions. - The harness reads the `rate_limited` record instead of parsing error text, and stops at once on `out_of_credit`. Every hybrid-on setup pins `retrieval.hybrid_sparse_backend: bm25`, as the protocol defines it. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index 7356ce87..72a3fa71 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -425,7 +425,7 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 14.2 Write failing tests that each answer row records `completion_calls`, the evidence characters, and the truncated passages and characters dropped, computed with OMRS's own `_fit_and_measure` on the same rows and settings. Prove they fail, then implement in `answers35.py`. Verify: the tests pass, and a row with cut evidence is still saved. - [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score, both chosen sizes, the A3 cell and the configuration hash. Then (A4, pre-registration section 14.1) tests that: index cost is the number of chunks times the embedding dimension, summed over the three core datasets, read from the sealed indexes with no score; the configuration hash is the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings with the SHA-256 of `halves.json` as one field, and the same inputs give the same hash twice; the any-page sensitivity repeats step 3 with any-page nDCG@10 on FinanceBench, records whether the cell changes, and never changes the registered cell. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. - [ ] 14.5 Ragas `AnswerCorrectness` (design D33). Write failing tests with a stub LLM and stub embeddings: the `ragas` stage scores five metrics, `AnswerCorrectness` gets the question, the answer and the dataset's reference answer (never the contexts), uses the frozen judge and the local 4B embedder, and records the weights; an answer without a reference records "not judged" and sends no call; resume skips saved rows. Prove they fail, then implement in `ragas35.py` with Ragas 0.4.3. Verify: the tests pass, and the version check still refuses another Ragas version. -- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B (150 with no Legal draw when drop step 1 applies, PR 11.2 and 17), each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 labels the sample and L2 labels 30% of it (task 15.8). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). +- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B (150 with no Legal draw when drop step 1 applies, PR 11.2 and 17), each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 and L2 both label the whole sample, and the gate uses the agreed label (task 15.8; O4). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). - [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome; a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups, with the PR 7 sensitivity on questions at 19,456 tokens or more in all three; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). ## 15. Methods-review fixes (pre-freeze amendment A2, 2026-11-13; design D35) @@ -453,7 +453,7 @@ Each test is proved to fail on the current code before the fix. - type I error simulation, now for this method: with no planted effect and unequal error variance across cells, the rejection rate at 0.05 stays near 0.05 over 200 simulations for the GEE (FreshStack-sized data) and for the sign-flip test (84 documents, about 1.8 questions each); a fixture with complete data gives the same estimate as the cell-means contrast of the full-factorial model. - The `stats` dependency group exists (decided by the operator, 2026-11-13; added on 2026-11-14 with `statsmodels` 0.15.0 in `pyproject.toml` and `uv.lock`, and `{include-group = "stats"}` in the `dev` group). The wrapper and its tests are not built yet, so this box stays open. A test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. - [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the L2 subset is 30% of each set with seeds 20261122, 20261123 and 20261124 (225 in all); no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); L2 labels 30% of the claim-kind and faithfulness sets (30 and 135, seeds 20261122 and 20261123) and all 200 correctness answers (365 in all; 315 after drop step 1; seed 20261124 retired, O4); the gate reference for a fully double-labelled set is the agreed label; no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 (per set, and per core dataset for correctness) comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.9 Hosted-call provenance (M7). Write failing tests: each answer, Ragas, GLM-5.3 and classifier row records `model_returned` and `provider_returned` from the stubbed reply; the join counts rows whose returned model differs from the requested one per role. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.10 Reranker rank check (M8). Write failing tests for `rerank_check35.py`: 200 pairs drawn with seed 20261117 are the same twice; Spearman below 0.95 marks the file "not pinned". Prove they fail, then implement. Run it on the test pod for each size (task 9.2 step 2a). Verify: the tests pass and `protocol.md` lists the correlation per file. - [ ] 15.11 MDE from the pilot (M1; A4, pre-registration section 10). Write `mde35.py` on saved files only: per core dataset and contrast type, the SD of the per-question contrast scores from the retrieval-only pilot (task 16.5, P5; an adjacent-level contrast uses the extreme-level SD as a stand-in, marked so), and on FinanceBench the design effect of document clustering; MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha 0.05/19 on FinanceBench and FreshStack and 0.05/8 on BioASQ, and n divided by the design effect on FinanceBench; answer-stage rows use the SD from the smoke and test-pod answer files named in task 16.6 (P6) and are labelled rough; each row is shown against the SESOI (0.02 nDCG@10, open item O2). Experiment 22 values are not used. Write failing tests on a fixture first. Verify: the tests pass and the MDE table of pre-registration section 10 is filled before the freeze. @@ -465,7 +465,7 @@ Each test is proved to fail on the current code before the fix. ## 16. Publication-review fixes (pre-freeze amendment A4, 2026-11-14; design D38) -`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 and 16.29 are code. Tasks 16.23 to 16.28 and 16.30 are the user's decisions O1 to O7. The owner decided O1, O5, O6 and O7 on 2026-11-14; O2, O3 and O4 stay open. Record each answer, never guess it. Every task closes before G7. +`preregistration.md` governs; "PR n" below means its section n. Tasks 16.1 to 16.16 are the pre-freeze tasks P1 to P16 of PR 23, in the same order. Tasks 16.17 to 16.22 and 16.29 are code. Tasks 16.23 to 16.28 and 16.30 are the user's decisions O1 to O7. The owner decided O1 to O7 on 2026-11-14; no owner decision is open. Record each answer, never guess it. Every task closes before G7. - [ ] 16.1 P1. FreshStack full corpus. Ask the user to approve the download of `freshstack/corpus-oct-2024`, config `langchain`, at a pinned revision. Then download it into `~/Development/DATA/omrs/` and record the revision, the row count (49,514 expected) and the corpus tree hash in `plan.json` and in the pins table (PR 16). Verify: the loader of task 13.5 accepts the corpus, and its check refuses a changed row count. - [ ] 16.2 P2. Time one full-corpus FreshStack index build on the chosen GPU type (part of task 9.2 or 9.3 when possible). Update the estimate in PR 17 from the measured time before relying on it. Verify: the compute table and the estimate name the measured hours; the "30 to 40 pod hours" reviewer figure is no longer the basis. @@ -475,7 +475,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.6 P6. Name the smoke and test-pod answer files that feed the H8a and H8b MDE rows, with their SHA-256. Verify: PR 10 names them and marks those rows rough. - [ ] 16.7 P7. Restate the checker 2-point tolerance against the H8a and H8b faithfulness MDE: the bar stays only if it is at most half that MDE. Verify: PR 11.1 and `protocol.md` section 4.4 give the restated rule; the "about 4 points" figure is not used. - [ ] 16.8 P8. Write one label guideline per set (claim kinds, faithfulness, correctness, Part 2 correctness), with worked examples drawn from outside every sample. Faithfulness follows the LLM-AggreFact task definition. Fix each guideline before its first label. Verify: the guidelines are committed before any label, and no worked example is in a sample (a test of `labels35.py` checks the IDs). -- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Confirm the pilot seed before the pilot runs (task 16.5). Seed 20261120 (audit) is used only if O3 (task 16.25) is yes; confirm it then. Verify: PR 16 and `plan.json` hold the confirmed values. +- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Confirm the pilot seed before the pilot runs (task 16.5). Seed 20261120 (audit) is used only if the O3 trigger fires (task 16.25); confirm it then. Seed 20261124 is retired (O4). Verify: PR 16 and `plan.json` hold the confirmed values. - [ ] 16.10 P10. Record in `plan.json`: the tokenizer revision; the answer-model host, reasoning mode, output cap and context length (task 14.1a); the Ragas judge and classifier host and settings; the GLM-5.3 settings. Verify: each "Missing: P10" cell of PR 16 points to a `plan.json` key. - [ ] 16.11 P11. Record in `plan.json`: the query instruction (empty string), the reranker instruction text, and the hybrid settings (RRF constant, per-side fusion depth, BM25 tokeniser and parameters). Write a failing test that the run record of a search stage holds the same values, then implement. Verify: the test passes. - [ ] 16.12 P12. Run `split35.py` (task 15.5), write `halves.json`, record its SHA-256 in `plan.json`, and list every smoke and pre-freeze question ID with its half in the existing-data table (PR 5). Verify: the hash in `plan.json` equals the file, and every listed ID has a half. @@ -490,9 +490,9 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.21 Release kit (PR 19). Write failing tests for a re-judge script that rebuilds evidence text from the pinned dataset revisions and the released unit and chunk IDs, on a fixture; and for an export that writes ranked lists in TREC run format, never corpus text. Prove they fail, then implement. Verify: the tests pass, and the release table of PR 19 names each artefact and its licence. - [ ] 16.22 Optional `e8b-c2048` rebuild check (PR 15). Run it on FinanceBench only when the cost projection after the search stage is at or below the estimate (a cost-only decision, recorded). Report the share of identical ranked lists and the nDCG@10 change; it is not a gate. Verify: the decision and, if run, the result are in the run record. - [ ] 16.23 O1 (decided by the owner on 2026-11-14): a file upload to the registered OSF project time-stamps the A3 configuration hash. The answer is recorded in PR 14.2 and PR 23. Verify: recorded before G7. -- [ ] 16.24 O2 (user decides). The SESOI value (proposed: 0.02 nDCG@10). Record the answer in PR 10. Verify: recorded before the pilot results are read. -- [ ] 16.25 O3 (user decides). Whether the optional blind relevance audit for H3 and C1 runs (about 300 to 400 L1 labels, seed 20261120, descriptive). Record the answer. If yes, add its sample and page to `labels35.py` with tests first. Verify: the answer is recorded before G7. -- [ ] 16.26 O4 (user decides). L2's name, role and relation to the project. Record it in the run record only, never in public files. Verify: no public file holds the name. +- [ ] 16.24 O2 (decided by the owner on 2026-11-14): the SESOI is 0.02 nDCG@10, with the written justification in PR 10; intervals are also read against 0.01 and 0.05. Recorded in PR 10 and PR 23 before any pilot result is read. Verify: recorded before the pilot results are read. +- [ ] 16.25 O3 (decided by the owner on 2026-11-14): no blind relevance audit by default. It runs only if the PR 11.3 trigger fires (mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2). Write failing tests for the trigger, then implement it; only if it fires, add the audit sample and page (seed 20261120) to `labels35.py` with tests first. Verify: tests pass, and the trigger values are in the report. +- [ ] 16.26 O4 (decided by the owner on 2026-11-14): L2 is an author or colleague, labels every correctness answer and every Part 2 label, and 30% of the faithfulness and claim-kind sets (PR 11.3). Before the first label, record L2's name, role and relation to the project in the run record only (P8). Verify: no public file holds the name. - [ ] 16.27 O5 (decided by the owner on 2026-11-14): no ethics query. The study has no human participants, and the labellers are members of the research team, not study subjects, so no ethics review is sought. This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation. The answer is recorded in PR 11.3 and PR 23. Verify: recorded before any label. - [ ] 16.28 O6 (decided by the owner on 2026-11-14): BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack). The answer is recorded in PR 7, 8 and 23. Verify: recorded before G7. - [ ] 16.29 Uncited segments (PR 6). Write failing tests: the join counts per answer the prose segments with no valid ordinal (the segments `split_claims` drops) and gives `uncited_segment_share` per setup and dataset; `claimkinds` also sorts uncited segments with the frozen classifier (question and segment only); the sensitivity counts uncited fact and mixed segments as unsupported; a fixture answer with one supported cited fact and one uncited fact scores 1.0 primary and 0.5 sensitivity. Prove they fail, then implement. Verify: the tests pass. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md index f8da804c..cd669ea6 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md @@ -178,13 +178,13 @@ FinanceBench page credit: (for `evidence_recall_20k`, superseded by A4, D0.15: a **B2. H6 confirmatory on long-document sets only (blocker).** (Rows, endpoints and the holds rule: `preregistration.md` section 7, A4, D0.15.) H6 is now: chunk size changes unit-level nDCG@10 on FinanceBench and FreshStack. It holds when at least one chunk-size contrast counts on both datasets. BioASQ results on chunk size are a reported sanity check, not evidence. Legal RAG Bench is exploratory throughout and is removed from every confirmatory hypothesis. -**M1. Minimum detectable effect from data.** (Superseded by A4, D0.15: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the MDE table; SESOI 0.02 nDCG@10 waits for the operator.) Before the freeze, compute the standard deviation of per-question paired differences per dataset and per contrast, from Experiment 22 (FreshStack, per-query nDCG@10) and the pre-freeze smoke and test-pod data (all four datasets). Compute the MDE with `min_detectable_effect` at the Holm-adjusted alpha of the worst rank (0.05 / the number of contrasts per dataset), power 0.80. Separate power checks: the H7 interaction (difference of differences) and any equivalence claim (TOST). After B2 no equivalence claim remains in a confirmatory hypothesis, so TOST power is computed only if one is added. The numbers and their sources go into `protocol.md` before the freeze. The earlier MDE figures (about 4, 2 and 5 points) were guesses and are withdrawn. +**M1. Minimum detectable effect from data.** (Superseded by A4, D0.15: a retrieval-only pilot on 40 seeded half-A questions per core dataset fills the MDE table; SESOI 0.02 nDCG@10, decided by the owner on 2026-11-14 (O2).) Before the freeze, compute the standard deviation of per-question paired differences per dataset and per contrast, from Experiment 22 (FreshStack, per-query nDCG@10) and the pre-freeze smoke and test-pod data (all four datasets). Compute the MDE with `min_detectable_effect` at the Holm-adjusted alpha of the worst rank (0.05 / the number of contrasts per dataset), power 0.80. Separate power checks: the H7 interaction (difference of differences) and any equivalence claim (TOST). After B2 no equivalence claim remains in a confirmatory hypothesis, so TOST power is computed only if one is added. The numbers and their sources go into `protocol.md` before the freeze. The earlier MDE figures (about 4, 2 and 5 points) were guesses and are withdrawn. **M2. One mixed model per dataset, then planned contrasts.** (Superseded by A2c and A4, D0.15: no mixed model; per-question contrast scores with GEE or the FinanceBench document-level sign-flip test; Holm families fixed at 19, 19 and 8 after the second round.) Per dataset, one linear mixed model: `ndcg10 ~ C(embedder) * C(chunk) * C(reranker) * C(hybrid)` with a random intercept per question, fitted with `statsmodels` `MixedLM` (REML), the version pinned in `uv.lock` (a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13)). The planned contrasts of H1 to H7 are estimated from the model as follow-ups. Holm runs across every H1 to H8 contrast of one dataset. The paired permutation test and paired bootstrap stay as a robustness check. FinanceBench bootstraps resample whole documents (`paired_bootstrap(groups=doc_name)`, `experiments/_lib/stats_d6.py:140-160`). Cross-dataset rule: a hypothesis holds overall when it is supported on at least 2 of the 3 confirmatory datasets (FinanceBench, FreshStack, BioASQ); H6 needs both long-document sets. Legal is exploratory and never counts. **M3. Split-half selection.** (Amended by A4, D0.15: FinanceBench is split by document; the others by question.) Each dataset's questions are split into halves A and B with seed 20261115 (stratified by dataset). The embedder and reranker size for the answer stage are chosen on half A only, with the D0.5 rule. All answer-stage headline results are reported on half B. The answer stage still runs on all questions; half A results are secondary. Search-stage hypotheses use all questions. -**M4. Human validation.** (Superseded by A4, D0.15: L1 labels 750 items, L2 225; gates and samples in `preregistration.md` section 11.) Two labellers, recorded as L1 (the operator) and L2 (the second labeller). No names in public records. +**M4. Human validation.** (Superseded by A4, D0.15: L1 labels 750 items, L2 365 (O4); gates and samples in `preregistration.md` section 11.) Two labellers, recorded as L1 (the operator) and L2 (the second labeller). No names in public records. | Set | Size (L1) | Drawn from | Stratified by | L2 share | |---|---:|---|---|---:| @@ -470,7 +470,7 @@ Decision (operator, 2026-11-13). The master plan is `docs/proposals/first-public - **Hand-off.** Part 2 runs on half B only, once per system. Experiment 35 does not run Part 2, does not run RAGFlow, and does not draw labels from RAGFlow answers. Part 2 has its own protocol (C1, C2, fairness rules, labels). - **Token-matched control (PR 209 review, 2026-11-13).** The factorial cell `c2048` x 10 keeps its 10-chunk evidence. A separate 15th answer setup, `c2048-t20k`, cuts evidence at 20,480 real Qwen3 tokens at 2,048-token chunks, reranker on and hybrid on, beside `c512-t20k` and `c1024-t20k`. Each chunk size has a token-matched cell, and the token-matched comparison of H8 uses the three `-t20k` setups. Registered before the freeze. It amends D0.13 N2, which cut the `c2048` comparison by the same rule. Cost: the answer stage grows by about 1/14 (about 7%). Counted from the readiness design D35 table (two token-matched setups: +36 pod hours, +$47 hosted), one more setup adds about $24 of hosted calls (answers, five Ragas metrics, judges and classifier) and about 18 pod hours (about $16 at $0.89 per hour, about 6 hours per replicate pod), so about $40 in all. The estimate moves from about $650 to about $690. The ceiling stays $975 (1.5 x the old estimate, 1.4 x the new one). At 3 times slower reranking the total is about $910, still under it. - **FinanceBench unit in Part 2 (2026-11-13).** Part 2 scores C1 at page level for every FinanceBench question and both systems, the unit used here (first page of a chunk). Pre-freeze gate and document-level fallback are in the master plan, Step 3 rule 3, and protocol section 0.12 item 4. -- **Counts.** The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each. The 3 runs stay as in D0.12. (Labels superseded by A4, D0.15: 750 L1 and 225 L2.) +- **Counts.** The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each. The 3 runs stay as in D0.12. (Labels superseded by A4, D0.15: 750 L1 and 365 L2 after O4.) - **Framework comparison.** Out of Experiment 35 scope. Part 2 holds it. ### D0.15 Pre-freeze amendment A4: publication review fixes (2026-11-14) @@ -483,7 +483,7 @@ Decision (operator, 2026-11-14). A publication-grade methods review gave 30 find - F1: all 9 FreshStack indexes use the full public LangChain corpus (49,514 documents, pinned revision). The 10,000-document subset is retired. alpha-nDCG@10 is secondary if the nugget labels allow it. - F4, F7, F10: no mixed model. Per-question contrast scores. On FreshStack, BioASQ and Legal the decision test is defined by its closed form (mean; Mancl and DeRouen bias-reduced variance; t on n minus 1), with `statsmodels` GEE as the implementation checked against it (`preregistration.md` section 9 point 3); and a document-level sign-flip test on FinanceBench. Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; second round C3). A gated row stays in its family with p = 1. No equivalence claim, no TOST, no McNemar test as a main test. - F5, F22, F23: H8 splits into H8a (crossed setups) and H8b (token-matched setups). Both are confirmatory. Every hypothesis has scope S, numbered rows and a holds rule. H7 is confirmatory on FinanceBench and FreshStack. After the second round (C3, C4), H6, H7, H8a and H8b hold only on FinanceBench and FreshStack, and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs. Legal RAG Bench is exploratory in every hypothesis. - - F3, F8, F19 (counts after drop step 1: L1 700, L2 210; second CodeRabbit round): L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 225. Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. + - F3, F8, F19 (counts after drop step 1: L1 700, L2 315 after O4; second CodeRabbit round): L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 365 (O4, 2026-11-14). Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. - F12: `evidence_recall_20k` gives any-page credit and is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11). nDCG@10 stays the primary metric of the study. First-page credit is a sensitivity. - F13: a second closed-book run uses the RAG prompt with empty evidence. - F14, F21: FinanceBench is split by document. Order: score half A, write `selection.json`, time-stamp the configuration hash on OSF, then score half B. The RQ1 retrieval headline is the A3 cell on half B. "Sealed" applies to indexes only. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md index 8d2dc082..b830f9c2 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md @@ -26,7 +26,7 @@ A publication-grade methods review gave 30 findings, F1 to F30. The operator app - FreshStack indexes use the full public LangChain corpus (49,514 documents). The 10,000-document subset is retired. - H8 splits into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory. Every hypothesis is scoped, with numbered rows and a holds rule. Legal RAG Bench is exploratory everywhere. - No mixed model. Per-question contrasts with GEE, and a document-level sign-flip test on FinanceBench. Holm families fixed at 19, 19 and 8 (second round C3; BioASQ rows 9 to 19 are sanity checks). -- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items and L2 labels 225. +- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items and L2 labels 365 (every correctness answer, 30% of the other two sets; O4). - `evidence_recall_20k` gives any-page credit. A second closed-book run uses the RAG prompt. FinanceBench is split by document. - The `openai/gpt-6.1-terra` cross-check is removed. Budget about $730 plus the pilot, ceiling $975, with a results-blind drop order that no longer holds E1 (E1 has its own ceiling). diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index f260c272..f439b3ef 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -153,7 +153,7 @@ Each faithfulness checker SHALL pass when Cohen's kappa against the human label ### Requirement: Human labels follow the registered counts -L1 SHALL label 750 items (claim kinds 100, faithfulness 450 from all questions, correctness 200 from half B), and L2 SHALL label a 30% subset of each set (225 items); after drop step 1, correctness 150, L1 700 and L2 210. The claim classifier SHALL reach agreement of at least 0.90 and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. +L1 SHALL label 750 items (claim kinds 100, faithfulness 450 from all questions, correctness 200 from half B), and L2 SHALL label 30% of claim kinds and faithfulness and all correctness answers (365 items); after drop step 1, correctness 150, L1 700 and L2 315. The claim classifier SHALL reach agreement of at least 0.90 and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. #### Scenario: A label file is released - **WHEN** human labels are released diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index 9e9d6029..3ef9810f 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -66,7 +66,7 @@ Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4 - [x] 4B.1 Write amendment A2 into this change (design D0.12, specs), `protocol.md` (section 0.10) and `plan.json` (amendment A2, metrics, hypotheses). Verify: both `openspec validate --strict` runs and `omrs experiments check` pass. Done 2026-11-13. - [ ] 4B.2 Compute the minimum detectable effects from data (M1, readiness task 15.11) and write the table with its sources into `protocol.md` section 0.10. Verify: the table is in a commit before the freeze commit. (Amended by A4: the table lives in `preregistration.md` section 10; see 4D.5.) - [ ] 4B.3 Pin every reranker GGUF with its SHA-256 (M8) only after its Spearman check (at least 0.95 on 200 seeded pairs) passes on the test pod. Verify: `plan.json` runtime holds three reranker entries, `load_runtime_block` accepts them, and `protocol.md` lists each correlation. -- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 225; see 5.7.) +- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 365; see 5.7.) - [ ] 4B.5 Run the closed-book run (M6) on replicate pod 1 and report correctness by retrieval hit or miss. Verify: the report table names its `output/` file. (Amended by A4: two closed-book runs; see 5.8.) - [ ] 4B.5a Run the oracle run (design D0.12, readiness tasks 15.12 and 15.13) on replicate pod 1 and report it beside closed-book as ceiling and floor, descriptive only. Verify: the table names its `output/` file and states the no-distractor caveat. - [ ] 4B.6 Deposit `protocol.md` and `plan.json` on OSF at the freeze commit (M9; the operator does the deposit). Verify: a commit after the freeze, before the first measured run, adds the OSF link and the freeze commit hash to `protocol.md`, and the deposit holds no personal data beyond the operator name. (Amended by A4: `preregistration.md` is the main file of the deposit; the OSF link and freeze commit go into `plan.json` `frozen`.) @@ -93,8 +93,8 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 4D.5 Fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the operator's approval, run the retrieval-only pilot on 40 seeded half-A questions per core dataset, and fill the MDE table (P5, P6); if the operator declines, fill it from assumed SDs of 0.15 to 0.25, labelled as assumed. Restate the checker 2-point tolerance against the table (P7). Verify: the MDE table has one row per dataset and row group, with its SD source. - [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Verify: the guidelines are committed before the freeze. (The labelling itself is task 5.7.) - [ ] 4D.7 Fix the second closed-book run (RAG prompt, empty evidence) and the length-controlled check (300 claims) in `plan.json` and the pre-registration. Verify: both are listed before the freeze. (The runs are task 5.8.) -- [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if O3 is yes); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. -- [ ] 4D.9 Get the operator's answers to O2, O3 and O4 (SESOI value, optional blind audit, L2 identity). The owner decided O1 (OSF file upload time-stamps the hash), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only) on 2026-11-14. Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. +- [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if the O3 trigger fires; 20261124 retired); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. +- [ ] 4D.9 Owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open): O1 (OSF file upload time-stamps the hash), O2 (SESOI 0.02 nDCG@10, justified in `preregistration.md` section 10), O3 (no blind audit unless the section 11.3 unjudged-share trigger fires), O4 (L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets; name in the run record only), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. - [ ] 4D.10 Check before the freeze that no live text in this change, `protocol.md` section 0 or `plan.json` contradicts `preregistration.md`. Verify: every hit for "MixedLM", "included whole", "gpt-6.1-terra", "operator decides" and "equivalence" is marked superseded or removed. ## 5. Run @@ -105,7 +105,7 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 5.4 Run the determinism check for every arm, and the rebuild check (4B baseline index rebuilt once per dataset in a fresh process). Verify: the output states match or mismatch per arm and dataset, and for each rebuild. - [ ] 5.5 Run Ragas on each arm where the dataset supports each metric. Verify: unsupported metrics show "not measured". - [ ] 5.6 Check about 50 judged items by hand. Verify: the agreement is computed with the measure frozen in 1.10. -- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 225, with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. +- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 365 (claim kinds 30, faithfulness 135, correctness 200; O4), with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. - [ ] 5.8 (A4, moved from 4D.7.) Run the second closed-book run (RAG prompt, empty evidence) beside the first, and the length-controlled check on 300 claims. Verify: each table names its `output/` file and is marked descriptive or sensitivity. ## 6. Analyse and report diff --git a/openspec/changes/rag-benchmark-readiness-flow/design.md b/openspec/changes/rag-benchmark-readiness-flow/design.md index 0b03b415..00a2b9e4 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/design.md +++ b/openspec/changes/rag-benchmark-readiness-flow/design.md @@ -91,7 +91,7 @@ E. Arm smoke run on the run machine; pod stopped after it | E2. A4 pre-freeze items: FreshStack full corpus pin and timing, | halves and MDE pilot on half A, P1 to P16, - | open items O2, O3 and O4 (O1, O5, O6, O7 decided) + | owner decisions O1 to O7 recorded (none open) | (with Part 2 steps 7.1, 7.2 and 7.2a) F. Freeze (G7) | diff --git a/openspec/changes/rag-benchmark-readiness-flow/proposal.md b/openspec/changes/rag-benchmark-readiness-flow/proposal.md index 672917c6..31817353 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/proposal.md +++ b/openspec/changes/rag-benchmark-readiness-flow/proposal.md @@ -15,7 +15,7 @@ Without one recorded order, two branches can touch the same specs at once, the l - A task order only (tasks.md groups 0 to 8) with gates. No code and no capability deltas. - The order (operator decisions 2026-11-06): cuts, then router, then one live switch (no re-index: the live collections are test data, operator 2026-11-07), then the pipeline sweep on RunPod, then Experiment 35. The page routing panel follows the router and can run beside the sweep, because it touches no runtime and no result. -- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the user's open items O2, O3 and O4; O1, O5, O6 and O7 are decided by the owner on 2026-11-14) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). +- Amendment A4 (2026-11-14): `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35. Group 6 gains the pre-freeze items of its section 23 (the FreshStack full-corpus pin and timing, the halves and the MDE pilot on half A, P1 to P16, and the owner decisions O1 to O7, all decided by the owner on 2026-11-14, so none is open) and follows its order of work (score half A, time-stamp the selection hash on OSF, then score half B). Group 7 lists what the Part 2 protocol must hold and adds step 7.2a, the matched RAGFlow search decision (design D6). ## Capabilities diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 0516ea7a..47a047f5 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -86,9 +86,9 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35 ("PR n" means its section n). "RDY 16.N" means task 16.N in `exp35-main-run-readiness/tasks.md`. Steps 6.10a to 6.10d run after step 6.10 and before the freeze. - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. -- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. Then RDY 16.24 (O2): ask the user to confirm or change the SESOI and record the answer and its date in PR 10, before any pilot result is read and before the MDE table is filled (with the pilot or the fallback). Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the O2 answer is dated before any pilot result is read, the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. +- [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. RDY 16.24 (O2) is done: the owner set the SESOI at 0.02 nDCG@10 on 2026-11-14, recorded with its justification and date in PR 10 before any pilot result is read and before the MDE table is filled. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the O2 answer is dated before any pilot result is read, the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. - [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Open items (the user decides; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 the SESOI (0.02 nDCG@10 proposed; asked in step 6.10b before any pilot result is read); O3 whether the optional blind relevance audit runs; O4 L2's name, role and relation to the project (run record only); O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. O2, O3 and O4 stay open. Verify: PR 23 shows each item decided. +- [ ] 6.10d Owner decisions (all decided by the owner on 2026-11-14; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, justified in PR 10; O3 decided by the owner on 2026-11-14: no blind relevance audit unless the PR 11.3 unjudged-share trigger fires; O4 decided by the owner on 2026-11-14: L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets, and L2's name goes in the run record only, before the first label; O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. No owner decision is open. Verify: PR 23 shows each item decided. ### F. Freeze From 728b57394f4b1d4f6ca12c024944a4b4a389abba Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 19:09:12 +0000 Subject: [PATCH 7/9] docs(exp35): limit the audit trigger to datasets with judged negatives The unjudged share on a positives-only dataset equals 1 minus precision@10, so the trigger would fire whenever one side retrieves better. The trigger now applies only where the qrels list judged non-relevant units. For positives-only datasets the owner chooses at P4 between a fixed small audit and no audit. --- experiments/35-full-rag-benchmark-2026-11-01/plan.json | 2 +- .../35-full-rag-benchmark-2026-11-01/preregistration.md | 8 ++++---- experiments/35-full-rag-benchmark-2026-11-01/protocol.md | 2 +- openspec/changes/exp35-main-run-readiness/design.md | 2 +- openspec/changes/exp35-main-run-readiness/tasks.md | 4 ++-- .../experiment-35-full-document-rag-benchmark/tasks.md | 4 ++-- openspec/changes/rag-benchmark-readiness-flow/tasks.md | 4 ++-- 7 files changed, 13 insertions(+), 13 deletions(-) diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 702a2f2b..5e0f2f0a 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 365 (every correctness answer and every Part 2 label, 30% of the faithfulness and claim-kind sets; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2); O4, L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 315 after O4; no confirmatory row changes).", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 365 (every correctness answer and every Part 2 label, 30% of the faithfulness and claim-kind sets; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), a trigger defined only on datasets whose qrels list judged non-relevant units, while for positives-only datasets the owner decides at P4 between a fixed small audit and no audit; O4, L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 315 after O4; no confirmatory row changes).", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index 6186cf09..d56c8fff 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -195,7 +195,7 @@ Why: a chance value makes an undefined resample count against a pass, and point - **Blinding.** Labellers never see scores, verdicts, setup names or chunk sizes. Cited passage length can still hint at chunk size; the report says so. A training round on practice items comes first and is not evidence. - **Ethics.** The study has no human participants. It uses public datasets, and the labellers are members of the research team, not study subjects, so no ethics review is sought (O5, decided by the owner on 2026-11-14). This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation (section 20). - **Conflict of interest.** The authors built OMRS. Safeguards: this pre-registration and its deviation log; selection on half A only; Holm families fixed now; blind labels with L2 as primary labeller in Part 2; published RAGFlow settings and maintainer correspondence; release of every output (section 19). The paper states the conflict. -- **Blind relevance audit for H3 and C1 (O3, decided by the owner on 2026-11-14).** No audit by default. The audit runs only if this trigger, fixed now, fires: the mean unjudged share in the top 10 units differs by more than 5 percentage points (0.5 units per question) (a) between hybrid on and hybrid off on FreshStack, marginal over the other factors, or (b) between OMRS and RAGFlow on any core dataset in Part 2. The trigger uses the absolute gap, so it is blind to which side scores higher. Trigger (a) is computed on all questions after step 5 of section 14.2, trigger (b) after the Part 2 runs. A unit is unjudged when the qrels hold no judgement for it. Where P4 finds that a dataset's qrels hold positives only, "unjudged" means "not in the gold set". The share is then 1 minus precision@10, so a side that finds more gold units also has a smaller share. Why: an unjudged unit scores as not relevant, so the bias a setup can suffer is bounded by how many more unjudged units it retrieves; with equal shares both sides are treated alike. The 5-point value is a judgement. Rough bound: if a quarter of a 0.5-unit gap were truly relevant, one side would miss about 0.125 relevant units per question. Each such unit is worth 1/log2(rank + 1) over IDCG, on average about 0.45 over IDCG across ranks 1 to 10. So the bias can reach about 0.06 nDCG@10 on a question with one gold unit, about 0.02 with five and about 0.01 with ten. If the trigger fires, L1 labels about 300 to 400 unjudged top-3 units (seed 20261120); the audit is descriptive and changes no decision. The report gives both trigger values. +- **Blind relevance audit for H3 and C1 (O3, decided by the owner on 2026-11-14).** No audit by default. The audit runs only if this trigger, fixed now, fires: the mean unjudged share in the top 10 units differs by more than 5 percentage points (0.5 units per question) (a) between hybrid on and hybrid off on FreshStack, marginal over the other factors, or (b) between OMRS and RAGFlow on any core dataset in Part 2. The trigger is defined only on a dataset whose qrels list judged non-relevant units (P4 checks this per dataset). The trigger uses the absolute gap, so it is blind to which side scores higher. Trigger (a) is computed on all questions after step 5 of section 14.2, trigger (b) after the Part 2 runs. Kept: the share exists only once each setup has retrieved its units, and the rule is fixed now and uses the absolute gap, so computing it then adds no choice. On such a dataset a unit is unjudged when the qrels hold no judgement for it. On a dataset whose qrels hold positives only, the unjudged share cannot be told apart from retrieval quality, so the trigger is not used there. For such a dataset the owner decides before the freeze, when P4 reports, between (i) a fixed small audit that runs whatever the results are, and (ii) no audit, with the pooling bias stated as a limitation. The decision is recorded with its date before any main-run score exists. Why: an unjudged unit scores as not relevant, so the bias a setup can suffer is bounded by how many more unjudged units it retrieves; with equal shares both sides are treated alike. The 5-point value is a judgement. Rough bound: if a quarter of a 0.5-unit gap were truly relevant, one side would miss about 0.125 relevant units per question. Each such unit is worth 1/log2(rank + 1) over IDCG, on average about 0.45 over IDCG across ranks 1 to 10. So the bias can reach about 0.06 nDCG@10 on a question with one gold unit, about 0.02 with five and about 0.01 with ten. If the trigger fires, L1 labels about 300 to 400 unjudged top-3 units (seed 20261120); the audit is descriptive and changes no decision. The report gives both trigger values. ## 12. Missing outcomes @@ -370,7 +370,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists ## 21. Reporting - Decisions per row and per hypothesis, estimates with unadjusted 95% intervals, SESOI and MDE lines, and the gated rows marked "not testable". -- **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed as defined; the audit trigger then uses the share not in the gold set (section 11.3). +- **Unjudged share** at 10 per setup and dataset, and condensed-list nDCG@10 (unjudged units removed; Sakai 2007) as a sensitivity for H1 to H5, where the qrels list judged non-relevant units (P4). For a dataset whose qrels hold positives only, the report states that neither can be computed as defined; the audit trigger is not used there (section 11.3). - Realised chunk-length distribution per chunk-size level and dataset. - Sensitivity tables: GEE against permutation disagreements; FinanceBench page rules; missing-outcome bounds; closed-book subset; length-controlled check; A3 cell under any-page credit; H8b on questions with at least 95% of the token budget in all three `-t20k` setups, with realised token totals per setup; faithfulness with uncited factual segments as unsupported, beside the uncited-segment share. - **Compute table**: GPU type, pod hours per stage including every pre-freeze pod (test pod, GPU timing, pilot, smoke runs), hosted tokens and cost per role, an energy estimate (GPU hours times board power; Strubell et al. 2019; Lacoste et al. 2019), `resent_calls`, error rows per stage and role, and every `out_of_credit` stop. @@ -397,7 +397,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - O1. Decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 configuration hash. - O2. Decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, with the written justification of section 10, recorded before any pilot result is read. -- O3. Decided by the owner on 2026-11-14: no relevance audit by default; it runs only if the registered unjudged-share trigger fires (section 11.3). +- O3. Decided by the owner on 2026-11-14: no relevance audit by default; it runs only if the registered unjudged-share trigger fires (section 11.3). This covers datasets with judged negatives. - O4. Decided by the owner on 2026-11-14: L2 is an author or colleague and labels every correctness answer and every Part 2 label, and a 30% subset of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1; section 11.3). The name goes in the run record only (P8). - O5. Decided by the owner on 2026-11-14: no ethics query (section 11.3). - O6. Decided by the owner on 2026-11-14: BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8, 46 contrasts); H8a and H8b hold on FinanceBench and FreshStack only. @@ -408,7 +408,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - P1. Download the full FreshStack LangChain corpus at a pinned revision (user approval at build time); record the revision, the row count and the tree hash. - P2. Time one full-corpus FreshStack index build; update the estimate before relying on it. - P3. Check whether FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's scorer; pin the scorer or record "not available". -- P4. Check, per dataset, whether the qrels list judged non-relevant units. +- P4. Check, per dataset, whether the qrels list judged non-relevant units. Then, for each dataset with positives only, the owner decides before the freeze between (i) a fixed small audit that runs whatever the results are, and (ii) no audit with the pooling bias stated as a limitation. Record the decision with its date before any main-run score exists (section 11.3). - P5. Fix the pilot cell list (at most the four H7 corner indexes per core dataset), project the pilot cost from the P2 timing, get the user's approval, run the pilot, and fill the MDE table (or apply the registered fallback of section 10). - P6. Name the smoke and test-pod answer files that feed the H8 MDE rows. - P7. Restate the checker 2-point tolerance against the MDE table. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index d94eb93f..529c8432 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -234,7 +234,7 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | --- | --- | --- | | O1 | A file upload to the registered OSF project time-stamps the A3 configuration hash | 14.2, 23 | | O2 | SESOI 0.02 nDCG@10, with a written justification (user terms, design lower bound, cost, published Qwen3 effect sizes, not set by power, sensitivity at 0.01 and 0.05). Recorded before any pilot result is read | 10, 23 | -| O3 | No blind relevance audit by default. It runs only if the mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2 | 11.3, 23 | +| O3 | No blind relevance audit by default. It runs only if the mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2. The trigger applies only where the qrels list judged non-relevant units; for positives-only datasets the owner decides at P4 between a fixed small audit and no audit | 11.3, 23 | | O4 | L2 (author or colleague) labels every correctness answer and every Part 2 label, and 30% of the faithfulness (135) and claim-kind (30) sets. L2 365, or 315 after drop step 1. Seed 20261124 retired. The name goes in the run record only | 11.3, 16, 23 | | O5 | No ethics query: the study has no human participants, and the labellers are members of the research team. The question reopens as a deviation if a paid external annotator is used | 11.3, 23 | | O7 | The primary faithfulness measure stays faithfulness of cited factual claims, because OMRS answers are built to cite their evidence. `uncited_segment_share` stays a co-outcome. The uncited-segment version stays a sensitivity analysis only | 6, 23 | diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index a8610802..3ebbe2e1 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -622,7 +622,7 @@ Written before the freeze; no measured run has started. A publication-grade meth - **FreshStack full corpus (F1; PR 4).** All 9 FreshStack indexes are built on the full public LangChain corpus (49,514 documents) at a pinned revision (P1). The download needs the user's approval at build time. The 10,000-document subset of Experiments 9a and 22 is retired for Experiment 35, and the continuity queries, continuity documents and overlay runbook are not used. No result is compared with the FreshStack leaderboard unless the corpus and the metric are the same. alpha-nDCG@10 is a secondary FreshStack metric if the nugget labels allow it (P3). One full-corpus build is timed before the estimate is relied on (P2). - **Hypotheses and families (F4, F5, F22, F23; PR 2, 7, 8).** Rows 1 to 19 of PR 8. H8 is split into H8a (rows 12 to 15, the 12 crossed setups) and H8b (rows 16 to 19, the three `-t20k` setups); both are confirmatory. "Holds" is defined per hypothesis; H4 tests simple effects; H7 is confirmatory on FinanceBench and FreshStack and a sanity check on BioASQ. Row 11 (H7) is a difference of differences with coefficients +1, -1, -1, +1 on the four corner cells after averaging over reranker and hybrid; the mean-minus-mean recipe does not apply to it (PR 9 point 1). Every question stays in H8b; "chunk size itself" also needs the registered sensitivity on questions where all three `-t20k` setups reach at least 95% of 20,480 tokens (PR 7), and realised evidence token totals are recorded per question and setup. Every hypothesis is scoped to scope S. Holm families are fixed now: 19 (FinanceBench), 19 (FreshStack), 8 (BioASQ; second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks); Legal has no family. H6, H7, H8a and H8b hold on FinanceBench and FreshStack; a two-sided hypothesis needs the same row with the same sign on each dataset its rule needs (C4). A row that fails a gate stays in its family with p = 1 and is reported "not testable", so no gate decided on main-run labels changes a family size. - **Analysis (F7, F10; PR 9).** No MixedLM. The per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells; the estimate is its plain mean. FreshStack, BioASQ and Legal: the closed-form test of PR 9 point 3 (mean of d_qk; Mancl and DeRouen bias-reduced variance, sum of squared residuals x (n / (n minus 1)) squared / n squared; t on n minus 1 degrees of freedom), implemented with `statsmodels` GEE on an intercept (independence working correlation, question clusters, `cov_type="bias_reduced"`) and checked against the closed form by the known-answer test of task 15.6. FinanceBench: document-level sign-flip permutation test, 100,000 draws, seed 20261101, interval by inverting the test. Registered sensitivity checks, never decisive: a paired permutation test by question (10,000 shuffles) on FreshStack and BioASQ, and the GEE with document clusters on FinanceBench; disagreements go in one table. No McNemar test and no equivalence test. The type I error simulation test now covers this method (task 15.6). -- **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). O3 (2026-11-14): the blind relevance audit runs only if the PR 11.3 unjudged-share trigger fires. +- **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). O3 (2026-11-14): the blind relevance audit runs only if the PR 11.3 unjudged-share trigger fires, and the trigger applies only where the qrels list judged non-relevant units. For a positives-only dataset the owner chooses at P4 between a fixed small audit and no audit. - **Missing outcomes (F9; PR 12).** Every question stays in its dataset's denominator, and every exclusion is counted per setup and row. A short list scores missing ranks as not relevant. A failed answer call follows the resend rule (D3); an answer still missing at run close scores 0 for correctness and number match and has no factual claim. Faithfulness uses complete cases per row, with two sensitivity analyses (no-claim answers scored 1, and 0). A row still unjudged at run close is counted and scored 0 in a sensitivity analysis. - **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. Every undefined resample is left out and counted once, whichever level lacks a supported claim; over 500 of 10,000 makes those rows not testable (PR 11.2). A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window of each cited chunk, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 365 (700 and 315 after drop step 1; O4: L2 labels every correctness answer and every Part 2 label, 30% of the other two sets), with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. - **Closed-book runs (F13; PR 13).** Two single runs: own-knowledge prompt (floor) and the RAG prompt with an empty evidence block. Contamination sensitivity: rows 14, 15, 18 and 19 re-run on the questions both runs answer wrongly. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index 72a3fa71..c616897e 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -470,7 +470,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.1 P1. FreshStack full corpus. Ask the user to approve the download of `freshstack/corpus-oct-2024`, config `langchain`, at a pinned revision. Then download it into `~/Development/DATA/omrs/` and record the revision, the row count (49,514 expected) and the corpus tree hash in `plan.json` and in the pins table (PR 16). Verify: the loader of task 13.5 accepts the corpus, and its check refuses a changed row count. - [ ] 16.2 P2. Time one full-corpus FreshStack index build on the chosen GPU type (part of task 9.2 or 9.3 when possible). Update the estimate in PR 17 from the measured time before relying on it. Verify: the compute table and the estimate name the measured hours; the "30 to 40 pod hours" reviewer figure is no longer the basis. - [ ] 16.3 P3. Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's own scorer. If yes, pin the scorer (version or commit) and write failing tests that a FreshStack secondary column holds it, then implement. If no, record "not available" in PR 6. Verify: PR 6 and `plan.json` say which. -- [ ] 16.4 P4. Check, per dataset, whether the qrels list judged non-relevant units. Record the answer. Then write failing report tests: the unjudged share at 10 per setup and dataset; condensed-list nDCG@10 (unjudged units removed) as a sensitivity analysis for H1 to H5 where non-relevant judgements exist; and the statement "neither can be computed" where the qrels hold positives only. Prove they fail, then implement. Verify: the tests pass and PR 21 matches the recorded answer. +- [ ] 16.4 P4. Check, per dataset, whether the qrels list judged non-relevant units. Record the answer. For each positives-only dataset, the owner then decides before the freeze: a fixed small audit that runs whatever the results are, or no audit with the pooling bias stated as a limitation; record it with its date before any main-run score exists. Then write failing report tests: the unjudged share at 10 per setup and dataset; condensed-list nDCG@10 (unjudged units removed) for H1 to H5 where non-relevant judgements exist; and the statement "neither can be computed" where the qrels hold positives only. Prove they fail, then implement. Verify: the tests pass and PR 21 matches the recorded answer. - [ ] 16.5 P5. MDE pilot. Fix the pilot cell list before the pilot runs: at most 4 indexes per core dataset, the H7 corner cells, built with the pinned settings and not reused. Project the pilot cost from the P2 timing, add it to the PR 17 estimate and ask the user. If approved: confirm seed 20261118 (task 16.9), draw 40 questions per core dataset from half A with it (after task 16.12), run retrieval only on the full-corpus indexes, with no hosted call, after a price quote and the user's go, run `mde35.py` (task 15.11) and fill the MDE table of PR 10. Verify: every pilot question ID is in half A, the cell list has a commit dated before the pilot, the table is filled, and the pilot pod hours are in the compute table. If declined: fill the table from the PR 10 fallback (assumed SDs 0.15 to 0.25, labelled assumed). Verify: the declined answer and its date are recorded, every row shows the assumed range and names PR 10 as its source, and no pilot pod hours appear. - [ ] 16.6 P6. Name the smoke and test-pod answer files that feed the H8a and H8b MDE rows, with their SHA-256. Verify: PR 10 names them and marks those rows rough. - [ ] 16.7 P7. Restate the checker 2-point tolerance against the H8a and H8b faithfulness MDE: the bar stays only if it is at most half that MDE. Verify: PR 11.1 and `protocol.md` section 4.4 give the restated rule; the "about 4 points" figure is not used. @@ -491,7 +491,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.22 Optional `e8b-c2048` rebuild check (PR 15). Run it on FinanceBench only when the cost projection after the search stage is at or below the estimate (a cost-only decision, recorded). Report the share of identical ranked lists and the nDCG@10 change; it is not a gate. Verify: the decision and, if run, the result are in the run record. - [ ] 16.23 O1 (decided by the owner on 2026-11-14): a file upload to the registered OSF project time-stamps the A3 configuration hash. The answer is recorded in PR 14.2 and PR 23. Verify: recorded before G7. - [ ] 16.24 O2 (decided by the owner on 2026-11-14): the SESOI is 0.02 nDCG@10, with the written justification in PR 10; intervals are also read against 0.01 and 0.05. Recorded in PR 10 and PR 23 before any pilot result is read. Verify: recorded before the pilot results are read. -- [ ] 16.25 O3 (decided by the owner on 2026-11-14): no blind relevance audit by default. It runs only if the PR 11.3 trigger fires (mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2). Write failing tests for the trigger, then implement it; only if it fires, add the audit sample and page (seed 20261120) to `labels35.py` with tests first. Verify: tests pass, and the trigger values are in the report. +- [ ] 16.25 O3 (decided by the owner on 2026-11-14): no blind relevance audit by default. It runs only if the PR 11.3 trigger fires (mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), and only on a dataset whose qrels list judged negatives. Write failing tests for the trigger, then implement it; only if it fires, add the audit sample and page (seed 20261120) to `labels35.py` with tests first. Verify: tests pass, and the trigger values are in the report. - [ ] 16.26 O4 (decided by the owner on 2026-11-14): L2 is an author or colleague, labels every correctness answer and every Part 2 label, and 30% of the faithfulness and claim-kind sets (PR 11.3). Before the first label, record L2's name, role and relation to the project in the run record only (P8). Verify: no public file holds the name. - [ ] 16.27 O5 (decided by the owner on 2026-11-14): no ethics query. The study has no human participants, and the labellers are members of the research team, not study subjects, so no ethics review is sought. This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation. The answer is recorded in PR 11.3 and PR 23. Verify: recorded before any label. - [ ] 16.28 O6 (decided by the owner on 2026-11-14): BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack). The answer is recorded in PR 7, 8 and 23. Verify: recorded before G7. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index 3ef9810f..fbe53f8b 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -89,12 +89,12 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 4D.1 Write amendment A4 into this change (proposal, design D0.15, specs), `plan.json` (amendment A4, hypotheses, decision rule, metrics, gates, references) and `protocol.md` section 0.13. Verify: `openspec validate experiment-35-full-document-rag-benchmark --strict` and `omrs experiments check` pass. - [ ] 4D.2 Download the full FreshStack LangChain corpus at a pinned revision after the operator approves (P1). Record the revision, row count (49,514) and tree hash. Verify: protocol section 4.1 and the pins table hold the three values. - [ ] 4D.3 Time one full-corpus FreshStack index build and update the estimate (P2). Verify: the estimate in `preregistration.md` section 17 names the measured hours. -- [ ] 4D.4 Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's scorer (P3), and whether each dataset's qrels list judged non-relevant units (P4). Verify: the scorer is pinned or recorded as "not available", and each dataset has one line. +- [ ] 4D.4 Check whether the FreshStack nugget labels allow alpha-nDCG@10 with the benchmark's scorer (P3), and whether each dataset's qrels list judged non-relevant units (P4), and for each positives-only dataset record the owner choice between a fixed small audit and no audit. Verify: the scorer is pinned or recorded as "not available", and each dataset has one line. - [ ] 4D.5 Fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the operator's approval, run the retrieval-only pilot on 40 seeded half-A questions per core dataset, and fill the MDE table (P5, P6); if the operator declines, fill it from assumed SDs of 0.15 to 0.25, labelled as assumed. Restate the checker 2-point tolerance against the table (P7). Verify: the MDE table has one row per dataset and row group, with its SD source. - [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Verify: the guidelines are committed before the freeze. (The labelling itself is task 5.7.) - [ ] 4D.7 Fix the second closed-book run (RAG prompt, empty evidence) and the length-controlled check (300 claims) in `plan.json` and the pre-registration. Verify: both are listed before the freeze. (The runs are task 5.8.) - [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if the O3 trigger fires; 20261124 retired); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. -- [ ] 4D.9 Owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open): O1 (OSF file upload time-stamps the hash), O2 (SESOI 0.02 nDCG@10, justified in `preregistration.md` section 10), O3 (no blind audit unless the section 11.3 unjudged-share trigger fires), O4 (L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets; name in the run record only), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. +- [ ] 4D.9 Owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open): O1 (OSF file upload time-stamps the hash), O2 (SESOI 0.02 nDCG@10, justified in `preregistration.md` section 10), O3 (no blind audit unless the section 11.3 unjudged-share trigger fires; it covers datasets with judged negatives), O4 (L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets; name in the run record only), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. - [ ] 4D.10 Check before the freeze that no live text in this change, `protocol.md` section 0 or `plan.json` contradicts `preregistration.md`. Verify: every hit for "MixedLM", "included whole", "gpt-6.1-terra", "operator decides" and "equivalence" is marked superseded or removed. ## 5. Run diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 47a047f5..08e467b9 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -87,8 +87,8 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. - [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. RDY 16.24 (O2) is done: the owner set the SESOI at 0.02 nDCG@10 on 2026-11-14, recorded with its justification and date in PR 10 before any pilot result is read and before the MDE table is filled. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the O2 answer is dated before any pilot result is read, the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. -- [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Owner decisions (all decided by the owner on 2026-11-14; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, justified in PR 10; O3 decided by the owner on 2026-11-14: no blind relevance audit unless the PR 11.3 unjudged-share trigger fires; O4 decided by the owner on 2026-11-14: L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets, and L2's name goes in the run record only, before the first label; O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. No owner decision is open. Verify: PR 23 shows each item decided. +- [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4, with the owner choice between a fixed small audit and no audit for each positives-only dataset), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. +- [ ] 6.10d Owner decisions (all decided by the owner on 2026-11-14; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, justified in PR 10; O3 decided by the owner on 2026-11-14: no blind relevance audit unless the PR 11.3 unjudged-share trigger fires, which covers datasets with judged negatives only; O4 decided by the owner on 2026-11-14: L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets, and L2's name goes in the run record only, before the first label; O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. No owner decision is open. Verify: PR 23 shows each item decided. ### F. Freeze From 3a7e1f56b968181a4b8381bc9f8e9f9a78f33a29 Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 19:22:30 +0000 Subject: [PATCH 8/9] docs(exp35): second labeller labels every item, with a staged fallback --- docs/proposals/first-publication-benchmark.md | 2 +- .../plan.json | 2 +- .../preregistration.md | 33 +++++++++++++------ .../protocol.md | 7 ++-- .../exp35-main-run-readiness/design.md | 8 ++--- .../changes/exp35-main-run-readiness/tasks.md | 8 ++--- .../design.md | 6 ++-- .../proposal.md | 2 +- .../specs/evaluation-observability/spec.md | 6 +++- .../tasks.md | 4 +-- .../rag-benchmark-readiness-flow/tasks.md | 2 +- 11 files changed, 49 insertions(+), 31 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index 666c46a2..41296660 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -108,7 +108,7 @@ The answer stage uses the embedder and reranker size chosen on half A (pre-regis - Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; O7, decided by the owner on 2026-11-14: the primary measure stays cited factual claims, because OMRS answers are built to cite their evidence). - Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. -- Human check: two labellers score samples blind (L1 750 labels, L2 365; L1 700 and L2 315 if drop step 1 removes Legal answers). L2 labels every correctness answer and every Part 2 label, and 30% of the faithfulness and claim-kind sets (O4). Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). +- Human check: two labellers score samples blind (L1 750 labels, L2 750; 700 each if drop step 1 removes Legal answers). By default L2 labels every item L1 labels and every Part 2 label (O4, revised on 2026-11-14), because faithfulness is analysed per chunk size and each correctness gate has only 50 answers. Only if L2 cannot commit the time, a staged fallback has L2 label a seeded 30% of the faithfulness and claim-kind sets first (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70. The plan is chosen and dated before the freeze, from L2's availability alone. Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. ### Hypotheses H1 to H8b diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 5e0f2f0a..567987c3 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 365 (every correctness answer and every Part 2 label, 30% of the faithfulness and claim-kind sets; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), a trigger defined only on datasets whose qrels list judged non-relevant units, while for positives-only datasets the owner decides at P4 between a fixed small audit and no audit; O4, L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 315 after O4; no confirmatory row changes).", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 750 in the full plan (every item L1 labels and every Part 2 label), or 365 at the staged fallback's first stage (a seeded 30% of the faithfulness and claim-kind sets, with the rest of a set added when kappa on its subset is below 0.70; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets used only in the staged plan, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), a trigger defined only on datasets whose qrels list judged non-relevant units, while for positives-only datasets the owner decides at P4 between a fixed small audit and no audit; O4 (revised the same day), L2 is an author or colleague and labels every item L1 labels and every Part 2 label (L1 750 and L2 750, or 700 each after drop step 1); only if L2 cannot commit the time, a staged fallback labels a seeded 30% of the faithfulness and claim-kind sets first (L2 365, or 315 after drop step 1) and the rest of a set when kappa on its subset is below 0.70; the plan is chosen and dated before the freeze from L2's availability alone, and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 700 in the full plan or 315 at the staged first stage, after O4; no confirmatory row changes).", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index d56c8fff..2d11f6b8 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -184,13 +184,26 @@ Why: a chance value makes an undefined resample count against a pass, and point - **Faithfulness gate, per checker.** Cohen's kappa (Sim and Wright 2005) of the checker against the human label: point estimate at least 0.60 and lower 95% bound at least 0.40. Sample: 450 claims from the crossed setups, 150 per chunk size, 50 per core dataset within each chunk size, at most one claim per answer, seed 20261116. The claims are drawn from answers to all questions (both halves), because H8a and H8b are tested on all questions. The correctness sample below stays on half B. The gate is pooled over the three core datasets because 150 claims per dataset are too few for a per-dataset gate; kappa per checker and per core dataset is reported with its interval as descriptive and decides nothing. "Can't tell" counts as not supported (the binary benchmark scale); kappa is also reported with "can't tell" items left out. If GLM-5.3 fails, rows 12, 13, 16 and 17 are not testable on every dataset. - **Differential-error rule.** "Unsupported" is the positive class. Specificity is the share of human-supported claims the checker calls supported. The rule uses GLM-5.3, the main checker; Clef-Flash specificity is reported only. If the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, H8a and H8b faithfulness (rows 12, 13, 16, 17) is exploratory and not testable. Sensitivity per chunk size is reported. -- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. If drop step 1 (section 17) removes Legal answer generation, the Legal answers are not drawn: the sample is 150 answers, the L1 total is 700, L2 labels all 150 and the L2 total is 315. Legal has no gate, so no confirmatory row changes. The Ragas score is cut at 0.5 (fixed before labelling). The human label for this gate is the agreed L1 and L2 label (section 11.3). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). +- **Correctness gate, per core dataset.** 50 questions per core dataset drawn from half B (seed 20261113); for each question, one answer from run 1 of an answer setup chosen uniformly at random by the same seed. 50 Legal answers, drawn the same way, are also labelled and reported without a gate. If drop step 1 (section 17) removes Legal answer generation, the Legal answers are not drawn: the sample is 150 answers and every label total falls by 50 (section 11.3). Legal has no gate, so no confirmatory row changes. The Ragas score is cut at 0.5 (fixed before labelling). The human label for this gate is the agreed L1 and L2 label (section 11.3). The gate passes on a dataset when kappa is at least 0.60 with a lower 95% bound of at least 0.40, computed with "can't tell" items scored both as correct and as not correct (both must pass). The continuous score is validated too: its AUC against the human label must have a lower 95% bound above 0.5. If the gate fails on a dataset, rows 14, 15, 18 and 19 are not testable there. Prevalence, raw agreement and PABAK are reported beside kappa (Feinstein and Cicchetti 1990; Byrt, Bishop and Carlin 1993). - **Length-controlled check (registered sensitivity, no gate).** 300 claims from `c2048` crossed answers (seed 20261119). GLM-5.3 and Clef-Flash judge each claim again against one 512-token window of each cited chunk: the chunk is split into consecutive 512-token windows, and the window that shares the most distinct lower-case claim words is kept (ties go to the earlier window). The report gives the verdict change rate with a Wilson interval. ### 11.3 Human labels -- **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller, an author or colleague (O4, decided by the owner on 2026-11-14). L2's name, role and relation to the project go in the run record before the first label (P8), never in public files. L2 labels every correctness answer and every Part 2 label. Reason: each correctness gate is per dataset on 50 answers, and 30% of 50 is too few to estimate human agreement per dataset; in Part 2, L1 is an OMRS author. L2 labels a 30% subset of the faithfulness set (135 of 450, seed 20261123) and of the claim-kind set (30 of 100, seed 20261122). Reason: each has one pooled gate, so a sample estimates agreement. Where both labellers label every item, the reference label for the gate is the agreed label after the disagreement rule. Kappa of L1 against L2 is reported per set, and per core dataset for correctness, with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. -- **Counts.** L1: claim kinds 100, faithfulness 450, correctness 200; 750 in all. L2: claim kinds 30, faithfulness 135, correctness 200; 365 in all. After drop step 1: correctness 150, L1 700, L2 315. Part 2 counts are set in the Part 2 protocol (P16), and L2 labels all of them. +- **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller, an author or colleague (O4, decided by the owner on 2026-11-14 and revised the same day). L2's name, role and relation to the project go in the run record before the first label (P8), never in public files. L2 labels every Part 2 item in both plans. +- **Full plan (default).** L2 labels every item L1 labels: claim kinds 100, faithfulness 450, correctness 200. Reasons: (1) the faithfulness sample is analysed by chunk size (150 per chunk size) and the differential-error rule compares 512 with 2,048, so human agreement must be measurable per chunk size, and a 30% subset would leave 45 per chunk size; (2) each correctness gate is per dataset on 50 answers; (3) refusal claims are rare, so a 30-claim subset cannot measure agreement on the refusal class; (4) the labels are released, and agreed labels are more reusable. +- **Staged fallback.** It applies only if L2 cannot commit the time for the full plan. L2 first labels a seeded 30% subset of the faithfulness set (135 claims, stratified like the full sample: 45 per chunk size, 15 per core dataset within each; seed 20261123) and of the claim-kind set (30 of 100; seed 20261122). L2 then labels the remaining 70% of a set (315 faithfulness claims, or 70 claim kinds) when the human-to-human kappa on its subset is below 0.70 overall or, for faithfulness, below 0.70 within any chunk-size level. Correctness and Part 2 stay at 100% in both plans. The value 0.70 is a judgement (a rule of thumb above the 0.60 checker gate), not a cited standard. +- **Choice of plan.** The owner chooses the plan and records it with its date before the freeze (P8), from L2's availability alone and never from results. Seeds 20261122 and 20261123 are used only in the staged plan. +- **Reference label and agreement.** Where both labellers labelled an item, the reference label for a gate is the agreed label after the disagreement rule. Where only L1 labelled it (staged plan, no escalation), it is L1's label. Kappa of L1 against L2 is reported per set, per chunk size for faithfulness and per core dataset for correctness, with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. +- **Counts.** After drop step 1, correctness is 150 and every total falls by 50. Part 2 counts are set in the Part 2 protocol (P16), and L2 labels all of them on top of these. + +| Set | L1 | L2 full plan | L2 staged, first stage | L2 staged, after both escalations | +| --- | ---: | ---: | ---: | ---: | +| Claim kinds | 100 | 100 | 30 | 100 | +| Faithfulness | 450 | 450 | 135 | 450 | +| Correctness | 200 | 200 | 200 | 200 | +| Total | 750 | 750 | 365 | 750 | +| Total after drop step 1 | 700 | 700 | 315 | 700 | + - **Guidelines.** One written guideline per set (claim kinds, faithfulness, correctness, and Part 2 correctness), with worked examples drawn from outside every sample, fixed before the first label and released (P8). Faithfulness follows the LLM-AggreFact task definition (MiniCheck): a claim is supported if and only if the cited passages support it, judged as the checker sees it, on meaning not grammar. - **Blinding.** Labellers never see scores, verdicts, setup names or chunk sizes. Cited passage length can still hint at chunk size; the report says so. A training round on practice items comes first and is not evidence. - **Ethics.** The study has no human participants. It uses public datasets, and the labellers are members of the research team, not study subjects, so no ethics review is sought (O5, decided by the owner on 2026-11-14). This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation (section 20). @@ -302,9 +315,9 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti | 20261119 | Length-controlled check (300 claims) | | 20261120 | Relevance audit, only if its trigger fires (O3, section 11.3) | | 20261121 | Factorial smoke run and test pod: setups and 5 questions per dataset | -| 20261122 | L2 subset, claim kinds (30 of 100) | -| 20261123 | L2 subset, faithfulness (135 of 450) | -| 20261124 | Retired, not reused: was the L2 correctness subset; L2 now labels every correctness answer (O4) | +| 20261122 | L2 subset, claim kinds (30 of 100); staged plan only (section 11.3) | +| 20261123 | L2 subset, faithfulness (135 of 450); staged plan only (section 11.3) | +| 20261124 | Retired, not reused: was the L2 correctness subset; L2 labels every correctness answer in both plans (O4) | | 20261125 | Faithfulness-gate kappa bootstrap (each checker, pooled and per dataset) | | 20261126 | Differential-error rule bootstrap (GLM-5.3 specificity difference) | | 20261127 | Correctness-gate kappa bootstrap | @@ -322,7 +335,7 @@ The word "sealed" applies to indexes only. Half B is held out from every selecti - **Estimate**: about **$730** on an RTX 4090 at $0.89 per hour (estimate). It is the A3 estimate of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 extra pod hours, about $30 to $40, not timed: P2), the second closed-book run (about $3 hosted and about 1 pod hour, by analogy with the first) and the pilot. The pilot cost is not yet estimated: it is projected from the P2 timing, added here before the freeze and approved by the user before the pod starts (section 10). Search cost barely changes with corpus size, because the rerank pool stays 100. - **Ceiling**: **$975**, pods and hosted calls together (1.5 x the A2 estimate, about 1.34 x this one). At 3 times slower reranking than assumed, the total is about $950 (estimate), still under the ceiling with little margin. The OpenRouter key carries a credit limit equal to the hosted share of the ceiling, so an overrun ends as a clean `out_of_credit` stop. - **Checks**: before the freeze (test pod extrapolation), on the build pods after the search stage, and at each stage end while the replicate pods run. Each projection uses billing and measured rates only, never scores. -- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. Step 1 also removes the 50 Legal correctness labels (section 11.2: correctness sample 150, L1 700, L2 315). E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. +- **Drop order at the ceiling** (results-blind, applied in order, each step recorded with the projection that triggered it): (1) Legal RAG Bench answer generation; (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. The owner approved this order on 2026-11-14. Step 1 also removes the 50 Legal correctness labels (section 11.2: correctness sample 150, L1 700, L2 700 in the full plan or 315 at the staged first stage). E1 is not in this order: it has its own ceiling (P16). No confirmatory item is dropped by choice. If the closed-book RAG-prompt run is dropped, the contamination sensitivity (section 13) uses the own-knowledge closed-book run alone, and the report says so. - **Stop rule.** The run is complete when every search setup has its run, every answer setup and dataset has 3 runs, the closed-book, oracle and length-controlled runs are done, G1 to G3 are checked, and every Ragas and checker row is judged or counted. A run that finishes every confirmatory item after one or more drop steps is "complete with registered drops": each drop is a numbered deviation (section 20). A run is "incomplete" only when a confirmatory item is missing. The run stops early only when the preflight fails, an execution-integrity gate (G1 to G3, section 15) fails, or a projection reaches the ceiling after the last drop step. A failed outcome-validation gate (faithfulness or correctness, section 11.2) does not stop the run: its rows stay in their families as "not testable" (p = 1), and the rest of the run continues. The classifier gate is decided before the freeze (section 11.1). - **Part 2 and E1** each get their own estimate and ceiling before their runs (P16). @@ -398,7 +411,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - O1. Decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 configuration hash. - O2. Decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, with the written justification of section 10, recorded before any pilot result is read. - O3. Decided by the owner on 2026-11-14: no relevance audit by default; it runs only if the registered unjudged-share trigger fires (section 11.3). This covers datasets with judged negatives. -- O4. Decided by the owner on 2026-11-14: L2 is an author or colleague and labels every correctness answer and every Part 2 label, and a 30% subset of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1; section 11.3). The name goes in the run record only (P8). +- O4. Decided by the owner on 2026-11-14 (revised the same day): L2 is an author or colleague. Full plan (default): L2 labels every item L1 labels and every Part 2 label (L1 750 and L2 750; 700 each after drop step 1). Staged fallback, only if L2 cannot commit the time: L2 first labels a seeded 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70. The plan is chosen and dated before the freeze, from L2's availability alone (section 11.3). The name goes in the run record only (P8). - O5. Decided by the owner on 2026-11-14: no ethics query (section 11.3). - O6. Decided by the owner on 2026-11-14: BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8, 46 contrasts); H8a and H8b hold on FinanceBench and FreshStack only. - O7. Decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims; the uncited-segment version is a sensitivity analysis only (section 6). @@ -412,8 +425,8 @@ After the freeze, every change is a dated, numbered deviation. The report lists - P5. Fix the pilot cell list (at most the four H7 corner indexes per core dataset), project the pilot cost from the P2 timing, get the user's approval, run the pilot, and fill the MDE table (or apply the registered fallback of section 10). - P6. Name the smoke and test-pod answer files that feed the H8 MDE rows. - P7. Restate the checker 2-point tolerance against the MDE table. -- P8. Write the label guidelines for each set, with worked examples outside every sample. Record L2's name, role and relation to the project in the run record before the first label. -- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16; 20261124 is retired), and the length-controlled sample size (300). The pilot seed is confirmed before the pilot runs (P5). Seed 20261120 (audit) is used only if the audit trigger fires (O3); confirm it then. +- P8. Write the label guidelines for each set, with worked examples outside every sample. Record L2's name, role and relation to the project in the run record before the first label. Record the chosen L2 plan (full or staged) with its date before the freeze, from L2's availability alone. +- P9. Confirm or change the new seeds 20261118, 20261119 and 20261121 to 20261135 (seeds table, section 16; 20261124 is retired; 20261122 and 20261123 are used only in the staged L2 plan), and the length-controlled sample size (300). The pilot seed is confirmed before the pilot runs (P5). Seed 20261120 (audit) is used only if the audit trigger fires (O3); confirm it then. - P10. Record the tokenizer revision, answer-model host, reasoning mode, output cap and context length, and the judge, classifier and GLM-5.3 settings (the Ragas judge and classifier seeds included) in `plan.json`. - P11. Record the query instruction, the reranker instruction text and the hybrid settings in `plan.json`. - P12. Write `halves.json`; record its SHA-256; list every smoke and pre-freeze question ID with its half. diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index 529c8432..bcdb3a69 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -165,7 +165,7 @@ Written before the freeze. No measured run has started. A publication-grade meth | F16 | Existing data declared | 5 | | F17 | Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop (E1 left the order, C6 below); "complete with registered drops" defined; "the operator decides" removed; Part 2 budget a pre-freeze task | 17 | | F18 | Release table per artefact and licence, re-judge kit | 19 | -| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 counts set (O4, decided 2026-11-14); no ethics query (O5, decided 2026-11-14) | 11.3 | +| F19 | Label guidelines per set, hidden system identity in Part 2, L2 primary in Part 2, conflict of interest; L2 counts set (O4, decided and revised 2026-11-14); no ethics query (O5, decided 2026-11-14) | 11.3 | | F20 | Pins table; index cost and configuration hash defined | 14.1, 16 | | F21 | FinanceBench split by document; pre-freeze samples from half A; order: score half A, hash, score half B | 4, 14.2 | | F22 | "Holds" defined per hypothesis; H7 confirmatory on FinanceBench and FreshStack, as H6; H4 named as simple effects | 7 | @@ -224,7 +224,7 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | Row 11 (H7) uses coefficients +1, -1, -1, +1 on the four corner cells after averaging over reranker and hybrid; the general mean-minus-mean recipe does not apply to it; known-answer fixture case (0.8, 0.2, 0.3, 0.7 gives 1.0); MDE on the same scale | 7, 9, 10 | | Short evidence pools: all 100 chunks are kept when they hold fewer than 20,480 tokens, and the realised token total is recorded per question and setup. Every question stays in H8b. "Chunk size itself" also needs a registered sensitivity on questions where all three `-t20k` setups reach at least 19,456 tokens (95%), with the count kept per dataset | 3, 7, 21 | | Every undefined resample is never drawn again and is counted once, whichever chunk-size level it lacks, also when the full sample is defined; the over-5% failure rule stays separate | 11.2 | -| Drop step 1 removes the 50 Legal correctness labels: correctness 150, L1 700, L2 210 (O4 later set L2 to 315); Legal has no gate, so no confirmatory row changes | 11.2, 16, 17 | +| Drop step 1 removes the 50 Legal correctness labels: correctness 150, L1 700, L2 210 (O4 later set L2 to 315, then to 700 in the full plan); Legal has no gate, so no confirmatory row changes | 11.2, 16, 17 | | OpenSpec only: O2 (SESOI) is asked in readiness flow step 6.10b, before any pilot result is read | none | | Rejected: move O5 before classifier-gate labelling. O5 was decided on 2026-11-14 (no ethics query), so the answer is already recorded before any label | none | @@ -235,7 +235,8 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | O1 | A file upload to the registered OSF project time-stamps the A3 configuration hash | 14.2, 23 | | O2 | SESOI 0.02 nDCG@10, with a written justification (user terms, design lower bound, cost, published Qwen3 effect sizes, not set by power, sensitivity at 0.01 and 0.05). Recorded before any pilot result is read | 10, 23 | | O3 | No blind relevance audit by default. It runs only if the mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2. The trigger applies only where the qrels list judged non-relevant units; for positives-only datasets the owner decides at P4 between a fixed small audit and no audit | 11.3, 23 | -| O4 | L2 (author or colleague) labels every correctness answer and every Part 2 label, and 30% of the faithfulness (135) and claim-kind (30) sets. L2 365, or 315 after drop step 1. Seed 20261124 retired. The name goes in the run record only | 11.3, 16, 23 | +| O4 (first decision) | Replaced by the revision below. L2 labelled every correctness answer and every Part 2 label, and 30% of the faithfulness (135) and claim-kind (30) sets. Seed 20261124 retired | 11.3, 16, 23 | +| O4 (revised the same day) | Full plan (default): L2 labels every item L1 labels and every Part 2 label (L1 750, L2 750; 700 each after drop step 1). Reason: faithfulness is analysed per chunk size (45 per size at 30%), correctness gates are per dataset on 50 answers, refusal claims are rare, and agreed labels are more reusable. Staged fallback, only if L2 cannot commit the time: L2 first labels a seeded 30% of the faithfulness (135, seed 20261123) and claim-kind (30, seed 20261122) sets (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level (0.70 is a judgement value, not a cited standard). Correctness and Part 2 stay at 100% in both plans. The plan is chosen and dated before the freeze from L2's availability alone, never from results. Seed 20261124 stays retired. The name goes in the run record only | 11.3, 16, 23 | | O5 | No ethics query: the study has no human participants, and the labellers are members of the research team. The question reopens as a deviation if a paid external annotator is used | 11.3, 23 | | O7 | The primary faithfulness measure stays faithfulness of cited factual claims, because OMRS answers are built to cite their evidence. `uncited_segment_share` stays a co-outcome. The uncited-segment version stays a sensitivity analysis only | 6, 23 | diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index 3ebbe2e1..a98fbff3 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -576,7 +576,7 @@ The operator approved every fix of an independent methods review (Exp 35 change - **Closed-book run (M6).** A `closedbook` stage: one run, all 1,160 questions, the answer model with no evidence (a fixed prompt that asks for an answer from the model's own knowledge), then `AnswerCorrectness` and the number match. It runs on replicate pod 1. **A4 (D38):** a second closed-book run uses the main RAG answer prompt with an empty evidence block, and the contamination sensitivity re-runs rows 14, 15, 18 and 19 on the questions both closed-book runs answer wrongly. - **Split halves (M3).** `split35.py` writes `halves.json` per dataset (seed 20261115). `select35.py` reads half A only. The report reads half B for answer-stage headlines. **A4 (D38):** FinanceBench is split by document, so no document sits in both halves; the other datasets are split by question. - **Clustered analysis (M2, A2c N4; replaced by A4, D38).** No MixedLM. For FreshStack, BioASQ and Legal the decision test is defined by its closed form (pre-registration section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster; the paired t-test variance times n / (n minus 1)); t reference on n minus 1 degrees of freedom. `experiments/_lib/stats_cluster.py` implements it with `statsmodels` `GEE` on per-question contrast scores (intercept only, Gaussian, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) and computes the p-value and interval from the t reference itself, because GEE results report a normal reference by default (`use_t=False` in the statsmodels source). The known-answer test of task 15.6 checks the GEE estimate and variance against the closed form; on a mismatch the closed form is used and the difference reported. The same module runs the document-level sign-flip permutation test (100,000 draws, seed 20261101) for FinanceBench; one method gives every estimate, interval and p-value (D38). It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock` (`statsmodels` 0.15.0, added 2026-11-14). The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. -- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set; O4 later retired 20261124), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750. O4 (2026-11-14): L2 labels 30% of claim kinds (30) and faithfulness (135) and all 200 correctness answers, so L2 365; the correctness gate uses the agreed label. +- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set; O4 later retired 20261124), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750. O4 (2026-11-14, revised the same day): in the full plan (default) L2 labels every item L1 labels, so L2 750; in the staged fallback L2 labels 30% of claim kinds (30) and faithfulness (135) and all 200 correctness answers first (365), then the rest of a set when kappa on its subset is below 0.70; the plan is chosen before the freeze from L2's availability alone; the gate reference is the agreed label, or L1's where only L1 labelled. - **Provenance (M7).** Each hosted call row records `model_returned` and `provider_returned` from the reply (OpenRouter `model` and `provider`; Z.AI `model`), beside the requested values. The join counts differences per role. - **Reranker check (M8).** `rerank_check35.py` scores 200 seeded pairs (seed 20261117) per size with llama-server and with the transformers recipe, and reports Spearman correlation; below 0.95 the file is not pinned. @@ -587,7 +587,7 @@ The operator approved every fix of an independent methods review (Exp 35 change | Depth 100 retrieval with `fetch_k` 100 passed, so the rerank pool stays 100 (without it, 300 and three times the rerank hours); dense and hybrid top 100 cost seconds | about 0 | $0 | | Two token-matched answer setups (6,960 answers at 2,048-level context: answers, five Ragas metrics, classifier; judges and Ragas on the replicate pods) | +36 (about +12 per replicate pod) | +$47 | | Closed-book run (1,160 short answers plus correctness) | +1 | +$3 | -| Human labels (400 L1, 120 L2; A4 raises them to 750 L1 and 365 L2, D38 and O4) | 0 | $0 | +| Human labels (400 L1, 120 L2; A4 raises them to 750 L1 and 750 L2 or 365 at the staged first stage, D38 and O4) | 0 | $0 | | **A2 total** | **about 378 pod hours, about $350 with the volume** | **about $290** | | Oracle run (D36) and A2c (no cost) | +4 | +$8 | | **Final total** | **about 382 pod hours, about $354 with the volume** | **about $298** | @@ -624,11 +624,11 @@ Written before the freeze; no measured run has started. A publication-grade meth - **Analysis (F7, F10; PR 9).** No MixedLM. The per-question contrast score d_qk is the mean over first-level cells minus the mean over second-level cells; the estimate is its plain mean. FreshStack, BioASQ and Legal: the closed-form test of PR 9 point 3 (mean of d_qk; Mancl and DeRouen bias-reduced variance, sum of squared residuals x (n / (n minus 1)) squared / n squared; t on n minus 1 degrees of freedom), implemented with `statsmodels` GEE on an intercept (independence working correlation, question clusters, `cov_type="bias_reduced"`) and checked against the closed form by the known-answer test of task 15.6. FinanceBench: document-level sign-flip permutation test, 100,000 draws, seed 20261101, interval by inverting the test. Registered sensitivity checks, never decisive: a paired permutation test by question (10,000 shuffles) on FreshStack and BioASQ, and the GEE with document clusters on FinanceBench; disagreements go in one table. No McNemar test and no equivalence test. The type I error simulation test now covers this method (task 15.6). - **Metrics (F11, F12, F24; PR 6, 21).** nDCG@10, Recall@10 and MRR@10 follow the trec_eval convention (ideal DCG over min(R_q, 10), plain mean over questions). `evidence_recall_20k` uses any-page credit on FinanceBench; first-page credit is a registered sensitivity analysis. The report gives the unjudged share at 10 per setup and dataset, and condensed-list nDCG@10 for H1 to H5 where the qrels list judged non-relevant units (P4). O3 (2026-11-14): the blind relevance audit runs only if the PR 11.3 unjudged-share trigger fires, and the trigger applies only where the qrels list judged non-relevant units. For a positives-only dataset the owner chooses at P4 between a fixed small audit and no audit. - **Missing outcomes (F9; PR 12).** Every question stays in its dataset's denominator, and every exclusion is counted per setup and row. A short list scores missing ranks as not relevant. A failed answer call follows the resend rule (D3); an answer still missing at run close scores 0 for correctness and number match and has no factual claim. Faithfulness uses complete cases per row, with two sensitivity analyses (no-claim answers scored 1, and 0). A row still unjudged at run close is counted and scored 0 in a sensitivity analysis. -- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. Every undefined resample is left out and counted once, whichever level lacks a supported claim; over 500 of 10,000 makes those rows not testable (PR 11.2). A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window of each cited chunk, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 365 (700 and 315 after drop step 1; O4: L2 labels every correctness answer and every Part 2 label, 30% of the other two sets), with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. +- **Judges and labels (F3, F8, F19, F26; PR 11).** Faithfulness gate per checker: kappa point estimate at least 0.60 and lower 95% bound at least 0.40, on 450 claims (150 per chunk size, 50 per core dataset within each, seed 20261116), drawn from answers to all questions in both halves because H8a and H8b use all questions; the correctness sample stays on half B. Every gate interval is a percentile bootstrap (10,000 resamples by question, own seed). Differential-error rule: if the 95% interval of the GLM-5.3 specificity difference (2,048 minus 512) excludes zero in either direction, rows 12, 13, 16 and 17 are not testable. Every undefined resample is left out and counted once, whichever level lacks a supported claim; over 500 of 10,000 makes those rows not testable (PR 11.2). A failed Clef-Flash gate marks its results not validated and changes no row. Correctness gate per core dataset: D33 as revised. Length-controlled check: 300 `c2048` claims (seed 20261119) judged again against the best-matching 512-token window of each cited chunk, verdict change rate with a Wilson interval; registered sensitivity, no gate. Gate decisions are recorded with a time before any H8a or H8b contrast result is computed. Label counts: L1 750, L2 750 in the full plan (700 each after drop step 1), or 365 at the staged first stage (315 after drop step 1; O4: L2 labels every correctness answer and every Part 2 label in both plans), with written guidelines per set (P8). The checker 2-point tolerance is restated against the MDE table (P7). The `openai/gpt-6.1-terra` cross-check is removed, because no cost estimate includes it; P15 checks that no file still names it. This change never ran it. - **Closed-book runs (F13; PR 13).** Two single runs: own-knowledge prompt (floor) and the RAG prompt with an empty evidence block. Contamination sensitivity: rows 14, 15, 18 and 19 re-run on the questions both runs answer wrongly. - **Halves, order of work and selection (F14, F20, F21, F27; PR 4, 14).** FinanceBench is split by document. Every pre-freeze sample drawn after `halves.json` exists (the pilot, the RAGFlow checks) comes from half A; earlier samples are listed with their IDs and halves (PR 5). Order: build and seal the 9 indexes and pass G2; run the 72 search setups on all questions; score half A only; apply the selection rule and write `selection.json`; time-stamp the configuration hash on OSF (O1); only then score half B and start the replicate pods. The harness enforces it (task 16.17). "Sealed" applies to indexes only. Index cost = chunks times embedding dimension, summed over the three core datasets. Configuration hash = SHA-256 of the canonical JSON of the selected cell's resolved settings, with the `halves.json` SHA-256 as one field. RQ1 retrieval headline = the A3 cell on half B; the 72-setup table is descriptive. The search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, without replacement, ascending sort). - **Pilot and MDE (F6; PR 10).** Retrieval only, before the freeze: 40 seeded half-A questions per core dataset (seed 20261118) on full-corpus indexes, through a cell list fixed before the pilot runs (P5). No hosted calls. SESOI 0.02 nDCG@10 (O2, decided by the owner on 2026-11-14, justified in PR 10). Experiment 22 values are not used. -- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation, which also removes the 50 Legal correctness labels (correctness 150, L1 700, L2 315; no gate, so no confirmatory row changes); (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. +- **Budget and drop order (F17, F28; PR 17, 21).** Estimate about $730 (estimate): the A3 figure of about $690 plus the full FreshStack corpus (reviewer estimate about 30 to 40 pod hours, about $30 to $40, not timed: P2), the second closed-book run and the pilot (projected from the P2 timing and approved before the pod; at most the 4 H7 corner indexes per core dataset, not reused). Ceiling $975. Drop order when a projection reaches the ceiling, results-blind and in order: (1) Legal RAG Bench answer generation, which also removes the 50 Legal correctness labels (correctness 150, L1 700, L2 700 in the full plan or 315 at the staged first stage; no gate, so no confirmatory row changes); (2) the length-controlled check; (3) the closed-book run with the RAG prompt; then stop. E1 has its own ceiling and is not in this order. A run with every confirmatory item done is "complete with registered drops". Part 2 and E1 get their own estimates and ceilings (P16). The compute table includes every pre-freeze pod. - **Pins, release, determinism, existing data (F16, F18, F20, F25; PR 5, 15, 16, 19).** One pins table (P10 to P14). A release table per artefact and licence, and a re-judge kit. One build per cell is assumed; an optional `e8b-c2048` rebuild check on FinanceBench runs only when the cost projection after the search stage is at or below the estimate, and is reported, not gated. The existing-data table lists every earlier look; P12 lists each smoke and pre-freeze question ID with its half. Harness consequences (tasks): `datasets35.py` loads the pinned full FreshStack corpus (13.5); `scoring35.py` gives any-page `evidence_recall_20k` and the formulas (15.1, 15.14); `split35.py` splits FinanceBench by document (15.5); `stats_cluster.py` and `factorial35.py` build the clustered analysis, the fixed families and the gated rows (14.4, 15.6, 15.7); `correct35.py` and `labels35.py` build the gates and the counts (14.6, 15.8, 16.18); the closed-book stage runs two prompts (15.4); the missing-outcome rules (8.9); the length-controlled check, the half-B guard, the unjudged-share tables and the report items (group 16). Pre-freeze tasks P1 to P16 and the owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open) are group 16. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index c616897e..d66c2368 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -425,7 +425,7 @@ Architecture invariant 11: one new file and one `register()` line, no branch on - [ ] 14.2 Write failing tests that each answer row records `completion_calls`, the evidence characters, and the truncated passages and characters dropped, computed with OMRS's own `_fit_and_measure` on the same rows and settings. Prove they fail, then implement in `answers35.py`. Verify: the tests pass, and a row with cut evidence is still saved. - [ ] 14.3 Selection rule (`select35.py`, design D31). Write failing tests on fixture scores: a gap of exactly 0.01 picks the smaller size; a gap of 0.011 picks the best; equal scores pick the smaller size; the pooled score is the plain mean of the FinanceBench, FreshStack and BioASQ dataset means (not question-weighted; amendment A3); a fixture that changes only the Legal RAG Bench scores changes neither the selected embedder nor the reranker size nor the cell; the four-factor step picks the cell in tie order reranker off, hybrid off, lowest index cost, smaller chunk size; the reranker size is chosen at the chosen embedder on reranker-on setups only; a missing setup stops and names it; `selection.json` lists every pooled score, both chosen sizes, the A3 cell and the configuration hash. Then (A4, pre-registration section 14.1) tests that: index cost is the number of chunks times the embedding dimension, summed over the three core datasets, read from the sealed indexes with no score; the configuration hash is the SHA-256 of the canonical JSON (keys sorted, no spaces, UTF-8) of the selected cell's resolved settings with the SHA-256 of `halves.json` as one field, and the same inputs give the same hash twice; the any-page sensitivity repeats step 3 with any-page nDCG@10 on FinanceBench, records whether the cell changes, and never changes the registered cell. Prove they fail, then implement. Verify: the tests pass, and the answer stage refuses to start without `selection.json`. - [ ] 14.5 Ragas `AnswerCorrectness` (design D33). Write failing tests with a stub LLM and stub embeddings: the `ragas` stage scores five metrics, `AnswerCorrectness` gets the question, the answer and the dataset's reference answer (never the contexts), uses the frozen judge and the local 4B embedder, and records the weights; an answer without a reference records "not judged" and sends no call; resume skips saved rows. Prove they fail, then implement in `ragas35.py` with Ragas 0.4.3. Verify: the tests pass, and the version check still refuses another Ragas version. -- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B (150 with no Legal draw when drop step 1 applies, PR 11.2 and 17), each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 and L2 both label the whole sample, and the gate uses the agreed label (task 15.8; O4). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). +- [ ] 14.6 Correctness gate per core dataset (design D33 as revised by D38; pre-registration section 11.2). Write failing tests for `correct35.py`: `sample` with seed 20261113 gives the same 200 answers twice, 50 questions per dataset (the three core datasets and Legal) from half B (150 with no Legal draw when drop step 1 applies, PR 11.2 and 17), each with the run-1 answer of an answer setup the seed picks uniformly, with no Ragas score in the output; the labelling page shows the question, the reference and the answer and saves a JSON file; `agree` cuts the Ragas score at 0.5 and gives per core dataset `cohen_kappa` with a percentile bootstrap 95% interval (10,000 resamples by question, seed 20261127), computed twice with "can't tell" scored as correct and as not correct; the gate passes on a dataset only when both computations give a point estimate of at least 0.60 and a lower bound of at least 0.40; the continuous score's AUC against the human label must have a lower 95% bound above 0.5 (same bootstrap, seed 20261128); prevalence, raw agreement and PABAK sit beside kappa; undefined statistics fail closed (PR 11.2: a one-class sample fails the gate, an undefined resample takes kappa 0 or AUC 0.5 and is counted, over 5% undefined fails the gate); Legal is reported with no gate; a failed gate marks rows 14, 15, 18 and 19 "not testable" on that dataset (p = 1, task 15.7); no pooled kappa decides anything. Prove they fail, then implement. Then, after the main run, L1 and L2 both label the whole sample (full plan), and the gate uses the reference label of pre-registration section 11.3 (task 15.8; O4). Verify: the tests pass, `protocol.md` holds the gate in a commit before labelling, and the gate is decided before any H8a or H8b contrast result is computed (C35 task 4A.4a). - [ ] 14.4 Factorial analysis (`factorial35.py`, design D31 and D38; pre-registration sections 7, 8 and 14.3). H6 and H7 on FinanceBench use `evidence_recall_20k` (any-page credit), with nDCG@10 under both page rules as sensitivity and their sign agreement per row reported; H8a and H8b use all questions; the RQ1 headline is the A3 cell on half B. Write failing tests on a fixture with a planted embedder effect: only H1's rows count; Holm runs within one dataset over the fixed family of task 15.7; a significant wrong-sign row does not count; "holds" follows the per-hypothesis rule of section 7 (H1 in full when both steps hold, in part when one does; H2, H3 and H5 on at least 2 of 3 core datasets; H4 per dataset when both rows count, overall on at least 2 of 3; H6, H7, H8a and H8b on both FinanceBench and FreshStack, H8a and H8b per outcome; a two-sided hypothesis holds only when the same row counts with the same sign on each dataset its rule needs, and opposite signs are reported as a dataset-dependent effect); H8a rows use the 12 crossed setups and H8b rows the 3 `-t20k` setups, with the PR 7 sensitivity on questions at 19,456 tokens or more in all three; BioASQ rows 9 to 19 carry `sanity_check: true`; answer scores average the 3 runs per question first; Legal rows carry `exploratory: true`; the RQ1 retrieval headline is the A3 cell's nDCG@10 on half B per core dataset with a 95% interval, and the 72-setup table on all questions is marked descriptive; the search-cost figure uses the exact expected-maximum estimator (adapted from Dodge et al. 2019, drawn without replacement) on pooled half-A scores sorted in ascending order, one curve over the 12 A3 cells and one over the 72 setups, and a RAGFlow curve only if the matched search ran. Prove they fail, then implement with `stats_cluster.py` (task 15.6) and the existing functions. Verify: the tests pass, and `git diff` shows no new function in `experiments/_lib/stats.py` or `stats_d6.py` other than `cohen_kappa` (task 8.8). ## 15. Methods-review fixes (pre-freeze amendment A2, 2026-11-13; design D35) @@ -453,7 +453,7 @@ Each test is proved to fail on the current code before the fix. - type I error simulation, now for this method: with no planted effect and unequal error variance across cells, the rejection rate at 0.05 stays near 0.05 over 200 simulations for the GEE (FreshStack-sized data) and for the sign-flip test (84 documents, about 1.8 questions each); a fixture with complete data gives the same estimate as the cell-means contrast of the full-factorial model. - The `stats` dependency group exists (decided by the operator, 2026-11-13; added on 2026-11-14 with `statsmodels` 0.15.0 in `pyproject.toml` and `uv.lock`, and `{include-group = "stats"}` in the `dev` group). The wrapper and its tests are not built yet, so this box stays open. A test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. - [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); L2 labels 30% of the claim-kind and faithfulness sets (30 and 135, seeds 20261122 and 20261123) and all 200 correctness answers (365 in all; 315 after drop step 1; seed 20261124 retired, O4); the gate reference for a fully double-labelled set is the agreed label; no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 (per set, and per core dataset for correctness) comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the plan is chosen and dated before the freeze (task 16.26); in the full plan (default) L2 labels every item L1 labels (750; 700 after drop step 1); in the staged fallback L2 labels a seeded 30% of the claim-kind and faithfulness sets first (30 and 135, seeds 20261122 and 20261123) and all 200 correctness answers (365 in all; 315 after drop step 1), then the other 70% of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level; seed 20261124 retired, O4; the gate reference is the agreed label where both labelled an item and L1's label where only L1 did; no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 (per set, and per core dataset for correctness) comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.9 Hosted-call provenance (M7). Write failing tests: each answer, Ragas, GLM-5.3 and classifier row records `model_returned` and `provider_returned` from the stubbed reply; the join counts rows whose returned model differs from the requested one per role. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.10 Reranker rank check (M8). Write failing tests for `rerank_check35.py`: 200 pairs drawn with seed 20261117 are the same twice; Spearman below 0.95 marks the file "not pinned". Prove they fail, then implement. Run it on the test pod for each size (task 9.2 step 2a). Verify: the tests pass and `protocol.md` lists the correlation per file. - [ ] 15.11 MDE from the pilot (M1; A4, pre-registration section 10). Write `mde35.py` on saved files only: per core dataset and contrast type, the SD of the per-question contrast scores from the retrieval-only pilot (task 16.5, P5; an adjacent-level contrast uses the extreme-level SD as a stand-in, marked so), and on FinanceBench the design effect of document clustering; MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha 0.05/19 on FinanceBench and FreshStack and 0.05/8 on BioASQ, and n divided by the design effect on FinanceBench; answer-stage rows use the SD from the smoke and test-pod answer files named in task 16.6 (P6) and are labelled rough; each row is shown against the SESOI (0.02 nDCG@10, open item O2). Experiment 22 values are not used. Write failing tests on a fixture first. Verify: the tests pass and the MDE table of pre-registration section 10 is filled before the freeze. @@ -475,7 +475,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.6 P6. Name the smoke and test-pod answer files that feed the H8a and H8b MDE rows, with their SHA-256. Verify: PR 10 names them and marks those rows rough. - [ ] 16.7 P7. Restate the checker 2-point tolerance against the H8a and H8b faithfulness MDE: the bar stays only if it is at most half that MDE. Verify: PR 11.1 and `protocol.md` section 4.4 give the restated rule; the "about 4 points" figure is not used. - [ ] 16.8 P8. Write one label guideline per set (claim kinds, faithfulness, correctness, Part 2 correctness), with worked examples drawn from outside every sample. Faithfulness follows the LLM-AggreFact task definition. Fix each guideline before its first label. Verify: the guidelines are committed before any label, and no worked example is in a sample (a test of `labels35.py` checks the IDs). -- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Confirm the pilot seed before the pilot runs (task 16.5). Seed 20261120 (audit) is used only if the O3 trigger fires (task 16.25); confirm it then. Seed 20261124 is retired (O4). Verify: PR 16 and `plan.json` hold the confirmed values. +- [ ] 16.9 P9. Ask the user to confirm or change the seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (seeds table of PR 16), and the length-controlled sample size of 300. Confirm the pilot seed before the pilot runs (task 16.5). Seed 20261120 (audit) is used only if the O3 trigger fires (task 16.25); confirm it then. Seed 20261124 is retired (O4); seeds 20261122 and 20261123 are used only in the staged L2 plan. Verify: PR 16 and `plan.json` hold the confirmed values. - [ ] 16.10 P10. Record in `plan.json`: the tokenizer revision; the answer-model host, reasoning mode, output cap and context length (task 14.1a); the Ragas judge and classifier host and settings; the GLM-5.3 settings. Verify: each "Missing: P10" cell of PR 16 points to a `plan.json` key. - [ ] 16.11 P11. Record in `plan.json`: the query instruction (empty string), the reranker instruction text, and the hybrid settings (RRF constant, per-side fusion depth, BM25 tokeniser and parameters). Write a failing test that the run record of a search stage holds the same values, then implement. Verify: the test passes. - [ ] 16.12 P12. Run `split35.py` (task 15.5), write `halves.json`, record its SHA-256 in `plan.json`, and list every smoke and pre-freeze question ID with its half in the existing-data table (PR 5). Verify: the hash in `plan.json` equals the file, and every listed ID has a half. @@ -492,7 +492,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.23 O1 (decided by the owner on 2026-11-14): a file upload to the registered OSF project time-stamps the A3 configuration hash. The answer is recorded in PR 14.2 and PR 23. Verify: recorded before G7. - [ ] 16.24 O2 (decided by the owner on 2026-11-14): the SESOI is 0.02 nDCG@10, with the written justification in PR 10; intervals are also read against 0.01 and 0.05. Recorded in PR 10 and PR 23 before any pilot result is read. Verify: recorded before the pilot results are read. - [ ] 16.25 O3 (decided by the owner on 2026-11-14): no blind relevance audit by default. It runs only if the PR 11.3 trigger fires (mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), and only on a dataset whose qrels list judged negatives. Write failing tests for the trigger, then implement it; only if it fires, add the audit sample and page (seed 20261120) to `labels35.py` with tests first. Verify: tests pass, and the trigger values are in the report. -- [ ] 16.26 O4 (decided by the owner on 2026-11-14): L2 is an author or colleague, labels every correctness answer and every Part 2 label, and 30% of the faithfulness and claim-kind sets (PR 11.3). Before the first label, record L2's name, role and relation to the project in the run record only (P8). Verify: no public file holds the name. +- [ ] 16.26 O4 (decided by the owner on 2026-11-14): L2 is an author or colleague. Full plan (default, revised the same day): L2 labels every item L1 labels and every Part 2 label. Staged fallback, only if L2 cannot commit the time: a seeded 30% of the faithfulness and claim-kind sets first, then the rest of a set when kappa on its subset is below 0.70 (PR 11.3). Before the freeze, record the chosen plan with its date, from L2's availability alone. Before the first label, record L2's name, role and relation to the project in the run record only (P8). Verify: no public file holds the name. - [ ] 16.27 O5 (decided by the owner on 2026-11-14): no ethics query. The study has no human participants, and the labellers are members of the research team, not study subjects, so no ethics review is sought. This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation. The answer is recorded in PR 11.3 and PR 23. Verify: recorded before any label. - [ ] 16.28 O6 (decided by the owner on 2026-11-14): BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack). The answer is recorded in PR 7, 8 and 23. Verify: recorded before G7. - [ ] 16.29 Uncited segments (PR 6). Write failing tests: the join counts per answer the prose segments with no valid ordinal (the segments `split_claims` drops) and gives `uncited_segment_share` per setup and dataset; `claimkinds` also sorts uncited segments with the frozen classifier (question and segment only); the sensitivity counts uncited fact and mixed segments as unsupported; a fixture answer with one supported cited fact and one uncited fact scores 1.0 primary and 0.5 sensitivity. Prove they fail, then implement. Verify: the tests pass. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md index cd669ea6..8559d8ae 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/design.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/design.md @@ -184,7 +184,7 @@ FinanceBench page credit: (for `evidence_recall_20k`, superseded by A4, D0.15: a **M3. Split-half selection.** (Amended by A4, D0.15: FinanceBench is split by document; the others by question.) Each dataset's questions are split into halves A and B with seed 20261115 (stratified by dataset). The embedder and reranker size for the answer stage are chosen on half A only, with the D0.5 rule. All answer-stage headline results are reported on half B. The answer stage still runs on all questions; half A results are secondary. Search-stage hypotheses use all questions. -**M4. Human validation.** (Superseded by A4, D0.15: L1 labels 750 items, L2 365 (O4); gates and samples in `preregistration.md` section 11.) Two labellers, recorded as L1 (the operator) and L2 (the second labeller). No names in public records. +**M4. Human validation.** (Superseded by A4, D0.15: L1 labels 750 items, L2 750 in the full plan or 365 at the staged first stage (O4); gates and samples in `preregistration.md` section 11.) Two labellers, recorded as L1 (the operator) and L2 (the second labeller). No names in public records. | Set | Size (L1) | Drawn from | Stratified by | L2 share | |---|---:|---|---|---:| @@ -470,7 +470,7 @@ Decision (operator, 2026-11-13). The master plan is `docs/proposals/first-public - **Hand-off.** Part 2 runs on half B only, once per system. Experiment 35 does not run Part 2, does not run RAGFlow, and does not draw labels from RAGFlow answers. Part 2 has its own protocol (C1, C2, fairness rules, labels). - **Token-matched control (PR 209 review, 2026-11-13).** The factorial cell `c2048` x 10 keeps its 10-chunk evidence. A separate 15th answer setup, `c2048-t20k`, cuts evidence at 20,480 real Qwen3 tokens at 2,048-token chunks, reranker on and hybrid on, beside `c512-t20k` and `c1024-t20k`. Each chunk size has a token-matched cell, and the token-matched comparison of H8 uses the three `-t20k` setups. Registered before the freeze. It amends D0.13 N2, which cut the `c2048` comparison by the same rule. Cost: the answer stage grows by about 1/14 (about 7%). Counted from the readiness design D35 table (two token-matched setups: +36 pod hours, +$47 hosted), one more setup adds about $24 of hosted calls (answers, five Ragas metrics, judges and classifier) and about 18 pod hours (about $16 at $0.89 per hour, about 6 hours per replicate pod), so about $40 in all. The estimate moves from about $650 to about $690. The ceiling stays $975 (1.5 x the old estimate, 1.4 x the new one). At 3 times slower reranking the total is about $910, still under it. - **FinanceBench unit in Part 2 (2026-11-13).** Part 2 scores C1 at page level for every FinanceBench question and both systems, the unit used here (first page of a chunk). Pre-freeze gate and document-level fallback are in the master plan, Step 3 rule 3, and protocol section 0.12 item 4. -- **Counts.** The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each. The 3 runs stay as in D0.12. (Labels superseded by A4, D0.15: 750 L1 and 365 L2 after O4.) +- **Counts.** The answer stage runs 15 setups (12 crossed plus 3 token-matched), 3 runs each. The 3 runs stay as in D0.12. (Labels superseded by A4, D0.15: 750 L1 and 750 L2 in the full plan, or 365 L2 at the staged first stage, after O4.) - **Framework comparison.** Out of Experiment 35 scope. Part 2 holds it. ### D0.15 Pre-freeze amendment A4: publication review fixes (2026-11-14) @@ -483,7 +483,7 @@ Decision (operator, 2026-11-14). A publication-grade methods review gave 30 find - F1: all 9 FreshStack indexes use the full public LangChain corpus (49,514 documents, pinned revision). The 10,000-document subset is retired. alpha-nDCG@10 is secondary if the nugget labels allow it. - F4, F7, F10: no mixed model. Per-question contrast scores. On FreshStack, BioASQ and Legal the decision test is defined by its closed form (mean; Mancl and DeRouen bias-reduced variance; t on n minus 1), with `statsmodels` GEE as the implementation checked against it (`preregistration.md` section 9 point 3); and a document-level sign-flip test on FinanceBench. Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; second round C3). A gated row stays in its family with p = 1. No equivalence claim, no TOST, no McNemar test as a main test. - F5, F22, F23: H8 splits into H8a (crossed setups) and H8b (token-matched setups). Both are confirmatory. Every hypothesis has scope S, numbered rows and a holds rule. H7 is confirmatory on FinanceBench and FreshStack. After the second round (C3, C4), H6, H7, H8a and H8b hold only on FinanceBench and FreshStack, and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs. Legal RAG Bench is exploratory in every hypothesis. - - F3, F8, F19 (counts after drop step 1: L1 700, L2 315 after O4; second CodeRabbit round): L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels 365 (O4, 2026-11-14). Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. + - F3, F8, F19 (counts after drop step 1: L1 700, L2 700 in the full plan or 315 at the staged first stage, after O4; second CodeRabbit round): L1 labels 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 labels the same 750 in the full plan, or 365 first in the staged fallback (O4, 2026-11-14, revised the same day). Faithfulness gate per checker and correctness gate per core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A differential-error rule and a length-controlled check guard H8a and H8b faithfulness. - F12: `evidence_recall_20k` gives any-page credit and is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11). nDCG@10 stays the primary metric of the study. First-page credit is a sensitivity. - F13: a second closed-book run uses the RAG prompt with empty evidence. - F14, F21: FinanceBench is split by document. Order: score half A, write `selection.json`, time-stamp the configuration hash on OSF, then score half B. The RQ1 retrieval headline is the A3 cell on half B. "Sealed" applies to indexes only. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md index b830f9c2..a58fd6ef 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md @@ -26,7 +26,7 @@ A publication-grade methods review gave 30 findings, F1 to F30. The operator app - FreshStack indexes use the full public LangChain corpus (49,514 documents). The 10,000-document subset is retired. - H8 splits into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory. Every hypothesis is scoped, with numbered rows and a holds rule. Legal RAG Bench is exploratory everywhere. - No mixed model. Per-question contrasts with GEE, and a document-level sign-flip test on FinanceBench. Holm families fixed at 19, 19 and 8 (second round C3; BioASQ rows 9 to 19 are sanity checks). -- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items and L2 labels 365 (every correctness answer, 30% of the other two sets; O4). +- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items. L2 labels the same 750 (full plan, default), or 365 first in a staged fallback (every correctness answer, a seeded 30% of the other two sets, then the rest of a set when kappa on its subset is below 0.70; O4). - `evidence_recall_20k` gives any-page credit. A second closed-book run uses the RAG prompt. FinanceBench is split by document. - The `openai/gpt-6.1-terra` cross-check is removed. Budget about $730 plus the pilot, ceiling $975, with a results-blind drop order that no longer holds E1 (E1 has its own ceiling). diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index f439b3ef..ce0292f7 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -153,7 +153,11 @@ Each faithfulness checker SHALL pass when Cohen's kappa against the human label ### Requirement: Human labels follow the registered counts -L1 SHALL label 750 items (claim kinds 100, faithfulness 450 from all questions, correctness 200 from half B), and L2 SHALL label 30% of claim kinds and faithfulness and all correctness answers (365 items); after drop step 1, correctness 150, L1 700 and L2 315. The claim classifier SHALL reach agreement of at least 0.90 and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. +L1 SHALL label 750 items (claim kinds 100, faithfulness 450, correctness 200). By default L2 SHALL label every item L1 labels. In a staged fallback, chosen and dated before the freeze from L2's availability alone, L2 SHALL label a seeded 30% of claim kinds and faithfulness first (365 items in all), then the rest of a set when kappa on its subset is below 0.70. The claim classifier SHALL reach agreement and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. + +#### Scenario: L2 cannot commit the time for the full plan +- **WHEN** the owner records the staged plan before the freeze +- **THEN** L2 SHALL label a seeded 30% of claim kinds and faithfulness first, and every correctness and Part 2 item #### Scenario: A label file is released - **WHEN** human labels are released diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index fbe53f8b..5e376058 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -66,7 +66,7 @@ Design D0.12. The code work is readiness task group 15. Where 4A tasks differ, 4 - [x] 4B.1 Write amendment A2 into this change (design D0.12, specs), `protocol.md` (section 0.10) and `plan.json` (amendment A2, metrics, hypotheses). Verify: both `openspec validate --strict` runs and `omrs experiments check` pass. Done 2026-11-13. - [ ] 4B.2 Compute the minimum detectable effects from data (M1, readiness task 15.11) and write the table with its sources into `protocol.md` section 0.10. Verify: the table is in a commit before the freeze commit. (Amended by A4: the table lives in `preregistration.md` section 10; see 4D.5.) - [ ] 4B.3 Pin every reranker GGUF with its SHA-256 (M8) only after its Spearman check (at least 0.95 on 200 seeded pairs) passes on the test pod. Verify: `plan.json` runtime holds three reranker entries, `load_runtime_block` accepts them, and `protocol.md` lists each correlation. -- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 365; see 5.7.) +- [ ] 4B.4 Label the four human-validation sets (M4): refusal gate 100, faithfulness 200, correctness 100 (L1), and 30% of each by L2. Write the disagreement rule into `protocol.md` before the first label. Verify: the kappa tables with intervals name their `output/` files, and no labeller name appears in any record. (Superseded by A4: L1 750 and L2 750, or 365 at the staged first stage; see 5.7.) - [ ] 4B.5 Run the closed-book run (M6) on replicate pod 1 and report correctness by retrieval hit or miss. Verify: the report table names its `output/` file. (Amended by A4: two closed-book runs; see 5.8.) - [ ] 4B.5a Run the oracle run (design D0.12, readiness tasks 15.12 and 15.13) on replicate pod 1 and report it beside closed-book as ceiling and floor, descriptive only. Verify: the table names its `output/` file and states the no-distractor caveat. - [ ] 4B.6 Deposit `protocol.md` and `plan.json` on OSF at the freeze commit (M9; the operator does the deposit). Verify: a commit after the freeze, before the first measured run, adds the OSF link and the freeze commit hash to `protocol.md`, and the deposit holds no personal data beyond the operator name. (Amended by A4: `preregistration.md` is the main file of the deposit; the OSF link and freeze commit go into `plan.json` `frozen`.) @@ -105,7 +105,7 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 5.4 Run the determinism check for every arm, and the rebuild check (4B baseline index rebuilt once per dataset in a fresh process). Verify: the output states match or mismatch per arm and dataset, and for each rebuild. - [ ] 5.5 Run Ragas on each arm where the dataset supports each metric. Verify: unsupported metrics show "not measured". - [ ] 5.6 Check about 50 judged items by hand. Verify: the agreement is computed with the measure frozen in 1.10. -- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 365 (claim kinds 30, faithfulness 135, correctness 200; O4), with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. +- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 750 (full plan, default; O4, revised 2026-11-14), or L2 365 first in the staged fallback (claim kinds 30, faithfulness 135, correctness 200, then the rest of a set when kappa on its subset is below 0.70), with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. - [ ] 5.8 (A4, moved from 4D.7.) Run the second closed-book run (RAG prompt, empty evidence) beside the first, and the length-controlled check on 300 claims. Verify: each table names its `output/` file and is marked descriptive or sensitivity. ## 6. Analyse and report diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 08e467b9..799b5798 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -88,7 +88,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime - [ ] 6.10a FreshStack full corpus (PR 4; F1). RDY 16.1 (P1): ask the user to approve the download, then pin the revision, row count (49,514) and tree hash. RDY 16.2 (P2): time one full-corpus index build and update the estimate in PR 17. RDY 16.3 (P3): alpha-nDCG@10 scorer or "not available". The 10,000-document subset and the overlay of step 6.1 are retired. On a pod, quote the price first and delete the pod when done. Verify: `plan.json` holds the pin, and the estimate names the measured build hours. - [ ] 6.10b Halves and MDE pilot (PR 4, PR 10; F6, F21). RDY 16.12 (P12): write `halves.json` (FinanceBench split by document) and list every smoke and pre-freeze question ID with its half. RDY 16.24 (O2) is done: the owner set the SESOI at 0.02 nDCG@10 on 2026-11-14, recorded with its justification and date in PR 10 before any pilot result is read and before the MDE table is filled. Then RDY 16.5 (P5): fix the pilot cell list (at most the four H7 corner indexes per core dataset, not reused), project the pilot cost from the P2 timing and get the user's approval; if approved, first confirm or change the pilot seed 20261118 (RDY 16.9, P9; the rest of P9 stays in step 6.10c), then run the retrieval-only pilot on 40 seeded half-A questions per core dataset (no hosted call), and fill the MDE table (or the PR 10 fallback if the user declines); RDY 16.6 (P6) and 16.7 (P7). Quote the pod price first. Verify: the O2 answer is dated before any pilot result is read, the seed answer is recorded before the pilot pod starts, every pilot question is in half A, and the MDE table is filled. - [ ] 6.10c Remaining pre-freeze tasks (PR 23). RDY 16.4 (P4, with the owner choice between a fixed small audit and no audit for each positives-only dataset), 16.8 to 16.11 (P8 to P11; the pilot seed of P9 is already confirmed in step 6.10b), 16.13 to 16.15 (P13 to P15), and 16.16 (P16, through steps 7.1, 7.2 and 7.2a). Also the existing readiness tasks PR 23 names: reranker pins, GPU choice, classifier gate, Legal revision, answer-model context length, test pod session. Verify: every "Missing" cell of the PR 16 pins table points to a recorded value. -- [ ] 6.10d Owner decisions (all decided by the owner on 2026-11-14; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, justified in PR 10; O3 decided by the owner on 2026-11-14: no blind relevance audit unless the PR 11.3 unjudged-share trigger fires, which covers datasets with judged negatives only; O4 decided by the owner on 2026-11-14: L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets, and L2's name goes in the run record only, before the first label; O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. No owner decision is open. Verify: PR 23 shows each item decided. +- [ ] 6.10d Owner decisions (all decided by the owner on 2026-11-14; RDY 16.23 to 16.28 and 16.30). O1 decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 hash; O2 decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, justified in PR 10; O3 decided by the owner on 2026-11-14: no blind relevance audit unless the PR 11.3 unjudged-share trigger fires, which covers datasets with judged negatives only; O4 decided by the owner on 2026-11-14: L2 is an author or colleague, labels every item L1 labels and every Part 2 label (a staged fallback labels a seeded 30% of the other two sets first, only if L2 cannot commit the time; revised the same day), and L2's name and the chosen plan go in the run record only, before the first label; O5 decided by the owner on 2026-11-14: no ethics query, because the study has no human participants and the labellers are members of the research team (a paid external annotator reopens the question as a deviation); O6 decided by the owner on 2026-11-14: BioASQ rows 12 to 19 as sanity checks (families 19 / 19 / 8, H8a and H8b on FinanceBench and FreshStack); O7 decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited-segment version stays a sensitivity analysis only (RDY 16.30). Record each answer; never guess one. No owner decision is open. Verify: PR 23 shows each item decided. ### F. Freeze From 1ea9318bdd139f7a818c4960ac814f1aafe536ab Mon Sep 17 00:00:00 2001 From: Aizat Hawari Date: Sat, 14 Nov 2026 19:47:04 +0000 Subject: [PATCH 9/9] docs(exp35): address PR 210 review round 3 - Staged fallback: an undefined subset kappa counts as below 0.70, so L2 labels the rest of that set (pre-registration 11.3 and mirrors). - Evaluation-observability spec: per-chunk-size escalation and the undefined case as scenarios, matching 11.3. - Task 4D.9 and the readiness proposal: full plan (L2 750) is the default; 365 is only the staged first stage. - Readiness flow: delete the factorial smoke pod and its disk at G7 at the latest, with a check that none remains. - protocol.md 0.13: CodeRabbit round 3 note. --- docs/proposals/first-publication-benchmark.md | 2 +- .../35-full-rag-benchmark-2026-11-01/plan.json | 2 +- .../preregistration.md | 4 ++-- .../35-full-rag-benchmark-2026-11-01/protocol.md | 11 ++++++++++- openspec/changes/exp35-main-run-readiness/design.md | 2 +- openspec/changes/exp35-main-run-readiness/proposal.md | 2 +- openspec/changes/exp35-main-run-readiness/tasks.md | 4 ++-- .../proposal.md | 2 +- .../specs/evaluation-observability/spec.md | 10 +++++++++- .../tasks.md | 4 ++-- .../changes/rag-benchmark-readiness-flow/design.md | 2 +- .../changes/rag-benchmark-readiness-flow/tasks.md | 6 +++--- 12 files changed, 34 insertions(+), 17 deletions(-) diff --git a/docs/proposals/first-publication-benchmark.md b/docs/proposals/first-publication-benchmark.md index 41296660..341eb601 100644 --- a/docs/proposals/first-publication-benchmark.md +++ b/docs/proposals/first-publication-benchmark.md @@ -108,7 +108,7 @@ The answer stage uses the embedder and reranker size chosen on half A (pre-regis - Faithfulness of cited factual claims: claim checks. GLM-5.3 is the main checker; Clef-Flash (local) is the second. The share of answer segments with no valid citation is a co-outcome, and a sensitivity analysis counts uncited factual segments as unsupported (pre-registration section 6; O7, decided by the owner on 2026-11-14: the primary measure stays cited factual claims, because OMRS answers are built to cite their evidence). - Correctness: Ragas `AnswerCorrectness` with `openai/gpt-oss-120b` as judge, plus a number match on FinanceBench. There is no cross-check judge: `openai/gpt-6.1-terra` is removed, because no cost estimate includes it. -- Human check: two labellers score samples blind (L1 750 labels, L2 750; 700 each if drop step 1 removes Legal answers). By default L2 labels every item L1 labels and every Part 2 label (O4, revised on 2026-11-14), because faithfulness is analysed per chunk size and each correctness gate has only 50 answers. Only if L2 cannot commit the time, a staged fallback has L2 label a seeded 30% of the faithfulness and claim-kind sets first (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70. The plan is chosen and dated before the freeze, from L2's availability alone. Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). +- Human check: two labellers score samples blind (L1 750 labels, L2 750; 700 each if drop step 1 removes Legal answers). By default L2 labels every item L1 labels and every Part 2 label (O4, revised on 2026-11-14), because faithfulness is analysed per chunk size and each correctness gate has only 50 answers. Only if L2 cannot commit the time, a staged fallback has L2 label a seeded 30% of the faithfulness and claim-kind sets first (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined. The plan is chosen and dated before the freeze, from L2's availability alone. Each checker must pass a faithfulness gate, and the correctness judge must pass a gate on each core dataset: kappa point estimate at least 0.60 and lower 95% bound at least 0.40. A row whose measure fails its gate stays in its Holm family as "not testable". Every gate interval is a percentile bootstrap (10,000 resamples by question). A differential-error rule makes H8a and H8b faithfulness exploratory if GLM-5.3 specificity differs between 512 and 2,048 chunks in either direction. If Clef-Flash fails its gate, its results are reported as not validated and change no confirmatory row (pre-registration section 11). - Other Ragas metrics are secondary. Ragas is not the main faithfulness measure. ### Hypotheses H1 to H8b diff --git a/experiments/35-full-rag-benchmark-2026-11-01/plan.json b/experiments/35-full-rag-benchmark-2026-11-01/plan.json index 567987c3..31884d65 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/plan.json +++ b/experiments/35-full-rag-benchmark-2026-11-01/plan.json @@ -380,7 +380,7 @@ "id": "A4", "date": "2026-11-14", "before_first_run": true, - "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 750 in the full plan (every item L1 labels and every Part 2 label), or 365 at the staged fallback's first stage (a seeded 30% of the faithfulness and claim-kind sets, with the rest of a set added when kappa on its subset is below 0.70; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets used only in the staged plan, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), a trigger defined only on datasets whose qrels list judged non-relevant units, while for positives-only datasets the owner decides at P4 between a fixed small audit and no audit; O4 (revised the same day), L2 is an author or colleague and labels every item L1 labels and every Part 2 label (L1 750 and L2 750, or 700 each after drop step 1); only if L2 cannot commit the time, a staged fallback labels a seeded 30% of the faithfulness and claim-kind sets first (L2 365, or 315 after drop step 1) and the rest of a set when kappa on its subset is below 0.70; the plan is chosen and dated before the freeze from L2's availability alone, and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 700 in the full plan or 315 at the staged first stage, after O4; no confirmatory row changes).", + "summary": "Publication review fixes F1 to F30, approved by the operator on 2026-11-14, and the second-round fixes C1 to C24 ruled on the same day. From A4 on, preregistration.md governs the live design and wins over protocol.md, this file and the OpenSpec changes; earlier amendments are history. Hypotheses: H1 to H7 scoped (scope S) with numbered rows and a holds rule each; H4 named as simple effects; H6, H7, H8a and H8b hold only on FinanceBench and FreshStack (H8a and H8b: open item O6, decided by the operator on 2026-11-14), and a two-sided hypothesis needs the same row with the same sign on every dataset its rule needs; H8 split into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory; every null is a zero effect; Legal RAG Bench exploratory everywhere, H8a and H8b correctness included. Analysis: the mixed model is dropped; per-question contrast scores with an intercept-only GEE (bias-reduced covariance, t on G minus 1) on FreshStack, BioASQ and Legal, and a document-level sign-flip test (100,000 draws, seed 20261101) on FinanceBench; Holm families fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ; rows 9 to 19 are sanity checks there), 46 contrasts; a gated row stays in its family with p = 1. Metrics: nDCG@10 is the primary metric of the study; evidence_recall_20k is the confirmatory endpoint for H6 and H7 on FinanceBench (rows 9 to 11) with any-page credit, so both carry primary: true in metrics (first-page credit is a sensitivity); alpha-nDCG@10 is secondary on FreshStack if its nugget labels allow (P3). FreshStack: all 9 indexes rebuilt on the full public LangChain corpus (49,514 documents, pinned revision, P1); the 10,000-document subset is retired. Labels and gates: L1 750 (claim kinds 100, faithfulness 450 at 150 per chunk size from answers to all questions, correctness 200 at 50 per dataset), L2 750 in the full plan (every item L1 labels and every Part 2 label), or 365 at the staged fallback's first stage (a seeded 30% of the faithfulness and claim-kind sets, with the rest of a set added when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined; O4); faithfulness gate per checker (pooled over the core datasets) and correctness gate per core dataset on kappa point estimate (at least 0.60) and lower 95% bound (at least 0.40), plus an AUC check on the continuous correctness score; every gate interval a percentile bootstrap with 10,000 resamples by question and its own seed; differential-error rule on GLM-5.3 in either direction; a length-controlled check on 300 claims. Runs: two closed-book runs (own-knowledge prompt, and the RAG prompt with empty evidence) beside the oracle run; H8a and H8b correctness rows re-run on questions both closed-book runs answer wrongly. Halves and order: FinanceBench split by document, the other datasets by question (seed 20261115); score half A, write selection.json, time-stamp the A3 configuration hash on OSF (O1, decided by the owner on 2026-11-14), then score half B; the RQ1 retrieval headline is the A3 cell on half B. Selection terms: index cost is stored vector values (chunks times embedding dimension) summed over the three core datasets; the configuration hash is the SHA-256 of the canonical JSON of the selected cell's resolved settings plus the halves.json SHA-256. The openai/gpt-6.1-terra cross-check is removed, because no cost estimate included it; its reference entry is deleted. New seeds 20261118 (pilot), 20261119 (length-controlled check) and 20261121 to 20261135 (smoke sample, L2 subsets used only in the staged plan, gate bootstraps, RAGFlow samples, Part 2 labels; preregistration.md section 16) wait for confirmation (P9); 20261120 (audit) is used only if the O3 unjudged-share trigger fires; 20261124 is retired. Budget: estimate about $730 plus the pilot (the full FreshStack build is not timed yet, P2; the pilot cost is projected from P2 and approved before the pod starts), ceiling $975 unchanged. Results-blind drop order at the ceiling: (1) Legal RAG Bench answer generation, (2) the length-controlled check, (3) the closed-book run with the RAG prompt, then stop; E1 has its own ceiling; a run that finishes every confirmatory item after drops is complete with registered drops, each a numbered deviation; no operator choice. Part 2 and E1 get their own estimate and ceiling (P16). The frozen list stays empty until the freeze; the pins table in preregistration.md section 16 fills it (answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; FreshStack and Legal revisions; halves.json and prompt template hashes; the evidence_recall_20k tolerance; the freeze commit and OSF link). These are pre-freeze tasks P1 to P16 and are not filled here; no owner decision is open. After the freeze, every change is a dated, numbered deviation, added to this list and uploaded to OSF. CodeRabbit round 1 on PR 210 (2026-11-14, before the freeze): faithfulness_glm_factual is named as faithfulness of cited factual claims; uncited_segment_share is a co-outcome, and a registered sensitivity counts uncited factual segments as unsupported (O7, decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims, and the uncited version stays a sensitivity analysis only); undefined gate statistics fail closed (preregistration.md section 11.2); the run stops early only on preflight, G1 to G3 or the ceiling after the last drop step, and a failed outcome-validation gate leaves its rows not testable; the 2-of-3 same-sign rule is stated once for C1 and C2; Part 2 C2 uses complete pairs with a sensitivity that scores unjudged rows 0; the half-B guard checks the hash of the current selection file. Owner decisions of 2026-11-14: O1, a file upload to the registered OSF project time-stamps the A3 configuration hash; O5, no ethics query, because the study has no human participants and every labeller is an author or colleague (a paid external annotator reopens the question as a deviation); O7, the primary faithfulness measure stays cited factual claims, because OMRS answers are built to cite their evidence; O2, SESOI 0.02 nDCG@10 with a written justification (preregistration.md section 10), recorded before any pilot result is read; O3, no relevance audit unless the mean unjudged share in the top 10 differs by more than 5 points (hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), a trigger defined only on datasets whose qrels list judged non-relevant units, while for positives-only datasets the owner decides at P4 between a fixed small audit and no audit; O4 (revised the same day), L2 is an author or colleague and labels every item L1 labels and every Part 2 label (L1 750 and L2 750, or 700 each after drop step 1); only if L2 cannot commit the time, a staged fallback labels a seeded 30% of the faithfulness and claim-kind sets first (L2 365, or 315 after drop step 1) and the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined; the plan is chosen and dated before the freeze from L2's availability alone, and the name goes in the run record only. CodeRabbit round 2 (PR 210): row 11 (H7) uses coefficients +1, -1, -1, +1 on its four corner cells, and its MDE uses that scale; every question stays in H8b, realised evidence token totals are recorded per question and setup, and 'chunk size itself' also needs the registered sensitivity on questions where all three -t20k setups reach at least 95% of 20,480 tokens; every undefined resample is left out (or takes its chance value) and counted once, separately from the over-5% failure rule; if drop step 1 applies, no Legal correctness answer is drawn (correctness 150, L1 700, L2 700 in the full plan or 315 at the staged first stage, after O4; no confirmatory row changes).", "where": "preregistration.md (governs from A4); protocol.md section 0.13; change experiment-35-full-document-rag-benchmark design D0.15 and tasks 4D" } ] diff --git a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md index 2d11f6b8..3a2d4b8c 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/preregistration.md @@ -191,7 +191,7 @@ Why: a chance value makes an undefined resample count against a pass, and point - **Labellers.** L1 is the operator, an OMRS author. L2 is a second labeller, an author or colleague (O4, decided by the owner on 2026-11-14 and revised the same day). L2's name, role and relation to the project go in the run record before the first label (P8), never in public files. L2 labels every Part 2 item in both plans. - **Full plan (default).** L2 labels every item L1 labels: claim kinds 100, faithfulness 450, correctness 200. Reasons: (1) the faithfulness sample is analysed by chunk size (150 per chunk size) and the differential-error rule compares 512 with 2,048, so human agreement must be measurable per chunk size, and a 30% subset would leave 45 per chunk size; (2) each correctness gate is per dataset on 50 answers; (3) refusal claims are rare, so a 30-claim subset cannot measure agreement on the refusal class; (4) the labels are released, and agreed labels are more reusable. -- **Staged fallback.** It applies only if L2 cannot commit the time for the full plan. L2 first labels a seeded 30% subset of the faithfulness set (135 claims, stratified like the full sample: 45 per chunk size, 15 per core dataset within each; seed 20261123) and of the claim-kind set (30 of 100; seed 20261122). L2 then labels the remaining 70% of a set (315 faithfulness claims, or 70 claim kinds) when the human-to-human kappa on its subset is below 0.70 overall or, for faithfulness, below 0.70 within any chunk-size level. Correctness and Part 2 stay at 100% in both plans. The value 0.70 is a judgement (a rule of thumb above the 0.60 checker gate), not a cited standard. +- **Staged fallback.** It applies only if L2 cannot commit the time for the full plan. L2 first labels a seeded 30% subset of the faithfulness set (135 claims, stratified like the full sample: 45 per chunk size, 15 per core dataset within each; seed 20261123) and of the claim-kind set (30 of 100; seed 20261122). L2 then labels the remaining 70% of a set (315 faithfulness claims, or 70 claim kinds) when the human-to-human kappa on its subset is below 0.70 overall or, for faithfulness, below 0.70 within any chunk-size level. An undefined subset kappa fails closed: it counts as below 0.70, so L2 labels the rest of that set (for example, when both labellers mark all 30 claim kinds as fact; same reason as section 11.2). Correctness and Part 2 stay at 100% in both plans. The value 0.70 is a judgement (a rule of thumb above the 0.60 checker gate), not a cited standard. - **Choice of plan.** The owner chooses the plan and records it with its date before the freeze (P8), from L2's availability alone and never from results. Seeds 20261122 and 20261123 are used only in the staged plan. - **Reference label and agreement.** Where both labellers labelled an item, the reference label for a gate is the agreed label after the disagreement rule. Where only L1 labelled it (staged plan, no escalation), it is L1's label. Kappa of L1 against L2 is reported per set, per chunk size for faithfulness and per core dataset for correctness, with a bootstrap interval (section 11.2 method). Disagreement rule: L1 and L2 discuss; if they cannot agree, the item is "can't tell" and counted. - **Counts.** After drop step 1, correctness is 150 and every total falls by 50. Part 2 counts are set in the Part 2 protocol (P16), and L2 labels all of them on top of these. @@ -411,7 +411,7 @@ After the freeze, every change is a dated, numbered deviation. The report lists - O1. Decided by the owner on 2026-11-14: a file upload to the registered OSF project time-stamps the A3 configuration hash. - O2. Decided by the owner on 2026-11-14: the SESOI is 0.02 nDCG@10, with the written justification of section 10, recorded before any pilot result is read. - O3. Decided by the owner on 2026-11-14: no relevance audit by default; it runs only if the registered unjudged-share trigger fires (section 11.3). This covers datasets with judged negatives. -- O4. Decided by the owner on 2026-11-14 (revised the same day): L2 is an author or colleague. Full plan (default): L2 labels every item L1 labels and every Part 2 label (L1 750 and L2 750; 700 each after drop step 1). Staged fallback, only if L2 cannot commit the time: L2 first labels a seeded 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70. The plan is chosen and dated before the freeze, from L2's availability alone (section 11.3). The name goes in the run record only (P8). +- O4. Decided by the owner on 2026-11-14 (revised the same day): L2 is an author or colleague. Full plan (default): L2 labels every item L1 labels and every Part 2 label (L1 750 and L2 750; 700 each after drop step 1). Staged fallback, only if L2 cannot commit the time: L2 first labels a seeded 30% of the faithfulness and claim-kind sets (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined. The plan is chosen and dated before the freeze, from L2's availability alone (section 11.3). The name goes in the run record only (P8). - O5. Decided by the owner on 2026-11-14: no ethics query (section 11.3). - O6. Decided by the owner on 2026-11-14: BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8, 46 contrasts); H8a and H8b hold on FinanceBench and FreshStack only. - O7. Decided by the owner on 2026-11-14: the primary measure stays faithfulness of cited factual claims; the uncited-segment version is a sensitivity analysis only (section 6). diff --git a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md index bcdb3a69..b5650982 100644 --- a/experiments/35-full-rag-benchmark-2026-11-01/protocol.md +++ b/experiments/35-full-rag-benchmark-2026-11-01/protocol.md @@ -228,6 +228,15 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | OpenSpec only: O2 (SESOI) is asked in readiness flow step 6.10b, before any pilot result is read | none | | Rejected: move O5 before classifier-gate labelling. O5 was decided on 2026-11-14 (no ethics query), so the answer is already recorded before any label | none | +**CodeRabbit round 3 (PR 210, 2026-11-14).** Four threads on commit 3a7e1f56, all fixed before the freeze: + +| Change | Pre-registration section | +| --- | --- | +| Staged fallback: an undefined subset kappa (for example, both labellers mark all 30 claim kinds as fact) counts as below 0.70, so L2 labels the rest of that set | 11.3, 23 | +| OpenSpec only: the evaluation-observability spec states the per-chunk-size escalation and the undefined case as scenarios, matching section 11.3 | none | +| OpenSpec only: task 4D.9 and the readiness proposal state the full plan (L2 750) as the default and 365 only as the staged first stage | none | +| OpenSpec only: the readiness flow deletes the factorial smoke pod and its disk at G7 at the latest (task 6.11), with a check that no smoke pod or volume remains | none | + **Owner decisions (2026-11-14).** Six open items are closed before the freeze (O6 is in the table above). No owner decision is open; the pre-freeze tasks P1 to P16 remain. | Item | Decision | Pre-registration section | @@ -236,7 +245,7 @@ C9, C10, C11, C20 and C23 change only the OpenSpec changes and the proposal (A4 | O2 | SESOI 0.02 nDCG@10, with a written justification (user terms, design lower bound, cost, published Qwen3 effect sizes, not set by power, sensitivity at 0.01 and 0.05). Recorded before any pilot result is read | 10, 23 | | O3 | No blind relevance audit by default. It runs only if the mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2. The trigger applies only where the qrels list judged non-relevant units; for positives-only datasets the owner decides at P4 between a fixed small audit and no audit | 11.3, 23 | | O4 (first decision) | Replaced by the revision below. L2 labelled every correctness answer and every Part 2 label, and 30% of the faithfulness (135) and claim-kind (30) sets. Seed 20261124 retired | 11.3, 16, 23 | -| O4 (revised the same day) | Full plan (default): L2 labels every item L1 labels and every Part 2 label (L1 750, L2 750; 700 each after drop step 1). Reason: faithfulness is analysed per chunk size (45 per size at 30%), correctness gates are per dataset on 50 answers, refusal claims are rare, and agreed labels are more reusable. Staged fallback, only if L2 cannot commit the time: L2 first labels a seeded 30% of the faithfulness (135, seed 20261123) and claim-kind (30, seed 20261122) sets (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level (0.70 is a judgement value, not a cited standard). Correctness and Part 2 stay at 100% in both plans. The plan is chosen and dated before the freeze from L2's availability alone, never from results. Seed 20261124 stays retired. The name goes in the run record only | 11.3, 16, 23 | +| O4 (revised the same day) | Full plan (default): L2 labels every item L1 labels and every Part 2 label (L1 750, L2 750; 700 each after drop step 1). Reason: faithfulness is analysed per chunk size (45 per size at 30%), correctness gates are per dataset on 50 answers, refusal claims are rare, and agreed labels are more reusable. Staged fallback, only if L2 cannot commit the time: L2 first labels a seeded 30% of the faithfulness (135, seed 20261123) and claim-kind (30, seed 20261122) sets (L2 365, or 315 after drop step 1), then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined (it then counts as below 0.70; 0.70 is a judgement value, not a cited standard). Correctness and Part 2 stay at 100% in both plans. The plan is chosen and dated before the freeze from L2's availability alone, never from results. Seed 20261124 stays retired. The name goes in the run record only | 11.3, 16, 23 | | O5 | No ethics query: the study has no human participants, and the labellers are members of the research team. The question reopens as a deviation if a paid external annotator is used | 11.3, 23 | | O7 | The primary faithfulness measure stays faithfulness of cited factual claims, because OMRS answers are built to cite their evidence. `uncited_segment_share` stays a co-outcome. The uncited-segment version stays a sensitivity analysis only | 6, 23 | diff --git a/openspec/changes/exp35-main-run-readiness/design.md b/openspec/changes/exp35-main-run-readiness/design.md index a98fbff3..e90f5efc 100644 --- a/openspec/changes/exp35-main-run-readiness/design.md +++ b/openspec/changes/exp35-main-run-readiness/design.md @@ -576,7 +576,7 @@ The operator approved every fix of an independent methods review (Exp 35 change - **Closed-book run (M6).** A `closedbook` stage: one run, all 1,160 questions, the answer model with no evidence (a fixed prompt that asks for an answer from the model's own knowledge), then `AnswerCorrectness` and the number match. It runs on replicate pod 1. **A4 (D38):** a second closed-book run uses the main RAG answer prompt with an empty evidence block, and the contamination sensitivity re-runs rows 14, 15, 18 and 19 on the questions both closed-book runs answer wrongly. - **Split halves (M3).** `split35.py` writes `halves.json` per dataset (seed 20261115). `select35.py` reads half A only. The report reads half B for answer-stage headlines. **A4 (D38):** FinanceBench is split by document, so no document sits in both halves; the other datasets are split by question. - **Clustered analysis (M2, A2c N4; replaced by A4, D38).** No MixedLM. For FreshStack, BioASQ and Legal the decision test is defined by its closed form (pre-registration section 9 point 3): estimate = mean of d_qk; variance = sum of squared residuals x (n / (n minus 1)) squared / n squared (Mancl and DeRouen bias-reduced sandwich, intercept only, one row per cluster; the paired t-test variance times n / (n minus 1)); t reference on n minus 1 degrees of freedom. `experiments/_lib/stats_cluster.py` implements it with `statsmodels` `GEE` on per-question contrast scores (intercept only, Gaussian, independence working correlation, clusters = questions, `cov_type="bias_reduced"`) and computes the p-value and interval from the t reference itself, because GEE results report a normal reference by default (`use_t=False` in the statsmodels source). The known-answer test of task 15.6 checks the GEE estimate and variance against the closed form; on a mismatch the closed form is used and the difference reported. The same module runs the document-level sign-flip permutation test (100,000 draws, seed 20261101) for FinanceBench; one method gives every estimate, interval and p-value (D38). It adds `statsmodels` as a PEP 735 dependency group `stats` in `[dependency-groups]` of `pyproject.toml`, the same mechanism as the `ragas` group; never an extra and never in the app install; `dev` includes it with `{include-group = "stats"}` so the fast suite runs its tests (decided by the operator, 2026-11-13), version pinned in `uv.lock` (`statsmodels` 0.15.0, added 2026-11-14). The `AGENTS.md` hard boundary on new dependencies is met: the operator decided this on 2026-11-13. The experiments rule "use `_lib/stats.py`, write no new statistics code" is kept by calling the library, not writing an estimator; the wrapper is tested against a fixture with a known effect. -- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set; O4 later retired 20261124), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750. O4 (2026-11-14, revised the same day): in the full plan (default) L2 labels every item L1 labels, so L2 750; in the staged fallback L2 labels 30% of claim kinds (30) and faithfulness (135) and all 200 correctness answers first (365), then the rest of a set when kappa on its subset is below 0.70; the plan is chosen before the freeze from L2's availability alone; the gate reference is the agreed label, or L1's where only L1 labelled. +- **Labelling (M4).** One labelling page for three sets (claim kinds, faithfulness verdicts, correctness), labeller id `L1` or `L2` chosen on the page, never a name. `labels35.py` draws the samples (seeds 20261112, 20261116, 20261113), the L2 subsets (30%, seed plus 1; A4 second round: seeds 20261122 to 20261124, one per set; O4 later retired 20261124), and computes kappa with bootstrap intervals (`cohen_kappa` and `paired_bootstrap` resampling; A4 second round: percentile bootstrap by question, 10,000 resamples, seeds of PR 16). **A4 (D38):** the sizes are claim kinds 100, faithfulness 450 (150 per chunk size, 50 per core dataset within each, from answers to all questions in both halves) and correctness 200 (50 per dataset, half B): L1 750. O4 (2026-11-14, revised the same day): in the full plan (default) L2 labels every item L1 labels, so L2 750; in the staged fallback L2 labels 30% of claim kinds (30) and faithfulness (135) and all 200 correctness answers first (365), then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined; the plan is chosen before the freeze from L2's availability alone; the gate reference is the agreed label, or L1's where only L1 labelled. - **Provenance (M7).** Each hosted call row records `model_returned` and `provider_returned` from the reply (OpenRouter `model` and `provider`; Z.AI `model`), beside the requested values. The join counts differences per role. - **Reranker check (M8).** `rerank_check35.py` scores 200 seeded pairs (seed 20261117) per size with llama-server and with the transformers recipe, and reports Spearman correlation; below 0.95 the file is not pinned. diff --git a/openspec/changes/exp35-main-run-readiness/proposal.md b/openspec/changes/exp35-main-run-readiness/proposal.md index 681ff491..cd2c37d7 100644 --- a/openspec/changes/exp35-main-run-readiness/proposal.md +++ b/openspec/changes/exp35-main-run-readiness/proposal.md @@ -29,7 +29,7 @@ Two problems sit in the OMRS app, not in the harness. A rate-limited answer come - Datasets (design D29): a Legal RAG Bench loader, the 203-query FreshStack set on the full public corpus (A4), and a FinanceBench number-match check. - Operator decisions of 2026-11-13: the FreshStack corpus drops the continuity documents and the overlay runbook; BioASQ and Legal passages are written as `.md`, so all four datasets count Qwen3 tokens (test-checked); the Ragas stage adds `AnswerCorrectness` (Ragas 0.4.3) against reference answers on all four datasets, checked against human labels by Cohen's kappa (design D33; A4 makes it a gate per core dataset on 50 labels each, D38); every answer is one round (`answer.context_window` 131,072, `answer.max_rounds` 1); each search setup runs once (design D34). Budget then: about $650, ceiling $975. - Methods-review fixes of 2026-11-13 (pre-freeze amendment A2; design D35; task group 15): unit-level scoring on the top 100 chunks collapsed to 10 distinct units; token-matched answer setups (512 x 40, 1,024 x 20; 14 answer setups, 15 with `c2048-t20k` added by A3); a closed-book run; split-half selection; `statsmodels` as an experiment-only dependency (approved); two labellers with a 30% L2 overlap; returned model and provider per hosted call; reranker Spearman check before pinning; MDE from data. (The A2 mixed model, the 400 L1 items and the $650 estimate are replaced by A4 below.) -- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 365 labels with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open) are tasks. +- Publication-review fixes of 2026-11-14 (pre-freeze amendment A4; design D38; task group 16). The new file `experiments/35-full-rag-benchmark-2026-11-01/preregistration.md` governs Experiment 35; where this change and that file disagree, the file wins. The change builds: FreshStack indexes on the full public corpus (49,514 documents, pinned revision; the 10,000-document subset retired); H8 split into H8a and H8b; one clustered analysis (GEE with independence working correlation and bias-reduced covariance, t on G minus 1; a document-level sign-flip test on FinanceBench), with no mixed model and Holm families fixed at 19, 19 and 8 (second round C3 and O6, decided by the owner on 2026-11-14: BioASQ rows 9 to 19 are sanity checks) with gated rows kept at p = 1; FinanceBench halves split by document; the order "score half A, write `selection.json`, time-stamp the hash on OSF, then score half B"; any-page credit for `evidence_recall_20k`; correctness and faithfulness gates on point estimate and lower bound, a differential-error rule and a length-controlled check; a second closed-book run with the RAG prompt and a contamination sensitivity; missing-outcome rules; L1 750 and L2 750 labels by default (L2 365 first only in the staged fallback, O4) with written guidelines; a retrieval-only MDE pilot on half A; the results-blind drop order at the ceiling; pins, release and report items. The `openai/gpt-6.1-terra` cross-check is removed. Budget: about $730 (estimate), ceiling $975. Pre-freeze tasks P1 to P16 and the owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open) are tasks. - Answer context budget (design D30): a test proves that the largest evidence set of every answer setup synthesises without the earlier context overflow, and each answer row records its rounds, evidence characters and truncation. - Selection rule and factorial statistics (design D31): a script applies the pre-registered embedder and reranker selection rule to saved search scores, and the analysis computes the marginal contrasts of H1 to H8b with the existing statistics functions. - The harness reads the `rate_limited` record instead of parsing error text, and stops at once on `out_of_credit`. Every hybrid-on setup pins `retrieval.hybrid_sparse_backend: bm25`, as the protocol defines it. diff --git a/openspec/changes/exp35-main-run-readiness/tasks.md b/openspec/changes/exp35-main-run-readiness/tasks.md index d66c2368..a9118e80 100644 --- a/openspec/changes/exp35-main-run-readiness/tasks.md +++ b/openspec/changes/exp35-main-run-readiness/tasks.md @@ -453,7 +453,7 @@ Each test is proved to fail on the current code before the fix. - type I error simulation, now for this method: with no planted effect and unequal error variance across cells, the rejection rate at 0.05 stays near 0.05 over 200 simulations for the GEE (FreshStack-sized data) and for the sign-flip test (84 documents, about 1.8 questions each); a fixture with complete data gives the same estimate as the cell-means contrast of the full-factorial model. - The `stats` dependency group exists (decided by the operator, 2026-11-13; added on 2026-11-14 with `statsmodels` 0.15.0 in `pyproject.toml` and `uv.lock`, and `{include-group = "stats"}` in the `dev` group). The wrapper and its tests are not built yet, so this box stays open. A test checks that `uv export --no-dev --no-group stats` lists no `statsmodels`. Prove they fail, then implement. Verify: the tests pass, `uv.lock` pins the version, and `tests/test_dependency_floors.py` passes. - [ ] 15.7 Holm families and gated rows (A4, pre-registration sections 7 and 8). Write failing tests for `factorial35.py`: the contrast table holds rows 1 to 19 with the signs and endpoints of section 8; family sizes are fixed at 19 (FinanceBench), 19 (FreshStack) and 8 (BioASQ, rows 1 to 8); BioASQ rows 9 to 19 are sanity checks with unadjusted 95% intervals and no decision; Legal has no family; a row that fails its gate or the differential-error rule stays in its family with p = 1, takes the last Holm rank, and is marked "not testable", so no gate changes a family size; a directional row counts only with Holm p below 0.05 and the predicted sign. Prove they fail, then implement. Verify: the tests pass. -- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the plan is chosen and dated before the freeze (task 16.26); in the full plan (default) L2 labels every item L1 labels (750; 700 after drop step 1); in the staged fallback L2 labels a seeded 30% of the claim-kind and faithfulness sets first (30 and 135, seeds 20261122 and 20261123) and all 200 correctness answers (365 in all; 315 after drop step 1), then the other 70% of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level; seed 20261124 retired, O4; the gate reference is the agreed label where both labelled an item and L1's label where only L1 did; no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 (per set, and per core dataset for correctness) comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. +- [ ] 15.8 Labelling for two labellers (M4; A4, pre-registration section 11.3). Extend the page of task 8.4 to three sets (claim kinds, faithfulness verdicts, correctness) with a labeller choice `L1` or `L2` and no name field. Write failing tests for `labels35.py`: seeds 20261112, 20261116 and 20261113 give the sizes 100, 450 and 200 (L1: 750 in all; 150 and 700 after drop step 1) with strata dataset; core dataset and chunk size (150 per chunk size, 50 per core dataset within each, at most one claim per answer, crossed setups only, drawn from answers to all questions in both halves); and dataset (50 questions per dataset, half B only, one run-1 answer of a setup picked uniformly); the plan is chosen and dated before the freeze (task 16.26); in the full plan (default) L2 labels every item L1 labels (750; 700 after drop step 1); in the staged fallback L2 labels a seeded 30% of the claim-kind and faithfulness sets first (30 and 135, seeds 20261122 and 20261123) and all 200 correctness answers (365 in all; 315 after drop step 1), then the other 70% of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined; seed 20261124 retired, O4; the gate reference is the agreed label where both labelled an item and L1's label where only L1 did; no score, verdict, setup name or chunk size reaches the page; kappa per checker, per dataset and L1 against L2 (per set, and per core dataset for correctness) comes with percentile bootstrap 95% intervals (10,000 resamples by question; seeds 20261125 and 20261129, PR 16); the disagreement rule makes an unresolved item "can't tell" and counts it. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.9 Hosted-call provenance (M7). Write failing tests: each answer, Ragas, GLM-5.3 and classifier row records `model_returned` and `provider_returned` from the stubbed reply; the join counts rows whose returned model differs from the requested one per role. Prove they fail, then implement. Verify: the tests pass. - [ ] 15.10 Reranker rank check (M8). Write failing tests for `rerank_check35.py`: 200 pairs drawn with seed 20261117 are the same twice; Spearman below 0.95 marks the file "not pinned". Prove they fail, then implement. Run it on the test pod for each size (task 9.2 step 2a). Verify: the tests pass and `protocol.md` lists the correlation per file. - [ ] 15.11 MDE from the pilot (M1; A4, pre-registration section 10). Write `mde35.py` on saved files only: per core dataset and contrast type, the SD of the per-question contrast scores from the retrieval-only pilot (task 16.5, P5; an adjacent-level contrast uses the extreme-level SD as a stand-in, marked so), and on FinanceBench the design effect of document clustering; MDE = (z at 1 minus alpha/2 plus z at 0.80) x SD / square root of n, with alpha 0.05/19 on FinanceBench and FreshStack and 0.05/8 on BioASQ, and n divided by the design effect on FinanceBench; answer-stage rows use the SD from the smoke and test-pod answer files named in task 16.6 (P6) and are labelled rough; each row is shown against the SESOI (0.02 nDCG@10, open item O2). Experiment 22 values are not used. Write failing tests on a fixture first. Verify: the tests pass and the MDE table of pre-registration section 10 is filled before the freeze. @@ -492,7 +492,7 @@ Each test is proved to fail on the current code before the fix. - [ ] 16.23 O1 (decided by the owner on 2026-11-14): a file upload to the registered OSF project time-stamps the A3 configuration hash. The answer is recorded in PR 14.2 and PR 23. Verify: recorded before G7. - [ ] 16.24 O2 (decided by the owner on 2026-11-14): the SESOI is 0.02 nDCG@10, with the written justification in PR 10; intervals are also read against 0.01 and 0.05. Recorded in PR 10 and PR 23 before any pilot result is read. Verify: recorded before the pilot results are read. - [ ] 16.25 O3 (decided by the owner on 2026-11-14): no blind relevance audit by default. It runs only if the PR 11.3 trigger fires (mean unjudged share in the top 10 differs by more than 5 points: hybrid on against off on FreshStack, or OMRS against RAGFlow on a core dataset in Part 2), and only on a dataset whose qrels list judged negatives. Write failing tests for the trigger, then implement it; only if it fires, add the audit sample and page (seed 20261120) to `labels35.py` with tests first. Verify: tests pass, and the trigger values are in the report. -- [ ] 16.26 O4 (decided by the owner on 2026-11-14): L2 is an author or colleague. Full plan (default, revised the same day): L2 labels every item L1 labels and every Part 2 label. Staged fallback, only if L2 cannot commit the time: a seeded 30% of the faithfulness and claim-kind sets first, then the rest of a set when kappa on its subset is below 0.70 (PR 11.3). Before the freeze, record the chosen plan with its date, from L2's availability alone. Before the first label, record L2's name, role and relation to the project in the run record only (P8). Verify: no public file holds the name. +- [ ] 16.26 O4 (decided by the owner on 2026-11-14): L2 is an author or colleague. Full plan (default, revised the same day): L2 labels every item L1 labels and every Part 2 label. Staged fallback, only if L2 cannot commit the time: a seeded 30% of the faithfulness and claim-kind sets first, then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, in any chunk-size level, or is undefined (PR 11.3). Before the freeze, record the chosen plan with its date, from L2's availability alone. Before the first label, record L2's name, role and relation to the project in the run record only (P8). Verify: no public file holds the name. - [ ] 16.27 O5 (decided by the owner on 2026-11-14): no ethics query. The study has no human participants, and the labellers are members of the research team, not study subjects, so no ethics review is sought. This holds while every labeller is an author or colleague. If a paid external annotator is ever used, the question is reopened as a deviation. The answer is recorded in PR 11.3 and PR 23. Verify: recorded before any label. - [ ] 16.28 O6 (decided by the owner on 2026-11-14): BioASQ rows 12 to 19 are sanity checks (families 19 / 19 / 8; H8a and H8b on FinanceBench and FreshStack). The answer is recorded in PR 7, 8 and 23. Verify: recorded before G7. - [ ] 16.29 Uncited segments (PR 6). Write failing tests: the join counts per answer the prose segments with no valid ordinal (the segments `split_claims` drops) and gives `uncited_segment_share` per setup and dataset; `claimkinds` also sorts uncited segments with the frozen classifier (question and segment only); the sensitivity counts uncited fact and mixed segments as unsupported; a fixture answer with one supported cited fact and one uncited fact scores 1.0 primary and 0.5 sensitivity. Prove they fail, then implement. Verify: the tests pass. diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md index a58fd6ef..40500738 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/proposal.md @@ -26,7 +26,7 @@ A publication-grade methods review gave 30 findings, F1 to F30. The operator app - FreshStack indexes use the full public LangChain corpus (49,514 documents). The 10,000-document subset is retired. - H8 splits into H8a (12 crossed setups) and H8b (3 token-matched setups), both confirmatory. Every hypothesis is scoped, with numbered rows and a holds rule. Legal RAG Bench is exploratory everywhere. - No mixed model. Per-question contrasts with GEE, and a document-level sign-flip test on FinanceBench. Holm families fixed at 19, 19 and 8 (second round C3; BioASQ rows 9 to 19 are sanity checks). -- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items. L2 labels the same 750 (full plan, default), or 365 first in a staged fallback (every correctness answer, a seeded 30% of the other two sets, then the rest of a set when kappa on its subset is below 0.70; O4). +- Faithfulness and correctness gates on the kappa point estimate and its lower bound. L1 labels 750 items. L2 labels the same 750 (full plan, default), or 365 first in a staged fallback (every correctness answer, a seeded 30% of the other two sets, then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined; O4). - `evidence_recall_20k` gives any-page credit. A second closed-book run uses the RAG prompt. FinanceBench is split by document. - The `openai/gpt-6.1-terra` cross-check is removed. Budget about $730 plus the pilot, ceiling $975, with a results-blind drop order that no longer holds E1 (E1 has its own ceiling). diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md index ce0292f7..1bbb1e03 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/specs/evaluation-observability/spec.md @@ -153,12 +153,20 @@ Each faithfulness checker SHALL pass when Cohen's kappa against the human label ### Requirement: Human labels follow the registered counts -L1 SHALL label 750 items (claim kinds 100, faithfulness 450, correctness 200). By default L2 SHALL label every item L1 labels. In a staged fallback, chosen and dated before the freeze from L2's availability alone, L2 SHALL label a seeded 30% of claim kinds and faithfulness first (365 items in all), then the rest of a set when kappa on its subset is below 0.70. The claim classifier SHALL reach agreement and refusal recall of at least 0.90 on 100 L1-labelled smoke claims before the freeze. +L1 SHALL label 750 items (claim kinds 100, faithfulness 450, correctness 200); by default L2 SHALL label all 750. A staged fallback, dated before the freeze from L2's availability alone, SHALL have L2 label a seeded 30% of claim kinds and faithfulness first (365), then the rest of a set when its subset kappa is below 0.70 overall or in any faithfulness chunk-size level. Before the freeze the claim classifier SHALL reach at least 0.90 agreement and refusal recall on 100 L1-labelled smoke claims. #### Scenario: L2 cannot commit the time for the full plan - **WHEN** the owner records the staged plan before the freeze - **THEN** L2 SHALL label a seeded 30% of claim kinds and faithfulness first, and every correctness and Part 2 item +#### Scenario: Subset kappa is low in one chunk-size level only +- **WHEN** in the staged plan the human kappa on the faithfulness subset is 0.70 or more overall but below 0.70 within one chunk-size level +- **THEN** L2 SHALL label the remaining 315 faithfulness claims + +#### Scenario: Subset kappa is undefined +- **WHEN** in the staged plan the human kappa on a subset is undefined, for example because both labellers mark all 30 claim kinds as fact +- **THEN** the kappa SHALL count as below 0.70, and L2 SHALL label the rest of that set (70 claim kinds or 315 faithfulness claims) + #### Scenario: A label file is released - **WHEN** human labels are released - **THEN** they SHALL carry the roles L1 and L2 only, and no labeller name SHALL appear diff --git a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md index 5e376058..31218780 100644 --- a/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md +++ b/openspec/changes/experiment-35-full-document-rag-benchmark/tasks.md @@ -94,7 +94,7 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 4D.6 Write the label guidelines for each set with worked examples from outside every sample (P8). Verify: the guidelines are committed before the freeze. (The labelling itself is task 5.7.) - [ ] 4D.7 Fix the second closed-book run (RAG prompt, empty evidence) and the length-controlled check (300 claims) in `plan.json` and the pre-registration. Verify: both are listed before the freeze. (The runs are task 5.8.) - [ ] 4D.8 Fill the pins (P9 to P14): confirm the new seeds of the `preregistration.md` section 16 seeds table (20261120 only if the O3 trigger fires; 20261124 retired); record answer-model host, reasoning mode, output cap and context length; judge, classifier and GLM-5.3 settings; query instruction, reranker instruction and hybrid settings; `halves.json` and its SHA-256 with every pre-freeze question ID and half; prompt template hashes; the `evidence_recall_20k` tolerance. Verify: every "Missing" row of `preregistration.md` section 16 has a value, and `plan.json` `frozen` lists them at the freeze. -- [ ] 4D.9 Owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open): O1 (OSF file upload time-stamps the hash), O2 (SESOI 0.02 nDCG@10, justified in `preregistration.md` section 10), O3 (no blind audit unless the section 11.3 unjudged-share trigger fires; it covers datasets with judged negatives), O4 (L2 is an author or colleague, labels every correctness answer and every Part 2 label and 30% of the other two sets; name in the run record only), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. +- [ ] 4D.9 Owner decisions O1 to O7 (all decided by the owner on 2026-11-14; none open): O1 (OSF file upload time-stamps the hash), O2 (SESOI 0.02 nDCG@10, justified in `preregistration.md` section 10), O3 (no blind audit unless the section 11.3 unjudged-share trigger fires; it covers datasets with judged negatives), O4 (revised the same day: L2 is an author or colleague; full plan by default, L2 labels every item L1 labels and every Part 2 label, 750; only if L2 cannot commit the time, the staged fallback labels a seeded 30% of the faithfulness and claim-kind sets first, 365; name in the run record only), O5 (no ethics query: no human participants, labellers are team members; a paid external annotator reopens it), O6 (BioASQ rows 12 to 19 as sanity checks) and O7 (primary faithfulness stays cited factual claims; uncited-segment version is a sensitivity only). Verify: each answer is recorded with its date before the freeze, and no L2 name appears in a public file. - [ ] 4D.10 Check before the freeze that no live text in this change, `protocol.md` section 0 or `plan.json` contradicts `preregistration.md`. Verify: every hit for "MixedLM", "included whole", "gpt-6.1-terra", "operator decides" and "equivalence" is marked superseded or removed. ## 5. Run @@ -105,7 +105,7 @@ Design D0.15. Protocol section 0.13. `preregistration.md` governs; P and O numbe - [ ] 5.4 Run the determinism check for every arm, and the rebuild check (4B baseline index rebuilt once per dataset in a fresh process). Verify: the output states match or mismatch per arm and dataset, and for each rebuild. - [ ] 5.5 Run Ragas on each arm where the dataset supports each metric. Verify: unsupported metrics show "not measured". - [ ] 5.6 Check about 50 judged items by hand. Verify: the agreement is computed with the measure frozen in 1.10. -- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 750 (full plan, default; O4, revised 2026-11-14), or L2 365 first in the staged fallback (claim kinds 30, faithfulness 135, correctness 200, then the rest of a set when kappa on its subset is below 0.70), with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. +- [ ] 5.7 (A4, moved from 4D.6.) Label L1 750 items (claim kinds 100, faithfulness 450 from answers to all questions, correctness 200 from half B) and L2 750 (full plan, default; O4, revised 2026-11-14), or L2 365 first in the staged fallback (claim kinds 30, faithfulness 135, correctness 200, then the rest of a set when kappa on its subset is below 0.70 overall or, for faithfulness, within any chunk-size level, or is undefined), with the guidelines of task 4D.6. Verify: the gate results are recorded with their time before any H8a or H8b contrast result. - [ ] 5.8 (A4, moved from 4D.7.) Run the second closed-book run (RAG prompt, empty evidence) beside the first, and the length-controlled check on 300 claims. Verify: each table names its `output/` file and is marked descriptive or sensitivity. ## 6. Analyse and report diff --git a/openspec/changes/rag-benchmark-readiness-flow/design.md b/openspec/changes/rag-benchmark-readiness-flow/design.md index 00a2b9e4..f06917a4 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/design.md +++ b/openspec/changes/rag-benchmark-readiness-flow/design.md @@ -103,7 +103,7 @@ Reasons: - The downloads go before the pod. The pod's Part C timing counts tokens on all three corpora, so it needs them. - The arm smoke run goes after the harness. Every arm needs the section 2 code. Archived task 10.6 and Exp 35 task 4.1 are the same check, so one run does both. -- The smoke run uses the run machine. Corpus and indexes stay in place for section 5. A stopped pod bills only its disk while the operator decides at G7. +- The smoke run uses the run machine. A stopped pod still bills its disk, so the smoke pod and its disk are deleted once their outputs are copied and checked, at G7 at the latest (task 6.11). Section 5 builds its indexes on new pods (task 6.12). - `pre-exp35-harness-and-answer-fixes` was archived with group 10 open. This group owns those tasks now. The detail stays in that change's design D13. - Step E2 (added 2026-11-14, amendment A4) holds the pre-freeze items of `preregistration.md` section 23. The pilot and every pre-freeze sample drawn after the halves exist come from half A, so the halves are written first; earlier samples are listed with their halves (P12). Step G follows the order of work of pre-registration section 14.2: no half-B score exists before the configuration hash is time-stamped on OSF. diff --git a/openspec/changes/rag-benchmark-readiness-flow/tasks.md b/openspec/changes/rag-benchmark-readiness-flow/tasks.md index 799b5798..83ab9826 100644 --- a/openspec/changes/rag-benchmark-readiness-flow/tasks.md +++ b/openspec/changes/rag-benchmark-readiness-flow/tasks.md @@ -79,7 +79,7 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime ### E. Smoke run on the run machine -- [ ] 6.10 One smoke run that does archived 10.6 and Exp 35 task 4.1 together: a seeded sample of the factorial setups (Exp 35 amendment A1), 5 seeded questions per dataset, twice, through the claim checkers GLM-5.3 (main) and Clef-Flash (second), and `chat`. Granite Guardian is dropped. Run it on the machine that task 1.5 names. On a pod, quote the price first, and stop the pod after the smoke run (only its disk bills) until G7. Verify: zero errors, a verdict per strategy for every claim, complete call records, an identical second run, and the budget of Exp 35 task 4A.5 (ceiling $975) not exceeded on extrapolation. History: the first smoke run, under the old six-arm design, was done 2026-11-10/11 on pod walsi2c5gfb8xc: all 6 arms, 3 datasets, 5 questions, 2 runs, all three checkers; retrieval identical in 18 of 18 pairs. Report: DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/prefreeze-report-2026-11-11.md. Pod deleted. Open: the extrapolation check needs the budget of Exp 35 task 4A.5. The FreshStack-only smoke rerun is done (2026-11-11, pod 2elh5ji2og6d74, deleted): Clef-Flash passes the guard (ADR-119 A7), and its two harness errors are fixed in PR #207. Tick this task only when the factorial smoke run above passes every Verify check, and after 4A.5 and the zero-error recheck of FreshStack questions 77536364 and 77762811 (Exp 35 task 4.1 (c)). The old six-arm smoke run does not count for the factorial setups (PR 209 review, 2026-11-13). +- [ ] 6.10 One smoke run that does archived 10.6 and Exp 35 task 4.1 together: a seeded sample of the factorial setups (Exp 35 amendment A1), 5 seeded questions per dataset, twice, through the claim checkers GLM-5.3 (main) and Clef-Flash (second), and `chat`. Granite Guardian is dropped. Run it on the machine that task 1.5 names. On a pod, quote the price first, and stop the pod after the smoke run (only its disk bills). Copy its outputs to `DATA/omrs/` and check them against `SHA256SUMS`, then delete the pod and its disk. Step 6.11 (G7) is the latest point for this delete. Verify: zero errors, a verdict per strategy for every claim, complete call records, an identical second run, and the budget of Exp 35 task 4A.5 (ceiling $975) not exceeded on extrapolation. History: the first smoke run, under the old six-arm design, was done 2026-11-10/11 on pod walsi2c5gfb8xc: all 6 arms, 3 datasets, 5 questions, 2 runs, all three checkers; retrieval identical in 18 of 18 pairs. Report: DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/prefreeze-report-2026-11-11.md. Pod deleted. Open: the extrapolation check needs the budget of Exp 35 task 4A.5. The FreshStack-only smoke rerun is done (2026-11-11, pod 2elh5ji2og6d74, deleted): Clef-Flash passes the guard (ADR-119 A7), and its two harness errors are fixed in PR #207. Tick this task only when the factorial smoke run above passes every Verify check, and after 4A.5 and the zero-error recheck of FreshStack questions 77536364 and 77762811 (Exp 35 task 4.1 (c)). The old six-arm smoke run does not count for the factorial setups (PR 209 review, 2026-11-13). ### E2. Amendment A4 pre-freeze items (steps 6.10a to 6.10d added 2026-11-14; design D6) @@ -92,11 +92,11 @@ Follow the steps in order (design D5). "Exp 35 task N" means a task in `experime ### F. Freeze -- [ ] 6.11 GATE G7: the user approves the freeze and the run budget (estimate about $730, ceiling $975, PR 17). Steps 6.10a to 6.10d and 7.1 to 7.2a are done. Exp 35 tasks 4.2 and 4.3: commit `preregistration.md`, `protocol.md` and `plan.json` (`frozen` filled), then remove the DRAFT status. The OSF deposit (Exp 35 task 4B.6) is shared with group 7 step 7.3. It holds `preregistration.md` as the main file, with the proposal, `protocol.md` (history), `plan.json` and the Part 2 protocol. Do not deposit before step 7.1 is done. After the freeze every change is a deviation (design D6). +- [ ] 6.11 GATE G7: the user approves the freeze and the run budget (estimate about $730, ceiling $975, PR 17). Steps 6.10a to 6.10d and 7.1 to 7.2a are done. Exp 35 tasks 4.2 and 4.3: commit `preregistration.md`, `protocol.md` and `plan.json` (`frozen` filled), then remove the DRAFT status. The OSF deposit (Exp 35 task 4B.6) is shared with group 7 step 7.3. It holds `preregistration.md` as the main file, with the proposal, `protocol.md` (history), `plan.json` and the Part 2 protocol. Do not deposit before step 7.1 is done. If the step 6.10 smoke pod still exists, copy and check its outputs, then delete the pod and its disk before the freeze. After the freeze every change is a deviation (design D6). Verify: a live RunPod read shows no smoke pod and no smoke volume. ### G. Run and report -- [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (O1, decided 2026-11-14); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step E pod was deleted, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. +- [ ] 6.12 Exp 35 search and answer stages (tasks 4A.6 to 4A.8, which replace the old section 5 text), in the order of work of PR 14.2: (1) build and seal the 9 indexes (FreshStack on the full corpus); G2 passes before any search writes into the shared output folder; (2) run the 72 search setups per dataset once each, on all questions; (3) score half A only, apply the selection rule (step 7.4) and write `selection.json`; (4) time-stamp the configuration hash on OSF (O1, decided 2026-11-14); (5) only then score half B and start the replicate pods: the 15 answer setups (12 crossed plus 3 token-matched), 3 runs each, the two closed-book runs, the oracle run and the length-controlled check. The step 6.10 smoke pod and its disk were deleted by step 6.11, so create nine index pods (task 4A.6) and three replicate pods (task 4A.7) on the machine type that task 1.5 names. For each pod, quote the hourly price first, restore the models, corpora and pod scripts from `DATA/omrs/experiments/35-full-rag-benchmark-2026-11-01/`, check each file against its `SHA256SUMS` before the first setup, and delete the pod when its runs and its copy are done. When a cost projection reaches the ceiling, apply the results-blind drop order of PR 17; no confirmatory item is dropped by choice. Verify: the OSF time-stamp is older than the first half-B score. - [ ] 6.13 Exp 35 section 6 (tasks 6.1 to 6.3): statistics as PR 9 (clustered method, Holm family sizes 19, 19 and 8 as fixed in PR 8; gated rows at p = 1), the reporting items of PR 21 (unjudged share, sensitivity tables, compute table with every pre-freeze pod), `report.md` with the deviation log (design D6), `experiment.json` status. - [ ] 6.14 Exp 35 section 7 (tasks 7.1 to 7.5): the release table of PR 19 (answers, evidence IDs, ranked lists, human labels and verdicts per licence, never corpus text) and the re-judge kit, copy to `DATA/omrs/`, Zenodo (one archive per dataset licence), identifier search. Merge and archive Exp 35 on the user's word.