Skip to content

Correct FiQA's 0.2291: it is the gate, not the analysis - #84

Merged
manuc66 merged 1 commit into
mainfrom
docs/fiqa-gate-correction
Nov 18, 2026
Merged

manuc66 merged 1 commit into
mainfrom
docs/fiqa-gate-correction

Conversation

@manuc66

@manuc66 manuc66 commented Nov 18, 2026

Copy link
Copy Markdown
Owner

What this corrects

The FiQA page said the stock-PostgreSQL BM25 row's 0.2291 was "the same ~0.007 analysis gap the other corpora show, not a gate cost". That is wrong. Measured at k1=0.9/b=0.4, linear gains, depth 1,000, default arithmetic:

FiQA-2018, english nDCG@10
ungated, exact 0.2430
gate 5,000 0.2417
gate 2,000 0.2363
gate 1,000 (default) 0.2291
reference analysis, in memory, exact 0.2378
published (reference conventions) 0.2361

Two facts follow:

  • The 0.2291 is the rare-term gate's recall bound, not the analyzer. Ungated, PostgreSQL's analysis reads 0.2430 — above the reference analysis (0.2378) and the published 0.2361. There is no analysis gap to close on FiQA; the gate is the cost.
  • The default budget is tuned for NFCorpus and SciFact, where it is free. On FiQA, a short-query corpus, it costs ~0.014; --backend-gate-budget recovers it (2,000 → 0.2363, 5,000 → 0.2417). The default stays 1,000, and the trade is now stated.

The published 0.2361 remains reproduced by the in-memory uax29 row under the reference's conventions, with an index byte-identical to Lucene 10.5.2's. That claim is unchanged.

Files

docs/evaluation.md, docs/backends.md (the gate section gains the FiQA side of the trade), docs/index.md, README.md, the changelog. No code change.

This corrects a claim merged in #80/#82, not a regression.

Ungated, the same english configuration reads 0.2430 (k1=0.9/b=0.4,
linear, depth 1,000), above the reference analysis's 0.2378 and the
published 0.2361. The 0.2291 the page attributed to a ~0.007 analysis gap is
the rare-term gate's recall bound; the budget moves it (1,000 -> 0.2291,
2,000 -> 0.2363, 5,000 -> 0.2417). The default stays 1,000, which is free on
NFCorpus and SciFact; the trade is stated on the evaluation and backend
pages, and the published 0.2361 remains the in-memory uax29 row's number.
@codecov-commenter

codecov-commenter commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.27%. Comparing base (1f145cd) to head (c00c27b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #84   +/-   ##
=======================================
  Coverage   89.26%   89.27%           
=======================================
  Files         271      271           
  Lines       21461    21461           
  Branches     4635     4635           
=======================================
+ Hits        19158    19160    +2     
+ Misses       1373     1372    -1     
+ Partials      930      929    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit 6612a11 into main Nov 18, 2026
16 checks passed
@manuc66
manuc66 deleted the docs/fiqa-gate-correction branch November 18, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants