From 9042ff24cc1728f456a1a85ce85514bd78a00192 Mon Sep 17 00:00:00 2001 From: manuc66 Date: Tue, 17 Nov 2026 07:10:40 +0100 Subject: [PATCH] Name the reference conventions FiQA's aligned figure needs The previous paragraphs aligned the analysis and the operating point only, and read 0.237 against 0.2361. The reference's aligned configuration also uses uax29, linear gains, query-term frequency, its index statistics, its score rounding, the omitted saturation constant and single precision: with those the in-memory BM25 reads 0.236 against the published 0.2361 (paper 0.236). The PostgreSQL row stays 0.2291 because it cannot use uax29, which is the analysis gap the other corpora already document. The harness's own closing note still pointed at --analyzer english for the published figures; it now names the full set of flags the aligned rows use. --- bench/LexiSharp.Eval/Program.cs | 6 ++++-- docs/backends.md | 8 +++++--- docs/evaluation.md | 11 +++++++---- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/bench/LexiSharp.Eval/Program.cs b/bench/LexiSharp.Eval/Program.cs index fc28862..9b2fa3d 100644 --- a/bench/LexiSharp.Eval/Program.cs +++ b/bench/LexiSharp.Eval/Program.cs @@ -1207,8 +1207,10 @@ void CloseRerankCache() Console.WriteLine( "Note: absolute scores depend on the analysis, which is why the run states it. The " + "relative ordering of the configs above is meaningful within a run; the comparison to " + - "the published reference is meaningful only under the analysis the reference used " + - "(--analyzer english for the published figures)."); + "the published reference is meaningful only under the analysis and conventions the " + + "reference used: --analyzer uax29, --ndcg-gain linear, --reference-bm25 0.9,0.4, " + + "--query-term-frequency, --reference-index-statistics, --reference-score-rounding, " + + "--omit-saturation-constant, --single-precision-bm25 and --top-k 1000."); if (backendLane is not null) { diff --git a/docs/backends.md b/docs/backends.md index b836233..3326036 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -386,9 +386,11 @@ NFCorpus for speed; raising it changes nothing here. financial Q&A with short queries) is not part of the pinned three. Under this table's analysis — `simple` + `unaccent`, k1=1.5/b=0.75 — the gate reads **0.2061** nDCG@10 and the ungated pool **0.1077**. That is not comparable with Anserini's published **0.2361**, which is Lucene's English -analysis at k1=0.9/b=0.4: align both (`--analyzer english --reference-bm25 0.9,0.4`, -`--backend-config english --backend-bm25 0.9,0.4`) and the in-memory BM25 reads 0.237 against -0.2361, this engine **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident: +analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer uax29`, linear gains, +k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation +constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the +paper's 0.236: the figure is reproduced. This engine cannot use `uax29`; with its closest +configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident: the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does not. diff --git a/docs/evaluation.md b/docs/evaluation.md index 4a6a59d..3633fb5 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -525,10 +525,13 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10): no stemming, no stop list — the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate and **0.1077** without it at pool 1,000. Those are **not** comparable with Anserini's published 0.2361: that figure is Lucene's English analysis at k1=0.9/b=0.4, and this row is PostgreSQL's - `simple` configuration at 1.5/0.75. Aligning both (`--analyzer english --reference-bm25 0.9,0.4`, - and `--backend-config english --backend-bm25 0.9,0.4` for the SQL row), the in-memory BM25 reads - **0.237** against the published **0.2361**, and the PostgreSQL row reads **0.2291** — the same - ~0.007 analysis gap the other corpora show, not a gate cost. The gate's direction is SciFact's: + `simple` configuration at 1.5/0.75. Under the reference's own conventions — `--analyzer uax29`, linear + gains, k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted + saturation constant, single precision, depth 1,000 — the in-memory BM25 reads **0.236** against + the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was + published at. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`, + k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate + cost. The gate's direction is SciFact's: the ungated candidates are chosen by coverage and truncated, the gated ones by idf. - **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable** with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric