diff --git a/bench/LexiSharp.Eval/Program.cs b/bench/LexiSharp.Eval/Program.cs index fc28862..9b2fa3d 100644 --- a/bench/LexiSharp.Eval/Program.cs +++ b/bench/LexiSharp.Eval/Program.cs @@ -1207,8 +1207,10 @@ void CloseRerankCache() Console.WriteLine( "Note: absolute scores depend on the analysis, which is why the run states it. The " + "relative ordering of the configs above is meaningful within a run; the comparison to " + - "the published reference is meaningful only under the analysis the reference used " + - "(--analyzer english for the published figures)."); + "the published reference is meaningful only under the analysis and conventions the " + + "reference used: --analyzer uax29, --ndcg-gain linear, --reference-bm25 0.9,0.4, " + + "--query-term-frequency, --reference-index-statistics, --reference-score-rounding, " + + "--omit-saturation-constant, --single-precision-bm25 and --top-k 1000."); if (backendLane is not null) { diff --git a/docs/backends.md b/docs/backends.md index b836233..3326036 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -386,9 +386,11 @@ NFCorpus for speed; raising it changes nothing here. financial Q&A with short queries) is not part of the pinned three. Under this table's analysis — `simple` + `unaccent`, k1=1.5/b=0.75 — the gate reads **0.2061** nDCG@10 and the ungated pool **0.1077**. That is not comparable with Anserini's published **0.2361**, which is Lucene's English -analysis at k1=0.9/b=0.4: align both (`--analyzer english --reference-bm25 0.9,0.4`, -`--backend-config english --backend-bm25 0.9,0.4`) and the in-memory BM25 reads 0.237 against -0.2361, this engine **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident: +analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer uax29`, linear gains, +k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation +constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the +paper's 0.236: the figure is reproduced. This engine cannot use `uax29`; with its closest +configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident: the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does not. diff --git a/docs/evaluation.md b/docs/evaluation.md index 4a6a59d..3633fb5 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -525,10 +525,13 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10): no stemming, no stop list — the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate and **0.1077** without it at pool 1,000. Those are **not** comparable with Anserini's published 0.2361: that figure is Lucene's English analysis at k1=0.9/b=0.4, and this row is PostgreSQL's - `simple` configuration at 1.5/0.75. Aligning both (`--analyzer english --reference-bm25 0.9,0.4`, - and `--backend-config english --backend-bm25 0.9,0.4` for the SQL row), the in-memory BM25 reads - **0.237** against the published **0.2361**, and the PostgreSQL row reads **0.2291** — the same - ~0.007 analysis gap the other corpora show, not a gate cost. The gate's direction is SciFact's: + `simple` configuration at 1.5/0.75. Under the reference's own conventions — `--analyzer uax29`, linear + gains, k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted + saturation constant, single precision, depth 1,000 — the in-memory BM25 reads **0.236** against + the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was + published at. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`, + k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate + cost. The gate's direction is SciFact's: the ungated candidates are chosen by coverage and truncated, the gated ones by idf. - **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable** with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric