Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions bench/LexiSharp.Eval/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1207,8 +1207,10 @@ void CloseRerankCache()
Console.WriteLine(
"Note: absolute scores depend on the analysis, which is why the run states it. The " +
"relative ordering of the configs above is meaningful within a run; the comparison to " +
"the published reference is meaningful only under the analysis the reference used " +
"(--analyzer english for the published figures).");
"the published reference is meaningful only under the analysis and conventions the " +
"reference used: --analyzer uax29, --ndcg-gain linear, --reference-bm25 0.9,0.4, " +
"--query-term-frequency, --reference-index-statistics, --reference-score-rounding, " +
"--omit-saturation-constant, --single-precision-bm25 and --top-k 1000.");

if (backendLane is not null)
{
Expand Down
8 changes: 5 additions & 3 deletions docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,9 +386,11 @@ NFCorpus for speed; raising it changes nothing here.
financial Q&A with short queries) is not part of the pinned three. Under this table's analysis —
`simple` + `unaccent`, k1=1.5/b=0.75 — the gate reads **0.2061** nDCG@10 and the ungated pool
**0.1077**. That is not comparable with Anserini's published **0.2361**, which is Lucene's English
analysis at k1=0.9/b=0.4: align both (`--analyzer english --reference-bm25 0.9,0.4`,
`--backend-config english --backend-bm25 0.9,0.4`) and the in-memory BM25 reads 0.237 against
0.2361, this engine **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident:
analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer uax29`, linear gains,
k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation
constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the
paper's 0.236: the figure is reproduced. This engine cannot use `uax29`; with its closest
configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident:
the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does
not.

Expand Down
11 changes: 7 additions & 4 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -525,10 +525,13 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10):
no stemming, no stop list — the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate
and **0.1077** without it at pool 1,000. Those are **not** comparable with Anserini's published
0.2361: that figure is Lucene's English analysis at k1=0.9/b=0.4, and this row is PostgreSQL's
`simple` configuration at 1.5/0.75. Aligning both (`--analyzer english --reference-bm25 0.9,0.4`,
and `--backend-config english --backend-bm25 0.9,0.4` for the SQL row), the in-memory BM25 reads
**0.237** against the published **0.2361**, and the PostgreSQL row reads **0.2291** — the same
~0.007 analysis gap the other corpora show, not a gate cost. The gate's direction is SciFact's:
`simple` configuration at 1.5/0.75. Under the reference's own conventions — `--analyzer uax29`, linear
gains, k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted
saturation constant, single precision, depth 1,000 — the in-memory BM25 reads **0.236** against
the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was
published at. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`,
k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate
cost. The gate's direction is SciFact's:
the ungated candidates are chosen by coverage and truncated, the gated ones by idf.
- **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable**
with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric
Expand Down
Loading