Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,13 @@

### Fixed

- **The FiQA page attributed the stock-PostgreSQL BM25 row's 0.2291 to its analysis.** It is the
rare-term gate: ungated, the same `english` configuration reads **0.2430** at k1=0.9/b=0.4 with
linear gains and depth 1,000, above the reference analysis's 0.2378 in memory and above the
published 0.2361. The gate's budget is the lever on short-query corpora: 1,000 → 0.2291,
2,000 → 0.2363, 5,000 → 0.2417, exact → 0.2430. The default stays 1,000, which is free on
NFCorpus and SciFact; the trade is now stated.

- **The evaluation harness measured a rounded run in the engine's tie-break, not in the run reader's.**
The four-decimal rounding reproduces what the reference writes down, and a written run is read back
by `trec_eval`, which ignores the rank column and breaks an exact score tie by document number
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,9 @@ linear classifier): [text analysis →](docs/text-analysis.md).
removes the bound. The rare-term gate is on by default and is what makes the path usable: it
reads ~59 ms p50 on a 5,000-document corpus with 13-term queries, against ~265 ms with the gate
off. It is a recall bound — a document holding none of the query's rare terms cannot be returned
— measured on two scientific corpora. The ParadeDB engine needs `pg_search`, which no major
managed service lets you install; the `tsvector`, BM25 and `pg_trgm` engines do not.
— measured on two scientific corpora and FiQA, where the default budget costs ~0.014 nDCG@10;
`--backend-gate-budget` is the lever there. The ParadeDB engine needs `pg_search`, which no
major managed service lets you install; the `tsvector`, BM25 and `pg_trgm` engines do not.
- SQL backends are measured on ranking for the engines that rank on their own; sparse and fuzzy
engines are covered by integration tests only.

Expand Down
28 changes: 14 additions & 14 deletions docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -382,21 +382,21 @@ by; on SciFact that truncation is the whole 0.10 gap. The gate chooses by idf in
candidate set is small enough not to truncate. Lowering the budget trades a little quality on
NFCorpus for speed; raising it changes nothing here.

FiQA-2018 shows the other side of that trade. At k1=0.9/b=0.4 with `english`, the gate reads 0.2291
at the default 1,000, 0.2363 at 2,000 and 0.2417 at 5,000, against 0.2430 ungated: the default is
tuned for the two corpora above, where it is free, and a short-query corpus pays for it. The budget
is a cost bound, and this is what it costs.

**A corpus of another nature.** FiQA-2018 (`--dataset fiqa`; 57,638 documents, 648 test queries,
financial Q&A with short queries) is not part of the pinned three. Under this table's analysis —
`simple` + `unaccent`, k1=1.5/b=0.75 — the gate reads **0.2061** nDCG@10 and the ungated pool
**0.1077**. That is not comparable with Anserini's published **0.2361**, which is Lucene's English
analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer uax29`, linear gains,
k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation
constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the
paper's 0.236: the figure is reproduced, and the index is the reference's: 5,288,635 terms, with a
per-document term stream byte-identical to Lucene 10.5.2's `EnglishAnalyzer`. This engine cannot use `uax29`; with its closest
configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The gap was decomposed: a PostgreSQL
configuration with the reference's 33-word stop list and Snowball reads 0.2196, so the stop list is
not the cause (the longer stock list helps) and the remainder is the stemmer and the parser — not
replaceable on stock PostgreSQL. The pattern is SciFact's rather than a scientific-corpus accident:
the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does
not.
financial Q&A with short queries) is not part of the pinned three. At k1=0.9/b=0.4, linear gains,
depth 1,000 and the library's default arithmetic, the `english` row reads **0.2291** at the default
gate and **0.2430** ungated: the difference is the gate's recall bound, not the analyzer — ungated,
PostgreSQL's analysis is above the reference analysis (0.2378 in memory) and above the published
0.2361, a different analysis under the reference conventions. The budget is the lever:
1,000 → 0.2291, 2,000 → 0.2363, 5,000 → 0.2417, exact → 0.2430; the default is tuned for the two
scientific corpora. The published 0.2361 remains reproduced by the in-memory `uax29` row under the
reference's conventions, with an index byte-identical to Lucene 10.5.2's `EnglishAnalyzer`
(5,288,635 terms), and multi-code-point emoji sequences are the one knowingly unimplemented rule.

**What it costs, in time.** The evaluation harness reports quality and not time on purpose; the
[PostgreSQL lane](benchmarks.md#postgresql-lane-postgres-what-a-search-costs-on-stock-postgresql)
Expand Down
33 changes: 13 additions & 20 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -521,26 +521,19 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10):
than by idf — on SciFact that truncation is the whole 0.10 difference. The numbers behind the
choice, and what it costs in milliseconds, are on the [backend guide](backends.md#bm25-over-stock-postgresql-postgresbm25searchengine).
- **The gate on a corpus of another nature.** FiQA-2018 is a selectable dataset (`--dataset fiqa`)
but not part of `--dataset all` or the pinned comparison. Under the library's default analysis —
no stemming, no stop list — the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate
and **0.1077** without it at pool 1,000. Those are **not** comparable with Anserini's published
0.2361: that figure is Lucene's English analysis at k1=0.9/b=0.4, and this row is PostgreSQL's
`simple` configuration at 1.5/0.75. Under the reference's own conventions — `--analyzer uax29`, linear
gains, k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted
saturation constant, single precision, depth 1,000 — the in-memory BM25 reads **0.236** against
the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was
published at. The index is the reference's, not merely a close one: `--fingerprint` reports
5,288,635 terms, and the per-document term stream is byte-identical to Lucene 10.5.2's
`EnglishAnalyzer` over the corpus (Extended_Pictographic symbols, one term per Han ideograph,
combining marks, format characters, the 255-unit token chop, U+202F). Multi-code-point emoji
sequences are the one knowingly unimplemented rule — FiQA contains none, and a test pins the
divergence. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`,
k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate
cost. Where the gap sits was measured rather than assumed: a PostgreSQL configuration carrying the
reference's 33-word stop list and its Snowball stemmer reads **0.2196**, so the stop list is not
the cause — the longer stock list helps here — and what remains is the stemmer and the parser,
which stock PostgreSQL cannot replace without an extension. The gate's direction is SciFact's:
the ungated candidates are chosen by coverage and truncated, the gated ones by idf.
but not part of `--dataset all` or the pinned comparison. At k1=0.9/b=0.4, linear gains, depth
1,000 and the library's default arithmetic, the stock-PostgreSQL BM25 row with `english` reads
**0.2291** at the default gate (1,000 rows) and **0.2430** ungated. The 0.2291 this page used to
attribute to a ~0.007 analysis gap is the gate's recall bound, not the analyzer: ungated,
PostgreSQL's analysis is above the reference analysis (0.2378 in memory) and above the published
0.2361, which is a different analysis under the reference conventions. The budget is the lever:
500 → 0.2297, 1,000 → 0.2291, 2,000 → 0.2363, 5,000 → 0.2417, exact → 0.2430. The default is
tuned for NFCorpus and SciFact, where it is free; short-query corpora pay for it, and
`--backend-gate-budget` is the knob. The stop list is not behind the difference either: at the
same gate budget, a PostgreSQL configuration with the reference's 33-word list reads 0.2196, so
the longer stock list helps. The published 0.2361 remains reproduced by the in-memory `uax29` row
under the reference's conventions, with an index byte-identical to Lucene 10.5.2's; multi-code-point
emoji sequences are the one knowingly unimplemented rule.
- **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable**
with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric
columns compare either way. The analysis is PostgreSQL's own, not this library's tokenizer, which
Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,8 @@ twice. Every figure below is reproducible with a command on the
term, and OR-ing them (`--backend-match any`) recovers **0.0988** on SciFact but only **0.1881**
on NFCorpus, so the rest of that gap is the ranker. The stock-PostgreSQL BM25 engine closes it
with no extension: **0.3072** and **0.6665** at its default candidate gate, **0.2999** and
**0.5636** with the gate off; on FiQA, the corpus of another nature, **0.2061** against
**0.1077**. pgvector at `ef_search` 10/40/160 reads **0.2882 / 0.2960 / 0.3013** on
**0.5636** with the gate off; on FiQA, the corpus of another nature, **0.2291** at the default
gate against **0.2430** ungated at k1=0.9/b=0.4 — the gate is the lever there. pgvector at `ef_search` 10/40/160 reads **0.2882 / 0.2960 / 0.3013** on
NFCorpus against the exact in-memory dense row's **0.304**, and **0.5998 / 0.6528 / 0.6550** on
SciFact against **0.655**: that gap is the index's cost in nDCG. ArguAna produces no row from any
of the three: the text engines reject 2 of its 1,406 test queries, and the vector engine rejects
Expand Down
Loading