diff --git a/CHANGELOG.md b/CHANGELOG.md index 25b8459..ebeb0f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -86,6 +86,13 @@ ### Fixed +- **The FiQA page attributed the stock-PostgreSQL BM25 row's 0.2291 to its analysis.** It is the + rare-term gate: ungated, the same `english` configuration reads **0.2430** at k1=0.9/b=0.4 with + linear gains and depth 1,000, above the reference analysis's 0.2378 in memory and above the + published 0.2361. The gate's budget is the lever on short-query corpora: 1,000 → 0.2291, + 2,000 → 0.2363, 5,000 → 0.2417, exact → 0.2430. The default stays 1,000, which is free on + NFCorpus and SciFact; the trade is now stated. + - **The evaluation harness measured a rounded run in the engine's tie-break, not in the run reader's.** The four-decimal rounding reproduces what the reference writes down, and a written run is read back by `trec_eval`, which ignores the rank column and breaks an exact score tie by document number diff --git a/README.md b/README.md index 1305201..a5dc41a 100644 --- a/README.md +++ b/README.md @@ -124,8 +124,9 @@ linear classifier): [text analysis →](docs/text-analysis.md). removes the bound. The rare-term gate is on by default and is what makes the path usable: it reads ~59 ms p50 on a 5,000-document corpus with 13-term queries, against ~265 ms with the gate off. It is a recall bound — a document holding none of the query's rare terms cannot be returned - — measured on two scientific corpora. The ParadeDB engine needs `pg_search`, which no major - managed service lets you install; the `tsvector`, BM25 and `pg_trgm` engines do not. + — measured on two scientific corpora and FiQA, where the default budget costs ~0.014 nDCG@10; + `--backend-gate-budget` is the lever there. The ParadeDB engine needs `pg_search`, which no + major managed service lets you install; the `tsvector`, BM25 and `pg_trgm` engines do not. - SQL backends are measured on ranking for the engines that rank on their own; sparse and fuzzy engines are covered by integration tests only. diff --git a/docs/backends.md b/docs/backends.md index 782b51b..5b3c8c6 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -382,21 +382,21 @@ by; on SciFact that truncation is the whole 0.10 gap. The gate chooses by idf in candidate set is small enough not to truncate. Lowering the budget trades a little quality on NFCorpus for speed; raising it changes nothing here. +FiQA-2018 shows the other side of that trade. At k1=0.9/b=0.4 with `english`, the gate reads 0.2291 +at the default 1,000, 0.2363 at 2,000 and 0.2417 at 5,000, against 0.2430 ungated: the default is +tuned for the two corpora above, where it is free, and a short-query corpus pays for it. The budget +is a cost bound, and this is what it costs. + **A corpus of another nature.** FiQA-2018 (`--dataset fiqa`; 57,638 documents, 648 test queries, -financial Q&A with short queries) is not part of the pinned three. Under this table's analysis — -`simple` + `unaccent`, k1=1.5/b=0.75 — the gate reads **0.2061** nDCG@10 and the ungated pool -**0.1077**. That is not comparable with Anserini's published **0.2361**, which is Lucene's English -analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer uax29`, linear gains, -k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation -constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the -paper's 0.236: the figure is reproduced, and the index is the reference's: 5,288,635 terms, with a -per-document term stream byte-identical to Lucene 10.5.2's `EnglishAnalyzer`. This engine cannot use `uax29`; with its closest -configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The gap was decomposed: a PostgreSQL -configuration with the reference's 33-word stop list and Snowball reads 0.2196, so the stop list is -not the cause (the longer stock list helps) and the remainder is the stemmer and the parser — not -replaceable on stock PostgreSQL. The pattern is SciFact's rather than a scientific-corpus accident: -the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does -not. +financial Q&A with short queries) is not part of the pinned three. At k1=0.9/b=0.4, linear gains, +depth 1,000 and the library's default arithmetic, the `english` row reads **0.2291** at the default +gate and **0.2430** ungated: the difference is the gate's recall bound, not the analyzer — ungated, +PostgreSQL's analysis is above the reference analysis (0.2378 in memory) and above the published +0.2361, a different analysis under the reference conventions. The budget is the lever: +1,000 → 0.2291, 2,000 → 0.2363, 5,000 → 0.2417, exact → 0.2430; the default is tuned for the two +scientific corpora. The published 0.2361 remains reproduced by the in-memory `uax29` row under the +reference's conventions, with an index byte-identical to Lucene 10.5.2's `EnglishAnalyzer` +(5,288,635 terms), and multi-code-point emoji sequences are the one knowingly unimplemented rule. **What it costs, in time.** The evaluation harness reports quality and not time on purpose; the [PostgreSQL lane](benchmarks.md#postgresql-lane-postgres-what-a-search-costs-on-stock-postgresql) diff --git a/docs/evaluation.md b/docs/evaluation.md index c3bc10a..971e8e0 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -521,26 +521,19 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10): than by idf — on SciFact that truncation is the whole 0.10 difference. The numbers behind the choice, and what it costs in milliseconds, are on the [backend guide](backends.md#bm25-over-stock-postgresql-postgresbm25searchengine). - **The gate on a corpus of another nature.** FiQA-2018 is a selectable dataset (`--dataset fiqa`) - but not part of `--dataset all` or the pinned comparison. Under the library's default analysis — - no stemming, no stop list — the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate - and **0.1077** without it at pool 1,000. Those are **not** comparable with Anserini's published - 0.2361: that figure is Lucene's English analysis at k1=0.9/b=0.4, and this row is PostgreSQL's - `simple` configuration at 1.5/0.75. Under the reference's own conventions — `--analyzer uax29`, linear - gains, k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted - saturation constant, single precision, depth 1,000 — the in-memory BM25 reads **0.236** against - the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was - published at. The index is the reference's, not merely a close one: `--fingerprint` reports - 5,288,635 terms, and the per-document term stream is byte-identical to Lucene 10.5.2's - `EnglishAnalyzer` over the corpus (Extended_Pictographic symbols, one term per Han ideograph, - combining marks, format characters, the 255-unit token chop, U+202F). Multi-code-point emoji - sequences are the one knowingly unimplemented rule — FiQA contains none, and a test pins the - divergence. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`, - k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate - cost. Where the gap sits was measured rather than assumed: a PostgreSQL configuration carrying the - reference's 33-word stop list and its Snowball stemmer reads **0.2196**, so the stop list is not - the cause — the longer stock list helps here — and what remains is the stemmer and the parser, - which stock PostgreSQL cannot replace without an extension. The gate's direction is SciFact's: - the ungated candidates are chosen by coverage and truncated, the gated ones by idf. + but not part of `--dataset all` or the pinned comparison. At k1=0.9/b=0.4, linear gains, depth + 1,000 and the library's default arithmetic, the stock-PostgreSQL BM25 row with `english` reads + **0.2291** at the default gate (1,000 rows) and **0.2430** ungated. The 0.2291 this page used to + attribute to a ~0.007 analysis gap is the gate's recall bound, not the analyzer: ungated, + PostgreSQL's analysis is above the reference analysis (0.2378 in memory) and above the published + 0.2361, which is a different analysis under the reference conventions. The budget is the lever: + 500 → 0.2297, 1,000 → 0.2291, 2,000 → 0.2363, 5,000 → 0.2417, exact → 0.2430. The default is + tuned for NFCorpus and SciFact, where it is free; short-query corpora pay for it, and + `--backend-gate-budget` is the knob. The stop list is not behind the difference either: at the + same gate budget, a PostgreSQL configuration with the reference's 33-word list reads 0.2196, so + the longer stock list helps. The published 0.2361 remains reproduced by the in-memory `uax29` row + under the reference's conventions, with an index byte-identical to Lucene 10.5.2's; multi-code-point + emoji sequences are the one knowingly unimplemented rule. - **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable** with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric columns compare either way. The analysis is PostgreSQL's own, not this library's tokenizer, which diff --git a/docs/index.md b/docs/index.md index 0782d6d..b5f60d0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -119,8 +119,8 @@ twice. Every figure below is reproducible with a command on the term, and OR-ing them (`--backend-match any`) recovers **0.0988** on SciFact but only **0.1881** on NFCorpus, so the rest of that gap is the ranker. The stock-PostgreSQL BM25 engine closes it with no extension: **0.3072** and **0.6665** at its default candidate gate, **0.2999** and - **0.5636** with the gate off; on FiQA, the corpus of another nature, **0.2061** against - **0.1077**. pgvector at `ef_search` 10/40/160 reads **0.2882 / 0.2960 / 0.3013** on + **0.5636** with the gate off; on FiQA, the corpus of another nature, **0.2291** at the default + gate against **0.2430** ungated at k1=0.9/b=0.4 — the gate is the lever there. pgvector at `ef_search` 10/40/160 reads **0.2882 / 0.2960 / 0.3013** on NFCorpus against the exact in-memory dense row's **0.304**, and **0.5998 / 0.6528 / 0.6550** on SciFact against **0.655**: that gap is the index's cost in nDCG. ArguAna produces no row from any of the three: the text engines reject 2 of its 1,406 test queries, and the vector engine rejects