Skip to content

Document the two tsvector limits on term frequency - #85

Merged
manuc66 merged 1 commit into
mainfrom
docs/tsvector-limits
Nov 18, 2026
Merged

manuc66 merged 1 commit into
mainfrom
docs/tsvector-limits

Conversation

@manuc66

@manuc66 manuc66 commented Nov 18, 2026

Copy link
Copy Markdown
Owner

What this adds

Two limits of the tsvector column the BM25 engine stores, stated on the backend page:

  • a position is 16 bits, so a document past 16,383 indexed tokens has its later positions clamped;
  • a lexeme stores at most 255 positions per document, so a term repeated more often has its frequency capped.

Both apply to the Database analysis and to any client-side vector, because they are properties of the tsvector type. No corpus measured here approaches the position bound, but FiQA has a document where over occurs more than 255 times: its tf is capped there, the vocabulary and document frequencies are unaffected, and BM25's saturation makes the score effect small. Where the in-memory scorer sees a tf past 255, this engine does not.

Measured on the test server: nentry for repeat('x ', 300) is 255, and the maximum position in a 17,000-token document is 16,383. Docs only, no code change.

A position is 16 bits and a lexeme stores at most 255 positions per
document, so a document past 16,383 indexed tokens has its later positions
clamped and a term repeated more than 255 times has its frequency capped.
FiQA has such a document for 'over'; the vocabulary and document frequencies
are unaffected and BM25's saturation makes the effect small, but the cap is
stated rather than discovered.
@codecov-commenter

codecov-commenter commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.18%. Comparing base (6612a11) to head (9fada9a).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #85   +/-   ##
=======================================
  Coverage   89.18%   89.18%           
=======================================
  Files         271      271           
  Lines       21461    21461           
  Branches     4635     4635           
=======================================
  Hits        19141    19141           
  Misses       1388     1388           
  Partials      932      932           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit 7e0fbff into main Nov 18, 2026
16 checks passed
@manuc66
manuc66 deleted the docs/tsvector-limits branch November 18, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants