[chrikrah]

Motivation

Second bug of #821, as @vishal-bala invited there. _tokenize_and_escape_query checks _stopwords and _text_weights after escaping. The word the user typed never matches: it's is tested as it\'s against a set that contains it's, and a don't weight key never applies.

Changes

Membership runs on the raw normalized word. Escaping feeds only the clause. The normalization expression stays byte-identical. Three tests cover the stopword, the weight key, and the clause AggregateHybridQuery sends.

before  _tokenize_and_escape_query("it's the fox")   "it\'s | fox"
after                                               "fox"
before  text_weights={"don't": 2.0}, "don't jump"    "don\'t | jump"
after                                               "don\'t=>{$weight:2.0} | jump"

Notes

Only the shared helper changes, so TextQuery keeps its own tokenizer copy and gets this behavior once #831 moves it onto the helper. The two share no lines, so either can merge first.

Release Notes

AggregateHybridQuery and HybridQuery now match stopwords and text weights against the word the user typed, punctuation included. Contractions like it's drop from scoring and weighted keys like don't apply, so results change for queries with such words.

Reviewer Brief

Start at _tokenize_and_escape_query: membership on word, escape output feeds only the clause. Falsification: with the fix reverted the three new tests fail; with it restored the touched suites are green.

$ pytest -q --noconftest tests/unit/test_full_text_query_helper.py tests/unit/test_text_query_weights.py tests/unit/test_query_types.py tests/unit/test_aggregation_types.py tests/unit/test_hybrid_types.py tests/unit/test_filter.py   # base 87d5c17, with this patch
288 passed, 1 skipped, 115 warnings in 0.71s

$ pytest -q --noconftest tests/unit/test_full_text_query_helper.py   # fix reverted, new tests kept
3 failed in 0.25s
# not run: live Redis, the integration suite

Fixes #821 (the second bug; #831 covers the first).