Repository navigation
FIX: fold non-ASCII case in the finding title search - #4002
Open
Manohar Paturi (ManoharPaturi) wants to merge 7 commits into
Open
Manohar Paturi (ManoharPaturi) wants to merge 7 commits into
Manohar Paturi (ManoharPaturi) wants to merge 7 commits into
Conversation
SQLite lower() only folds ASCII, so a stored value like CAFE with an accent was found by an exact-case search but missed by a lowercase one. The search now folds the column with the Unicode-aware UnicodeLower used by the harm category filters and folds the pattern in Python. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
str.lower is context sensitive for Greek final sigma, so lowering the term separately from the stored text lost verbatim matches like AS in ASTRO. Both sides now fold through a per character UnicodeCaseFold, and a Greek final sigma regression covers the pair. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
The docstring used a Sphinx reST role that PyRIT's MyST docs do not render, which the pre-commit rest role guard rejects; plain double backticks replace it. A per dialect compile test covers the SQLite registered casefold fallback with its ASCII fast path and the SQL Server LOWER compilation. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
The SQLite casefold compiler and UnicodeLower now build their CASE expression through one helper so the BLOB length guard and ASCII fast path cannot drift, the casefold docstring scopes the sharp s expansion to SQLite, the mssql compile is pinned exactly, a statement level test proves the real search folds both sides in SQL on each dialect, and a sharp s regression pins the SQLite expansion. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
The decorator had landed on the shared helper, whose keyword-only signature cannot take the compiler arguments, so every SQLite UnicodeLower query raised TypeError and analytics requests returned 500. The decorator is back on _sqlite_unicode_lower and the helper takes its arguments keyword only. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
The test built its own select, so folding the pattern in Python while SQL Server mixed foldings passed every test. It now compiles the statement the seed search actually produces through _seed_example_filters for both dialects and asserts the fold wraps column and pattern exactly twice, which fails if either side folds in Python. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
The title search compiled to lower(title) LIKE lower(pattern) on SQLite, whose lower() only folds ASCII, so a CAFE titled finding was found by the exact case and missed by the lowercase query. Both sides now fold through the per character UnicodeCaseFold the seed value search already uses, and the literal wildcard test keeps its escaping assertions. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #4001
The finding title search compiled to
lower(title) LIKE lower(pattern)on SQLite, whoselower()only folds ASCII letters, so a finding titledCAFÉ security reviewwas found by searchingCAFÉbut missed by searchingcafé, while the same data matched both directions on Azure SQL.Both sides now fold through the per character
UnicodeCaseFold, the same machinery the seed value search uses for exactly this reason, with the pattern folded in SQL through the same element so neither backend can mix foldings by lowering one side in Python. Wildcard and bracket escaping is unchanged.This branch sits on top of #3665, which introduces
UnicodeCaseFold; it can be rebased onto main the moment that lands.Tests and Documentation
Added a regression storing one accented title and one literal wildcard title, asserting the accented title matches in both fold directions and
100%stays a literal query. It fails without the change. The literal wildcard test keeps its existing assertions and now also confirms the fold appears in the compiled SQL for SQL Server. The findings, memory interface, and analytics migration suites pass (816 tests) and ruff is clean.No documentation change needed, so I did not run JupyText.