[Manohar Paturi (ManoharPaturi)]

Description

Fixes #3664

value_search is documented to match stored values containing the text while ignoring case, but on SQLite the filter compiled to lower(value) LIKE lower(pattern) and SQLite's lower() only folds ASCII letters. A seed stored as CAFÉ au lait tactics was found by searching CAFÉ but missed by searching café, while the same data matched both on Azure SQL.

The search now folds the column with the existing Unicode-aware UnicodeLower (the same machinery the harm category filters already use for exactly this reason) and folds the pattern in Python, so both backends apply the same folding. Wildcard escaping is unchanged.

Tests and Documentation

Added a regression next to the harm filter's non-ASCII folding test, covering the accented pair in both directions plus an escaped wildcard value. It fails on main and passes with the change. The full memory interface suite passes (759 tests) and ruff is clean.

No documentation change needed, so I did not run JupyText.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Manohar Paturi (ManoharPaturi)]

Good catch on the final sigma, and the casefold suggestion is exactly right: ΑΣ lower()s to ας while ΑΣΤΡΟ lower()s to αστρο, so my two sided lower lost a match that main's accidental case sensitive equality used to catch. Both sides now fold through a new UnicodeCaseFold in d873599 (casefold registered on SQLite connections, ASCII fast path reused, SQL Server LOWER which is per character, and UnicodeLower left untouched for the profile keys), with the pattern folded in SQL through the same element so nothing maps independently of the other side. A Greek final sigma regression stores ΑΣΤΡΟ and searches both ΑΣ and ασ; it fails on the previous head and passes now. The seed example and sqlite suites pass (121 tests) and ruff is clean.

[Manohar Paturi (ManoharPaturi)]

Both review items are in d572272.

The docstring now uses plain double backticks, and that was also the pre-commit failure: the repo's rest role guard rejects :class: because the docs render with MyST, so the same line broke pre-commit (ubuntu/macos). Verified locally against the guard itself now, not just ruff.

The per dialect compile test you asked for covers the mssql LOWER( compilation and the SQLite side in the same case: the ASCII fast path stays plain lower( with the registered pyrit_analytics_casefold fallback in the compiled expression. The migration suite passes at 16 tests and the memory interface tree at 776.

On the other two red jobs: main-job (ubuntu, 3.14, dev) failed on test_xss_javascript_uri_near_miss_does_not_backtrack with the near miss taking 1.116 seconds, a wall clock bound on a scorer this PR never touches, and coverage failed only on its per file gate for my new analytics_sql.py lines, which this compile test now covers.

[hannahwestra25]

Not blocking this PR: analytics search (AttackAnalyticsQueryCompiler._search_condition) still uses UnicodeLower on both sides, so ΑΣ misses ΑΣΤΡΟ there. Worth a separate issue? A shared helper would fix both call sites:

# pyrit/memory/analytics_sql.py (add `import re`)
def unicode_contains(*, value: ColumnElement[Any], search: str) -> ColumnElement[bool]:
    """
    Match a literal substring, folding case per character on both sides.

    Returns:
        ColumnElement[bool]: A case-insensitive substring predicate.
    """
    escaped = re.sub(r"([\\%_\[])", r"\\\1", search)
    return UnicodeCaseFold(value).like(UnicodeCaseFold(literal(f"%{escaped}%", UnicodeText())), escape="\\")


# attack_analytics_query.py, _search_condition:
#     return unicode_contains(value=label, search=search)
# memory_interface.py, value_search filter (drop `pattern` and the now-unused
# UnicodeCaseFold/UnicodeText imports):
#     unicode_contains(value=SeedEntry.value, search=value_search),

[Manohar Paturi (ManoharPaturi)]

All four points are in e323861.

The mssql compile is pinned exactly (LOWER(:param_1)), so a wrapper around the fallback can no longer pass. A statement level test compiles the real two sided search for both dialects and asserts the fold marker appears exactly twice, column and pattern, so folding the pattern in Python while SQL Server mixes LOWER and casefold can no longer slip through. The casefold docstring now scopes the sharp s expansion to SQLite's registered function and says SQL Server compiles to plain LOWER where expansions follow the collation, and a sharp s regression next to the final sigma one pins STRASSE matching Straße on SQLite. The duplicated CASE expression moved into one shared helper used by both fold elements, so the BLOB length guard and ASCII fast path live in a single place.

The seed example suite passes with the new regressions and the migration suite at 17. One preexisting note: tests/unit/memory/memory_interface/test_interface_attack_results.py shows 11 failures on my machine, but no file in this PR executes in those tests and they fail identically with the casefold commits reverted, so they look environmental rather than related.

[hannahwestra25]

Bumping the follow-up from my last review body in case it got lost: analytics _search_condition still uses UnicodeLower on both sides, so ΑΣ misses ΑΣΤΡΟ there. Not blocking, and a separate issue is fine. Same fix as the seed search:

@@ pyrit/memory/attack_analytics_query.py imports @@
+    UnicodeCaseFold,
     UnicodeLower,
     UnicodeStrip,
@@ _search_condition @@
-        return UnicodeLower(label).like(UnicodeLower(literal(f"%{escaped}%", UnicodeText())), escape="\\")
+        return UnicodeCaseFold(label).like(UnicodeCaseFold(literal(f"%{escaped}%", UnicodeText())), escape="\\")

[Manohar Paturi (ManoharPaturi)]

You were right and this one is on me: the helper refactor left the decorator on _sqlite_fold_case, whose keyword only signature cannot take the element and compiler, so every SQLite UnicodeLower query raised TypeError and took the analytics endpoint down with it. Fixed in f5d8374: the decorator is back on _sqlite_unicode_lower, the helper takes value and fallback_function keyword only per the style guide, and both call sites use the keyword form. Verified both elements compile to their CASE expressions with the registered fallbacks on the sqlite dialect, then ran all of tests/unit/memory as you asked: 1926 passed, 1 skipped, and the single failure is the preexisting test_operation_id_filter_uses_operation_id_index[id_and_legacy_name] case, which fails identically on the pristine tree and touches nothing in this PR.

[Manohar Paturi (ManoharPaturi)]

Both items are in: the compiler fix landed as f5d8374 (decorator back on _sqlite_unicode_lower, helper keyword-only per the style guide), and the statement test is reworked in c00d012.

The test no longer builds its own select. It calls _seed_example_filters directly with a value search and compiles the statement the seed search actually produces for both dialects, asserting the fold marker appears exactly twice, column and pattern. Mutation checked: switching the pattern to literal(pattern.casefold(), ...) in memory_interface drops the count to one on SQLite and the test fails, which is exactly the Python-side folding it existed to catch. The synthetic statement test is gone.

All of tests/unit/memory runs green at 1926 passed; the single failure in test_attack_analytics is the preexisting operation_id_index case that fails identically on the untouched tree, and I confirmed it fails the same way with my changes stashed.

[hannahwestra25]

Analytics facet search on SQLite still misses exact matches: _search_condition lowers both sides, so ΑΣ becomes %ας% but ΑΣΤΡΟ becomes αστρο. Using UnicodeCaseFold fixes all three facet paths; patch and test below (test fails on head). Would you mind updating?

diff --git a/pyrit/memory/attack_analytics_query.py b/pyrit/memory/attack_analytics_query.py
--- a/pyrit/memory/attack_analytics_query.py
+++ b/pyrit/memory/attack_analytics_query.py
@@ -55,6 +55,7 @@ from pyrit.memory.analytics_sql import (
     JsonIsArray,
     JsonObjectAggregate,
     JsonScalar,
+    UnicodeCaseFold,
     UnicodeLower,
     UnicodeStrip,
 )
@@ -689,7 +690,7 @@ class AttackAnalyticsQueryCompiler:
             ColumnElement[bool]: A case-insensitive substring predicate.
         """
         escaped = search.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_").replace("[", "\\[")
-        return UnicodeLower(label).like(UnicodeLower(literal(f"%{escaped}%", UnicodeText())), escape="\\")
+        return UnicodeCaseFold(label).like(UnicodeCaseFold(literal(f"%{escaped}%", UnicodeText())), escape="\\")
 
     def results(self, *, limit: int, after: DecodedKeysetCursor | None = None) -> Select[*tuple[Any, ...]]:
         """
diff --git a/tests/unit/memory/test_attack_analytics.py b/tests/unit/memory/test_attack_analytics.py
--- a/tests/unit/memory/test_attack_analytics.py
+++ b/tests/unit/memory/test_attack_analytics.py
@@ -236,6 +236,31 @@ async def test_non_ascii_case_variants_share_group_filter_and_facet_keys(sqlite_
         assert (await connection.execute(select(UnicodeLower(literal("\u00c9COLE"))))).scalar_one() == "\u00e9cole"
 
 
+@pytest.mark.parametrize("search", ["\u0391\u03a3", "\u039f\u03a3"])
+@pytest.mark.parametrize(
+    "dimension",
+    [
+        AttackAnalyticsDimension(name="label", label_key="case"),
+        AttackAnalyticsDimension(name="targeted_harm_category"),
+        AttackAnalyticsDimension(name="attack_type"),
+    ],
+    ids=["label", "category", "attack_type"],
+)
+async def test_facet_search_matches_greek_sigma_verbatim(
+    *, sqlite_instance: SQLiteMemory, dimension: AttackAnalyticsDimension, search: str
+) -> None:
+    astros = "\u0391\u03a3\u03a4\u03a1\u039f\u03a3"
+    await sqlite_instance.add_attack_results_to_memory_async(
+        attack_results=[
+            make_result(labels={"case": astros}, categories=[astros], attack_class_name=astros),
+            make_result(index=2, labels={"case": "other"}, categories=["other"], attack_class_name="Other"),
+        ]
+    )
+    query = AttackAnalyticsFacetQuery(dimension=dimension, search=search)
+    facet = await AttackAnalyticsReader(memory=sqlite_instance).facets_async(query=query, control=control())
+    assert [item.label.casefold() for item in facet.items] == [astros.casefold()]
+
+
 @pytest.mark.parametrize(
     ("value", "fallback"),
     [