[kriszyp]

⊙ Problem

Match null and omitted values the same way whether a condition leads the scan or filters changes which records REST, Resource API and search_by_conditions conditions return for null and omitted values. The reference never said how conditions treat those values, and REST did not document =between= at all.

🤔 Does the brief's "SQL is unchanged" hold?

Partly. <>, NOT IN and IS NULL are unchanged. A range compared with NULL did change: it used to fail on a typed attribute, or return NULL rows from BETWEEN NULL AND x. Now it matches no rows. The SQL page and the release note say so.

💡 Solution

• REST querying gets a Null and Omitted Values section: the rules, a six-record sample, and the records each query matches.
• REST documents =between= and its gele, gelt, gtle and gtlt forms.
• The Resource API, SQL, operations API, schema and query-optimization pages link to it and say what differs there.
• The sorting docs say a sort on the attribute of a non-null ne runs in memory (except on the primary key), and limit does not reduce that work.
• A 5.4 release-note entry with upgrade impact.

Behavior new in 5.4 has a v5.4.0 badge and, on each page that explains it, a pending-release note.

⚖️ Alternatives

• Put the rules on the Resource API page — the examples are REST URLs, and REST is the page that lacked between syntax.
• A standalone null-values page — about 30 lines of rules, split away from the condition syntax they describe.

🤔 Publishing the range aliases

=gele=, =gelt=, =gtle= and =gtlt= become documented REST syntax. They already parse, and each was checked, including values equal to a bound. Documented syntax is hard to retire, so drop those rows if they should stay internal.

🔧 Changes

• reference/rest/querying.md — the =between= and range-alias operator rows, the Null Queries clarification that ==null misses omitted attributes, the new Null and Omitted Values section with its rules and sample results, and the in-memory sort paragraph. Look hardest at the rules list: every other page links to it.
• reference/resources/resource-api.md — a null and omitted values subsection under conditions with two search() examples, the indexed-sort bullet pointing at its exception, and the sort exception for conditions that cannot read the index.
• reference/resources/query-optimization.md — the not_equal row, the Unaligned sort bullet with the primary-key exception, and the v5.4.0 sort note.
• reference/operations-api/sql.md — a NULL and Omitted Values subsection, and a pointer to it from the legacy-engine changes list.
• reference/operations-api/operations.md — links from search_by_conditions and get_analytics.
• reference/database/schema.md — the null-indexing note says ?category=null does not match omitted attributes.
• release-notes/v5-lincoln/5.4.md — the release-note entry with upgrade impact.

✅ Verification

• Built harper at the PR head (b4fa7a988) and at its merge-base (81ddff7b5). A script asserted every published example on an indexed and an unindexed copy of the sample table, over REST, Table.search(), search_by_conditions and SQL: 96/96 pass on the head. On the merge-base 43 fail, each showing documented old behavior.
• explain: true shows in-memory sorting for not_equal sorted on the same attribute, and none for the primary key or a range.
• npm run build, format:check, lint, typecheck and test:examples pass.

What the merge-base returned instead

• Indexed discount=ne=5: 2, 5 (null and omitted dropped); the same through search() and search_by_conditions.
• ge=null, between=(null,10), gele/gelt with a null lower bound: also returned the stored null (record 3), indexed and unindexed.
• tags=ne=sale: indexed 2 only; unindexed 2, 3, 4, 5, 6 (whole-array comparison). Unindexed tags=ne=null included [].
• Unindexed discount=ne=5&sort(discount) and an unindexed u=lt=3&sort(u): primary-key order.
• SQL on a typed attribute: discount >= NULL and BETWEEN NULL AND 10 failed with Invalid value for attribute discount. On an untyped attribute, BETWEEN NULL AND 3 returned the NULL row.

Refs HarperFast/harper#3925

Depends-on: HarperFast/harper#3925

PR QA: skipped @ d4c017a — HarperFast/documentation PR; PR-time QA covers harper and harper-pro

_{🤖 Generated by Claude Opus 5.5 (Claude Code); posted via @kriszyp.}

Release target: v5.4 — documents harper PR #3925, which is milestoned v5.4

Related PRs: #784 overlaps (edits the same search_by_conditions line in operations.md and inserts at the same point after conditions in resource-api.md), #779 overlaps (adjacent edits to the resource-api.md sort section and the 5.4 release notes), 3 others independent
Complexity: medium

Origin — the dispatch brief this PR was written from

Document how query conditions treat null and omitted values (harper v5.4)

Document how REST and Resource-API query conditions treat null and omitted attribute values, after Match null and omitted values the same way whether a condition leads the scan or filters (harper v5.4).

What changed (applies whichever condition leads, indexed or not):

• ne with a non-null value (?b=ne=5) matches rows where the attribute is null or omitted. Before, an indexed ne that was the only condition (or led the query) dropped those rows.
• No range comparator (lt, le, gt, ge, between and its gele/gelt/gtlt/gtle variants) ever matches a null or omitted value. A null lower bound means "no lower bound": ?b=between=(null,3) returns the same rows as ?b=le=3, and ?b=ge=null returns every row with a non-null b. Before, ge=null and between=(null,x) also returned null rows.
• On an array attribute, ne v matches a row when some element is not v; a row whose only elements equal v (['x'], ['x','x']) does not match, an empty array does not match ne null, and a null or omitted array matches ne v.
• A sort on the same attribute as a condition that cannot use that attribute's index (an unindexed attribute, a negated condition, a non-null ne) now returns rows in sort order. Before, such results came back in primary-key order.
• SQL is unchanged: <> and NOT IN exclude NULL, IS NULL matches null and omitted, and a range compared with a NULL literal matches no rows.

Where: the REST querying reference (comparison operators, ne, between, null values, sorting) and the Resource API search conditions reference. Add a short "null and missing values" note with the rules above and one or two examples; keep it in the v5.4 docs, with a release-note line for the changed ne, null-bound range and array ne behavior.

Verify every example against a v5.4 build (or the PR branch) before publishing.

_{Dispatch: task docs-harper-3925 · queued by task:harper-3731-null-omitted-scan-filter-parity · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Ready-For-Promotion: d4c017a urgency=normal late=none}

_{Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex; adjudicated=domain; declined=cursor-grok,muse; rounds=5; full=1 @ d4c017a}

_{Review-Attention: skim ~1m @ d4c017a}

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-783

This preview will update automatically when you push new commits.

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-783

This preview will update automatically when you push new commits.

[github-actions]

🧹 Preview Cleanup

The preview deployment for this PR has been removed.