Document how query conditions treat null and omitted values (5.4)

reference/database/schema.md

@@ -358,7 +358,7 @@ type Product @table {
 
 If the field value is an array, each element in the array is individually indexed, enabling queries by any individual value.
 
-Null values are indexed by default (added in v4.3.0), enabling queries like `GET /Product/?category=null`.
+Null values are indexed by default (Added in: v4.3.0), enabling queries like `GET /Product/?category=null`. That query matches records that store `null`, not records that omit the attribute; see [Null and Omitted Values](../rest/querying.md#null-and-omitted-values).
 
 ### `@embed`
 

reference/operations-api/operations.md

@@ -301,7 +301,7 @@ Returns records with a matching value on any attribute. Supports wildcards (e.g.
 
 ### `search_by_conditions`
 
-Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits.
+Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. Conditions match null and omitted values by the rules in [Null and Omitted Values](../rest/querying.md#null-and-omitted-values).
 
 ```json
 {
@@ -2425,7 +2425,7 @@ Detailed documentation: [Analytics Operations](../analytics/operations.md)
 
 ### `get_analytics`
 
-Retrieves analytics data. Supports `start_time`/`end_time` (Unix ms), `get_attributes`, and `conditions` (same format as `search_by_conditions`).
+Retrieves analytics data. Supports `start_time`/`end_time` (Unix ms), `get_attributes`, and `conditions` (same format as `search_by_conditions`, including its [null and omitted value rules](../rest/querying.md#null-and-omitted-values)).
 
 ```json
 {

reference/operations-api/sql.md

@@ -97,6 +97,16 @@ WHERE d.owner_name IN ('Kyle', 'Zach')
 ORDER BY d.dog_name
 ```
 
+### NULL and Omitted Values
+
+`WHERE` treats an omitted attribute as `NULL`:
+
+- `IS NULL` matches records that store `null` and records that omit the attribute. `IS NOT NULL` matches the rest.
+- `<>` and `NOT IN` with non-null values exclude `NULL` and omitted values.
+- <VersionBadge type="changed" version="v5.4.0" /> With the default `auto` engine, a range (`<`, `<=`, `>`, `>=`, `BETWEEN`) compared with a `NULL` literal matches no rows. Before v5.4.0, it could fail with an `Invalid value for attribute` error, and `BETWEEN NULL AND x` could return rows whose value is `NULL`. This change is pending release; see [Match null and omitted values the same way whether a condition leads the scan or filters](https://github.com/HarperFast/harper/pull/3925).
+
+Use `IS NULL` and `IS NOT NULL` to test for `NULL`. REST and Resource API conditions follow different rules: `ne` with a non-null value matches null and omitted values, and a `null` lower bound means no lower bound. See [Null and Omitted Values](../rest/querying.md#null-and-omitted-values).
+
 ---
 
 ## Query Performance
@@ -155,7 +165,7 @@ For queries the new engine plans (the default under `auto`), a handful of non-st
 - **`OFFSET` past the end of a result returns no rows.** The legacy engine could still return rows when the `OFFSET` exceeded the number of matches.
 - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER(<null>)` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`).
 
-These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior.
+These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values).
 
 ### Practical guidance
 

reference/resources/query-optimization.md

@@ -36,15 +36,15 @@ When a query is executed, conditions are evaluated against the database. Indexed
 
 ### Index Performance Characteristics
 
-| Operator                                                             | Uses index         | Notes                                                                    |
-| -------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------ |
-| `equals`                                                             | Yes                | Fast lookup in sorted index                                              |
-| `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal` | Yes                | Range scan in sorted index; narrower range = faster                      |
-| `starts_with`                                                        | Yes                | Prefix search in sorted index                                            |
-| `not_equal`                                                          | No                 | Full scan required (unless combined with selective indexed condition)    |
-| `contains`                                                           | No                 | Full scan required                                                       |
-| `ends_with`                                                          | No                 | Full scan required                                                       |
-| `!= null`                                                            | Yes (special case) | Can use indexes to find non-null records; only helpful for sparse fields |
+| Operator                                                             | Uses index         | Notes                                                                                                                                           |
+| -------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| `equals`                                                             | Yes                | Fast lookup in sorted index                                                                                                                     |
+| `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal` | Yes                | Range scan in sorted index; narrower range = faster                                                                                             |
+| `starts_with`                                                        | Yes                | Prefix search in sorted index                                                                                                                   |
+| `not_equal` with a non-null value                                    | No                 | Full scan required (unless combined with selective indexed condition); a `sort` on the same attribute runs in memory, except on the primary key |
+| `contains`                                                           | No                 | Full scan required                                                                                                                              |
+| `ends_with`                                                          | No                 | Full scan required                                                                                                                              |
+| `!= null`                                                            | Yes (special case) | Can use indexes to find non-null records; only helpful for sparse fields                                                                        |
 
 **Rule of thumb**: Use `equals`, range operators, and `starts_with` on indexed fields. Avoid `contains`, `ends_with`, and `not_equal` as the sole or first condition in large datasets.
 
@@ -169,10 +169,12 @@ type Brand @table {
 
 Sorting can significantly impact query performance.
 
-- **Aligned sort and index**: If the sort attribute is the same indexed field used in the primary condition, Harper can use the index to retrieve results already in order — very fast.
-- **Unaligned sort**: If the sort is on a different field than the condition, or the sort field is not indexed, Harper must retrieve and sort all matching records. For large result sets this can be slow, and it also **defeats streaming** (see below).
+- **Aligned sort and index**: If the sort attribute is the same indexed field used in the primary condition, and that condition reads the field's index (such as `equals`, a range, or `starts_with`), Harper can use the index to retrieve results already in order — very fast.
+- **Unaligned sort**: If the sort is on a different field than the condition, the sort field is not indexed, or the condition on it cannot read its index (such as `not_equal` with a non-null value on an attribute other than the primary key), Harper must retrieve and sort all matching records. For large result sets this can be slow, and it also **defeats streaming** (see below). A `limit` does not reduce this work: Harper sorts every match before returning the page.
 
-Best practice: sort on the same indexed field you are filtering on, or sort on a secondary indexed field with a narrow enough condition to produce a manageable result set.
+<VersionBadge type="changed" version="v5.4.0" /> Before v5.4.0, a sort on the same attribute as an unindexed condition or a `not_equal` condition could return records in primary-key order; Harper now sorts them. This change is pending release; see [Null and Omitted Values](../rest/querying.md#null-and-omitted-values).
+
+Best practice: sort on the same indexed field you are filtering on with an index-reading condition, or sort on a secondary indexed field with a narrow enough condition to produce a manageable result set.
 
 ## Streaming
 

reference/resources/resource-api.md

@@ -1068,6 +1068,25 @@ Product.search({ conditions: [{ attribute: ['brand', 'name'], value: 'Harper' }]
 
 <VersionBadge version="v4.3.0" />
 
+#### Null and omitted values
+
+<VersionBadge type="changed" version="v5.4.0" />
+
+:::note Pending release
+
+Targeted for v5.4.0; not released yet. See [Match null and omitted values the same way whether a condition leads the scan or filters](https://github.com/HarperFast/harper/pull/3925).
+
+:::
+
+Conditions treat a stored `null` and an omitted attribute the same way, except `equals` with a `null` value, which matches a stored `null` only. `not_equal` with a non-null value matches null and omitted values, and `not_equal` with `null` matches values that are present and not `null`. No range comparator matches a null or omitted value, and a `null` lower bound means no lower bound. On an array attribute, `not_equal` matches a record when at least one element differs. These rules hold whether or not the attribute is indexed, and whichever condition runs first. See [Null and Omitted Values](../rest/querying.md#null-and-omitted-values) for the full rules and the sample data used here:
+
+```javascript
+// ids 2, 3, 4, 5 and 6: the null and omitted discounts match
+Product.search({ conditions: [{ attribute: 'discount', comparator: 'not_equal', value: 5 }] });
+// ids 1 and 2: the same records as less_than_equal 10
+Product.search({ conditions: [{ attribute: 'discount', comparator: 'between', value: [null, 10] }] });
+```
+
 ### `operator`
 
 Top-level `and` (default) or `or` for the `conditions` array.
@@ -1233,7 +1252,7 @@ Sort order object:
 
 For top-level collection queries, Harper uses an index to provide sort order, so a `sort` needs one of:
 
-- An `@indexed` sort `attribute`. Harper aligns the scan with the index automatically — **no condition is required**, and the condition (if any) does not have to be on the sort attribute.
+- An `@indexed` sort `attribute`. Harper aligns the scan with the index automatically — **no condition is required**, and the condition (if any) does not have to be on the sort attribute. A condition on the sort attribute that cannot read its index is the exception; see below.
 - At least one `conditions` entry (on **any** attribute) when the sort `attribute` is not indexed. Harper filters by the condition and then orders the result set in memory.
 
 Only the combination of a non-indexed sort `attribute` **and** zero conditions is rejected:
@@ -1253,6 +1272,8 @@ Product.search({
 
 Alternatively, pass `allowFullScan: true` to permit an unconditional ordered scan, or — if order doesn't matter — omit `sort` entirely and `search({})` will iterate without an index requirement.
 
+<VersionBadge type="changed" version="v5.4.0" /> An indexed sort `attribute` does not guarantee an index-ordered scan. When the condition Harper runs first is on the sort attribute but cannot read that attribute's index, as with `not_equal` and a non-null value on an attribute other than the primary key, Harper collects and sorts every matching record in memory before applying `offset` and `limit`. Before v5.4.0, such a query, or one whose condition was on an unindexed sort attribute, could return records in primary-key order. See [Sorting](./query-optimization.md#sorting).
+
 <VersionBadge type="changed" version="v5.4.0" /> When `attribute` is a path array that traverses a to-many relationship, each parent's sort key is the list of its related records' values in relationship order: the related records' primary-key order in the foreign-key index for a `@relationship(to: ...)` field (not sorted by the target attribute or reduced to a min/max), or the stored foreign-key array order for a `@relationship(from: ...)` array. Top-level `conditions` on a relationship restrict which parents match and which children `select` returns, but do not narrow the related records loaded into each parent's sort key. Lists compare element by element (`['alpha']` < `['alpha', 'delta']` < `['bravo']`), and `descending: true` reverses that order. A parent with no related records has an empty list (`[]`), a related record that omits the sort attribute (or an unresolved ID in a `@relationship(from: ...)` array without `filterMissing`) produces `undefined` at that position, and an explicit `null` attribute produces `null`: both `null` and `undefined` sort before booleans, numbers, and strings (`[]` and `[undefined]` compare as equal because reading past the end of a shorter array yields `undefined`). Across multiple to-many hops, each hop produces a list in relationship order, so two to-many hops compare a nested list of lists per parent (`[]` < `[[], ['a']]` < `[['b']]`). The index-or-condition rule above checks `@indexed` on the target attribute at the end of the path. List-per-parent sorting runs in memory when the target attribute is not `@indexed` (with a condition or `allowFullScan: true`), when a more selective condition on another attribute leads the query, or when `enforceExecutionOrder: true` is set, loading the related records for every matched parent; if the target attribute is `@indexed` and no more selective condition on another attribute leads the query, Harper instead drives the scan from the related table's index join (yielding each parent at its first matching index entry and, when unconditional, omitting parents with no related records). Note that, unlike a nested `select` sort (`select: [{ name: 'products', sort: { attribute: 'price' } }]`), which checks read permissions on the child sort attribute and orders children within each parent, a top-level `sort` orders the parent records and does not currently enforce `attribute_permissions` on the related sort attribute, so restricting read access to a child attribute does not prevent ordering parent records by it:
 
 ```javascript