diff --git a/reference/database/schema.md b/reference/database/schema.md index 7035be5bf..0f100bf0d 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -358,7 +358,7 @@ type Product @table { If the field value is an array, each element in the array is individually indexed, enabling queries by any individual value. -Null values are indexed by default (added in v4.3.0), enabling queries like `GET /Product/?category=null`. +Null values are indexed by default (Added in: v4.3.0), enabling queries like `GET /Product/?category=null`. That query matches records that store `null`, not records that omit the attribute; see [Null and Omitted Values](../rest/querying.md#null-and-omitted-values). ### `@embed` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 981103058..61ba89a48 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -301,7 +301,7 @@ Returns records with a matching value on any attribute. Supports wildcards (e.g. ### `search_by_conditions` -Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. +Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. Conditions match null and omitted values by the rules in [Null and Omitted Values](../rest/querying.md#null-and-omitted-values). ```json { @@ -2425,7 +2425,7 @@ Detailed documentation: [Analytics Operations](../analytics/operations.md) ### `get_analytics` -Retrieves analytics data. Supports `start_time`/`end_time` (Unix ms), `get_attributes`, and `conditions` (same format as `search_by_conditions`). +Retrieves analytics data. Supports `start_time`/`end_time` (Unix ms), `get_attributes`, and `conditions` (same format as `search_by_conditions`, including its [null and omitted value rules](../rest/querying.md#null-and-omitted-values)). ```json { diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 12a5ce71b..4667acff2 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -97,6 +97,16 @@ WHERE d.owner_name IN ('Kyle', 'Zach') ORDER BY d.dog_name ``` +### NULL and Omitted Values + +`WHERE` treats an omitted attribute as `NULL`: + +- `IS NULL` matches records that store `null` and records that omit the attribute. `IS NOT NULL` matches the rest. +- `<>` and `NOT IN` with non-null values exclude `NULL` and omitted values. +- With the default `auto` engine, a range (`<`, `<=`, `>`, `>=`, `BETWEEN`) compared with a `NULL` literal matches no rows. Before v5.4.0, it could fail with an `Invalid value for attribute` error, and `BETWEEN NULL AND x` could return rows whose value is `NULL`. This change is pending release; see [Match null and omitted values the same way whether a condition leads the scan or filters](https://github.com/HarperFast/harper/pull/3925). + +Use `IS NULL` and `IS NOT NULL` to test for `NULL`. REST and Resource API conditions follow different rules: `ne` with a non-null value matches null and omitted values, and a `null` lower bound means no lower bound. See [Null and Omitted Values](../rest/querying.md#null-and-omitted-values). + --- ## Query Performance @@ -155,7 +165,7 @@ For queries the new engine plans (the default under `auto`), a handful of non-st - **`OFFSET` past the end of a result returns no rows.** The legacy engine could still return rows when the `OFFSET` exceeded the number of matches. - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). -These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. +These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. For how `WHERE` compares `NULL` and omitted values, including the v5.4.0 change to ranges compared with `NULL`, see [NULL and Omitted Values](#null-and-omitted-values). ### Practical guidance diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index 3da6e188b..14ef7641e 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -36,15 +36,15 @@ When a query is executed, conditions are evaluated against the database. Indexed ### Index Performance Characteristics -| Operator | Uses index | Notes | -| -------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------ | -| `equals` | Yes | Fast lookup in sorted index | -| `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal` | Yes | Range scan in sorted index; narrower range = faster | -| `starts_with` | Yes | Prefix search in sorted index | -| `not_equal` | No | Full scan required (unless combined with selective indexed condition) | -| `contains` | No | Full scan required | -| `ends_with` | No | Full scan required | -| `!= null` | Yes (special case) | Can use indexes to find non-null records; only helpful for sparse fields | +| Operator | Uses index | Notes | +| -------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| `equals` | Yes | Fast lookup in sorted index | +| `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal` | Yes | Range scan in sorted index; narrower range = faster | +| `starts_with` | Yes | Prefix search in sorted index | +| `not_equal` with a non-null value | No | Full scan required (unless combined with selective indexed condition); a `sort` on the same attribute runs in memory, except on the primary key | +| `contains` | No | Full scan required | +| `ends_with` | No | Full scan required | +| `!= null` | Yes (special case) | Can use indexes to find non-null records; only helpful for sparse fields | **Rule of thumb**: Use `equals`, range operators, and `starts_with` on indexed fields. Avoid `contains`, `ends_with`, and `not_equal` as the sole or first condition in large datasets. @@ -169,10 +169,12 @@ type Brand @table { Sorting can significantly impact query performance. -- **Aligned sort and index**: If the sort attribute is the same indexed field used in the primary condition, Harper can use the index to retrieve results already in order — very fast. -- **Unaligned sort**: If the sort is on a different field than the condition, or the sort field is not indexed, Harper must retrieve and sort all matching records. For large result sets this can be slow, and it also **defeats streaming** (see below). +- **Aligned sort and index**: If the sort attribute is the same indexed field used in the primary condition, and that condition reads the field's index (such as `equals`, a range, or `starts_with`), Harper can use the index to retrieve results already in order — very fast. +- **Unaligned sort**: If the sort is on a different field than the condition, the sort field is not indexed, or the condition on it cannot read its index (such as `not_equal` with a non-null value on an attribute other than the primary key), Harper must retrieve and sort all matching records. For large result sets this can be slow, and it also **defeats streaming** (see below). A `limit` does not reduce this work: Harper sorts every match before returning the page. -Best practice: sort on the same indexed field you are filtering on, or sort on a secondary indexed field with a narrow enough condition to produce a manageable result set. + Before v5.4.0, a sort on the same attribute as an unindexed condition or a `not_equal` condition could return records in primary-key order; Harper now sorts them. This change is pending release; see [Null and Omitted Values](../rest/querying.md#null-and-omitted-values). + +Best practice: sort on the same indexed field you are filtering on with an index-reading condition, or sort on a secondary indexed field with a narrow enough condition to produce a manageable result set. ## Streaming diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 7bc962206..1753992c0 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1068,6 +1068,25 @@ Product.search({ conditions: [{ attribute: ['brand', 'name'], value: 'Harper' }] +#### Null and omitted values + + + +:::note Pending release + +Targeted for v5.4.0; not released yet. See [Match null and omitted values the same way whether a condition leads the scan or filters](https://github.com/HarperFast/harper/pull/3925). + +::: + +Conditions treat a stored `null` and an omitted attribute the same way, except `equals` with a `null` value, which matches a stored `null` only. `not_equal` with a non-null value matches null and omitted values, and `not_equal` with `null` matches values that are present and not `null`. No range comparator matches a null or omitted value, and a `null` lower bound means no lower bound. On an array attribute, `not_equal` matches a record when at least one element differs. These rules hold whether or not the attribute is indexed, and whichever condition runs first. See [Null and Omitted Values](../rest/querying.md#null-and-omitted-values) for the full rules and the sample data used here: + +```javascript +// ids 2, 3, 4, 5 and 6: the null and omitted discounts match +Product.search({ conditions: [{ attribute: 'discount', comparator: 'not_equal', value: 5 }] }); +// ids 1 and 2: the same records as less_than_equal 10 +Product.search({ conditions: [{ attribute: 'discount', comparator: 'between', value: [null, 10] }] }); +``` + ### `operator` Top-level `and` (default) or `or` for the `conditions` array. @@ -1233,7 +1252,7 @@ Sort order object: For top-level collection queries, Harper uses an index to provide sort order, so a `sort` needs one of: -- An `@indexed` sort `attribute`. Harper aligns the scan with the index automatically — **no condition is required**, and the condition (if any) does not have to be on the sort attribute. +- An `@indexed` sort `attribute`. Harper aligns the scan with the index automatically — **no condition is required**, and the condition (if any) does not have to be on the sort attribute. A condition on the sort attribute that cannot read its index is the exception; see below. - At least one `conditions` entry (on **any** attribute) when the sort `attribute` is not indexed. Harper filters by the condition and then orders the result set in memory. Only the combination of a non-indexed sort `attribute` **and** zero conditions is rejected: @@ -1253,6 +1272,8 @@ Product.search({ Alternatively, pass `allowFullScan: true` to permit an unconditional ordered scan, or — if order doesn't matter — omit `sort` entirely and `search({})` will iterate without an index requirement. + An indexed sort `attribute` does not guarantee an index-ordered scan. When the condition Harper runs first is on the sort attribute but cannot read that attribute's index, as with `not_equal` and a non-null value on an attribute other than the primary key, Harper collects and sorts every matching record in memory before applying `offset` and `limit`. Before v5.4.0, such a query, or one whose condition was on an unindexed sort attribute, could return records in primary-key order. See [Sorting](./query-optimization.md#sorting). + When `attribute` is a path array that traverses a to-many relationship, each parent's sort key is the list of its related records' values in relationship order: the related records' primary-key order in the foreign-key index for a `@relationship(to: ...)` field (not sorted by the target attribute or reduced to a min/max), or the stored foreign-key array order for a `@relationship(from: ...)` array. Top-level `conditions` on a relationship restrict which parents match and which children `select` returns, but do not narrow the related records loaded into each parent's sort key. Lists compare element by element (`['alpha']` < `['alpha', 'delta']` < `['bravo']`), and `descending: true` reverses that order. A parent with no related records has an empty list (`[]`), a related record that omits the sort attribute (or an unresolved ID in a `@relationship(from: ...)` array without `filterMissing`) produces `undefined` at that position, and an explicit `null` attribute produces `null`: both `null` and `undefined` sort before booleans, numbers, and strings (`[]` and `[undefined]` compare as equal because reading past the end of a shorter array yields `undefined`). Across multiple to-many hops, each hop produces a list in relationship order, so two to-many hops compare a nested list of lists per parent (`[]` < `[[], ['a']]` < `[['b']]`). The index-or-condition rule above checks `@indexed` on the target attribute at the end of the path. List-per-parent sorting runs in memory when the target attribute is not `@indexed` (with a condition or `allowFullScan: true`), when a more selective condition on another attribute leads the query, or when `enforceExecutionOrder: true` is set, loading the related records for every matched parent; if the target attribute is `@indexed` and no more selective condition on another attribute leads the query, Harper instead drives the scan from the related table's index join (yielding each parent at its first matching index entry and, when unconditional, omitting parents with no related records). Note that, unlike a nested `select` sort (`select: [{ name: 'products', sort: { attribute: 'price' } }]`), which checks read permissions on the child sort attribute and orders children within each parent, a top-level `sort` orders the parent records and does not currently enforce `attribute_permissions` on the related sort attribute, so restricting read access to a child attribute does not prevent ordering parent records by it: ```javascript diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 72faec949..b7cbce1a4 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -29,31 +29,38 @@ GET /Product/?category=software&inStock=true -Query for null values or non-null values: +Query for records that store `null`, or for records whose value is present and not `null`: ```http GET /Product/?discount=null +GET /Product/?discount=ne=null ``` +`discount=null` matches records that store `null` in `discount`. A record that omits `discount` matches neither query. See [Null and Omitted Values](#null-and-omitted-values) for how other comparators treat null and omitted values. + Note: Only indexes created in v4.3.0 or later support null indexing. Existing indexes must be rebuilt (removed and re-added) to support null queries. ## Comparison Operators (FIQL) Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompub-fiql-00) syntax for comparison operators: -| Operator | Meaning | -| -------------------- | -------------------------------------- | -| `==` | Equal | -| `=lt=` | Less than | -| `=le=` | Less than or equal | -| `=gt=` | Greater than | -| `=ge=` | Greater than or equal | -| `=ne=`, `!=` | Not equal | -| `=ct=` | Contains (strings) | -| `=sw=`, `==*` | Starts with (strings) | -| `=ew=` | Ends with (strings) | -| `=`, `===` | Strict equality (no type conversion) | -| `!==` | Strict inequality (no type conversion) | +| Operator | Meaning | +| ----------------------------------------- | --------------------------------------- | +| `==` | Equal | +| `=lt=` | Less than | +| `=le=` | Less than or equal | +| `=gt=` | Greater than | +| `=ge=` | Greater than or equal | +| `=ne=`, `!=` | Not equal | +| `=between=(low,high)`, `=gele=(low,high)` | At least `low` and at most `high` | +| `=gelt=(low,high)` | At least `low` and less than `high` | +| `=gtle=(low,high)` | Greater than `low` and at most `high` | +| `=gtlt=(low,high)` | Greater than `low` and less than `high` | +| `=ct=` | Contains (strings) | +| `=sw=`, `==*` | Starts with (strings) | +| `=ew=` | Ends with (strings) | +| `=`, `===` | Strict equality (no type conversion) | +| `!==` | Strict inequality (no type conversion) | Full-text indexes support `=matches=`, `=matches_all=`, `=matches_phrase=`, `=matches_prefix=`, `=matches_fuzzy=`, and `=matches_fuzzy_prefix=`. Each has a negated form: `=not_matches=`, `=not_matches_all=`, `=not_matches_phrase=`, `=not_matches_prefix=`, `=not_matches_fuzzy=`, and `=not_matches_fuzzy_prefix=`. Combine a negated condition with a positive full-text condition on the same index; a structured filter alone does not qualify. These use the full-text index rather than ordinary string comparisons. See [Full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging). @@ -64,6 +71,7 @@ GET /Product/?price=gt=100 GET /Product/?price=le=20 GET /Product/?name==Keyboard* GET /Product/?category=software&price=gt=100&price=lt=200 +GET /Product/?price=between=(100,200) ``` For date fields, colons must be URL-encoded as `%3A`: @@ -128,6 +136,60 @@ If the attribute specifies a type in the schema (e.g., `Float`), values are alwa For strict operators (`=`, `===`, `!==`), no automatic type conversion is applied — the value is decoded as a URL-encoded string, and the attribute type (if declared in the schema) dictates type conversion. +### Null and Omitted Values + + + +:::note Pending release + +Targeted for v5.4.0; not released yet. See [Match null and omitted values the same way whether a condition leads the scan or filters](https://github.com/HarperFast/harper/pull/3925). + +::: + +A record can store `null` in an attribute, or omit the attribute. Comparators treat a null value and an omitted value the same way, except `==null`, which matches a stored `null` only. The rules hold whether or not the attribute is indexed, and whichever condition Harper runs first: + +- `ne` (`!=`) with a non-null value matches null and omitted values. +- No range comparator (`lt`, `le`, `gt`, `ge`, `between`, `gele`, `gelt`, `gtle`, `gtlt`) matches a null or omitted value. +- A `null` lower bound means no lower bound: `ge=null` and `gt=null` match every non-null value, and `between=(null,10)` returns the same records as `le=10`. A `null` upper bound, `lt=null` and `le=null` match nothing. +- On an array attribute, `ne=v` matches a record when at least one element is not `v`. An array whose elements all equal `v` (`["sale"]`, `["sale", "sale"]`) does not match, an empty array matches no `ne` condition, and a null or omitted array matches `ne=v` for a non-null `v`. + +For example, with this schema and data: + +```graphql +type Product @table @export { + id: ID @primaryKey + discount: Int @indexed + tags: [String] @indexed +} +``` + +| `id` | `discount` | `tags` | +| ---- | ---------- | ------------------ | +| 1 | `5` | `["sale"]` | +| 2 | `10` | `["sale", "new"]` | +| 3 | `null` | `["sale", "sale"]` | +| 4 | omitted | `[]` | +| 5 | `15` | `null` | +| 6 | `20` | omitted | + +| Query | Matching `id`s | +| ------------------------------------------ | -------------- | +| `GET /Product/?discount=ne=5` | 2, 3, 4, 5, 6 | +| `GET /Product/?discount=ne=null` | 1, 2, 5, 6 | +| `GET /Product/?discount=null` | 3 | +| `GET /Product/?discount=lt=12` | 1, 2 | +| `GET /Product/?discount=ge=null` | 1, 2, 5, 6 | +| `GET /Product/?discount=between=(null,10)` | 1, 2 | +| `GET /Product/?discount=between=(10,null)` | none | +| `GET /Product/?tags=ne=sale` | 2, 5, 6 | +| `GET /Product/?tags=ne=null` | 1, 2, 3 | + +The table lists which records match, not their order. Sorting puts null and omitted values first, or last with `-`, with no guaranteed order between the two. `GET /Product/?discount=ne=5&sort(discount)` returns 3 and 4 in either order, then 2, 5, 6. + +Before v5.4.0, results depended on the query plan: an indexed `ne` that led the query dropped null and omitted values, `ge=null` and `between=(null,x)` matched stored `null` values, and `ne` compared an unindexed array as a whole. + +SQL follows different rules: `<>` and `NOT IN` exclude null and omitted values, and `IS NULL` matches both. See [NULL and omitted values in SQL](../operations-api/sql.md#null-and-omitted-values). + ## Unions (OR Logic) Use `|` instead of `&` to combine conditions with OR logic: @@ -206,6 +268,8 @@ GET /Product/?rating=gt=3&sort(+name) GET /Product/?sort(+rating,-price) ``` + A sort also applies when a condition on the sort attribute cannot read that attribute's index, as with an unindexed attribute, or `ne` with a non-null value on an attribute other than the primary key (`?discount=ne=5&sort(discount)`). Harper then collects and sorts every matching record in memory before applying `limit()`, so a small page does not reduce the work. Before v5.4.0, such a query could return records in primary-key order. See [Sorting](../resources/query-optimization.md#sorting) for performance guidance. + ## Pagination and Total Count diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 916e1b5af..25dc1efb4 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -20,6 +20,20 @@ Full-text indexes using fulltext 0.5.0 with current field-based declarations, or These corrections are not released yet. +## Null and Omitted Values in Query Conditions + +[Match null and omitted values the same way whether a condition leads the scan or filters](https://github.com/HarperFast/harper/pull/3925) makes REST, Resource API, and `search_by_conditions` conditions return the same records for null and omitted values whichever condition leads the query, and whether or not the attribute is indexed: + +- `ne` with a non-null value (`?discount=ne=5`) matches records whose value is null or omitted. Before, an indexed `ne` that led the query dropped those records. +- No range comparator matches a null or omitted value, and a `null` lower bound means no lower bound: `?discount=between=(null,10)` returns the same records as `?discount=le=10`, and `?discount=ge=null` returns every record with a non-null `discount`. Before, `ge=null` and `between=(null,x)` also returned stored `null` values. +- On an array attribute, `ne=v` matches a record when at least one element is not `v`, and a null or omitted array matches `ne=v` for a non-null `v`. Before, an unindexed `ne` compared the whole array, so `["x", "x"]` and `[]` matched `ne=x`, and `[]` matched `ne=null`; now none of them do. +- A sort on the same attribute as a condition that cannot read that attribute's index, such as a condition on an unindexed attribute or a non-null `ne`, now returns records in sort order instead of primary-key order. +- In SQL, a range compared with a `NULL` literal (`discount >= NULL`, `discount BETWEEN NULL AND 10`) now matches no rows. Before, it could fail with an `Invalid value for attribute` error, or `BETWEEN NULL AND x` could return rows whose value is `NULL`. `<>`, `NOT IN`, and `IS NULL` are unchanged. + +**Upgrade impact:** `ne` queries can return more records, and `ge=null` or `between=(null,x)` queries fewer. A sorted `ne` query such as `?discount=ne=5&sort(discount)&limit(10)` now reads every record and sorts in memory, where it used to read its page from the index and skip null and omitted values, so it takes longer on a large table. See [Null and Omitted Values](/reference/v5/rest/querying#null-and-omitted-values), [NULL and omitted values in SQL](/reference/v5/operations-api/sql#null-and-omitted-values), and [Sorting](/reference/v5/resources/query-optimization#sorting). + +This change is not released yet. + ## Replication ### Dedicated Replication Threads