diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index b5c774e9a..a072a6d51 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -301,7 +301,7 @@ Returns records with a matching value on any attribute. Supports wildcards (e.g. ### `search_by_conditions` -Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. Conditions match null and omitted values by the rules in [Null and Omitted Values](../rest/querying.md#null-and-omitted-values). +Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. A condition that pairs an `attribute` with nested `conditions` (the Resource API's [relationship groups](../resources/resource-api.md#conditions-on-a-to-many-relationship)) is not supported. Full-text conditions use comparators such as `matches` and `matches_prefix`; see [full-text query execution](../resources/query-optimization.md#full-text-search-work-and-paging) for paging costs and limits. Conditions match null and omitted values by the rules in [Null and Omitted Values](../rest/querying.md#null-and-omitted-values). ```json { diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 1d93fc8ed..bd0e8e1aa 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1031,10 +1031,10 @@ Array of condition objects to filter records. Each condition: | Property | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | +| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`). With `conditions`, a relationship name or path; see [relationship groups](#conditions-on-a-to-many-relationship) | | `value` | The value to match | | `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal`, or a [full-text comparator](../full-text-search/querying.md#match-modes) | -| `conditions` | Nested conditions array | +| `conditions` | Nested conditions array; with `attribute`, conditions on the related table that one related record must meet (no `comparator` or `value`) | | `operator` | `and` (default) or `or` for the nested `conditions` | | `fields` | Full-text source fields to search; every name must belong to the selected full-text index | | `includeHighlights` | Include configured full-text highlights | @@ -1089,6 +1089,34 @@ Product.search({ conditions: [{ attribute: 'discount', comparator: 'not_equal', Product.search({ conditions: [{ attribute: 'discount', comparator: 'between', value: [null, 10] }] }); ``` +#### Conditions on a to-many relationship + + + +Each condition whose path traverses a to-many relationship is met by some related record, which can differ from the one that meets another condition. To require one related record to meet several conditions, use a relationship group (Added in: v5.4.0): a condition whose `attribute` names the relationship (or a path ending in one), with a `conditions` array on the related table and no `comparator` or `value`. Its `operator` defaults to `and`; with `or`, one related record must meet at least one of the conditions. + +```javascript +const brands = []; +for await (const brand of Brand.search({ + conditions: [ + { + attribute: 'products', + conditions: [ + { attribute: 'name', value: 'Keyboard' }, + { attribute: 'price', comparator: 'greater_than_equal', value: 30 }, + ], + }, + ], + select: ['id', 'name', { name: 'products', select: ['name', 'price'] }], +})) { + brands.push(brand); +} +``` + +On the [sample brand and product data](../rest/querying.md#filtering-and-paging-each-relationship), `brands` contains Acme with only its Keyboard. With `'Cable'` in place of `'Keyboard'`, `brands` is empty: no product is a Cable that costs at least 30. As two separate conditions, `{ attribute: ['products', 'name'], value: 'Cable' }` and `{ attribute: ['products', 'price'], comparator: 'greater_than_equal', value: 30 }` match Acme, and selecting `products` returns all three of its products, since each meets one of the conditions. + +A group's conditions can be groups themselves, including another relationship group; the [REST bracket form](../rest/querying.md#conditions-one-related-record-must-meet) describes nesting and paths. With [`checkPermission: true`](#checkpermission), a group needs read permission on the related table and on each related attribute its conditions name, as a path does; otherwise the search throws an error with `statusCode` 403. On a to-one relationship, a group matches the same records as separate conditions. A group without conditions, with a `comparator` or `value`, on an attribute that is not a relationship, or with a full-text comparator in its conditions throws an error with `statusCode` 400. The operations API's `search_by_conditions` does not support relationship groups. + ### `operator` Top-level `and` (default) or `or` for the `conditions` array. @@ -1148,13 +1176,35 @@ When a field is defined as a [relationship](../database/schema.md#relationships) A to-many relationship resolves to an array of records; depending on access pattern you may need to `await` the property before iterating it. -**Join behavior:** selecting a relationship without an outer condition on it behaves as a **LEFT JOIN**: parents with no related row remain present, with the relationship omitted or empty. A top-level `conditions` entry on a related attribute (for example, `attribute: ['author', 'name']`) behaves as an **INNER JOIN**: it removes parents without a matching related row and restricts the selected related records. A `conditions` array inside a nested `select` object only filters the displayed children; it does not remove a parent. +**Join behavior:** selecting a relationship without an outer condition on it behaves as a **LEFT JOIN**: parents with no related row remain present, with the relationship omitted or empty. A top-level `conditions` entry on a related attribute (for example, `attribute: ['author', 'name']`) joined with `and` behaves as an **INNER JOIN**: it removes parents without a matching related row and restricts the selected related records. With several such entries on one to-many relationship, the selection keeps the related records that meet at least one of them; a [relationship group](#conditions-on-a-to-many-relationship) keeps the records that meet the whole group. A `conditions` array inside a nested `select` object only filters the displayed children; it does not remove a parent. + +**A related-attribute condition inside `or` decides parents only.** It does not restrict which related records `select` returns for a matching parent: + +```javascript +const brands = []; +for await (const brand of Brand.search({ + conditions: [ + { + operator: 'or', + conditions: [ + { attribute: ['products', 'name'], value: 'Cable' }, + { attribute: 'id', value: 44 }, + ], + }, + ], + select: ['id', { name: 'products', select: ['name'] }], +})) { + brands.push(brand); +} +``` + +`brands` contains Acme with all three of its products and Studio Gear with its Stand and Monitor. Without the `or` group, the Cable condition would restrict Acme's `products` to the Cable. A condition joined with `and` beside the `or` group still restricts the selection. In v5.3.5, this search returned Acme with only its Cable and Studio Gear with an empty `products` array. #### Constraining related records :::note Pending release -Targeted for v5.4.0; not released yet. See [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775). +Targeted for v5.4.0; not released yet. See [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279), [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775), and [Make each relationship condition existential, and add relationship[...] for conditions one related record must meet](https://github.com/HarperFast/harper/pull/3927). ::: @@ -1191,7 +1241,7 @@ The single-record example returns: } ``` -**Outer relationship conditions also restrict nested results.** You do not need to repeat an outer product-price condition inside the `products` selection. For example, this returns only brands with products costing at least 30, and includes their cheapest qualifying product: +**Outer relationship conditions joined with `and` also restrict nested results.** You do not need to repeat an outer product-price condition inside the `products` selection. For example, this returns only brands with products costing at least 30, and includes their cheapest qualifying product: ```javascript const brands = []; @@ -1238,7 +1288,7 @@ Add a nested `conditions` array only to narrow these inherited candidates furthe For production collections, also set a top-level `limit` to bound parents. Use a unique final sort key for both parent and child order, and remember that a small child limit does not bound sorting work. See [parent pagination and stable sorting](../rest/querying.md#bound-parent-pages-and-use-stable-sorting) for an example and performance guidance. -These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Top-level `conditions` on that relationship still restrict the child candidates available to the nested selection, so an identical condition need not be repeated. +These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children, and one that traverses a further relationship follows the [same rules](#conditions-on-a-to-many-relationship) as a top-level condition; descendant filtering requires a selection on that descendant. Top-level `and` conditions on that relationship still restrict the child candidates available to the nested selection, to the children that meet at least one of them, so a single condition need not be repeated; a condition inside an `or` group does not restrict them. See [Filtering and Paging Each Relationship](../rest/querying.md#filtering-and-paging-each-relationship) for missing-value behavior, authorization, cache-local eligibility and ordering, scan costs, and unsupported constraints. @@ -1276,7 +1326,7 @@ Alternatively, pass `allowFullScan: true` to permit an unconditional ordered sca An indexed sort `attribute` does not guarantee an index-ordered scan. When the condition Harper runs first is on the sort attribute but cannot read that attribute's index, as with `not_equal` and a non-null value on an attribute other than the primary key, Harper collects and sorts every matching record in memory before applying `offset` and `limit`. Before v5.4.0, such a query, or one whose condition was on an unindexed sort attribute, could return records in primary-key order. See [Sorting](./query-optimization.md#sorting). - When `attribute` is a path array that traverses a to-many relationship, each parent's sort key is the list of its related records' values in relationship order: the related records' primary-key order in the foreign-key index for a `@relationship(to: ...)` field (not sorted by the target attribute or reduced to a min/max), or the stored foreign-key array order for a `@relationship(from: ...)` array. Top-level `conditions` on a relationship restrict which parents match and which children `select` returns, but do not narrow the related records loaded into each parent's sort key. Lists compare element by element (`['alpha']` < `['alpha', 'delta']` < `['bravo']`), and `descending: true` reverses that order. A parent with no related records has an empty list (`[]`), a related record that omits the sort attribute (or an unresolved ID in a `@relationship(from: ...)` array without `filterMissing`) produces `undefined` at that position, and an explicit `null` attribute produces `null`: both `null` and `undefined` sort before booleans, numbers, and strings (`[]` and `[undefined]` compare as equal because reading past the end of a shorter array yields `undefined`). Across multiple to-many hops, each hop produces a list in relationship order, so two to-many hops compare a nested list of lists per parent (`[]` < `[[], ['a']]` < `[['b']]`). The index-or-condition rule above checks `@indexed` on the target attribute at the end of the path. List-per-parent sorting runs in memory when the target attribute is not `@indexed` (with a condition or `allowFullScan: true`), when a more selective condition on another attribute leads the query, or when `enforceExecutionOrder: true` is set, loading the related records for every matched parent; if the target attribute is `@indexed` and no more selective condition on another attribute leads the query, Harper instead drives the scan from the related table's index join (yielding each parent at its first matching index entry and, when unconditional, omitting parents with no related records). Note that, unlike a nested `select` sort (`select: [{ name: 'products', sort: { attribute: 'price' } }]`), which checks read permissions on the child sort attribute and orders children within each parent, a top-level `sort` orders the parent records and does not currently enforce `attribute_permissions` on the related sort attribute, so restricting read access to a child attribute does not prevent ordering parent records by it: + When `attribute` is a path array that traverses a to-many relationship, each parent's sort key is the list of its related records' values in relationship order: the related records' primary-key order in the foreign-key index for a `@relationship(to: ...)` field (not sorted by the target attribute or reduced to a min/max), or the stored foreign-key array order for a `@relationship(from: ...)` array. Top-level `conditions` on a relationship restrict which parents match and (when joined with `and`) which children `select` returns, but do not narrow the related records loaded into each parent's sort key. Lists compare element by element (`['alpha']` < `['alpha', 'delta']` < `['bravo']`), and `descending: true` reverses that order. A parent with no related records has an empty list (`[]`), a related record that omits the sort attribute (or an unresolved ID in a `@relationship(from: ...)` array without `filterMissing`) produces `undefined` at that position, and an explicit `null` attribute produces `null`: both `null` and `undefined` sort before booleans, numbers, and strings (`[]` and `[undefined]` compare as equal because reading past the end of a shorter array yields `undefined`). Across multiple to-many hops, each hop produces a list in relationship order, so two to-many hops compare a nested list of lists per parent (`[]` < `[[], ['a']]` < `[['b']]`). The index-or-condition rule above checks `@indexed` on the target attribute at the end of the path. List-per-parent sorting runs in memory when the target attribute is not `@indexed` (with a condition or `allowFullScan: true`), when a more selective condition on another attribute leads the query, or when `enforceExecutionOrder: true` is set, loading the related records for every matched parent; if the target attribute is `@indexed` and no more selective condition on another attribute leads the query, Harper instead drives the scan from the related table's index join (yielding each parent at its first matching index entry and, when unconditional, omitting parents with no related records). Note that, unlike a nested `select` sort (`select: [{ name: 'products', sort: { attribute: 'price' } }]`), which checks read permissions on the child sort attribute and orders children within each parent, a top-level `sort` orders the parent records and does not currently enforce `attribute_permissions` on the related sort attribute, so restricting read access to a child attribute does not prevent ordering parent records by it: ```javascript // On the Brand/Product sample in REST Querying, orders qualifying brands by diff --git a/reference/rest/overview.md b/reference/rest/overview.md index 0f60b27c6..85219790b 100644 --- a/reference/rest/overview.md +++ b/reference/rest/overview.md @@ -213,6 +213,8 @@ DELETE /MyTable/?status=archived Deletes all records matching `status=archived`. + Separate conditions through a to-many relationship may each be met by a different related record. When one related record must meet all of them, put them in brackets, as in `DELETE /Brand/?products[name=Cable&price=ge=30]`. Before v5.4.0, brackets after a relationship name applied their conditions to the queried table, so on an earlier release this request can delete the wrong records; check a bracketed filter with a `GET` first. See [Conditions on a To-Many Relationship](./querying.md#conditions-on-a-to-many-relationship). + ## Content Types Harper supports multiple content types for both request bodies and responses. Use the `Content-Type` header for request bodies and the `Accept` header to request a specific response format. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 3e366fac2..0c6b69b75 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -200,7 +200,7 @@ GET /Product/?rating=5|featured=true ## Grouping -Use parentheses or square brackets to control order of operations: +Use parentheses or square brackets to control order of operations. Brackets directly after a relationship name mean [something else](#conditions-one-related-record-must-meet): ```http GET /Product/?rating=5|(price=gt=100&price=lt=200) @@ -218,7 +218,7 @@ Constructing from JavaScript: let url = `/Product/?rating=5&[${tags.map(encodeURIComponent).join('|')}]`; ``` -These root-level brackets group conditions. Brackets after a relationship name inside `select()`, such as `select(id,products[limit(1)])`, instead form a [nested relationship query](#filtering-and-paging-each-relationship). +These root-level brackets group conditions. Brackets directly after a relationship name among the conditions, such as `products[name=Cable&price=ge=30]`, instead hold conditions that [one related record must meet](#conditions-one-related-record-must-meet). Brackets after a relationship name inside `select()`, such as `select(id,products[limit(1)])`, form a [nested relationship query](#filtering-and-paging-each-relationship). Groups can be nested for complex conditions: @@ -363,6 +363,8 @@ GET /Product/?brand.name=Microsoft GET /Brand/?products.name=Keyboard ``` +To combine several conditions on one to-many relationship, see [Conditions on a To-Many Relationship](#conditions-on-a-to-many-relationship). + ### Nested Select with Joins Relationship attributes are not included by default. Use `select()` to include them: @@ -373,19 +375,19 @@ GET /Product/?brand.name=Microsoft&select(name,brand{name}) GET /Product/?name=Keyboard&select(name,brand{name,id}) ``` -Selecting a relationship without an outer dot-syntax filter on it behaves as a LEFT JOIN: parents remain present even when there are no related records. For a singular foreign key that is null or references a missing record, the relationship property is omitted. An outer filter such as `products.price=ge=number:30` instead removes parents without a qualifying product and also narrows the selected products. A filter inside `select(...,products[...])` only narrows the projected products; it does not remove a parent. See the [worked scenarios](#filtering-and-paging-each-relationship) below. +Selecting a relationship without an outer dot-syntax filter on it behaves as a LEFT JOIN: parents remain present even when there are no related records. For a singular foreign key that is null or references a missing record, the relationship property is omitted. An outer filter joined with `&`, such as `products.price=ge=number:30`, instead removes parents without a qualifying product and also narrows the selected products; with several such filters on one relationship, the selection keeps the products that meet [any one of them](#what-select-returns). An outer filter inside an OR only decides which parents match. A filter inside `select(...,products[...])` only narrows the projected products; it does not remove a parent. See the [worked scenarios](#filtering-and-paging-each-relationship) below. ### Filtering and Paging Each Relationship :::note Pending release -Targeted for v5.4.0; not released yet. See [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775). +Targeted for v5.4.0; not released yet. See [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279), [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775), and [Make each relationship condition existential, and add relationship[...] for conditions one related record must meet](https://github.com/HarperFast/harper/pull/3927). ::: Use square brackets inside `select()` to apply conditions, sort, and pagination to each parent's related records. -**An outer relationship filter also narrows that relationship's nested results.** For example, `products.price=ge=number:30` selects brands with a qualifying product and restricts their selected products to those costing at least 30. **You do not need to repeat that price filter inside `products[...]`.** Add a nested condition only when you want to narrow the already-eligible products further without removing their parents. +**An outer relationship filter joined with `&` also narrows that relationship's nested results.** A filter inside an OR [does not](#what-select-returns). For example, `products.price=ge=number:30` selects brands with a qualifying product and restricts their selected products to those costing at least 30. **You do not need to repeat that price filter inside `products[...]`.** Add a nested condition only when you want to narrow the already-eligible products further without removing their parents. The following scenarios use this self-contained schema, including string product IDs and an indexed price field. In a fresh application, use this schema instead of the earlier Brand/Product example; do not combine the two definitions: @@ -603,7 +605,7 @@ With the sample data, the response is: The outer `sort(name,id)` orders brands; the nested `sort(price,id)` orders products within each brand. Use a unique field such as `id` as the final sort key so equal names or prices have a deterministic order. This does not prevent records moving between offset pages when data changes between requests. - An outer sort can also traverse a to-many relationship — for example, `GET /Brand/?products.price=ge=number:30&sort(products.name,id)` — to order parent records by their related values, unlike a nested selection sort (`products[sort(price)]`), which orders the children within each parent and not the parents. Each parent's sort key is the list of its related records' values in relationship order: the related records' primary-key order in the foreign-key index for a `@relationship(to: ...)` field (not sorted by the target attribute or reduced to a min/max), or the stored foreign-key array order for a `@relationship(from: ...)` array. An outer relationship filter such as `products.price=ge=number:30` restricts which parents match and which children `select()` returns, but does not narrow the related records loaded into each parent's sort key: on the sample data above, Acme's `products.name` key is `['Cable', 'Keyboard', 'Dock']` (from `p1`, `p2`, `p3`) and Studio Gear's is `['Stand', 'Monitor']` (`p5`, `p6`), so `sort(products.name,id)` returns Acme (`42`) before Studio Gear (`44`) and `sort(-products.name,id)` reverses that order. Lists compare element by element (`['alpha']` < `['alpha', 'delta']` < `['bravo']`). A parent with no related records has an empty list (`[]`), a related record that omits the sort attribute (or an unresolved ID in a `@relationship(from: ...)` array without `filterMissing`) produces `undefined` at that position, and an explicit `null` attribute produces `null`: both `null` and `undefined` sort before booleans, numbers, and strings (`[]` and `[undefined]` compare as equal because reading past the end of a shorter array yields `undefined`). Across multiple to-many hops, each hop produces a list in relationship order, so two to-many hops compare a nested list of lists per parent (`[]` < `[[], ['a']]` < `[['b']]`). List-per-parent sorting runs in memory when the target attribute is not `@indexed` (with a condition or `allowFullScan: true`), when a more selective condition on another attribute leads the query, or when `enforceExecutionOrder: true` is set, loading the related records for every matched parent; if the target attribute is `@indexed` (such as `products.price`) and no more selective condition on another attribute leads the query, Harper instead drives the scan from the related table's index join (yielding each parent at its first matching index entry and, when unconditional, omitting parents with no related records). Note that, unlike a nested `select` sort, an outer `sort` does not currently enforce `attribute_permissions` on the related sort attribute, so restricting read access to a child attribute does not prevent ordering parent records by it. + An outer sort can also traverse a to-many relationship — for example, `GET /Brand/?products.price=ge=number:30&sort(products.name,id)` — to order parent records by their related values, unlike a nested selection sort (`products[sort(price)]`), which orders the children within each parent and not the parents. Each parent's sort key is the list of its related records' values in relationship order: the related records' primary-key order in the foreign-key index for a `@relationship(to: ...)` field (not sorted by the target attribute or reduced to a min/max), or the stored foreign-key array order for a `@relationship(from: ...)` array. An outer relationship filter such as `products.price=ge=number:30` restricts which parents match and (when joined with `&`) which children `select()` returns, but does not narrow the related records loaded into each parent's sort key: on the sample data above, Acme's `products.name` key is `['Cable', 'Keyboard', 'Dock']` (from `p1`, `p2`, `p3`) and Studio Gear's is `['Stand', 'Monitor']` (`p5`, `p6`), so `sort(products.name,id)` returns Acme (`42`) before Studio Gear (`44`) and `sort(-products.name,id)` reverses that order. Lists compare element by element (`['alpha']` < `['alpha', 'delta']` < `['bravo']`). A parent with no related records has an empty list (`[]`), a related record that omits the sort attribute (or an unresolved ID in a `@relationship(from: ...)` array without `filterMissing`) produces `undefined` at that position, and an explicit `null` attribute produces `null`: both `null` and `undefined` sort before booleans, numbers, and strings (`[]` and `[undefined]` compare as equal because reading past the end of a shorter array yields `undefined`). Across multiple to-many hops, each hop produces a list in relationship order, so two to-many hops compare a nested list of lists per parent (`[]` < `[[], ['a']]` < `[['b']]`). List-per-parent sorting runs in memory when the target attribute is not `@indexed` (with a condition or `allowFullScan: true`), when a more selective condition on another attribute leads the query, or when `enforceExecutionOrder: true` is set, loading the related records for every matched parent; if the target attribute is `@indexed` (such as `products.price`) and no more selective condition on another attribute leads the query, Harper instead drives the scan from the related table's index join (yielding each parent at its first matching index entry and, when unconditional, omitting parents with no related records). Note that, unlike a nested `select` sort, an outer `sort` does not currently enforce `attribute_permissions` on the related sort attribute, so restricting read access to a child attribute does not prevent ordering parent records by it. Keep relationship lookup fields and outer filter fields indexed, and project only the fields the client needs. The sample schema indexes `brandId`, `price`, and `Brand.name`. Nested sorting can use an unindexed child field because it orders the already-related candidates. A child sort may still collect and sort all eligible related records before returning its page; an index on `price` does not guarantee an index-ordered nested scan. Keep parent pages small and consider a separately paginated child endpoint for very large relationships. Unsorted nested limits can stop early, but are appropriate only when child order does not matter. @@ -647,23 +649,23 @@ Acme's matching products sort to Dock ($60), then Cable ($10), so `limit(1,2)` r #### Nested option reference -| Nested option | Behavior | -| ------------------ | ---------------------------------------------------------------------------------------------------------- | -| Conditions | Qualify children of the selected relationship; combine with `&` (AND), `\|` (OR), and [groups](#grouping). | -| `sort(price)` | Order that parent's eligible children before pagination. | -| `limit(end)` | Return at most `end` children per parent. | -| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | -| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | +| Nested option | Behavior | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Conditions | Qualify children of the selected relationship; combine with `&` (AND), `\|` (OR), and [groups](#grouping). Conditions through a further relationship follow the [top-level rules](#conditions-on-a-to-many-relationship). | +| `sort(price)` | Order that parent's eligible children before pagination. | +| `limit(end)` | Return at most `end` children per parent. | +| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | +| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | Use a non-negative safe-integer window; invalid bounds such as negative or fractional values return HTTP 400; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, pagination alone preserves missing target positions; adding conditions or a sort drops missing targets. Child conditions and sort fields remain subject to read permissions. -A condition that traverses another relationship qualifies the immediate child; it does not narrow that child's projected descendants. To filter or page grandchildren, put those constraints on the grandchild's own bracket selection. Outer join restrictions are still inherited. +A condition that traverses another relationship qualifies the immediate child; it does not narrow that child's projected descendants. It follows the same rules as a [top-level condition](#conditions-on-a-to-many-relationship): separate conditions on a grandchild relationship may each be met by a different grandchild, and a `relationship[...]` group requires one grandchild to meet all of them. To filter or page grandchildren, put those constraints on the grandchild's own bracket selection. Outer filters joined with `&` are still inherited; one inside an OR is not. For source-backed caches, eligibility and sort keys use local cached entries, as in a top-level table query. Returned children can refresh from source; a refreshed child that no longer matches, or has been deleted, does not consume offset or limit. A locally rejected child is not fetched to discover whether its source value now matches. Ordering is not recomputed from refreshed source values. Unsorted pagination can stop when the requested page is filled. Sorting or a predicate with few matches can still examine all related candidates; a small output limit does not guarantee a small scan. -Nested filters require a table-backed relationship. Filters on plain JSON projections or custom non-table resources, and full-text predicates inside a relationship selection, return HTTP 400 instead of being silently ignored. Projection-only selections and sorting/pagination of plain JSON arrays remain supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. +Nested filters require a table-backed relationship. Filters on plain JSON projections or custom non-table resources, and full-text predicates inside a relationship selection, return HTTP 400 instead of being silently ignored. Projection-only selections and sorting/pagination of plain JSON arrays remain supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. For how several outer conditions on one relationship combine in v5.4.0, see [Conditions on a To-Many Relationship](#conditions-on-a-to-many-relationship). The [Resource API](../resources/resource-api.md#constraining-related-records) accepts the same options in a nested selection object. @@ -686,6 +688,111 @@ GET /Product/?resellers.name=Cool Shop&select(id,name,resellers{name,id}) The array order of `resellerIds` is preserved when resolving the relationship. +### Conditions on a To-Many Relationship + + + +:::note Pending release + +Targeted for v5.4.0; not released yet. See [Make each relationship condition existential, and add relationship[...] for conditions one related record must meet](https://github.com/HarperFast/harper/pull/3927). + +::: + +The Brand examples in this section use the [sample brands and products](#filtering-and-paging-each-relationship): Acme sells a Cable for 10, a Keyboard for 35, and a Dock for 60; Budget Basics sells a Mouse for 15; Studio Gear sells a Stand for 45 and a Monitor for 120. + +#### Separate conditions may match different related records + +Each condition joined with `&` on a to-many relationship is met by some related record, which can differ from the one that meets another condition. Conditions on an array attribute work the same way. This request returns Acme, which has a Keyboard and has a Cable: + +```http +GET /Brand/?products.name=Keyboard&products.name=Cable +``` + +Likewise, `products.name=Cable&products.price=ge=number:30` returns Acme: it has a Cable, and it has products costing at least 30, though the Cable is not one of them. A collection `DELETE` matches records by the same rule. Before v5.4.0, whether such conditions had to be met by one related record depended on which condition Harper evaluated first. + +#### Conditions one related record must meet + + + +When one related record must meet several conditions, put them in brackets after the relationship name, like MongoDB's `$elemMatch`: + +```http +GET /Brand/?products[name=Cable&price=ge=30] +GET /Brand/?products[name=Keyboard&price=ge=30] +``` + +The first request returns `[]`: no product is a Cable that costs at least 30. The second returns Acme, whose Keyboard costs 35. Use this form for a collection `DELETE` that must not remove a parent whose conditions are met by different related records. + +Inside the brackets: + +- Conditions name attributes of the related table: `price`, not `products.price`. A value converts to that attribute's schema type, so `price=ge=30` needs no `number:` prefix; keep `number:` on a dotted path such as `products.price=ge=number:30`. +- Conditions combine with `&`, `|`, and parenthesized or bracketed groups. `products[name=Mouse|price=ge=100]` returns Budget Basics and Studio Gear. +- Brackets can nest another relationship and can follow a relationship path, as [the next section](#conditions-through-several-relationships) shows. + +The name before the brackets must be a relationship, the brackets must hold at least one condition, and no condition in them can use a full-text comparator such as `matches`; otherwise Harper returns HTTP 400. On a to-one relationship, which has one related record, brackets match the same records as separate conditions. Brackets after a relationship name inside `select()` keep their meaning as a [nested relationship query](#filtering-and-paging-each-relationship). + +Querying through a relationship, by a dotted path or in brackets, needs read permission on the related table and on each related attribute a condition names. Without it, Harper returns HTTP 403. + +#### Conditions through several relationships + +In this schema, each company has departments and each department has employees: + +```graphql +type Company @table @export { + id: ID @primaryKey + departments: [Department] @relationship(to: "companyId") +} + +type Department @table @export { + id: ID @primaryKey + companyId: ID @indexed + employees: [Employee] @relationship(to: "departmentId") +} + +type Employee @table @export { + id: ID @primaryKey + departmentId: ID @indexed + tag: String @indexed + level: Int @indexed +} +``` + +With these records: + +| Company | Department | Employee | Tag | Level | +| ------- | ---------- | -------- | ------ | ----- | +| c1 | d1 | e1 | oncall | 1 | +| c1 | d1 | e2 | mentor | 4 | +| c2 | d2 | e3 | oncall | 3 | +| c3 | d3 | e4 | oncall | 1 | +| c3 | d4 | e5 | mentor | 5 | + +where the brackets go decides which conditions one record must meet: + +| `GET /Company/?...` | Matches | Why | +| -------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------- | +| `departments.employees.tag=oncall&departments.employees.level=ge=number:3` | c1, c2, c3 | each condition may be met by a different department | +| `departments[employees.tag=oncall&employees.level=ge=number:3]` | c1, c2 | one department must meet both, possibly by different people | +| `departments[employees[tag=oncall&level=ge=3]]` | c2 | one employee must meet both | +| `departments.employees[tag=oncall&level=ge=3]` | c2 | the same as the previous row | + +#### What `select()` returns + +An outer condition on a relationship also narrows which related records `select()` returns for that relationship: + +- Conditions joined with `&` keep the related records that meet at least one of them. A bracketed group counts as one condition, met by a record that meets the whole group, including any `|` inside it. +- A condition inside an OR narrows nothing: it only decides which parents match. A condition joined with `&` beside the OR still narrows. + +```http +GET /Brand/?products.name=Keyboard&products.price=ge=number:30&select(id,products{name}) +GET /Brand/?products[name=Keyboard&price=ge=30]&select(id,products{name}) +GET /Brand/?(products.name=Cable|id=44)&select(id,products{name}) +``` + +The first request returns Acme with its Keyboard and Dock: the Keyboard meets both conditions, and the Dock meets the price condition. The second returns Acme with only its Keyboard. The third returns Acme with all three of its products and Studio Gear with its Stand and Monitor, while `products.name=Cable&select(id,products{name})` returns Acme with only its Cable. Adding `products.price=ge=number:30&` before the third request's OR returns Acme with its Keyboard and Dock (its Cable meets the OR) and Studio Gear with its Stand and Monitor. + +In v5.3.5, the first request returned Acme with only its Keyboard, and the third returned Acme with only its Cable and Studio Gear with an empty `products` array. + ## Property Access via URL diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 6d879f42c..25eacbc83 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -21,7 +21,7 @@ Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log b ## Nested Relationship Constraints -[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775) correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside `select` apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so an identical filter does not need to be repeated inside the nested selection. When a top-level sort across a to-many relationship (such as `sort(products.name)`) runs in memory — when the target attribute is not `@indexed`, a more selective condition on another attribute leads the query, or `enforceExecutionOrder: true` is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an `@indexed` target attribute leads the query, Harper continues to order via the related table's index join. +[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) and [Fix wrong rows, unfiltered selects and ignored sorts in relationship queries](https://github.com/HarperFast/harper/pull/3775) correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside `select` apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters joined with AND — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so a single filter does not need to be repeated inside the nested selection; see [Conditions on To-Many Relationships](#conditions-on-to-many-relationships) for several filters on one relationship and for filters inside an OR. When a top-level sort across a to-many relationship (such as `sort(products.name)`) runs in memory — when the target attribute is not `@indexed`, a more selective condition on another attribute leads the query, or `enforceExecutionOrder: true` is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an `@indexed` target attribute leads the query, Harper continues to order via the related table's index join. **Upgrade impact:** requests using nested constraints can return fewer children, and requests with top-level sorts across to-many relationships now return parents ordered by their related values. Unsupported filters on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Sorting and pagination of plain JSON arrays remain supported. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship) and [`sort`](/reference/v5/resources/resource-api#sort). @@ -41,6 +41,14 @@ These corrections are not released yet. This change is not released yet. +## Conditions on To-Many Relationships + +[Make each relationship condition existential, and add relationship[...] for conditions one related record must meet](https://github.com/HarperFast/harper/pull/3927) makes each AND condition on a to-many relationship be met by some related record, possibly a different one for each condition, as on an array attribute, regardless of which condition Harper evaluates first: `GET /Brand/?products.name=Keyboard&products.name=Cable` returns a brand that has both. To require one related record to meet several conditions, put them in brackets after the relationship name: `GET /Brand/?products[name=Keyboard&price=ge=30]` returns a brand with a Keyboard that costs at least 30. Programmatically, the group is `{ attribute: 'products', conditions: [...] }`. Groups can use `|`, nest another relationship (`departments[employees[tag=oncall&level=ge=3]]`), and follow a path (`departments.employees[tag=oncall&level=ge=3]`). `select()` of the relationship returns the related records that meet any one of the AND conditions, a group counting as one condition, and a relationship condition inside an OR decides which parents match without narrowing the selection. + +**Upgrade impact:** before 5.4, whether separate conditions on one to-many relationship had to be met by one related record depended on which condition Harper evaluated first. A query or collection `DELETE` with such conditions can now match more parents: use the bracket form where one related record must meet every condition, and check a destructive filter with a `GET` first. `select()` of the relationship can return more related records. A request with a relationship condition inside an OR can return more parents and more related records. Brackets after a relationship name among the conditions, such as `products[name=Cable]`, used to apply their conditions to the queried table; they now apply to the related records, and brackets after a name that is not a relationship return HTTP 400. See [Conditions on a To-Many Relationship](/reference/v5/rest/querying#conditions-on-a-to-many-relationship) and [relationship groups](/reference/v5/resources/resource-api#conditions-on-a-to-many-relationship). + +This change is not released yet. + ## Replication ### Sharding