From 8611d47c2127cf31d2d25a0eacafc2b0e93a6ce1 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 20:49:01 -0600 Subject: [PATCH 01/20] Document nested relationship query constraints --- reference/resources/resource-api.md | 27 ++++++++++++++++++++++++ reference/rest/querying.md | 32 +++++++++++++++++++++++++++++ release-notes/v5-lincoln/5.3.md | 8 ++++++++ 3 files changed, 67 insertions(+) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index da064a259..9b22a6976 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1114,6 +1114,33 @@ A to-many relationship resolves to an array of records; depending on access patt **Join behavior:** selecting a relationship _without_ filtering on it behaves as a **LEFT JOIN** — records with no related row are still returned (the relationship property is omitted or empty). Adding a condition on a related attribute (e.g. `attribute: ['author', 'name']`) behaves as an **INNER JOIN** — only records with a matching related row are returned. +#### Constraining related records + +Pending a 5.3 patch release; see [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). + +A relationship selection object accepts `conditions`, `operator`, `sort`, `offset`, and `limit`, with an optional `select` projection: + +```javascript +const brand = await Brand.get({ + id: 42, + select: [ + 'id', + { + name: 'products', + conditions: [{ attribute: 'price', comparator: 'greater_than_equal', value: 30 }], + sort: { attribute: 'price' }, + offset: 0, + limit: 1, + select: ['id', 'price'], + }, + ], +}); +``` + +These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. Neither changes which parents are returned. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Inherited outer join restrictions still apply. + +See [Filtering and Paging Each Relationship](../rest/querying.md#filtering-and-paging-each-relationship) for missing-value behavior, authorization, cache-local eligibility and ordering, scan costs, and unsupported constraints. + ### `sort` Sort order object: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index b8d7d0e66..1b5130d3f 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -272,6 +272,38 @@ GET /Product/?name=Keyboard&select(name,brand{name,id}) When selecting without a filter on the related table, this acts as a LEFT JOIN — the relationship property is omitted if the foreign key is null or references a non-existent record. +### Filtering and Paging Each Relationship + +Pending a 5.3 patch release; see [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). + +Use square brackets inside `select()` to apply conditions, sort, and pagination to each parent's related records. For a `Brand.products` relationship with `price` and `id` fields: + +```http +GET /Brand/?select(id,products[price=ge=number:30&sort(price)&limit(1)&select(id,price)]) +``` + +Each brand receives at most one product priced at 30 or more, ordered by price. A brand with no qualifying products remains in the result with an empty `products` array. By comparison, an outer condition such as `/Brand/?products.price=ge=number:30` filters which brands are returned. Existing outer relationship filters also constrain the related candidates available to a nested selection. + +| Nested option | Behavior | +| ------------------ | --------------------------------------------------------------------------------- | -------------- | +| Conditions | Qualify children of the selected relationship; combine with `&`, ` | `, and groups. | +| `sort(price)` | Order that parent's eligible children before pagination. | +| `limit(end)` | Return at most `end` children per parent. | +| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | +| `select(id,price)` | Project fields from the returned children. Omit it to return whole child records. | + +Use a non-negative safe-integer window; `limit(0)` produces an empty to-many selection without reading children. A missing singular relationship remains absent. For array-valued foreign keys, unsorted pagination preserves missing target positions; sorting drops missing targets. Child conditions and sort fields remain subject to read permissions. + +A condition that traverses another relationship qualifies the immediate child; it does not narrow that child's projected descendants. To filter or page grandchildren, put those constraints on the grandchild's own bracket selection. Outer join restrictions are still inherited. + +For source-backed caches, eligibility and sort keys use local cached entries, as in a top-level table query. Returned children can refresh from source; a refreshed child that no longer matches, or has been deleted, does not consume offset or limit. A locally rejected child is not fetched to discover whether its source value now matches. Ordering is not recomputed from refreshed source values. + +Unsorted pagination can stop when the requested page is filled. Sorting or a predicate with few matches can still examine all related candidates; a small output limit does not guarantee a small scan. + +Query constraints on plain JSON projections and full-text predicates inside a relationship selection are unsupported and return a client error. Projection-only JSON syntax remains supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. + +The [Resource API](../resources/resource-api.md#constraining-related-records) accepts the same options in a nested selection object. + ### Many-to-Many Relationships Many-to-many relationships can be modeled with an array of foreign key values, without a junction table: diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 195c6baa9..8fe57cf74 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -12,6 +12,14 @@ These notes summarize what changed relative to 5.2. The [Harper Pro v5.3.0 relea All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). +### Pending Patch: Nested Relationship Constraints + +[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. + +**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Existing cache-local candidate and ordering semantics remain unchanged. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). + +This correction is not released yet. Replace this pending heading with the shipping patch version when the core change is released. + ## Upgrade Notes Read these before upgrading a production node or cluster. From eab3d6f533ae688d5d4471f9618aab2546d60301 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 20:53:54 -0600 Subject: [PATCH 02/20] Clarify nested selection limits and repair option table --- reference/resources/resource-api.md | 2 ++ reference/rest/querying.md | 18 +++++++++--------- release-notes/v5-lincoln/5.3.md | 2 +- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 9b22a6976..db73dab63 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1137,6 +1137,8 @@ const brand = await Brand.get({ }); ``` +`operator` defaults to `and`; use `or` to combine the nested conditions as alternatives. + These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. Neither changes which parents are returned. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Inherited outer join restrictions still apply. See [Filtering and Paging Each Relationship](../rest/querying.md#filtering-and-paging-each-relationship) for missing-value behavior, authorization, cache-local eligibility and ordering, scan costs, and unsupported constraints. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 1b5130d3f..d1315ce54 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -284,15 +284,15 @@ GET /Brand/?select(id,products[price=ge=number:30&sort(price)&limit(1)&select(id Each brand receives at most one product priced at 30 or more, ordered by price. A brand with no qualifying products remains in the result with an empty `products` array. By comparison, an outer condition such as `/Brand/?products.price=ge=number:30` filters which brands are returned. Existing outer relationship filters also constrain the related candidates available to a nested selection. -| Nested option | Behavior | -| ------------------ | --------------------------------------------------------------------------------- | -------------- | -| Conditions | Qualify children of the selected relationship; combine with `&`, ` | `, and groups. | -| `sort(price)` | Order that parent's eligible children before pagination. | -| `limit(end)` | Return at most `end` children per parent. | -| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | -| `select(id,price)` | Project fields from the returned children. Omit it to return whole child records. | - -Use a non-negative safe-integer window; `limit(0)` produces an empty to-many selection without reading children. A missing singular relationship remains absent. For array-valued foreign keys, unsorted pagination preserves missing target positions; sorting drops missing targets. Child conditions and sort fields remain subject to read permissions. +| Nested option | Behavior | +| ------------------ | -------------------------------------------------------------------------------- | +| Conditions | Qualify children of the selected relationship; combine with AND, OR, and groups. | +| `sort(price)` | Order that parent's eligible children before pagination. | +| `limit(end)` | Return at most `end` children per parent. | +| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | +| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | + +Use a non-negative safe-integer window; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, unsorted pagination preserves missing target positions; sorting drops missing targets. Child conditions and sort fields remain subject to read permissions. A condition that traverses another relationship qualifies the immediate child; it does not narrow that child's projected descendants. To filter or page grandchildren, put those constraints on the grandchild's own bracket selection. Outer join restrictions are still inherited. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 8fe57cf74..cdde2a0e9 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -18,7 +18,7 @@ All patch release notes for 5.3.x are available on the [releases page](https://g **Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Existing cache-local candidate and ordering semantics remain unchanged. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). -This correction is not released yet. Replace this pending heading with the shipping patch version when the core change is released. +This correction is not released yet. ## Upgrade Notes From 636f05a32e8893a62043fdf2924ea9182cc32abf Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sun, 18 Oct 2026 20:56:34 -0600 Subject: [PATCH 03/20] Clarify filtered foreign keys and pending availability --- reference/resources/resource-api.md | 6 +++++- reference/rest/querying.md | 6 +++++- release-notes/v5-lincoln/5.3.md | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index db73dab63..b37bcd535 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1116,8 +1116,12 @@ A to-many relationship resolves to an array of records; depending on access patt #### Constraining related records +:::note Pending release + Pending a 5.3 patch release; see [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). +::: + A relationship selection object accepts `conditions`, `operator`, `sort`, `offset`, and `limit`, with an optional `select` projection: ```javascript @@ -1139,7 +1143,7 @@ const brand = await Brand.get({ `operator` defaults to `and`; use `or` to combine the nested conditions as alternatives. -These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. Neither changes which parents are returned. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Inherited outer join restrictions still apply. +These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Inherited outer join restrictions still apply. See [Filtering and Paging Each Relationship](../rest/querying.md#filtering-and-paging-each-relationship) for missing-value behavior, authorization, cache-local eligibility and ordering, scan costs, and unsupported constraints. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index d1315ce54..7462c8d2f 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -274,8 +274,12 @@ When selecting without a filter on the related table, this acts as a LEFT JOIN ### Filtering and Paging Each Relationship +:::note Pending release + Pending a 5.3 patch release; see [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). +::: + Use square brackets inside `select()` to apply conditions, sort, and pagination to each parent's related records. For a `Brand.products` relationship with `price` and `id` fields: ```http @@ -292,7 +296,7 @@ Each brand receives at most one product priced at 30 or more, ordered by price. | `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | | `select(id,price)` | Project fields from returned children; omit it to return whole child records. | -Use a non-negative safe-integer window; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, unsorted pagination preserves missing target positions; sorting drops missing targets. Child conditions and sort fields remain subject to read permissions. +Use a non-negative safe-integer window; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, pagination alone preserves missing target positions; adding conditions or a sort drops missing targets. Child conditions and sort fields remain subject to read permissions. A condition that traverses another relationship qualifies the immediate child; it does not narrow that child's projected descendants. To filter or page grandchildren, put those constraints on the grandchild's own bracket selection. Outer join restrictions are still inherited. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index cdde2a0e9..8ce2c83c2 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -16,7 +16,7 @@ All patch release notes for 5.3.x are available on the [releases page](https://g [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. -**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Existing cache-local candidate and ordering semantics remain unchanged. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). +**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; that top-level contract is unchanged. Refreshed nonmatches are excluded from the returned child page. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). This correction is not released yet. From e1a7fa44b0b3e5d5cec7b212f54924d2c06a39aa Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 06:57:18 -0600 Subject: [PATCH 04/20] Explain inherited relationship filters with verified responses --- reference/resources/resource-api.md | 57 ++++++++++ reference/rest/querying.md | 164 +++++++++++++++++++++++++++- release-notes/v5-lincoln/5.3.md | 2 +- 3 files changed, 219 insertions(+), 4 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index b37bcd535..ffdc24d1c 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1141,6 +1141,63 @@ const brand = await Brand.get({ }); ``` +The single-record example returns: + +```json +{ + "id": 42, + "products": [ + { + "id": "p2", + "price": 35 + } + ] +} +``` + +**Outer relationship conditions also restrict nested results.** You do not need to repeat an outer product-price condition inside the `products` selection. For example, this returns only brands with products costing at least 30, and includes their cheapest qualifying product: + +```javascript +const brands = []; +for await (const brand of Brand.search({ + conditions: [{ attribute: ['products', 'price'], comparator: 'greater_than_equal', value: 30 }], + select: ['id', 'name', { name: 'products', sort: { attribute: 'price' }, limit: 1, select: ['id', 'name', 'price'] }], +})) { + brands.push(brand); +} +``` + +Using the [sample brand and product data](../rest/querying.md#filtering-and-paging-each-relationship), `brands` contains: + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } +] +``` + +Add a nested `conditions` array only to narrow these inherited candidates further; it does not remove qualifying parents. See the [REST scenarios](../rest/querying.md#filtering-and-paging-each-relationship) for nested-only filtering, stricter child filters, and per-parent pagination. + `operator` defaults to `and`; use `or` to combine the nested conditions as alternatives. These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Inherited outer join restrictions still apply. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 7462c8d2f..ee5b8ad47 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -280,13 +280,171 @@ Pending a 5.3 patch release; see [Honor nested relationship filters and paginati ::: -Use square brackets inside `select()` to apply conditions, sort, and pagination to each parent's related records. For a `Brand.products` relationship with `price` and `id` fields: +Use square brackets inside `select()` to apply conditions, sort, and pagination to each parent's related records. + +**An outer relationship filter also narrows that relationship's nested results.** For example, `products.price=ge=number:30` selects brands with a qualifying product and restricts their selected products to those costing at least 30. **You do not need to repeat that price filter inside `products[...]`.** Add a nested condition only when you want to narrow the already-eligible products further without removing their parents. + +The following examples use a `Brand.products` relationship with this data: + +| Brand ID | Brand | Product ID | Product | Price | +| -------- | ------------- | ---------- | -------- | ----- | +| 42 | Acme | p1 | Cable | 10 | +| 42 | Acme | p2 | Keyboard | 35 | +| 42 | Acme | p3 | Dock | 60 | +| 43 | Budget Basics | p4 | Mouse | 15 | +| 44 | Studio Gear | p5 | Stand | 45 | +| 44 | Studio Gear | p6 | Monitor | 120 | + +Each response below contains only the requested fields. Parent order is shown for this sample; the nested `sort(price)` orders products within each brand, not the brands themselves. + +#### Keep every brand; show its cheapest product costing at least 30 + +The price filter is inside the relationship selection, so Budget Basics remains in the response with an empty product list. ```http -GET /Brand/?select(id,products[price=ge=number:30&sort(price)&limit(1)&select(id,price)]) +GET /Brand/?select(id,name,products[price=ge=number:30&sort(price)&limit(1)&select(id,name,price)]) +``` + +Response: + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 43, + "name": "Budget Basics", + "products": [] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } +] +``` + +#### Return only qualifying brands; inherit the price filter + +The outer price filter removes Budget Basics and also removes cheaper products from the nested candidates. The nested selection only needs sorting, a limit, and a projection; repeating the price filter is unnecessary. + +```http +GET /Brand/?products.price=ge=number:30&select(id,name,products[sort(price)&limit(1)&select(id,name,price)]) +``` + +Response: + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } +] +``` + +#### Use a stricter condition for the displayed products + +Both Acme and Studio Gear qualify because each has a product costing at least 30. The additional nested condition displays only products costing at least 100. Acme remains with an empty array: the nested condition does not remove a parent. It further narrows the inherited candidates; it cannot restore products excluded by the outer filter. + +```http +GET /Brand/?products.price=ge=number:30&select(id,name,products[price=ge=number:100&sort(price)&limit(1)&select(id,name,price)]) +``` + +Response: + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p6", + "name": "Monitor", + "price": 120 + } + ] + } +] +``` + +#### Page through each brand's qualifying products + +After the outer price filter and per-brand sort, `limit(1,2)` skips the first eligible product and returns the second for each qualifying brand. Acme returns Dock, not Keyboard; Studio Gear returns Monitor, not Stand. The offset applies independently to each brand. + +```http +GET /Brand/?products.price=ge=number:30&select(id,name,products[sort(price)&limit(1,2)&select(id,name,price)]) +``` + +Response: + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p3", + "name": "Dock", + "price": 60 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p6", + "name": "Monitor", + "price": 120 + } + ] + } +] ``` -Each brand receives at most one product priced at 30 or more, ordered by price. A brand with no qualifying products remains in the result with an empty `products` array. By comparison, an outer condition such as `/Brand/?products.price=ge=number:30` filters which brands are returned. Existing outer relationship filters also constrain the related candidates available to a nested selection. +#### Nested option reference | Nested option | Behavior | | ------------------ | -------------------------------------------------------------------------------- | diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 8ce2c83c2..66c1cd5d9 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -14,7 +14,7 @@ All patch release notes for 5.3.x are available on the [releases page](https://g ### Pending Patch: Nested Relationship Constraints -[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. +[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. **Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; that top-level contract is unchanged. Refreshed nonmatches are excluded from the returned child page. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). From bb52829382e08eec8c79b11bd6850c10a1a16b3a Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:05:20 -0600 Subject: [PATCH 05/20] Give relationship scenarios a reproducible schema --- reference/rest/querying.md | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index ee5b8ad47..78f1f6b90 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -114,7 +114,7 @@ Use parentheses or square brackets to control order of operations: GET /Product/?rating=5|(price=gt=100&price=lt=200) ``` -Square brackets are recommended when constructing queries from user input because standard URI encoding safely encodes `[` and `]` (but not `(`): +Square brackets are recommended when constructing queries from user input because standard URI encoding safely encodes `[` and `]` (but not `(`). Brackets after a relationship name inside `select()` instead form a [nested relationship query](#filtering-and-paging-each-relationship): ```http GET /Product/?rating=5&[tag=fast|tag=scalable|tag=efficient] @@ -284,7 +284,24 @@ Use square brackets inside `select()` to apply conditions, sort, and pagination **An outer relationship filter also narrows that relationship's nested results.** For example, `products.price=ge=number:30` selects brands with a qualifying product and restricts their selected products to those costing at least 30. **You do not need to repeat that price filter inside `products[...]`.** Add a nested condition only when you want to narrow the already-eligible products further without removing their parents. -The following examples use a `Brand.products` relationship with this data: +The following scenarios use this self-contained schema, including string product IDs and an indexed price field: + +```graphql +type Brand @table @export { + id: Int @primaryKey + name: String @indexed + products: [Product] @relationship(to: "brandId") +} + +type Product @table @export { + id: ID @primaryKey + brandId: Int @indexed + name: String + price: Int @indexed +} +``` + +Use this sample data: | Brand ID | Brand | Product ID | Product | Price | | -------- | ------------- | ---------- | -------- | ----- | @@ -446,13 +463,13 @@ Response: #### Nested option reference -| Nested option | Behavior | -| ------------------ | -------------------------------------------------------------------------------- | -| Conditions | Qualify children of the selected relationship; combine with AND, OR, and groups. | -| `sort(price)` | Order that parent's eligible children before pagination. | -| `limit(end)` | Return at most `end` children per parent. | -| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | -| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | +| Nested option | Behavior | +| ------------------ | --------------------------------------------------------------------------------------------- | +| Conditions | Qualify children of the selected relationship; combine with `&` (AND), `\|` (OR), and groups. | +| `sort(price)` | Order that parent's eligible children before pagination. | +| `limit(end)` | Return at most `end` children per parent. | +| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | +| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | Use a non-negative safe-integer window; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, pagination alone preserves missing target positions; adding conditions or a sort drops missing targets. Child conditions and sort fields remain subject to read permissions. From f3f07e8204dabb44353eccd26acc0f34f8e7bde5 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:09:15 -0600 Subject: [PATCH 06/20] Distinguish outer join filters from nested selections --- reference/resources/resource-api.md | 4 ++-- reference/rest/querying.md | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index ffdc24d1c..e91501d01 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1112,7 +1112,7 @@ When a field is defined as a [relationship](../database/schema.md#relationships) A to-many relationship resolves to an array of records; depending on access pattern you may need to `await` the property before iterating it. -**Join behavior:** selecting a relationship _without_ filtering on it behaves as a **LEFT JOIN** — records with no related row are still returned (the relationship property is omitted or empty). Adding a condition on a related attribute (e.g. `attribute: ['author', 'name']`) behaves as an **INNER JOIN** — only records with a matching related row are returned. +**Join behavior:** selecting a relationship without an outer condition on it behaves as a **LEFT JOIN**: parents with no related row remain present, with the relationship omitted or empty. A top-level `conditions` entry on a related attribute (for example, `attribute: ['author', 'name']`) behaves as an **INNER JOIN**: it removes parents without a matching related row and restricts the selected related records. A `conditions` array inside a nested `select` object only filters the displayed children; it does not remove a parent. #### Constraining related records @@ -1200,7 +1200,7 @@ Add a nested `conditions` array only to narrow these inherited candidates furthe `operator` defaults to `and`; use `or` to combine the nested conditions as alternatives. -These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Inherited outer join restrictions still apply. +These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Top-level `conditions` on that relationship still restrict the child candidates available to the nested selection, so an identical condition need not be repeated. See [Filtering and Paging Each Relationship](../rest/querying.md#filtering-and-paging-each-relationship) for missing-value behavior, authorization, cache-local eligibility and ordering, scan costs, and unsupported constraints. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 78f1f6b90..5f89fbaf6 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -114,7 +114,7 @@ Use parentheses or square brackets to control order of operations: GET /Product/?rating=5|(price=gt=100&price=lt=200) ``` -Square brackets are recommended when constructing queries from user input because standard URI encoding safely encodes `[` and `]` (but not `(`). Brackets after a relationship name inside `select()` instead form a [nested relationship query](#filtering-and-paging-each-relationship): +Square brackets are recommended when constructing queries from user input because standard URI encoding safely encodes `[` and `]` (but not `(`): ```http GET /Product/?rating=5&[tag=fast|tag=scalable|tag=efficient] @@ -126,6 +126,8 @@ Constructing from JavaScript: let url = `/Product/?rating=5&[${tags.map(encodeURIComponent).join('|')}]`; ``` +These root-level brackets group conditions. Brackets after a relationship name inside `select()`, such as `select(id,products[limit(1)])`, instead form a [nested relationship query](#filtering-and-paging-each-relationship). + Groups can be nested for complex conditions: ```http @@ -270,7 +272,7 @@ GET /Product/?brand.name=Microsoft&select(name,brand{name}) GET /Product/?name=Keyboard&select(name,brand{name,id}) ``` -When selecting without a filter on the related table, this acts as a LEFT JOIN — the relationship property is omitted if the foreign key is null or references a non-existent record. +Selecting a relationship without an outer dot-syntax filter on it behaves as a LEFT JOIN: parents remain present even when there are no related records. For a singular foreign key that is null or references a missing record, the relationship property is omitted. An outer filter such as `products.price=ge=number:30` instead removes parents without a qualifying product and also narrows the selected products. A filter inside `select(...,products[...])` only narrows the projected products; it does not remove a parent. See the [worked scenarios](#filtering-and-paging-each-relationship) below. ### Filtering and Paging Each Relationship From 6b7f12b7ffd7fe70f7a6d524b185fee25a6ab994 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:13:40 -0600 Subject: [PATCH 07/20] Clarify relationship scenario schema setup --- reference/rest/querying.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 5f89fbaf6..2c119d44a 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -286,7 +286,7 @@ Use square brackets inside `select()` to apply conditions, sort, and pagination **An outer relationship filter also narrows that relationship's nested results.** For example, `products.price=ge=number:30` selects brands with a qualifying product and restricts their selected products to those costing at least 30. **You do not need to repeat that price filter inside `products[...]`.** Add a nested condition only when you want to narrow the already-eligible products further without removing their parents. -The following scenarios use this self-contained schema, including string product IDs and an indexed price field: +The following scenarios use this self-contained schema, including string product IDs and an indexed price field. In a fresh application, use this schema instead of the earlier Brand/Product example; do not combine the two definitions: ```graphql type Brand @table @export { From 27b69088da4d1b35983dc80a0cad9cb1d4e5b24a Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:27:36 -0600 Subject: [PATCH 08/20] Guard relationship examples and document bounded stable pagination --- .github/workflows/validate.yaml | 3 + package.json | 1 + reference/resources/resource-api.md | 2 + reference/rest/querying.md | 41 ++++ .../nested-relationship-docs/examples.json | 228 ++++++++++++++++++ .../nested-relationship-docs/resources.js | 32 +++ .../nested-relationship-docs/schema.graphql | 12 + scripts/nested-relationship-examples.test.mjs | 64 +++++ 8 files changed, 383 insertions(+) create mode 100644 scripts/fixtures/nested-relationship-docs/examples.json create mode 100644 scripts/fixtures/nested-relationship-docs/resources.js create mode 100644 scripts/fixtures/nested-relationship-docs/schema.graphql create mode 100644 scripts/nested-relationship-examples.test.mjs diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index ecf599ece..35d6e83c2 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -20,6 +20,7 @@ jobs: - Lint Check - Format Check - Workflow Tests + - Documentation Examples include: - check: TypeScript Check command: typecheck @@ -29,6 +30,8 @@ jobs: command: format:check - check: Workflow Tests command: test:workflows + - check: Documentation Examples + command: test:examples steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 diff --git a/package.json b/package.json index 764829735..2474d86cf 100644 --- a/package.json +++ b/package.json @@ -20,6 +20,7 @@ "format:write": "npm run format -- --write", "lint": "echo 0;", "preview:pr": "node scripts/preview-pr.mjs", + "test:examples": "node --test scripts/nested-relationship-examples.test.mjs", "test:workflows": "node scripts/companion-check.test.mjs" }, "devDependencies": { diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index e91501d01..6b12d0671 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1200,6 +1200,8 @@ Add a nested `conditions` array only to narrow these inherited candidates furthe `operator` defaults to `and`; use `or` to combine the nested conditions as alternatives. +For production collections, also set a top-level `limit` to bound parents. Use a unique final sort key for both parent and child order, and remember that a small child limit does not bound sorting work. See [parent pagination and stable sorting](../rest/querying.md#bound-parent-pages-and-use-stable-sorting) for an example and performance guidance. + These options apply independently to each parent's children in both single-record `get()` and collection queries. `limit` is a count; `offset` is the number of eligible children to skip. These nested options do not remove parents: a parent with no matching children remains present with an empty to-many array. Nested conditions qualify immediate children; descendant filtering requires a selection on that descendant. Top-level `conditions` on that relationship still restrict the child candidates available to the nested selection, so an identical condition need not be repeated. See [Filtering and Paging Each Relationship](../rest/querying.md#filtering-and-paging-each-relationship) for missing-value behavior, authorization, cache-local eligibility and ordering, scan costs, and unsupported constraints. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 2c119d44a..2d3092f95 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -463,6 +463,47 @@ Response: ] ``` +#### Bound parent pages and use stable sorting + +The examples above focus on child selection. For a production collection, also limit the parents: a nested `limit(1)` bounds products per brand, not the number of brands. For example, return up to 20 qualifying brands with one product each: + +```http +GET /Brand/?products.price=ge=number:30&sort(name,id)&limit(0,20)&select(id,name,products[sort(price,id)&limit(1)&select(id,name,price)]) +``` + +With the sample data, the response is: + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } +] +``` + +The outer `sort(name,id)` orders brands; the nested `sort(price,id)` orders products within each brand. Use a unique field such as `id` as the final sort key so equal names or prices have a deterministic order. This does not prevent records moving between offset pages when data changes between requests. + +Keep relationship lookup fields and outer filter fields indexed, and project only the fields the client needs. The sample schema indexes `brandId`, `price`, and `Brand.name`. A child sort may still collect and sort all eligible related records before returning its page; an index on `price` does not guarantee an index-ordered nested scan. Keep parent pages small and consider a separately paginated child endpoint for very large relationships. Unsorted nested limits can stop early, but are appropriate only when child order does not matter. + #### Nested option reference | Nested option | Behavior | diff --git a/scripts/fixtures/nested-relationship-docs/examples.json b/scripts/fixtures/nested-relationship-docs/examples.json new file mode 100644 index 000000000..f2abf5eec --- /dev/null +++ b/scripts/fixtures/nested-relationship-docs/examples.json @@ -0,0 +1,228 @@ +{ + "brands": [ + { + "id": 42, + "name": "Acme" + }, + { + "id": 43, + "name": "Budget Basics" + }, + { + "id": 44, + "name": "Studio Gear" + } + ], + "products": [ + { + "id": "p1", + "brandId": 42, + "name": "Cable", + "price": 10 + }, + { + "id": "p2", + "brandId": 42, + "name": "Keyboard", + "price": 35 + }, + { + "id": "p3", + "brandId": 42, + "name": "Dock", + "price": 60 + }, + { + "id": "p4", + "brandId": 43, + "name": "Mouse", + "price": 15 + }, + { + "id": "p5", + "brandId": 44, + "name": "Stand", + "price": 45 + }, + { + "id": "p6", + "brandId": 44, + "name": "Monitor", + "price": 120 + } + ], + "programmatic": [ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } + ], + "single": { + "id": 42, + "products": [ + { + "id": "p2", + "price": 35 + } + ] + }, + "cases": [ + { + "key": "nested_only", + "url": "/Brand/?select(id,name,products[price=ge=number:30&sort(price)&limit(1)&select(id,name,price)])", + "response": [ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 43, + "name": "Budget Basics", + "products": [] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } + ] + }, + { + "key": "outer_inherited", + "url": "/Brand/?products.price=ge=number:30&select(id,name,products[sort(price)&limit(1)&select(id,name,price)])", + "response": [ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } + ] + }, + { + "key": "additional_nested", + "url": "/Brand/?products.price=ge=number:30&select(id,name,products[price=ge=number:100&sort(price)&limit(1)&select(id,name,price)])", + "response": [ + { + "id": 42, + "name": "Acme", + "products": [] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p6", + "name": "Monitor", + "price": 120 + } + ] + } + ] + }, + { + "key": "per_parent_offset", + "url": "/Brand/?products.price=ge=number:30&select(id,name,products[sort(price)&limit(1,2)&select(id,name,price)])", + "response": [ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p3", + "name": "Dock", + "price": 60 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p6", + "name": "Monitor", + "price": 120 + } + ] + } + ] + }, + { + "key": "bounded_parent_page", + "url": "/Brand/?products.price=ge=number:30&sort(name,id)&limit(0,20)&select(id,name,products[sort(price,id)&limit(1)&select(id,name,price)])", + "response": [ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p2", + "name": "Keyboard", + "price": 35 + } + ] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [ + { + "id": "p5", + "name": "Stand", + "price": 45 + } + ] + } + ] + } + ] +} diff --git a/scripts/fixtures/nested-relationship-docs/resources.js b/scripts/fixtures/nested-relationship-docs/resources.js new file mode 100644 index 000000000..ac146ed69 --- /dev/null +++ b/scripts/fixtures/nested-relationship-docs/resources.js @@ -0,0 +1,32 @@ +export class DocumentedQueries extends Resource { + static loadAsInstance = false; + async get() { + const Brand = tables.Brand; + const brand = await Brand.get({ + id: 42, + select: [ + 'id', + { + name: 'products', + conditions: [{ attribute: 'price', comparator: 'greater_than_equal', value: 30 }], + sort: { attribute: 'price' }, + offset: 0, + limit: 1, + select: ['id', 'price'], + }, + ], + }); + const brands = []; + for await (const brand of Brand.search({ + conditions: [{ attribute: ['products', 'price'], comparator: 'greater_than_equal', value: 30 }], + select: [ + 'id', + 'name', + { name: 'products', sort: { attribute: 'price' }, limit: 1, select: ['id', 'name', 'price'] }, + ], + })) { + brands.push(brand); + } + return { single: brand, programmatic: brands }; + } +} diff --git a/scripts/fixtures/nested-relationship-docs/schema.graphql b/scripts/fixtures/nested-relationship-docs/schema.graphql new file mode 100644 index 000000000..484e57d8d --- /dev/null +++ b/scripts/fixtures/nested-relationship-docs/schema.graphql @@ -0,0 +1,12 @@ +type Brand @table @export { + id: Int @primaryKey + name: String @indexed + products: [Product] @relationship(to: "brandId") +} + +type Product @table @export { + id: ID @primaryKey + brandId: Int @indexed + name: String + price: Int @indexed +} diff --git a/scripts/nested-relationship-examples.test.mjs b/scripts/nested-relationship-examples.test.mjs new file mode 100644 index 000000000..fa48c459f --- /dev/null +++ b/scripts/nested-relationship-examples.test.mjs @@ -0,0 +1,64 @@ +import { readFileSync } from 'node:fs'; +import { test } from 'node:test'; +import assert from 'node:assert'; +import ts from 'typescript'; + +const read = (path) => readFileSync(new URL(path, import.meta.url), 'utf8'); +// Keep these fixtures identical to integrationTests/resources/nested-relationship-docs in Harper core. +const fixture = './fixtures/nested-relationship-docs/'; +const examples = JSON.parse(read(fixture + 'examples.json')); +const rest = read('../reference/rest/querying.md') + .split('### Filtering and Paging Each Relationship')[1] + .split('### Many-to-Many Relationships')[0]; +const resource = read('../reference/resources/resource-api.md') + .split('#### Constraining related records')[1] + .split('### `sort`')[0]; +const blocks = (source, language) => + [...source.matchAll(new RegExp('```' + language + '\\n([\\s\\S]*?)```', 'g'))].map((match) => match[1].trim()); +const normalize = (text) => text.replace(/\s+/g, ' ').trim(); + +test('all five REST requests and complete responses match the executable core fixture', () => { + assert.deepStrictEqual( + blocks(rest, 'http'), + examples.cases.map((example) => 'GET ' + example.url) + ); + assert.deepStrictEqual( + blocks(rest, 'json').map(JSON.parse), + examples.cases.map((example) => example.response) + ); +}); + +test('the schema matches the schema loaded by the core HTTP tests', () => { + assert.strictEqual(normalize(blocks(rest, 'graphql')[0]), normalize(read(fixture + 'schema.graphql'))); +}); + +test('the sample-data table matches the seeded brands and products', () => { + const rows = [...rest.matchAll(/^\|\s*(\d+)\s*\|\s*([^|]+)\|\s*(p\d+)\s*\|\s*([^|]+)\|\s*(\d+)\s*\|$/gm)]; + assert.deepStrictEqual( + rows.map((row) => ({ id: row[3], brandId: Number(row[1]), name: row[4].trim(), price: Number(row[5]) })), + examples.products + ); + assert.deepStrictEqual( + [...new Map(rows.map((row) => [Number(row[1]), { id: Number(row[1]), name: row[2].trim() }])).values()], + examples.brands + ); +}); + +function javascriptTokens(source) { + const scanner = ts.createScanner(ts.ScriptTarget.Latest, true, ts.LanguageVariant.Standard, source); + const tokens = []; + while (scanner.scan() !== ts.SyntaxKind.EndOfFileToken) tokens.push(scanner.getTokenText()); + return JSON.stringify(tokens.filter((token, i) => token !== ',' || ![']', '}'].includes(tokens[i + 1]))).slice(1, -1); +} + +test('both programmatic requests and complete responses match the executed fixture resource', () => { + const snippets = blocks(resource, 'javascript'); + assert.strictEqual(snippets.length, 2); + const source = read(fixture + 'resources.js'); + for (const snippet of snippets) + assert( + javascriptTokens(source).includes(javascriptTokens(snippet)), + 'programmatic snippet drifted from the executable fixture' + ); + assert.deepStrictEqual(blocks(resource, 'json').map(JSON.parse), [examples.single, examples.programmatic]); +}); From d489d60966434a8c3c7fa0ca0cf8761bbdae6c3b Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:34:21 -0600 Subject: [PATCH 09/20] Fail docs CI when the core example contract diverges --- reference/rest/querying.md | 18 ++++----- release-notes/v5-lincoln/5.3.md | 2 +- scripts/nested-relationship-examples.test.mjs | 39 +++++++++++++++++++ 3 files changed, 49 insertions(+), 10 deletions(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 2d3092f95..ce19f6b03 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -159,7 +159,7 @@ GET /Product/?brand.name=Microsoft&select(name,brand{name}) ### `limit(end)` or `limit(start,end)` -Limit the number of results returned, with an optional starting offset. +Return records in the zero-based window from `start` (inclusive) to `end` (exclusive). `limit(end)` starts at zero; `limit(start,end)` returns at most `end - start` records. The same window semantics apply inside relationship selections. ```http GET /Product/?rating=gt=3&inStock=true&select(rating,name)&limit(20) @@ -506,15 +506,15 @@ Keep relationship lookup fields and outer filter fields indexed, and project onl #### Nested option reference -| Nested option | Behavior | -| ------------------ | --------------------------------------------------------------------------------------------- | -| Conditions | Qualify children of the selected relationship; combine with `&` (AND), `\|` (OR), and groups. | -| `sort(price)` | Order that parent's eligible children before pagination. | -| `limit(end)` | Return at most `end` children per parent. | -| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | -| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | +| Nested option | Behavior | +| ------------------ | ---------------------------------------------------------------------------------------------------------- | +| Conditions | Qualify children of the selected relationship; combine with `&` (AND), `\|` (OR), and [groups](#grouping). | +| `sort(price)` | Order that parent's eligible children before pagination. | +| `limit(end)` | Return at most `end` children per parent. | +| `limit(start,end)` | Skip `start` eligible children and return at most `end - start`. | +| `select(id,price)` | Project fields from returned children; omit it to return whole child records. | -Use a non-negative safe-integer window; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, pagination alone preserves missing target positions; adding conditions or a sort drops missing targets. Child conditions and sort fields remain subject to read permissions. +Use a non-negative safe-integer window; invalid bounds such as negative or fractional values return HTTP 400; `limit(0)` returns no children without reading them; an absent foreign key keeps its missing-value shape. A missing singular relationship remains absent. For array-valued foreign keys, pagination alone preserves missing target positions; adding conditions or a sort drops missing targets. Child conditions and sort fields remain subject to read permissions. A condition that traverses another relationship qualifies the immediate child; it does not narrow that child's projected descendants. To filter or page grandchildren, put those constraints on the grandchild's own bracket selection. Outer join restrictions are still inherited. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 66c1cd5d9..448467283 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -16,7 +16,7 @@ All patch release notes for 5.3.x are available on the [releases page](https://g [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. -**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; that top-level contract is unchanged. Refreshed nonmatches are excluded from the returned child page. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). +**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). This correction is not released yet. diff --git a/scripts/nested-relationship-examples.test.mjs b/scripts/nested-relationship-examples.test.mjs index fa48c459f..204045401 100644 --- a/scripts/nested-relationship-examples.test.mjs +++ b/scripts/nested-relationship-examples.test.mjs @@ -1,4 +1,5 @@ import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; import { test } from 'node:test'; import assert from 'node:assert'; import ts from 'typescript'; @@ -62,3 +63,41 @@ test('both programmatic requests and complete responses match the executed fixtu ); assert.deepStrictEqual(blocks(resource, 'json').map(JSON.parse), [examples.single, examples.programmatic]); }); + +async function readPeerFile(path) { + const response = await fetch( + 'https://raw.githubusercontent.com/HarperFast/harper/' + + peerRevision + + '/integrationTests/resources/nested-relationship-docs/' + + path, + { signal: AbortSignal.timeout(15_000) } + ); + assert.strictEqual(response.status, 200, 'Cannot read peer fixture: ' + response.url); + return response.text(); +} +let peerRevision = ''; +test( + 'paired documentation fixtures stay synchronized across repositories', + { skip: !process.env.CI && !process.env.HARPER_CORE_FIXTURE_ROOT }, + async () => { + const peerRoot = process.env.HARPER_CORE_FIXTURE_ROOT; + if (!peerRoot) { + const response = await fetch('https://api.github.com/repos/HarperFast/harper/pulls/3279', { + signal: AbortSignal.timeout(15_000), + }); + assert.strictEqual(response.status, 200, 'Cannot resolve companion PR revision'); + const pr = await response.json(); + peerRevision = pr.merged_at ? 'main' : pr.head.sha; + assert(peerRevision === 'main' || /^[a-f0-9]{40}$/.test(peerRevision)); + } + for (const name of ['examples.json', 'schema.graphql', 'resources.js']) { + const expected = readFileSync(new URL(name, new URL(fixture, import.meta.url)), 'utf8'); + const actual = peerRoot ? readFileSync(resolve(peerRoot, name), 'utf8') : await readPeerFile(name); + assert.strictEqual( + actual.replace(/\r\n/g, '\n'), + expected.replace(/\r\n/g, '\n'), + name + ' differs from the companion repository; update both fixtures together' + ); + } + } +); From 82fa9188031f47551290f1f812dc584bb1db5842 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:36:15 -0600 Subject: [PATCH 10/20] Make nested selection and sorting scope explicit --- reference/rest/querying.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index ce19f6b03..9e5c5e323 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -142,13 +142,14 @@ Harper supports special query functions using call syntax, included in the query Specify which properties to include in the response. -| Syntax | Returns | -| -------------------------------------- | ------------------------------------------- | -| `?select(property)` | Values of a single property directly | -| `?select(property1,property2)` | Objects with only the specified properties | -| `?select([property1,property2])` | Arrays of property values | -| `?select(property1,)` | Objects with a single specified property | -| `?select(property{subProp1,subProp2})` | Nested objects with specific sub-properties | +| Syntax | Returns | +| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `?select(property)` | Values of a single property directly | +| `?select(property1,property2)` | Objects with only the specified properties | +| `?select([property1,property2])` | Arrays of property values | +| `?select(property1,)` | Objects with a single specified property | +| `?select(property{subProp1,subProp2})` | Nested objects with specific sub-properties | +| `?select(id,products[limit(1)])` | Parents with a constrained relationship selection; see [relationship options](#filtering-and-paging-each-relationship) | **Examples**: @@ -502,7 +503,7 @@ With the sample data, the response is: The outer `sort(name,id)` orders brands; the nested `sort(price,id)` orders products within each brand. Use a unique field such as `id` as the final sort key so equal names or prices have a deterministic order. This does not prevent records moving between offset pages when data changes between requests. -Keep relationship lookup fields and outer filter fields indexed, and project only the fields the client needs. The sample schema indexes `brandId`, `price`, and `Brand.name`. A child sort may still collect and sort all eligible related records before returning its page; an index on `price` does not guarantee an index-ordered nested scan. Keep parent pages small and consider a separately paginated child endpoint for very large relationships. Unsorted nested limits can stop early, but are appropriate only when child order does not matter. +Keep relationship lookup fields and outer filter fields indexed, and project only the fields the client needs. The sample schema indexes `brandId`, `price`, and `Brand.name`. Nested sorting can use an unindexed child field because it orders the already-related candidates. A child sort may still collect and sort all eligible related records before returning its page; an index on `price` does not guarantee an index-ordered nested scan. Keep parent pages small and consider a separately paginated child endpoint for very large relationships. Unsorted nested limits can stop early, but are appropriate only when child order does not matter. #### Nested option reference From 4181ee9496176e2151b427609e5e24c6101c60f1 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:37:58 -0600 Subject: [PATCH 11/20] Enforce example checks through the required lint CI gate --- .github/workflows/validate.yaml | 3 --- package.json | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index 35d6e83c2..ecf599ece 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -20,7 +20,6 @@ jobs: - Lint Check - Format Check - Workflow Tests - - Documentation Examples include: - check: TypeScript Check command: typecheck @@ -30,8 +29,6 @@ jobs: command: format:check - check: Workflow Tests command: test:workflows - - check: Documentation Examples - command: test:examples steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 diff --git a/package.json b/package.json index 2474d86cf..2327c855e 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,7 @@ "format": "prettier .", "format:check": "npm run format -- --check", "format:write": "npm run format -- --write", - "lint": "echo 0;", + "lint": "npm run test:examples", "preview:pr": "node scripts/preview-pr.mjs", "test:examples": "node --test scripts/nested-relationship-examples.test.mjs", "test:workflows": "node scripts/companion-check.test.mjs" From 6d3cf4b4fd387879807705ec00d4820407df458e Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 07:46:37 -0600 Subject: [PATCH 12/20] Align top-level pagination and sorting explanations --- reference/resources/resource-api.md | 2 +- reference/rest/querying.md | 2 +- scripts/nested-relationship-examples.test.mjs | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 6b12d0671..9d0d5c3c9 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1216,7 +1216,7 @@ Sort order object: | `descending` | Sort descending if `true` (default: `false`) | | `next` | Secondary sort to resolve ties (same structure) | -Harper uses an index to provide sort order, so a `sort` needs one of: +For top-level collection queries, Harper uses an index to provide sort order, so a `sort` needs one of: - An `@indexed` sort `attribute`. Harper aligns the scan with the index automatically — **no condition is required**, and the condition (if any) does not have to be on the sort attribute. - At least one `conditions` entry (on **any** attribute) when the sort `attribute` is not indexed. Harper filters by the condition and then orders the result set in memory. diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 9e5c5e323..732c52d2d 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -182,7 +182,7 @@ GET /Product/?sort(+rating,-price) Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. -Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. +Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the page size (`end - start`, or `end` in the one-argument form) must be a non-negative integer no larger than **10,000**, and the window end (`offset + page size`) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. ### Requesting a count diff --git a/scripts/nested-relationship-examples.test.mjs b/scripts/nested-relationship-examples.test.mjs index 204045401..b4d525840 100644 --- a/scripts/nested-relationship-examples.test.mjs +++ b/scripts/nested-relationship-examples.test.mjs @@ -5,7 +5,7 @@ import assert from 'node:assert'; import ts from 'typescript'; const read = (path) => readFileSync(new URL(path, import.meta.url), 'utf8'); -// Keep these fixtures identical to integrationTests/resources/nested-relationship-docs in Harper core. +// The offline snapshot lets authors validate Markdown without a GitHub connection. const fixture = './fixtures/nested-relationship-docs/'; const examples = JSON.parse(read(fixture + 'examples.json')); const rest = read('../reference/rest/querying.md') From 840061f1ca26c5f82773d6172077a860390e1347 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 13:00:16 -0600 Subject: [PATCH 13/20] docs: demonstrate grouped OR relationship pages and harden fixture parity --- .github/workflows/validate.yaml | 5 + package.json | 2 +- reference/rest/querying.md | 38 +++++ .../nested-relationship-docs/examples.json | 27 ++++ .../github-fixture-fetch.mjs | 97 ++++++++++++ scripts/github-fixture-fetch.test.mjs | 141 ++++++++++++++++++ scripts/nested-relationship-examples.test.mjs | 45 ++---- 7 files changed, 320 insertions(+), 35 deletions(-) create mode 100644 scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs create mode 100644 scripts/github-fixture-fetch.test.mjs diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index ecf599ece..800133483 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -9,6 +9,9 @@ concurrency: group: validate-${{ github.event.pull_request.number }} cancel-in-progress: true +permissions: + contents: read + jobs: validate: name: ${{ matrix.check }} @@ -39,4 +42,6 @@ jobs: - name: Install dependencies run: npm ci - name: Run ${{ matrix.check }} + env: + GH_TOKEN: ${{ matrix.command == 'lint' && github.token || '' }} run: npm run ${{ matrix.command }} diff --git a/package.json b/package.json index 2327c855e..2c09cd930 100644 --- a/package.json +++ b/package.json @@ -20,7 +20,7 @@ "format:write": "npm run format -- --write", "lint": "npm run test:examples", "preview:pr": "node scripts/preview-pr.mjs", - "test:examples": "node --test scripts/nested-relationship-examples.test.mjs", + "test:examples": "node --test scripts/nested-relationship-examples.test.mjs scripts/github-fixture-fetch.test.mjs", "test:workflows": "node scripts/companion-check.test.mjs" }, "devDependencies": { diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 732c52d2d..5935736b2 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -505,6 +505,44 @@ The outer `sort(name,id)` orders brands; the nested `sort(price,id)` orders prod Keep relationship lookup fields and outer filter fields indexed, and project only the fields the client needs. The sample schema indexes `brandId`, `price`, and `Brand.name`. Nested sorting can use an unindexed child field because it orders the already-related candidates. A child sort may still collect and sort all eligible related records before returning its page; an index on `price` does not guarantee an index-ordered nested scan. Keep parent pages small and consider a separately paginated child endpoint for very large relationships. Unsorted nested limits can stop early, but are appropriate only when child order does not matter. +#### Combine OR filters with child sorting and pagination + +Group the OR conditions inside the relationship selection, then separate the sort, page, and projection options with `&`. This request selects products costing less than $30 **or** at least $60, sorts each brand's matches by descending price (then `id`), skips its first match, and returns at most one more: + +```http +GET /Brand/?sort(name,id)&limit(0,20)&select(id,name,products[[price=lt=number:30|price=ge=number:60]&sort(-price,id)&limit(1,2)&select(id,name,price)]) +``` + +The literal `|` is the OR operator in the query syntax; encode individual values rather than the assembled expression. The inner `[price=lt=number:30|price=ge=number:60]` groups only the price conditions; the following `sort()`, `limit()`, and `select()` options apply to the combined matches. + +```json +[ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p1", + "name": "Cable", + "price": 10 + } + ] + }, + { + "id": 43, + "name": "Budget Basics", + "products": [] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [] + } +] +``` + +Acme's matching products sort to Dock ($60), then Cable ($10), so `limit(1,2)` returns Cable. Budget Basics and Studio Gear each have only one match, leaving an empty child page after the offset. All three brands remain because these are nested filters; there is no outer relationship filter to remove parents. Use `limit(0,3)` to see the unskipped matches: Acme returns Dock and Cable, Budget Basics returns Mouse, and Studio Gear returns Monitor. + #### Nested option reference | Nested option | Behavior | diff --git a/scripts/fixtures/nested-relationship-docs/examples.json b/scripts/fixtures/nested-relationship-docs/examples.json index f2abf5eec..fb9ccd00e 100644 --- a/scripts/fixtures/nested-relationship-docs/examples.json +++ b/scripts/fixtures/nested-relationship-docs/examples.json @@ -223,6 +223,33 @@ ] } ] + }, + { + "key": "grouped_or_page", + "url": "/Brand/?sort(name,id)&limit(0,20)&select(id,name,products[[price=lt=number:30|price=ge=number:60]&sort(-price,id)&limit(1,2)&select(id,name,price)])", + "response": [ + { + "id": 42, + "name": "Acme", + "products": [ + { + "id": "p1", + "name": "Cable", + "price": 10 + } + ] + }, + { + "id": 43, + "name": "Budget Basics", + "products": [] + }, + { + "id": 44, + "name": "Studio Gear", + "products": [] + } + ] } ] } diff --git a/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs b/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs new file mode 100644 index 000000000..7b949cddb --- /dev/null +++ b/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs @@ -0,0 +1,97 @@ +import assert from 'node:assert'; +import { readFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { setTimeout as delay } from 'node:timers/promises'; + +export function githubHeaders(url, token) { + if (new URL(url).origin !== 'https://api.github.com') return {}; + return { + Accept: 'application/vnd.github+json', + ...(token ? { Authorization: 'Bearer ' + token } : {}), + }; +} + +export async function readGitHubText(url, { token, signal, timeout = 15_000 } = {}) { + for (let attempt = 1; attempt <= 3; attempt++) { + let response; + try { + response = await fetch(url, { + headers: githubHeaders(url, token), + signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(timeout)]) : AbortSignal.timeout(timeout), + }); + if (response.status === 200) return await response.text(); + } catch { + response = undefined; + if (signal?.aborted || attempt === 3) + throw new Error( + 'Cannot read fixture URL ' + url + ' after ' + attempt + ' attempt(s): network error or timeout' + ); + } + let wait = 250 * 2 ** (attempt - 1); + if (response) { + const retryable = response.status === 408 || response.status === 429 || response.status >= 500; + await response.body?.cancel(); + if (!retryable || attempt === 3) + throw new Error( + 'Cannot read fixture URL ' + url + ': HTTP ' + response.status + ' after ' + attempt + ' attempt(s)' + ); + const retryAfter = response.headers.get('retry-after'); + if (retryAfter !== null) { + const seconds = Number(retryAfter); + const requested = Number.isFinite(seconds) ? seconds * 1000 : Date.parse(retryAfter) - Date.now(); + if (Number.isFinite(requested)) wait = Math.max(0, requested); + } + if (wait > 2000) + throw new Error( + 'Cannot read fixture URL ' + url + ': HTTP ' + response.status + ' requires a longer Retry-After' + ); + } + try { + await delay(wait, undefined, { signal }); + } catch { + throw new Error('Fixture comparison deadline exceeded while reading ' + url); + } + } +} + +export function peerRevision(pr) { + assert(pr.merged_at || pr.state === 'open', 'Companion PR closed without merging; update the fixture pairing'); + const revision = pr.merged_at ? 'main' : pr.head?.sha; + assert(revision === 'main' || /^[a-f0-9]{40}$/.test(revision), 'Invalid companion revision'); + return revision; +} + +export async function verifyPeerFixtures({ + localRoot, + peerRoot, + repository, + pull, + peerPath, + token = process.env.GH_TOKEN || process.env.GITHUB_TOKEN, +}) { + const signal = AbortSignal.timeout(30_000); + let revision; + if (!peerRoot) { + assert(!process.env.CI || token, 'Fixture parity in CI requires GH_TOKEN or GITHUB_TOKEN'); + const metadata = await readGitHubText('https://api.github.com/repos/' + repository + '/pulls/' + pull, { + token, + signal, + }); + const pr = JSON.parse(metadata); + revision = peerRevision(pr); + } + for (const name of ['examples.json', 'schema.graphql', 'resources.js', 'github-fixture-fetch.mjs']) { + const expected = await readFile(resolve(localRoot, name), 'utf8'); + const actual = peerRoot + ? await readFile(resolve(peerRoot, name), 'utf8') + : await readGitHubText( + 'https://raw.githubusercontent.com/' + repository + '/' + revision + '/' + peerPath + '/' + name, + { signal } + ); + assert.strictEqual( + actual.replace(/\r\n/g, '\n'), + expected.replace(/\r\n/g, '\n'), + name + ' differs from the companion repository; update both fixtures together' + ); + } +} diff --git a/scripts/github-fixture-fetch.test.mjs b/scripts/github-fixture-fetch.test.mjs new file mode 100644 index 000000000..8c7a48b97 --- /dev/null +++ b/scripts/github-fixture-fetch.test.mjs @@ -0,0 +1,141 @@ +import { test } from 'node:test'; +import assert from 'node:assert'; +import { createServer } from 'node:http'; +import { + githubHeaders, + peerRevision, + readGitHubText, +} from './fixtures/nested-relationship-docs/github-fixture-fetch.mjs'; + +async function withServer(handler, run) { + const server = createServer(handler); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + try { + await run('http://127.0.0.1:' + server.address().port + '/fixture'); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); + } +} + +test('credentials are attached only to the exact GitHub API origin', () => { + const token = 'test-token-not-a-credential'; + assert.strictEqual( + githubHeaders('https://api.github.com/repos/HarperFast/harper', token).Authorization, + 'Bearer ' + token + ); + for (const url of [ + 'https://raw.githubusercontent.com/HarperFast/harper/main/file', + 'http://api.github.com/file', + 'https://api.github.com.example/file', + 'http://127.0.0.1/file', + ]) + assert.strictEqual(githubHeaders(url, token).Authorization, undefined); + assert.strictEqual(githubHeaders('https://api.github.com/repos/HarperFast/harper').Authorization, undefined); +}); + +test('companion revisions fail closed for closed-unmerged or malformed peers', () => { + const sha = 'a'.repeat(40); + assert.strictEqual(peerRevision({ state: 'open', head: { sha } }), sha); + assert.strictEqual(peerRevision({ state: 'closed', merged_at: '2026-10-19T00:00:00Z' }), 'main'); + assert.throws(() => peerRevision({ state: 'closed', head: { sha } }), /closed without merging/); + assert.throws(() => peerRevision({ state: 'open', head: { sha: 'invalid' } }), /Invalid companion revision/); +}); + +test('transient HTTP failures retry and return the successful body without leaking a token', async () => { + for (const status of [408, 429, 500, 503]) { + let attempts = 0; + await withServer( + (request, response) => { + assert.strictEqual(request.headers.authorization, undefined); + response.writeHead(++attempts < 3 ? status : 200, { 'Retry-After': '0' }); + response.end(attempts < 3 ? 'temporary' : 'fixture bytes'); + }, + async (url) => { + assert.strictEqual(await readGitHubText(url, { token: 'not-for-this-origin' }), 'fixture bytes'); + assert.strictEqual(attempts, 3); + } + ); + } +}); + +test('network disconnects retry within the same request budget', async () => { + let attempts = 0; + await withServer( + (request, response) => { + if (++attempts < 3) request.socket.destroy(); + else response.end('recovered'); + }, + async (url) => { + assert.strictEqual(await readGitHubText(url), 'recovered'); + assert.strictEqual(attempts, 3); + } + ); +}); + +test('permanent HTTP failures do not retry or expose response bodies', async () => { + for (const status of [401, 403, 404]) { + let attempts = 0; + await withServer( + (_request, response) => { + attempts++; + response.writeHead(status, { 'Retry-After': '0' }); + response.end('sensitive-response'); + }, + async (url) => { + await assert.rejects(readGitHubText(url, { token: 'sensitive-token' }), (error) => { + assert(error.message.includes('HTTP ' + status)); + assert(error.message.includes(url)); + assert(!/sensitive|Bearer|Authorization/.test(error.message)); + return true; + }); + assert.strictEqual(attempts, 1); + } + ); + } +}); + +test('transient failures stop after three attempts', async () => { + let attempts = 0; + await withServer( + (_request, response) => { + attempts++; + response.writeHead(503, { 'Retry-After': '0' }); + response.end('unavailable'); + }, + async (url) => { + await assert.rejects(readGitHubText(url), /HTTP 503 after 3 attempt/); + assert.strictEqual(attempts, 3); + } + ); +}); + +test('Retry-After beyond the comparison wait budget fails rather than retrying early', async () => { + let attempts = 0; + await withServer( + (_request, response) => { + attempts++; + response.writeHead(429, { 'Retry-After': '60' }); + response.end(); + }, + async (url) => { + await assert.rejects(readGitHubText(url), /longer Retry-After/); + assert.strictEqual(attempts, 1); + } + ); +}); + +test('the shared comparison signal terminates a stalled request without retries', async () => { + let attempts = 0; + const controller = new AbortController(); + await withServer( + () => { + attempts++; + controller.abort(); + }, + async (url) => { + await assert.rejects(readGitHubText(url, { signal: controller.signal }), /network error or timeout/); + assert.strictEqual(attempts, 1); + } + ); +}); diff --git a/scripts/nested-relationship-examples.test.mjs b/scripts/nested-relationship-examples.test.mjs index b4d525840..971ab1d8c 100644 --- a/scripts/nested-relationship-examples.test.mjs +++ b/scripts/nested-relationship-examples.test.mjs @@ -1,8 +1,9 @@ import { readFileSync } from 'node:fs'; -import { resolve } from 'node:path'; import { test } from 'node:test'; import assert from 'node:assert'; import ts from 'typescript'; +import { fileURLToPath } from 'node:url'; +import { verifyPeerFixtures } from './fixtures/nested-relationship-docs/github-fixture-fetch.mjs'; const read = (path) => readFileSync(new URL(path, import.meta.url), 'utf8'); // The offline snapshot lets authors validate Markdown without a GitHub connection. @@ -18,7 +19,7 @@ const blocks = (source, language) => [...source.matchAll(new RegExp('```' + language + '\\n([\\s\\S]*?)```', 'g'))].map((match) => match[1].trim()); const normalize = (text) => text.replace(/\s+/g, ' ').trim(); -test('all five REST requests and complete responses match the executable core fixture', () => { +test('all six REST requests and complete responses match the executable core fixture', () => { assert.deepStrictEqual( blocks(rest, 'http'), examples.cases.map((example) => 'GET ' + example.url) @@ -64,40 +65,16 @@ test('both programmatic requests and complete responses match the executed fixtu assert.deepStrictEqual(blocks(resource, 'json').map(JSON.parse), [examples.single, examples.programmatic]); }); -async function readPeerFile(path) { - const response = await fetch( - 'https://raw.githubusercontent.com/HarperFast/harper/' + - peerRevision + - '/integrationTests/resources/nested-relationship-docs/' + - path, - { signal: AbortSignal.timeout(15_000) } - ); - assert.strictEqual(response.status, 200, 'Cannot read peer fixture: ' + response.url); - return response.text(); -} -let peerRevision = ''; test( 'paired documentation fixtures stay synchronized across repositories', - { skip: !process.env.CI && !process.env.HARPER_CORE_FIXTURE_ROOT }, + { skip: !process.env.CI && !process.env.HARPER_CORE_FIXTURE_ROOT, timeout: 35_000 }, async () => { - const peerRoot = process.env.HARPER_CORE_FIXTURE_ROOT; - if (!peerRoot) { - const response = await fetch('https://api.github.com/repos/HarperFast/harper/pulls/3279', { - signal: AbortSignal.timeout(15_000), - }); - assert.strictEqual(response.status, 200, 'Cannot resolve companion PR revision'); - const pr = await response.json(); - peerRevision = pr.merged_at ? 'main' : pr.head.sha; - assert(peerRevision === 'main' || /^[a-f0-9]{40}$/.test(peerRevision)); - } - for (const name of ['examples.json', 'schema.graphql', 'resources.js']) { - const expected = readFileSync(new URL(name, new URL(fixture, import.meta.url)), 'utf8'); - const actual = peerRoot ? readFileSync(resolve(peerRoot, name), 'utf8') : await readPeerFile(name); - assert.strictEqual( - actual.replace(/\r\n/g, '\n'), - expected.replace(/\r\n/g, '\n'), - name + ' differs from the companion repository; update both fixtures together' - ); - } + await verifyPeerFixtures({ + localRoot: fileURLToPath(new URL(fixture, import.meta.url)), + peerRoot: process.env.HARPER_CORE_FIXTURE_ROOT, + repository: 'HarperFast/harper', + pull: 3279, + peerPath: 'integrationTests/resources/nested-relationship-docs', + }); } ); From ad165ca15875d1003eab7a2c67bc4045edd117ff Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 13:22:07 -0600 Subject: [PATCH 14/20] docs: correct test guidance and handle errored fixture bodies --- .github/workflows/claude-review.yml | 5 +++-- .github/workflows/gemini-review.yml | 5 +++-- AGENTS.md | 6 +++--- reference/rest/querying.md | 2 +- .../github-fixture-fetch.mjs | 10 +++++++++- scripts/github-fixture-fetch.test.mjs | 14 ++++++++++++++ 6 files changed, 33 insertions(+), 9 deletions(-) diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index f6f35da91..4728d432f 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -89,8 +89,9 @@ jobs: - **Redirects**: `historic-redirects.ts` keeps old URLs alive. Renaming or moving a page without a redirect breaks external links. - **Tooling**: `npm run typecheck` (tsc) and `format:check` (prettier). - `npm run lint` is a no-op stub (`echo 0`) — do not treat a green lint as - evidence of anything. + `npm run lint` runs documented-example and fixture-fetch tests, including + live companion-fixture parity in CI. It does not execute Harper; the + companion core suite owns native HTTP response verification. - Do not comment on formatting; prettier owns it. secrets: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} diff --git a/.github/workflows/gemini-review.yml b/.github/workflows/gemini-review.yml index db8acbff9..87029c47f 100644 --- a/.github/workflows/gemini-review.yml +++ b/.github/workflows/gemini-review.yml @@ -94,8 +94,9 @@ jobs: - **Redirects**: `historic-redirects.ts` keeps old URLs alive. Renaming or moving a page without a redirect breaks external links. - **Tooling**: `npm run typecheck` (tsc) and `format:check` (prettier). - `npm run lint` is a no-op stub (`echo 0`) — do not treat a green lint as - evidence of anything. + `npm run lint` runs documented-example and fixture-fetch tests, including + live companion-fixture parity in CI. It does not execute Harper; the + companion core suite owns native HTTP response verification. - Do not comment on formatting; prettier owns it. secrets: GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} diff --git a/AGENTS.md b/AGENTS.md index 84ba5391d..514abf132 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ Review `README.md` and `CONTRIBUTING.md` for all relevant repository information - Prettier is configured via `@harperfast/code-guidelines/prettier` (see `package.json`). - **Required before every commit/PR:** run `npm run format:write`, then confirm `npm run format:check` is clean. This is a hard gate, not advisory — CI fails any unformatted PR (see [CI](#ci)). - Formatting applies to **every file in the repo**, not just content. Prettier runs against the whole tree (`prettier .`) — `.ts`, `.md`, `.mdx`, `.json`, and files under `plans/`, `scripts/`, etc. Do not assume a directory is exempt because it is not user-facing docs. -- The lint script is a no-op (`echo 0`); do not expect `npm run lint` to catch issues. +- `npm run lint` runs `test:examples`: documented-example checks and fixture-fetch reliability tests. In CI, it also compares fixtures against the companion core revision; locally set `HARPER_CORE_FIXTURE_ROOT` to compare an offline core checkout. ## Content Style @@ -65,7 +65,7 @@ Prefer plain ASCII characters in Markdown unless a typographic character is genu ## Testing -- There is no automated test suite. Verification is done by running the dev server or build. +- `npm run test:examples` checks the documented requests/responses and fixture-fetch behavior; `npm run test:workflows` checks companion workflow scenarios. The companion core suite executes the request/response fixtures against Harper. - `npm run build` is the best end-to-end check — it will fail on broken links, missing imports, or MDX errors. - `npm run typecheck` checks TypeScript config files only. - Known issue: `npm run serve` (post-build preview) 404s on paths like `/docs/4.X` locally due to an upstream `serve-handler` bug. This is not a real breakage — see CONTRIBUTING.md for details. @@ -75,7 +75,7 @@ Prefer plain ASCII characters in Markdown unless a typographic character is genu ## CI -- CI runs `npm run build` and `npm run format:check` on pull requests. +- CI runs the production build, TypeScript Check, Lint Check, Format Check and Workflow Tests on pull requests. Lint Check receives a read-only token for the live companion-fixture comparison. - If CI fails on formatting, run `npm run format:write` locally and commit the result. ## Issues & Project Management diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 5935736b2..2ebeaea06 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -507,7 +507,7 @@ Keep relationship lookup fields and outer filter fields indexed, and project onl #### Combine OR filters with child sorting and pagination -Group the OR conditions inside the relationship selection, then separate the sort, page, and projection options with `&`. This request selects products costing less than $30 **or** at least $60, sorts each brand's matches by descending price (then `id`), skips its first match, and returns at most one more: +Always group OR conditions inside the relationship selection when combining them with query functions, then separate the sort, page, and projection options with `&`. This request selects products costing less than $30 **or** at least $60, sorts each brand's matches by descending price (then `id`), skips its first match, and returns at most one more: ```http GET /Brand/?sort(name,id)&limit(0,20)&select(id,name,products[[price=lt=number:30|price=ge=number:60]&sort(-price,id)&limit(1,2)&select(id,name,price)]) diff --git a/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs b/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs index 7b949cddb..81ada7ce9 100644 --- a/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs +++ b/scripts/fixtures/nested-relationship-docs/github-fixture-fetch.mjs @@ -11,6 +11,14 @@ export function githubHeaders(url, token) { }; } +export async function discardBody(response) { + try { + await response.body?.cancel(); + } catch { + // An errored discarded body must not replace the HTTP failure or suppress retries. + } +} + export async function readGitHubText(url, { token, signal, timeout = 15_000 } = {}) { for (let attempt = 1; attempt <= 3; attempt++) { let response; @@ -30,7 +38,7 @@ export async function readGitHubText(url, { token, signal, timeout = 15_000 } = let wait = 250 * 2 ** (attempt - 1); if (response) { const retryable = response.status === 408 || response.status === 429 || response.status >= 500; - await response.body?.cancel(); + await discardBody(response); if (!retryable || attempt === 3) throw new Error( 'Cannot read fixture URL ' + url + ': HTTP ' + response.status + ' after ' + attempt + ' attempt(s)' diff --git a/scripts/github-fixture-fetch.test.mjs b/scripts/github-fixture-fetch.test.mjs index 8c7a48b97..e2005d455 100644 --- a/scripts/github-fixture-fetch.test.mjs +++ b/scripts/github-fixture-fetch.test.mjs @@ -2,6 +2,7 @@ import { test } from 'node:test'; import assert from 'node:assert'; import { createServer } from 'node:http'; import { + discardBody, githubHeaders, peerRevision, readGitHubText, @@ -34,6 +35,19 @@ test('credentials are attached only to the exact GitHub API origin', () => { assert.strictEqual(githubHeaders('https://api.github.com/repos/HarperFast/harper').Authorization, undefined); }); +test('discarding an already errored response body preserves HTTP failure handling', async () => { + const response = new Response( + new ReadableStream({ + start(controller) { + controller.error(new Error('discarded stream failed')); + }, + }), + { status: 503 } + ); + await assert.rejects(response.body.cancel(), /discarded stream failed/); + await assert.doesNotReject(discardBody(response)); +}); + test('companion revisions fail closed for closed-unmerged or malformed peers', () => { const sha = 'a'.repeat(40); assert.strictEqual(peerRevision({ state: 'open', head: { sha } }), sha); From 71d0f8021c73077d4375c70c22df7f20aa2e35ff Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Mon, 19 Oct 2026 22:50:09 -0600 Subject: [PATCH 15/20] docs: call out unsupported custom resource constraints --- reference/rest/querying.md | 2 +- release-notes/v5-lincoln/5.3.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 2ebeaea06..6686a4b26 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -561,7 +561,7 @@ For source-backed caches, eligibility and sort keys use local cached entries, as Unsorted pagination can stop when the requested page is filled. Sorting or a predicate with few matches can still examine all related candidates; a small output limit does not guarantee a small scan. -Query constraints on plain JSON projections and full-text predicates inside a relationship selection are unsupported and return a client error. Projection-only JSON syntax remains supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. +Nested query constraints require a table-backed relationship. Constraints on plain JSON projections or custom non-table resources, and full-text predicates inside a relationship selection, return HTTP 400 instead of being silently ignored. Projection-only selections remain supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. The [Resource API](../resources/resource-api.md#constraining-related-records) accepts the same options in a nested selection object. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 448467283..ba9082511 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -16,7 +16,7 @@ All patch release notes for 5.3.x are available on the [releases page](https://g [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. -**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections and nested full-text predicates now return a client error instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). +**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). This correction is not released yet. From 0fd7579d3b33b973fc1708cbd10048c20b5e12cf Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 20 Oct 2026 16:19:15 -0600 Subject: [PATCH 16/20] docs: target nested relationship constraints for v5.4 --- reference/resources/resource-api.md | 2 +- reference/rest/querying.md | 2 +- release-notes/v5-lincoln/5.3.md | 8 -------- release-notes/v5-lincoln/5.4.md | 8 ++++++++ 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 9d0d5c3c9..8412c1b0e 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1118,7 +1118,7 @@ A to-many relationship resolves to an array of records; depending on access patt :::note Pending release -Pending a 5.3 patch release; see [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). +Targeted for v5.4.0; not released yet. See [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). ::: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 6686a4b26..02d50da6a 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -279,7 +279,7 @@ Selecting a relationship without an outer dot-syntax filter on it behaves as a L :::note Pending release -Pending a 5.3 patch release; see [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). +Targeted for v5.4.0; not released yet. See [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279). ::: diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index ba9082511..195c6baa9 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -12,14 +12,6 @@ These notes summarize what changed relative to 5.2. The [Harper Pro v5.3.0 relea All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). -### Pending Patch: Nested Relationship Constraints - -[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. - -**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). - -This correction is not released yet. - ## Upgrade Notes Read these before upgrading a production node or cluster. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index e1f638196..81a8ad785 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -8,6 +8,14 @@ title: '5.4' All patch release notes for 5.4.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.4&expanded=true). +## Nested Relationship Constraints + +[Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. + +**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). + +This correction is not released yet. + ## Replication ### Dedicated Replication Threads From b312ce0cb06577aea3f662848a5cf07f8e5d2771 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 24 Oct 2026 09:44:52 -0600 Subject: [PATCH 17/20] fix: separate documentation lint and example checks --- .github/workflows/validate.yaml | 5 +- .oxlintrc.json | 3 + CONTRIBUTING.md | 6 + package-lock.json | 160 ++++++++++++++++++++++++++ package.json | 3 +- scripts/rehype-docusaurus-to-llms.mjs | 2 +- scripts/verify-redirects.mjs | 2 +- 7 files changed, 177 insertions(+), 4 deletions(-) create mode 100644 .oxlintrc.json diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index 800133483..79f8ec840 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -23,6 +23,7 @@ jobs: - Lint Check - Format Check - Workflow Tests + - Example Tests include: - check: TypeScript Check command: typecheck @@ -32,6 +33,8 @@ jobs: command: format:check - check: Workflow Tests command: test:workflows + - check: Example Tests + command: test:examples steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 @@ -43,5 +46,5 @@ jobs: run: npm ci - name: Run ${{ matrix.check }} env: - GH_TOKEN: ${{ matrix.command == 'lint' && github.token || '' }} + GH_TOKEN: ${{ matrix.command == 'test:examples' && github.token || '' }} run: npm run ${{ matrix.command }} diff --git a/.oxlintrc.json b/.oxlintrc.json new file mode 100644 index 000000000..0ef3c520e --- /dev/null +++ b/.oxlintrc.json @@ -0,0 +1,3 @@ +{ + "ignorePatterns": ["static/js/reo.js"] +} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc6b96bed..35b36f16e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -61,6 +61,12 @@ npm run format:check CI runs `npm run format:check` on every pull request and will fail if anything is unformatted. If CI fails on formatting, run `npm run format:write` locally and commit the result. Prettier is configured via `@harperfast/code-guidelines/prettier` (see `package.json`). +## Code Checks + +Run `npm run lint` to lint JavaScript and TypeScript with oxlint. Warnings fail the check. The vendor-provided Reo loader (`static/js/reo.js`) is excluded. + +Run `npm run test:examples` separately to validate the executable nested-relationship examples and fixture-fetch helper. CI runs both commands as separate checks. + ## Site Organization This site is powered by Docusaurus. The documentation is split into four distinct sections, each serving a different purpose and configured as its own Docusaurus plugin instance. diff --git a/package-lock.json b/package-lock.json index 7b8b04dc7..ac3c3d0a8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -24,6 +24,7 @@ "@mdx-js/react": "3.1.1", "@signalwire/docusaurus-plugin-llms-txt": "1.2.2", "clsx": "2.1.1", + "oxlint": "1.31.0", "prettier": "3.8.1", "prism-react-renderer": "2.4.1", "react": "19.2.4", @@ -5540,6 +5541,130 @@ "node": ">=8.0.0" } }, + "node_modules/@oxlint/darwin-arm64": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/darwin-arm64/-/darwin-arm64-1.31.0.tgz", + "integrity": "sha512-HqoYNH5WFZRdqGUROTFGOdBcA9y/YdHNoR/ujlyVO53it+q96dujbgKEvlff/WEuo4LbDKBrKLWKTKvOd/VYdg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@oxlint/darwin-x64": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/darwin-x64/-/darwin-x64-1.31.0.tgz", + "integrity": "sha512-gNq+JQXBCkYKQhmJEgSNjuPqmdL8yBEX3v0sueLH3g5ym4OIrNO7ml1M7xzCs0zhINQCR9MsjMJMyBNaF1ed+g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@oxlint/linux-arm64-gnu": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/linux-arm64-gnu/-/linux-arm64-gnu-1.31.0.tgz", + "integrity": "sha512-cRmttpr3yHPwbrvtPNlv+0Zw2Oeh0cU902iMI4fFW9ylbW/vUAcz6DvzGMCYZbII8VDiwQ453SV5AA8xBgMbmw==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@oxlint/linux-arm64-musl": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/linux-arm64-musl/-/linux-arm64-musl-1.31.0.tgz", + "integrity": "sha512-0p7vn0hdMdNPIUzemw8f1zZ2rRZ/963EkK3o4P0KUXOPgleo+J9ZIPH7gcHSHtyrNaBifN03wET1rH4SuWQYnA==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@oxlint/linux-x64-gnu": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/linux-x64-gnu/-/linux-x64-gnu-1.31.0.tgz", + "integrity": "sha512-vNIbpSwQ4dwN0CUmojG7Y91O3CXOf0Kno7DSTshk/JJR4+u8HNVuYVjX2qBRk0OMc4wscJbEd7wJCl0VJOoCOw==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@oxlint/linux-x64-musl": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/linux-x64-musl/-/linux-x64-musl-1.31.0.tgz", + "integrity": "sha512-4avnH09FJRTOT2cULdDPG0s14C+Ku4cnbNye6XO7rsiX6Bprz+aQblLA+1WLOr7UfC/0zF+jnZ9K5VyBBJy9Kw==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@oxlint/win32-arm64": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/win32-arm64/-/win32-arm64-1.31.0.tgz", + "integrity": "sha512-mQaD5H93OUpxiGjC518t5wLQikf0Ur5mQEKO2VoTlkp01gqmrQ+hyCLOzABlsAIAeDJD58S9JwNOw4KFFnrqdw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@oxlint/win32-x64": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/@oxlint/win32-x64/-/win32-x64-1.31.0.tgz", + "integrity": "sha512-AS/h58HfloccRlVs7P3zbyZfxNS62JuE8/3fYGjkiRlR1ZoDxdqmz5QgLEn+YxxFUTMmclGAPMFHg9z2Pk315A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, "node_modules/@pkgr/core": { "version": "0.2.9", "resolved": "https://registry.npmjs.org/@pkgr/core/-/core-0.2.9.tgz", @@ -16849,6 +16974,41 @@ "node": ">= 0.8.0" } }, + "node_modules/oxlint": { + "version": "1.31.0", + "resolved": "https://registry.npmjs.org/oxlint/-/oxlint-1.31.0.tgz", + "integrity": "sha512-U+Z3VShi1zuLF2Hz/pm4vWJUBm5sDHjwSzj340tz4tS2yXg9H5PTipsZv+Yu/alg6Z7EM2cZPKGNBZAvmdfkQg==", + "dev": true, + "license": "MIT", + "bin": { + "oxc_language_server": "bin/oxc_language_server", + "oxlint": "bin/oxlint" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "funding": { + "url": "https://github.com/sponsors/Boshen" + }, + "optionalDependencies": { + "@oxlint/darwin-arm64": "1.31.0", + "@oxlint/darwin-x64": "1.31.0", + "@oxlint/linux-arm64-gnu": "1.31.0", + "@oxlint/linux-arm64-musl": "1.31.0", + "@oxlint/linux-x64-gnu": "1.31.0", + "@oxlint/linux-x64-musl": "1.31.0", + "@oxlint/win32-arm64": "1.31.0", + "@oxlint/win32-x64": "1.31.0" + }, + "peerDependencies": { + "oxlint-tsgolint": ">=0.8.1" + }, + "peerDependenciesMeta": { + "oxlint-tsgolint": { + "optional": true + } + } + }, "node_modules/p-cancelable": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/p-cancelable/-/p-cancelable-3.0.0.tgz", diff --git a/package.json b/package.json index 2c09cd930..3cde323d6 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,7 @@ "format": "prettier .", "format:check": "npm run format -- --check", "format:write": "npm run format -- --write", - "lint": "npm run test:examples", + "lint": "oxlint --deny-warnings .", "preview:pr": "node scripts/preview-pr.mjs", "test:examples": "node --test scripts/nested-relationship-examples.test.mjs scripts/github-fixture-fetch.test.mjs", "test:workflows": "node scripts/companion-check.test.mjs" @@ -39,6 +39,7 @@ "@mdx-js/react": "3.1.1", "@signalwire/docusaurus-plugin-llms-txt": "1.2.2", "clsx": "2.1.1", + "oxlint": "1.31.0", "prettier": "3.8.1", "prism-react-renderer": "2.4.1", "react": "19.2.4", diff --git a/scripts/rehype-docusaurus-to-llms.mjs b/scripts/rehype-docusaurus-to-llms.mjs index e55ad1a50..6f0af8d62 100644 --- a/scripts/rehype-docusaurus-to-llms.mjs +++ b/scripts/rehype-docusaurus-to-llms.mjs @@ -16,7 +16,7 @@ // 2. Hash-link anchors () are removed entirely. // These are Docusaurus's "direct link to this heading" UI affordances // and have no semantic value to an LLM. The default conversion emits -// noise like `[​](#anchor "Direct link to ...")` next to every heading. +// noise like `[\u200b](#anchor "Direct link to ...")` next to every heading. // // 3. Version badge spans (Added in: ...) // are collapsed into clean italic text. The React render leaves empty diff --git a/scripts/verify-redirects.mjs b/scripts/verify-redirects.mjs index 8e4a9e7de..96675906d 100644 --- a/scripts/verify-redirects.mjs +++ b/scripts/verify-redirects.mjs @@ -117,7 +117,7 @@ async function checkPath(path) { /** Worker-pool concurrency. */ async function runWithConcurrency(items, worker, limit) { - const results = new Array(items.length); + const results = Array.from({ length: items.length }); let i = 0; let done = 0; const total = items.length; From ae2366b04018e97aa4671e0aead81d2fc7668b08 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 24 Oct 2026 10:59:43 -0600 Subject: [PATCH 18/20] fix: preserve required example validation and update lint guidance --- .github/workflows/claude-review.yml | 7 ++++--- .github/workflows/gemini-review.yml | 7 ++++--- .github/workflows/validate.yaml | 10 +++++----- AGENTS.md | 3 ++- CONTRIBUTING.md | 2 +- 5 files changed, 16 insertions(+), 13 deletions(-) diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index 4728d432f..d237598b6 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -89,9 +89,10 @@ jobs: - **Redirects**: `historic-redirects.ts` keeps old URLs alive. Renaming or moving a page without a redirect breaks external links. - **Tooling**: `npm run typecheck` (tsc) and `format:check` (prettier). - `npm run lint` runs documented-example and fixture-fetch tests, including - live companion-fixture parity in CI. It does not execute Harper; the - companion core suite owns native HTTP response verification. + `npm run lint` runs oxlint with warnings treated as errors. The separate + `npm run test:examples` check runs documented-example and fixture-fetch + tests, including live companion-fixture parity in CI. It does not execute + Harper; the companion core suite owns native HTTP response verification. - Do not comment on formatting; prettier owns it. secrets: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} diff --git a/.github/workflows/gemini-review.yml b/.github/workflows/gemini-review.yml index 87029c47f..c49d4d805 100644 --- a/.github/workflows/gemini-review.yml +++ b/.github/workflows/gemini-review.yml @@ -94,9 +94,10 @@ jobs: - **Redirects**: `historic-redirects.ts` keeps old URLs alive. Renaming or moving a page without a redirect breaks external links. - **Tooling**: `npm run typecheck` (tsc) and `format:check` (prettier). - `npm run lint` runs documented-example and fixture-fetch tests, including - live companion-fixture parity in CI. It does not execute Harper; the - companion core suite owns native HTTP response verification. + `npm run lint` runs oxlint with warnings treated as errors. The separate + `npm run test:examples` check runs documented-example and fixture-fetch + tests, including live companion-fixture parity in CI. It does not execute + Harper; the companion core suite owns native HTTP response verification. - Do not comment on formatting; prettier owns it. secrets: GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index 79f8ec840..6b9b4fdfc 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -23,7 +23,6 @@ jobs: - Lint Check - Format Check - Workflow Tests - - Example Tests include: - check: TypeScript Check command: typecheck @@ -33,8 +32,6 @@ jobs: command: format:check - check: Workflow Tests command: test:workflows - - check: Example Tests - command: test:examples steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 @@ -45,6 +42,9 @@ jobs: - name: Install dependencies run: npm ci - name: Run ${{ matrix.check }} - env: - GH_TOKEN: ${{ matrix.command == 'test:examples' && github.token || '' }} run: npm run ${{ matrix.command }} + - name: Run example tests + if: matrix.command == 'lint' + env: + GH_TOKEN: ${{ github.token }} + run: npm run test:examples diff --git a/AGENTS.md b/AGENTS.md index 514abf132..31659256b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,8 @@ Review `README.md` and `CONTRIBUTING.md` for all relevant repository information - Prettier is configured via `@harperfast/code-guidelines/prettier` (see `package.json`). - **Required before every commit/PR:** run `npm run format:write`, then confirm `npm run format:check` is clean. This is a hard gate, not advisory — CI fails any unformatted PR (see [CI](#ci)). - Formatting applies to **every file in the repo**, not just content. Prettier runs against the whole tree (`prettier .`) — `.ts`, `.md`, `.mdx`, `.json`, and files under `plans/`, `scripts/`, etc. Do not assume a directory is exempt because it is not user-facing docs. -- `npm run lint` runs `test:examples`: documented-example checks and fixture-fetch reliability tests. In CI, it also compares fixtures against the companion core revision; locally set `HARPER_CORE_FIXTURE_ROOT` to compare an offline core checkout. +- `npm run lint` runs oxlint over JavaScript and TypeScript with warnings treated as errors. +- `npm run test:examples` separately runs documented-example checks and fixture-fetch reliability tests. In CI, it also compares fixtures against the companion core revision; locally set `HARPER_CORE_FIXTURE_ROOT` to compare an offline core checkout. ## Content Style diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 35b36f16e..b343b912e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -65,7 +65,7 @@ CI runs `npm run format:check` on every pull request and will fail if anything i Run `npm run lint` to lint JavaScript and TypeScript with oxlint. Warnings fail the check. The vendor-provided Reo loader (`static/js/reo.js`) is excluded. -Run `npm run test:examples` separately to validate the executable nested-relationship examples and fixture-fetch helper. CI runs both commands as separate checks. +Run `npm run test:examples` separately to validate the executable nested-relationship examples and fixture-fetch helper. CI runs both commands as separate steps in the required Lint Check job. ## Site Organization From ec09ffab214fd1dd366da86862fa81d9b65e4510 Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Sat, 24 Oct 2026 11:21:03 -0600 Subject: [PATCH 19/20] fix: align Node minimum with oxlint requirements --- CONTRIBUTING.md | 2 +- package-lock.json | 2 +- package.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b343b912e..6e557a183 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,7 +21,7 @@ You don't need to get type, area, or milestone exactly right when filing — mai Before contributing, please ensure you have the following installed: -- Node.js (version 22) +- Node.js (version 22.12.0 or later; `.node-version` selects Node 24) - npm (version 10 - included with Node.js) ## Set Up diff --git a/package-lock.json b/package-lock.json index ac3c3d0a8..af992b647 100644 --- a/package-lock.json +++ b/package-lock.json @@ -32,7 +32,7 @@ "typescript": "5.9.3" }, "engines": { - "node": ">=22" + "node": ">=22.12.0" } }, "node_modules/@ai-sdk/gateway": { diff --git a/package.json b/package.json index 3cde323d6..7baa0735e 100644 --- a/package.json +++ b/package.json @@ -59,7 +59,7 @@ ] }, "engines": { - "node": ">=22" + "node": ">=22.12.0" }, "prettier": "@harperfast/code-guidelines/prettier", "license": "Apache-2.0" From 4329248ea0a253e8b807d2216ae02e174e291a4f Mon Sep 17 00:00:00 2001 From: Kyle Bernhardy Date: Tue, 27 Oct 2026 07:54:42 -0600 Subject: [PATCH 20/20] Clarify nested filter validation alongside JSON array windows --- reference/rest/querying.md | 2 +- release-notes/v5-lincoln/5.4.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 02d50da6a..45474a83a 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -561,7 +561,7 @@ For source-backed caches, eligibility and sort keys use local cached entries, as Unsorted pagination can stop when the requested page is filled. Sorting or a predicate with few matches can still examine all related candidates; a small output limit does not guarantee a small scan. -Nested query constraints require a table-backed relationship. Constraints on plain JSON projections or custom non-table resources, and full-text predicates inside a relationship selection, return HTTP 400 instead of being silently ignored. Projection-only selections remain supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. +Nested filters require a table-backed relationship. Filters on plain JSON projections or custom non-table resources, and full-text predicates inside a relationship selection, return HTTP 400 instead of being silently ignored. Projection-only selections and sorting/pagination of plain JSON arrays remain supported. Earlier versions accepted nested filters and pagination but could silently ignore them and return all related children; this correction can reduce child arrays and reject previously ignored unsupported options. The [Resource API](../resources/resource-api.md#constraining-related-records) accepts the same options in a nested selection object. diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index 81a8ad785..8fb51dac4 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -12,7 +12,7 @@ All patch release notes for 5.4.x are available on the [releases page](https://g [Honor nested relationship filters and pagination](https://github.com/HarperFast/harper/pull/3279) corrects REST and programmatic nested selections that accepted filters or pagination but returned all related records. Filters, ordering, offset, and limit apply per parent in collection and single-record reads. Parents remain present when the nested selection has no matches. Outer relationship filters also constrain the selected children, so an identical filter does not need to be repeated inside the nested selection. -**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported constraints on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). +**Upgrade impact:** requests using nested constraints can return fewer children. Unsupported filters on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Sorting and pagination of plain JSON arrays remain supported. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See [Filtering and Paging Each Relationship](/reference/v5/rest/querying#filtering-and-paging-each-relationship). This correction is not released yet.