Skip to content

Page collection reads and restrict federation export to inspection - #194

Open
nirmalhk7 wants to merge 7 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/collection-pagination
Open

nirmalhk7 wants to merge 7 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/collection-pagination

Conversation

@nirmalhk7

Copy link
Copy Markdown
Contributor

What the change is

Adds bounded pages to conversation lists, the user directory, and read-only federation exports. JSON response shapes remain compatible; encrypted continuation tokens travel in response headers, and the bundled chat client follows them.

Dependency: stacked on #191 at eae4e15. The current combined diff includes that ownership PR; this PR’s own change is commit 4c58c3240377ac7faaa039733dfdc3a17c0cd901. Review #191 first and deploy private chat with #192.

Why it is needed

Single-page DynamoDB scans silently lose later records, while collecting every page in one response makes reads unbounded. Following more pages also increases the impact of existing privacy gaps: directory responses expose account fields, and federation export sends raw Users records and local dialog notes.

How the change works

Each request evaluates up to 50 records per table by default, with a maximum of 100. Cursors expire after one hour and bind the account, endpoint, limit and filters. Empty filtered pages retain their continuation. Conversation ownership applies on every page; directory responses allowlist public fields and include coarse location only with the existing sharing opt-in.

Federation export requires a distinct read-only key and omits account records, dialog narrative and scalar ratings. Outbound inspection uses explicitly configured peer read keys, refuses redirects, and requires declared pagination. Scheduled paths inspect all pages and report counts; automatic importing is disabled and lastSyncAt is not advanced. The existing importer cannot safely merge redacted summaries without source-aware conflict handling. Manual import and other account/admin hardening remain documented follow-up work.

Scans provide no snapshot or stable sort under concurrent changes. External clients must follow the pagination headers. Export-key configuration, field omissions, compatibility changes, and deployment dependencies are detailed in the guide below.

Where it is involved

Backend collection routes, conversation model, shared cursor/scanning helper, federation export/inspection paths, and the chat account client. API, privacy and rollout guide. CONFORMANCE.md records the application boundary and remaining federation work.

Verified locally: all 37 Node tests, 82 Jest tests, 13 chat-client tests, and the production UI build passed. New HTTP and client regressions failed before implementation. A real DynamoDB Local check passed continuation across the 1 MiB scan boundary, empty filtered pages, owner isolation, private projections, account-client traversal, read-only export counts, and preservation of local credentials and narrative. The same storage check is included in CI with a digest-pinned database image. Syntax, hardcoded-URL and diff whitespace checks passed. The commit carries a DCO sign-off.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Bound conversation, directory, and federation scans with encrypted continuation tokens. Keep owner and sharing filters on every page, and make bundled clients follow continuation explicitly.

Use a separate read-only export credential and explicit peer configuration. Exclude local accounts and narrative, and disable automatic merging of these summaries until source-aware reconciliation is defined. Document rollout dependencies and compatibility changes.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
@nirmalhk7
nirmalhk7 marked this pull request as ready for review October 28, 2026 20:19
Comment thread backend/routes/userRoutes.js Fixed
Comment thread backend/server.js Fixed
Apply the 1,000-per-minute limits before authentication and after
account verification. Adding an API key must not reset the account's
allowance. Bound the local storage-check server as well.

Keep pagination and private attachment checks intact. Merge current
main's retry-safe messages, retain both message-model exports and CORS
headers, and test retries against owned conversations. API-key-only
callers remain unable to send or replay private messages.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Move input checks and saved-receipt replay into focused helpers. Extract
verification database setup and worker cleanup without changing request
results, receipt keys or lifecycle ordering.

Add a regression for a missing stored assistant reply: replay must return
410 without repeating delivery or returning a partial success.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>

Copy link
Copy Markdown
Contributor Author

Resolved the conflicts with main in a7cc234.

For GET /users, the directory retains id, username, the existing role, and main’s locationType field/filter. Coarse location is returned only when locationPrivacy === true. Scalar reputationScore and reputationWeight are excluded: the repository’s covenant guidance requires full reputation distributions rather than a single score. This preserves useful structural discovery without exposing private account data or inventing a replacement reputation format. The rationale is also recorded in handler comments and the pagination guide. Location-type filters are bound into continuation cursors.

The earlier description’s separate export-key flow is superseded by #197’s per-peer HMAC authentication and signed-response verification. Bounded pages, encrypted cursors, account/narrative omissions, and scheduled inspection without automatic imports or lastSyncAt updates are preserved. Admin-only provisioning and conditional account writes remain intact.

Verification on this head passed: cd backend && npm test (68 Node tests and 119 Jest tests), cd frontend/chat-ui && npm test (13 tests), the chat production build, and DYNAMODB_ENDPOINT=http://127.0.0.1:18001 node scripts/check-collection-pagination.js against disposable DynamoDB Local tables. The storage check covers the 1 MiB continuation boundary, location-type filtering, privacy projections, HMAC export traversal, and unchanged local credentials/narrative. Hosted checks and the existing #191/#192 review and rollout dependencies remain separate gates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants