Skip to content

Authenticate conversation streams and retire the global feed - #192

Open
nirmalhk7 wants to merge 6 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/agrinet-authenticated-streams
Open

nirmalhk7 wants to merge 6 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/agrinet-authenticated-streams

Conversation

@nirmalhk7

@nirmalhk7 nirmalhk7 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Conversation streams require a verified account JWT and ownership before SSE headers or private events are sent. The unrestricted /events feed remains retired with HTTP 410. Browser streams use Authorization headers, close on account changes or token expiry, and stop retrying after authorization failures or oversized frames. Credentials stay out of URLs and browser storage.

This update integrates current main at 92cd98394ab830569b6316d7c2570ed4a8d2e48c, including merged #191 ownership and #197 security hardening. Published history is preserved through an append-only merge; the diff against main retains the stream change rather than duplicating those prerequisites. Shared chat throttles, conversation-scoped attachment lookup, safe file creation, retry receipts, and retryable conversation deletion are retained.

The stream route now applies the existing chat-route IP-attempt guard, verified account authentication, a Redis-backed account request limit, and createConcurrentStreamLimit({ maximum: 5 }) before opening the stream. The duplicate process-local capacity count is removed. Deleting a pending stream ends its response and releases capacity before a stalled ownership read completes. A disconnect while Redis acquisition is pending releases the late reservation without opening a stream. Message cleanup precedes parent deletion, preserving ownership for a failed-cleanup retry; successful parent deletion closes local streams.

Verification on merge commit 45a4fe06600ea42f1d2bbd4a54a758c36b4fe131:

  • The production-route regression first reproduced HTTP 200 despite five shared reservations. The pending-deletion check reproduced unreleased capacity, and the acquisition-race check reproduced dispatch after disconnect. All pass after their fixes.
  • cd backend && npm test: 74 Node tests pass; two optional Redis cases skip without explicit service URLs. All 103 Jest tests across 28 suites pass.
  • Both optional integrations pass against disposable loopback Redis: separate processes share the account request budget and five-stream cap; renewed tokens and conversation changes do not reset capacity; disconnect frees a slot on the other server. Removing the shared-cap guard in an isolated control makes the cross-process stream test fail with HTTP 200 instead of 429.
  • The real DynamoDB ownership script passes owner conditions, legacy isolation, multi-page listing, attachment lookup, and deletion using disposable tables.
  • All 20 chat UI tests and 10 planner tests pass. The Vite chat build, frontend lint, and Next.js production build pass.
  • Backend syntax, isolated-source hardcoded-URL and whitespace checks pass. Full-repository Qlty comparisons add no findings: nine existing smells and 13 existing check findings remain. Existing worktree build caches and dependency/configuration files were left untouched; tests ran in an isolated source copy with matching lockfile dependencies.

Deployment limits remain explicit. Multiple replicas require REDIS_URL, RATE_LIMIT_REDIS_REQUIRED=true, and the exact deployed chat origins in CORS_ALLOWED_ORIGINS. Shared capacity does not add cross-replica event delivery, deletion revocation, or account-token revocation. The existing reservation helper uses JWT expiry and an eight-day Redis key TTL rather than short renewable leases; process/store failure limits are documented. No production deployment verification is claimed.

The change follows NTARI's Bug-Build-Test guide, fork/PR contribution rules, per-commit DCO sign-offs and independent human review. CONFORMANCE.md binds the private application controls while keeping ledger/covenant/escrow stand-ins unchanged. Hosted checks, including CodeQL, must confirm the published head, and the earlier changes-requested review still requires human re-review. The author must not merge this PR. No Slack task or forwarding record is updated by this change.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Comment thread backend/server.js Outdated
req.on('close', () => sseClients.delete(res));
// Account-scoped streams share the same owner check as message reads/sends.
const conversationStreams = require('./lib/conversationStreams');
app.get('/events', chatAuth, (_req, res) => {
Comment thread backend/server.js Outdated
const payload = `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`;
clients.forEach(res => res.write(payload));
}
app.get('/stream/:conversationId', chatAuth, asyncHandler(conversationStreams.open));
Comment thread backend/server.js Outdated
// Middleware
app.use(authMiddleware);
app.use('/uploads', express.static(path.join(__dirname, 'uploads')));
app.use('/uploads', chatAuth, asyncHandler(require('./middleware/attachmentOwner')), express.static(path.join(__dirname, 'uploads'), {
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, Nirmal. Locking /stream and /events behind an account token plus conversation ownership closes a real hole: on main, anyone with a conversation ID can read its live chat. Registering the stream before the ownership read, revoking even pending streams on delete, closing on token expiry, and using the same 404 for foreign, legacy and missing chats are all careful work. The tests are strong: they caught 31 of the 40 bugs we planted.

What I tested:

  • 5154bdf via merge ref cfc4f03, plus a fresh merge onto current main 7524c48. That merge has the same code as your new head 4571d3a; only the order of two CONFORMANCE.md sections differs.
  • Results: backend 37/37 node --test and 73/73 Jest, chat-ui 13/13, vite build, frontend lint, test:planner and next build all pass. The two new backend test files also passed 8 more reruns.

Requesting changes because of one merge blocker:

  1. The CodeQL gate fails with 5 high js/missing-rate-limiting alerts: server.js:67, :70 and :83, conversationRoutes.js:8 and communicationRoutes.js:8. Main's ruleset blocks merges on these. Two are real:
    • /uploads now scans the whole Messages table on every request (message.js:69-84).
    • /stream has no cap on open streams per user.
      Please add express-rate-limit after chatAuth, keyed by req.user.userId, at least on /uploads and /stream. Cap open streams per user in conversationStreams.open (429 above about 5, decremented in cleanup()), and drop chatAuth from the 410 /events route. The alerts that moved over from main (#5, #15, #16) can be rate-limited too or dismissed by a maintainer with a reason.

Should fix (fine as follow-ups):

  1. Attachment lookup: serve files at /conversations/:id/attachments/:name behind conversationOwner and Query that one partition, or add a filePath GSI.

  2. Sign-in state: nothing calls setAccountToken yet. Please show a visible "sign in" state on 401 in Sidebar.jsx:29-36 and ChatWindow instead of only logging.

Nits:

  1. Merge order: main only allows squash or rebase merges, so once #191 lands this branch will conflict in CONFORMANCE.md and conversation-ownership.md. Please rebase then so the PR holds only the stream commit. While you're there, rename "A1/A2" to #191/#192 in authenticated-streams.md:3, :44 and conversation-ownership.md:47-48.

  2. Flaky test: conversationStreams.test.js:29 can fail near a second boundary. Sign with exp = Math.ceil(Date.now()/1000) + 1 and raise the 1500 ms timeout above 2 s.

  3. Attachment names with %, # or ? can't be downloaded (accountClient.mjs:43-50). Validate the raw name and request '/uploads/' + encodeURIComponent(name).

  4. A frame over 1 MiB makes the client reconnect every second. Treat it as a terminal error instead.

  5. Older issues for separate tickets, not this PR:

    • Deleting a chat with more than 25 messages leaves them behind (message.js:43-63).
    • Each token and message is sent twice on the same stream (communication_controller.js:13-29).
    • The ownership script isn't run in CI yet.

I'll re-test once the rebased head is up.

Apply account request budgets and cap pending plus active streams at
five per account. Release capacity on expiry, disconnect and deletion,
and return 410 for the retired global feed without authentication.

Carry shared ownership and attachment fixes. Stop oversized-frame
retries while preserving valid frames coalesced in one network chunk.
Add regression coverage and update deployment documentation.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>

Copy link
Copy Markdown
Collaborator

Please keep this PR dependent on #191 and do not merge it first. After #191 and security PR #197 are merged, rebase this branch onto the updated main and remove the duplicated #191 changes from this PR. Wire the stream route in this order: chatAuth → explicit account-keyed chat rate limiter → createConcurrentStreamLimit({ maximum: 5 }) → conversationStreams.open. Replace the process-local active-stream cap with this shared Redis-backed reservation helper; it releases reservations on close/finish and expires abandoned reservations. The legacy stream endpoint remains 410 until this authenticated wiring is in place. Also keep the CORS allowlist in sync with the actual deployed chat origin.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
@nirmalhk7
nirmalhk7 force-pushed the nirmal/agrinet-authenticated-streams branch from 4f34cd9 to c16fe74 Compare November 6, 2026 06:57
Preserve merged account, ownership, attachment and retry protections. Apply the shared reservation helper to authenticated streams, release pending deletions and late disconnected acquisitions, and keep private event delivery and revocation limits explicit.

Verification: 74 Node and 103 Jest tests pass; both optional Redis integrations pass explicitly. DynamoDB ownership checks, 20 chat UI tests, 10 planner tests, frontend lint and both builds pass. Full Qlty comparison adds no findings.
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants