Skip to content

security: prevent account takeover and secure federation - #197

Merged
NetworkTheoryAppliedResearchInstitute merged 83 commits into
mainfrom
security/account-federation-hardening
Nov 9, 2026
Merged

NetworkTheoryAppliedResearchInstitute merged 83 commits into
mainfrom
security/account-federation-hardening

Conversation

@csecrestjr

@csecrestjr csecrestjr commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This is a security-first remediation for the account takeover and federation data exposure identified during review of #193. It is based on current main and contains implementation and regression tests, not a patch artifact.

Account and user data

  • Make POST /users administrator-only using a verified JWT principal; the shared browser/API key does not qualify as admin identity.
  • Require an individual account JWT for private chat routes; shared API-key-only chat calls are rejected with 401.
  • Generate user IDs on the server and use DynamoDB conditional creates to prevent overwriting existing accounts.
  • Preserve public registration at /api/auth/register, also using server-generated IDs and conditional writes.
  • Separate public directory, authenticated-account, and administrator response DTOs. Omit password hashes, verification codes, precise coordinates, and unneeded private fields.
  • Protect /api/admin/users and /api/admin/keys with explicit administrator authorization and project only required fields.

Federation

  • Apply the protections to the actual mounted file backend/federation/federationRoutes.js.
  • Require per-peer HMAC request authentication; derive sourceNodeId from the authenticated peer rather than the request body. Apply an IP-based attempt budget before peer authentication and a peer-keyed budget after it.
  • Sign federation export responses and verify them against the operator-managed NodeRegistry identity.
  • Use explicit allowlists for user, listing, and transaction exports.
  • Strip credentials, contact data, roles, verification state, coordinates, and unknown fields from imported users; preserve authenticated source and remote-user identifiers; reject ID collisions with an atomic conditional write.
  • Correct the sync path/flow to use /federation/export and the correct /federation/import contract. The sync job verifies the peer response before importing.

Rate limiting and temporary fail-closed behavior

  • Add express-rate-limit and ioredis; limit by authenticated account, authenticated peer, or IP when no individual identity exists.
  • Provide a Redis-backed store for distributed request counters and a shared concurrent-stream reservation helper. Configure RATE_LIMIT_REDIS_REQUIRED=true for multi-instance deployments.
  • Add chat request limits after authentication.
  • Until the owner-checked chat changes are rebased and deployed, retire the legacy /events and /stream/:conversationId endpoints with 410 responses and temporarily disable /uploads downloads. Do not re-enable these until Restrict private chats and attachments to their owners #191/Authenticate conversation streams and retire the global feed #192 protections are present.
  • Document CORS origins and peer/Redis configuration in .env.example and backend/docs/security-hardening.md.

Regression tests added

  • User creation rejects caller-supplied IDs and non-admin callers; conditional writes and collision handling.
  • Public/admin response allowlists and secret omission.
  • Federation import sanitization, authenticated provenance, conditional collision rejection, missing/invalid peer authentication, minimal exports, and signed responses.
  • Per-account/IP request limits and concurrent stream-cap/disconnect-release behavior.

Verification status

  • Latest head: 6b3e2fd3f35bc8313a6e91f21ad7cdeccf4794e6 (updated 2026-11-07).
  • DCO: passed — GitHub reports “All commits are signed off!” on this exact head.
  • Backend regression, CodeQL (JavaScript/TypeScript and Actions), and hardcoded-URL checks: passed on this exact head.
  • The backend regression workflow and its whitespace check are green.
  • Merge blocker still open: backend/sync/importHelper.js still imports peer listings and transactions using unconditional whole-row DynamoDB put calls. There is no per-type allowlist, authenticated provenance stamp, origin-node skip, or collision-safe row handling for these records. The current importHelper.test.js still asserts those unconditional writes. Do not merge while scheduled federation import can overwrite local marketplace and escrow data.
  • CONFORMANCE.md has not yet been updated to record the federation admission/provenance rules and the listings/transactions import limitation.
  • The maintainer's November 4 CHANGES_REQUESTED review remains the active human review. After fixing the federation import boundary and updating its round-trip tests/conformance note, request a fresh review on the new head.
  • Additional pre-deployment follow-ups from that review remain to be addressed or explicitly tracked: trusted-proxy configuration for IP-based limits; use of the Redis-backed limiter consistently for auth/chat routes and accurate Redis failure handling; short renewable stream-reservation leases; and fixed HMAC known-answer/tampering tests.

Deployment and follow-up requirements

Review requested

Please independently review administrator authorization, peer credential provisioning, federation provenance semantics, export allowlists, and the Redis deployment assumptions. Do not merge until CI is green and a different authorized maintainer approves.

@csecrestjr
csecrestjr marked this pull request as ready for review November 3, 2026 16:55

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 3f8c128. Thanks, Calvin. This closes the account-takeover paths we found while reviewing #193, and most of it is solid. Two things block merge: the DCO check, and federation import of listings and transactions. Details below.

What we tested

  • Commit: head 3f8c128. Its merge ref 8525c54 is main 9373310 plus this branch, with no conflicts against main.
  • Baseline, main 9373310 run locally on Node 24: node:test 61/61, Jest 46/46 (19 suites).
  • This PR, from CI run 41678759143 (backend-regression, Node 22) on 8525c54: node:test 60/60, Jest 65/65 (25 suites), and git diff --check is clean. Jest 65/65 also reproduced locally on a copy of 3f8c128.
  • No new failures. The count changes make sense: five API-key chat retry tests were removed on purpose, one filter test went with the role/location filters, and there are 19 new Jest tests in 6 new suites.
  • CodeQL on the merge ref shows 0 open alerts, and the 20 missing-rate-limiting alerts are fixed.
  • We also ran the PR's own modules against an in-memory DynamoDB stand-in, and broke some protections on purpose to see whether the tests notice.
  • Frontend not run: no frontend files changed.

What's good

  • POST /users is admin-only with a verified JWT. It rejects body ids and unknown fields, generates the id on the server, creates conditionally, returns 409 on collision and returns the public DTO. It's well tested.
  • Register reads only four fields, so id, role and verified can't come from the body.
  • Public, account and admin DTOs are enforced twice, by ProjectionExpression and by allowlist.
  • requireAdmin covers all of /api/admin, and the shared key alone can't pass.
  • Federation:
    • HMAC over the raw body, compared with timingSafeEqual.
    • Per-peer secrets that fail closed.
    • sourceNodeId comes from the authenticated peer.
    • Signed export responses are checked against NodeRegistry.
    • Exports use allowlists.
    • Imported users are sanitized and created conditionally.
      Hashes and contact fields no longer leave the node.
  • /events, /stream and /uploads fail closed until #191/#192.
  • This adds the repo's first backend test workflow, and the PR body is upfront about deployment prerequisites.

Blocking

  1. DCO: none of the 83 commits has Signed-off-by. DCO is a required check with no bypass, and squash merging won't fix it. On your machine, as Calvin Secrest dev@calvinsecrest.com:
git switch security/account-federation-hardening   # at 3f8c128
git rebase --signoff 9373310
git rev-parse HEAD^{tree}   # should print 01f68149c3a88bece9b7a7edd790f57b8881626c
git push --force-with-lease=security/account-federation-hardening:3f8c1285d00edbaf482992d409882c516ebc921c origin security/account-federation-hardening
  • We tried this on a throwaway copy: no conflicts, same tree.
  • Please don't use the DCO app's "Set DCO to pass" button. It waives the sign-off instead of adding it.
  • Jodson needs to confirm that CLAUDE.md rule 4 (never force-push) allows this for an author's own unmerged branch. If it doesn't, squash onto a new branch instead (git switch -c security/account-federation-hardening-signed 9373310 && git merge --squash 3f8c128 && git commit -s) and open a PR that supersedes this one.
  • Push before anyone approves, because a push dismisses approvals.
  • Any code fixes should be signed commits (git commit -s). The tree check above only applies to a push that adds sign-offs and nothing else.
  1. Federation import still overwrites local listings and transactions, and with this PR it happens in normal operation.
  • backend/sync/importHelper.js:39-44 writes each peer listing and transaction whole: no allowlist, no sourceNodeId, no condition.
  • On main this never ran, because the sync job posted to a route that doesn't exist. This PR makes the 6-hourly sync import for real (federation/federationSyncJob.js:60) and narrows the export (federation/federationRoutes.js:38-43). The export also re-sends rows a node imported earlier.
  • Result: once two nodes sync both ways, each node's own marketplace transactions come back as {id, status} and replace the originals. That wipes buyerId, sellerId, the escrow flags and the dialog fields. Listings lose category, media, tags and availability.
  • After that, release-escrow refuses that transaction for good, and with #196 both parties lose access to it.
  • These writes happen before the users step, so they land even when the run then fails.
  • sync/importHelper.test.js:15-17 asserts the unconditional put.
  • This is the CLAUDE.md pair "drop provenance on federation import" and "silent last-write-wins without an audit trail", and it touches the escrow invariant.

Smallest fix: stop importing listings and transactions for now, and note that as an open item in CONFORMANCE.md. Fuller fix:

  • per-type allowlists that never accept escrow, party or dialog fields;
  • stamp sourceNodeId from the authenticated peer;
  • export originNodeId and skip rows that originated locally;
  • skip and log collisions row by row instead of throwing;
  • validate the whole payload before the first write.

Either way, replace the test assertion with a round trip: A exports, B imports, B exports, A imports, and A's rows come out unchanged.

Should fix (here, or in a tracked follow-up before any deployment)

  1. Repeat syncs stop at the first user already imported. importHelper.js:45-52 throws on an existing id, and the export re-sends every user, so the second sync with any peer fails:
  • users after the collision never import;
  • lastSyncAt stops updating (federationSyncJob.js:62-67);
  • POST /federation/import returns 409 after the listing and transaction writes have already happened.

In a two-way pair, a node's own users come back to it on the first pull. Fix:

  • treat a row with the same sourceNodeId/remoteUserId as already imported (skip it, or update guarded on sourceNodeId);
  • skip users that originated locally;
  • add a test that syncs twice with non-empty users. Today's sync tests only use users: [].
  1. trust proxy is never set (server.js:11). The HTTPS rule for peers means running behind nginx or another TLS proxy (docs/FEDERATION_GUIDE.md:20). Every IP-keyed limiter would then key on the proxy address, so login/register and federation attempts become one budget shared by all clients. express-rate-limit won't warn, because the limiters use custom keyGenerators. Add a TRUST_PROXY setting (hop count or CIDR, unset by default, refuse true), document it, and test it with X-Forwarded-For.

  2. The docs overstate Redis sharing.

  • authRoutes.js:10-17 and the inline chat limiters use the in-memory store. With REDIS_URL set, login/register limits are still per process.
  • docs/security-hardening.md:35 and .env.example:36-37 say the counters are shared and that startup fails without the store, but the code only checks that REDIS_URL is set.
  • When Redis is down, limited routes return 500, and ioredis logs unhandled error events.

Build these limiters with createRequestLimiter, each with its own prefix (that also fixes the mixed draft-6/draft-8 headers), or correct the docs. Add a redis 'error' listener.

  1. Tests that don't guard the new protections:
  • authRoutes.test.js is unchanged. Spreading req.body into register, or dropping its ConditionExpression or its 409 branch, all keep CI green. Add tests that a body id, role, verified or reputationScore is ignored, that the condition is set, and that a collision returns 409.
  • Removing the final HMAC comparison in verifyPeerResponse (federationPeerAuth.js:109) keeps all 65 Jest tests green, because the negative test changes the node id and the signature together. Dropping digest, path or method from both signer and verifier also passes. Add fixed known-answer signature vectors for requests and responses, plus negative tests: tampered body, path or method; another peer's secret; timestamp ±301 s; a response with the right node id but a bad signature; a stale response.
  • /conversations API-key rejection and the pre-auth limiters aren't tested through the real routes.
  1. Docs:
  • security-hardening.md:55 says no upload-creation endpoint exists, but POST /messages/:conversationId with file.data still writes under backend/uploads. The fix is pending in #185/#191.
  • :29 cites a Locations table that only exists in #193. Here, coordinates live on Users rows, and the export projection is what keeps them out.
  • :63 names #193's workflow. This PR adds backend-regression.
  • Please also add a CONFORMANCE.md note (CLAUDE.md:5) covering: how peers are admitted in each direction, users being create-only with provenance, the listings/transactions open item, and reputation still crossing as one number.
  1. streamConcurrency.js isn't mounted yet, but the docs tell #192 to use it. A reservation lasts until the token expires (7 days) and is released only when the stream closes, so a deploy or crash leaves slots taken for days, and logging in again doesn't help. Before #192 wires it, use a short lease renewed while the stream is open, capped at exp, and release on shutdown.

  2. chatRateLimit.js:4-10 always uses the prefix agrinet:chat, and rateLimit.js:62 exports an unused copy. With Redis, every route that uses it shares one counter with different limits, so 60 chat requests would also block /stream and /uploads. Require a per-route prefix and delete the unused export.

Federation policy: decided

  1. #194 set the opposite federation policy on the same files. No federation exists yet to decide between them, so Jodson has decided it as a founder-board bootstrap act (bylaws §16.1): this PR's stance stands, with scheduled import on and your HMAC peer auth. #194 should rebase onto this and keep its pagination. Because scheduled import stays on, item 2 has to be fixed before this merges.

Separately, reputationScore still crosses as one number, in the public /users DTO (models/user.js:23) and in the export (JFA line 8). That is a separate open question and isn't part of this decision.

Nits / follow-ups (non-blocking)

  • requireAdmin.js:4 trusts the role inside a 7-day token, and POST /users can grant admin with no record of who did it. Before any role-change route lands, check the stored role. Also, authMiddleware returns on x-api-key before it reads the bearer token, so an admin client that sends both (frontend/user_login.js does) gets 403. Have requireAdmin verify the bearer token itself, like chatAuth does, and add one test through the real server.js middleware order.
  • Registration doesn't enforce unique emails, and login takes the first match of one unpaginated Scan (authRoutes.js:55-60). This predates the PR and will get its own PR, which touches authRoutes.js and userRoutes.js and uses your 409 wording; whichever lands second rebases.
  • /trends and the AI insight helper also read Users rows outside the new DTOs. This predates the PR and will get its own PR, so nothing is needed here.
  • JWT_SECRET has no length check. Mirror the 32-character federation rule at startup, and pin HS256 in authMiddleware.
  • .env.example:31-32: the placeholder secrets are long enough to pass the 32-character check. Use empty values, or reject replace-with.
  • Federation signing, before any peer goes live:
    • require different secrets per direction (security-hardening.md:17,23 currently allow one);
    • add a version tag, both node ids and the sorted query string to the signed text (#194 adds query parameters);
    • add stale/future timestamp tests.
  • federationSyncJob.js:42-44: set maxRedirects: 0, validateStatus to accept only 200, and a generous maxContentLength.
  • Export sourceNodeId next to originNodeId/originUserId so downstream nodes can see a relay.
  • /deposit is mounted (server.js:16) before apiRateLimit. Move it into the route list.
  • security-hardening.yml: add a push trigger on main, timeout-minutes, and current action majors (CI warns about Node 20). Mock ioredis in a test to cover RedisRateLimitStore.
  • Older docs still describe x-api-key chat, live /events and /stream, and REDIS_HOST/PORT (backend/README.md, docs/API_TESTING.md, docs/FEDERATION_GUIDE.md). A pointer to security-hardening.md would do. Also document how the first admin gets created.
  • server.js lines 13 and 39 are 421 and 740 characters long. Please wrap them.
  • PR body: please say it supersedes #186 and the auth-route move in #183, credit Nirmal, and list the conflicting PRs.

Overlap
This branch merges cleanly with #196 and conflicts with #183, #186, #191, #192, #193 (.env.example only), #194, #195 (server.js) and #161. #186 is fully covered here. #183's serverAuth.test.js is worth porting with the public-DTO expectation, because nothing in this PR checks that register and login work through the real server.

We'll re-test whatever new head you push. Any approval will be pinned to that commit.

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
…rite

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
…rite

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
…rite

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
…mits

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
… contract

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
…cation

Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>
Signed-off-by: Calvin Secrest <dev@calvinsecrest.com>

Copy link
Copy Markdown
Collaborator Author

Follow-up verification on head 6b3e2fd3f35bc8313a6e91f21ad7cdeccf4794e6 (2026-11-07): DCO now passes (“All commits are signed off!”), and backend regression, CodeQL, and hardcoded-URL checks all pass on this head.

I rechecked the outstanding federation-import blocker from the November 4 review. It is still present: backend/sync/importHelper.js writes full peer listing and transaction objects with unconditional put calls before processing users; backend/sync/__tests__/importHelper.test.js still asserts those writes. This can overwrite local listing/escrow records during two-way scheduled sync. CONFORMANCE.md also does not yet record the federation import limitation/provenance rules.

Please keep #197 unmerged until that import path is disabled or made provenance-aware, allowlisted, collision-safe, validated before writes, and covered by the requested A→B→A round-trip regression test. Once pushed, request a fresh review on the new commit.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 6b3e2fd. Every commit now carries Calvin's sign-off. The code tree is identical to the head reviewed on 11-04 (01f6814), and DCO, CodeQL, hardcoded-urls and backend-regression all pass.

Review item 2 (federation import overwriting local listings and transactions) is waived as a merge condition, as a founder-board bootstrap act recorded in the governance registry (Entry 4, annotating Entry 3's 11-04 federation decision). It is still open and has to be fixed before the first peer node is registered. The other should-fix items from the 11-04 review stay open as follow-ups.

@NetworkTheoryAppliedResearchInstitute
NetworkTheoryAppliedResearchInstitute merged commit d9784c3 into main Nov 9, 2026
6 checks passed
NetworkTheoryAppliedResearchInstitute added a commit that referenced this pull request Nov 9, 2026
Resolve the backend/server.js conflict with #197 by taking main's
server.js and reapplying this branch's three changes: require the
JSON error handler, give the CORS refusal status 403 with an exposed
message, and register the handler after every route.

Full backend suite on the merge: node:test 64/64, Jest 65/65.

Signed-off-by: Jodson Graves <info@ntari.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NetworkTheoryAppliedResearchInstitute

Copy link
Copy Markdown
Collaborator

AWS verification (re: Calvin's Nov 7 Slack question on the operating conditions for this waiver)

Checked the NTARI AWS account across every region today (2026-11-10):

  • No DynamoDB tables exist in the account — there is no NodeRegistry table.
  • No Lambda functions, ECS clusters, EC2 instances, Elastic Beanstalk apps, Secrets Manager secrets, or SSM parameters exist anywhere in the account.
  • Current-month AWS spend is $0.00.
  • The only two IAM users in the account have been inactive since early 2026, each scoped only to read-only access on a Parallel Computing Service GPU cluster — unrelated to Agrinet's data layer, with no DynamoDB/Secrets Manager/env-var access of any kind.

So both conditions hold: no federation peers are registered, and no federation HMAC credentials are configured — because there is no live Agrinet AWS deployment to hold either right now.

Caveat: this covers the NTARI AWS account only. It does not cover NTARIHQ (the self-hosted environment NTARI has been migrating workloads to). If Agrinet's backend is actually running there, the equivalent check needs to be done against that environment before the first peer node is registered.

Checked by: Jodson Graves (via Claude), 2026-11-10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants