Skip to content

fix(pointer): harden pointer replication against thin views and floods - #240

Open
grumbach wants to merge 7 commits into
WithAutonomi:mainfrom
grumbach:fix/pointer-replication-hardening
Open

grumbach wants to merge 7 commits into
WithAutonomi:mainfrom
grumbach:fix/pointer-replication-hardening

Conversation

@grumbach

@grumbach grumbach commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Hardening for merged pointer replication (#231), from a review of it. Each change brings the code back in line with what ADR-0016 already says it does.

  • Repair on a thin view. Repair counted its quorum over however many members of the close group this node could see. A node still filling its routing table could see one, and adopt an owner-signed state nobody paid to store on that one peer's word. It now counts over the whole configured group, so members it cannot see are unanswered rather than a smaller quorum, and it does not repair at all while bootstrapping.
  • Serve floods. Every fetch and state request spawned a task before it waited for one of 32 serve permits, so nothing bounded how many waited. Requests are now admitted before a task exists, fairly across peers, as chunk fetches are: at most 128 outstanding and 16 per peer. A state request larger than an honest one is dropped before admission.
  • Honest peers under that limit. This node now keeps at most 8 pointer requests outstanding at any one peer, across repair, possession checks and pruning together. That is half the allowance a peer gives it, so its requests are never dropped as a flood and an honest peer is never judged on a request it never saw.
  • Capability set. Any sender, routed or not, was remembered as speaking pointers, with nothing bounding the set. Only routing-table peers are remembered now, and the set is capped.
  • Signature checks on the executor. A record fetched from a peer was verified on the async executor. It is now verified on the blocking pool, as ADR-0016 says every pointer signature check is.
  • Lost records. A commit for a record this node had lost accepted any state, so a write verified before the loss could roll the node back past it. It now takes the lost state or a newer one, never an older one.
  • Possession checks. One invalid record was charged twice, once by the fetch and once as missing; a peer that left the group while the check waited was still penalised; and a verification failure on this node was charged to the peer. Each answer is now judged once, membership is checked again before a penalty, and a failure of this node's own, including a request it never sent, charges nobody.
  • Waiting and shutdown. This node's own requests to one peer waited for a permit with no limit and past shutdown. A request now waits no longer than its own timeout and not past shutdown, and serve tasks stop waiting at shutdown; when shutdown and a permit are ready together, shutdown wins.
  • Hint floods. Every neighbour-sync request made this node scan every pointer it holds, before the request was admitted. Hints now answer only an admitted, still-fresh request from a routing-table peer, at most once per peer per shortest sync interval, at most eight at once.
  • Traffic logging. The six pointer messages were counted but never logged; they now are.

Not changed: a peer asking which state this node holds is answered from the index, not the file. The module does that on purpose, so a vote costs no disk read, and a lost file is caught on its first read. A possession check still penalises a peer that does not answer in time, as ADR-0003 does for chunks under the same admission limits.

Linear issue

Closes V2-1363

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

  • Wire: none. The same messages, answered or dropped under load as before.
  • Storage: none.
  • API: none. Only crate-internal functions changed.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

  • Rebased on 2026-10-02 onto main at 56b5770 (0.21.0, with main's Rust 1.99 lint updates), with no conflicts: the seven reviewed commits are unchanged. At head 5f5b457, locally on macOS with Rust 1.99.0: cargo test --lib --features test-utils 1,233 passed; cargo test --lib --no-default-features 1,188 passed; pointer_convergence 15; the e2e pointer, replication and fresh-offer tests, 50, passed over QUIC; cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --check, cargo doc with -D warnings and scripts/adr-governance.py against main all clean. The full CI matrix runs on this head. feat(pointer): keep the first final state, and look before taking one #239 merges this head, and its combined run is listed there.
  • a_thin_view_of_the_group_cannot_adopt_on_one_vote: one visible peer can no longer decide a repair. It fails when the quorum width goes back to the peers seen.
  • a_commit_after_a_loss_takes_nothing_older_than_what_was_lost: put_bytes of an older state after a loss is Stale, the lost state itself is restored. It fails when the commit rule is reverted.
  • outbound_permits_in_use_are_never_dropped, plus a compile-time check that the per-peer serve allowance is at least twice what an honest node asks of one peer.
  • a_possession_check_judges_each_answer_once, a_request_that_cannot_be_sent_in_time_is_given_up_not_queued (including shutdown winning over a free permit, every time) and a_peer_is_answered_with_hints_once_per_sync_interval. Each fails when its fix is reverted, which I checked.
  • Every pointer e2e test, 12 of them over QUIC, passes: repair by neighbour sync, a late joiner catching up, a lone unpaid state never adopted, pruning, possession checks, audits. The replication e2e tests pass too, 49 of them. pointer_convergence 15/15.
  • Before the rebase, every test step CI runs, locally on macOS at e7e6ac5: cargo test --lib --features test-utils 1,233 passed; e2e 110 passed, 3 ignored as on main; migration_reclaims_disk 2, migration_crash_safety 5, migration_shared_volume 5, storage_scale 2, webrtc_direct_devnet 3, poc_commitment_audit_attacks 19, poc_audit_handler_live 16, poc_bootstrap_stall 3, poc_shutdown_lmdb_drain 1, and pointer_convergence 15, all passing.
  • cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --check, cargo doc with --deny=warnings pass.
  • Dev testnet: not run. That gate is the release manager's call.

New dependency

none

ADR

https://github.com/WithAutonomi/ant-node/blob/main/docs/adr/ADR-0016-pointers-immutable-owner.md

Mitigation / rollback

Revert. Nothing is persisted or changed on the wire, so a node running the previous code behaves as before.

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head e7e6ac5fd8621877dbc1e60e88904a1007b5e474.

Approve: no introduced blockers found by parent review, independent code-review seat, or independent OpenRouter GLM-5.2 review.

Checked lost-record rollback floor, full-group repair threshold, bootstrap gating, bounded inbound/outbound work, cancellation, signature verification off the executor, and local versus remote fault attribution.

Verified Linux CI logs: 110 e2e passed (3 ignored), 19 audit attack PoCs passed, 16 live audit-handler tests passed. Code/build/test checks are green. A newly triggered metadata/template workflow had queued checks on the final check; this approval does not waive them. Local focused compilation did not yield a completed test result, so no local pass is claimed for this PR.

Non-blocking qualifications: responder admission takes a short async lock, so it is not literally wait-free; semaphore admission is non-blocking and expensive work remains in bounded tasks. Restricting reciprocal hints to admitted fresh syncs intentionally narrows delivery; watch convergence under churn.

#238 and #240 merge cleanly at Git level, but the combined tree has not been tested here. Re-run combined audit/repair coverage before rollout. Approval is not production-rollout sign-off.

- Repair counts its quorum over the whole configured close group, so a
  node that sees few members of it cannot adopt an owner-signed state
  nobody paid for on one peer's word, and it does not repair while it is
  still bootstrapping.
- Fetch and state requests are admitted fairly before a task exists, at
  most 128 outstanding and 16 per peer, as chunk fetches are. A state
  request larger than an honest one is dropped before admission.
- This node keeps at most 8 pointer requests outstanding at any one peer,
  across repair, possession checks and pruning, so it never exceeds the
  allowance a peer gives it and is never dropped as a flood.
- A peer is remembered as speaking pointers only while it is in the
  routing table, and that set is capped.
- A fetched record's signature is verified off the async executor, as
  ADR-0016 says every pointer signature check is.
- A commit for a record this node lost takes the lost state or a newer
  one, never an older one, so a write verified before the loss cannot
  roll the node back.
A possession check fetched the record from a peer, and a record that did
not verify was penalised by the fetch and then again by the check as a
record not held. The fetch now says whether it penalised an invalid
record, and the check does not add a second penalty for it.
…swers

A possession check looked up the close group once and then asked each
peer in turn. Asking waits on that peer's other requests and on the
peers before it, so a peer could leave the group before it was asked and
still be penalised for not holding the record. Membership and capability
are now checked again before a penalty.

A record this node could not check, because the verification task
failed here, was treated as no record and charged to the peer. It is
now its own outcome and charges nobody.

The judgement of one answer is a pure function with a unit test, which
fails if an invalid record, already charged by the fetch, is charged
again as missing.
…nterval

Every neighbour-sync request started a hint push before admission, so a
peer sending small sync requests, even refused ones, made this node scan
every pointer it holds, with a routing lookup per record, as often as it
liked. Hints now answer only an admitted request, at most once per peer
per shortest sync interval, which an honest peer never syncs faster
than, and at most eight answers run at once. A request that finds them
busy gets no hints this time; the next sync round sends them anyway.

The traffic summary counted the six pointer messages but never logged
them. It now does, in a fourth summary line.
…nd never refuse a new one

Hint answers ran before the neighbour-sync worker's freshness check, so
a request later shed as stale had already cost a scan of every pointer,
and any sender could be answered. A full map of answered peers also
refused every new peer, an honest bootstrapping one included, for a
whole sync interval. Answers now run only for a request that is
admitted and still fresh, only for a routing-table peer, and a full map
forgets the peer answered longest ago instead of refusing anyone.
…obody for it, and stop serving at shutdown

This node's own requests to one peer waited for a permit with no limit
and past shutdown, outside the request's timeout, so repeated paid
updates to one close peer could pile up possession checks waiting
forever and hold shutdown open. A request now waits for its permit no
longer than its own timeout and not past shutdown, and one never sent is
its own outcome: a possession check judges nobody on it.

Serve tasks waiting for a serve permit ignored shutdown, so requests
admitted just before it kept reading disk and answering after
cancellation. The wait now ends at shutdown, and the work behind it.
The waits added for shutdown picked either branch at random when a
permit was free and shutdown had already begun, so a request could
still be sent, or a stored record read and served, after cancellation.
Shutdown is now checked first and again once a permit is held. A test
asks repeatedly with a free permit after shutdown and requires nothing
be sent each time.
@grumbach
grumbach force-pushed the fix/pointer-replication-hardening branch from e7e6ac5 to 5f5b457 Compare October 2, 2026 04:21
grumbach added a commit that referenced this pull request Oct 2, 2026
Both change the pointer possession check. #240 judges a peer's answer in
judge_possession, after which the peer is charged only if it still owes the
record. The final state here adds one outcome to that judgement: a peer serving
a different final state from the one offered holds the other side of an
owner's fork, as the merge rule told it to, so it is named loudly and not
charged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants