Conversation
dirvine
left a comment
There was a problem hiding this comment.
Reviewed exact head e7e6ac5fd8621877dbc1e60e88904a1007b5e474.
Approve: no introduced blockers found by parent review, independent code-review seat, or independent OpenRouter GLM-5.2 review.
Checked lost-record rollback floor, full-group repair threshold, bootstrap gating, bounded inbound/outbound work, cancellation, signature verification off the executor, and local versus remote fault attribution.
Verified Linux CI logs: 110 e2e passed (3 ignored), 19 audit attack PoCs passed, 16 live audit-handler tests passed. Code/build/test checks are green. A newly triggered metadata/template workflow had queued checks on the final check; this approval does not waive them. Local focused compilation did not yield a completed test result, so no local pass is claimed for this PR.
Non-blocking qualifications: responder admission takes a short async lock, so it is not literally wait-free; semaphore admission is non-blocking and expensive work remains in bounded tasks. Restricting reciprocal hints to admitted fresh syncs intentionally narrows delivery; watch convergence under churn.
#238 and #240 merge cleanly at Git level, but the combined tree has not been tested here. Re-run combined audit/repair coverage before rollout. Approval is not production-rollout sign-off.
- Repair counts its quorum over the whole configured close group, so a node that sees few members of it cannot adopt an owner-signed state nobody paid for on one peer's word, and it does not repair while it is still bootstrapping. - Fetch and state requests are admitted fairly before a task exists, at most 128 outstanding and 16 per peer, as chunk fetches are. A state request larger than an honest one is dropped before admission. - This node keeps at most 8 pointer requests outstanding at any one peer, across repair, possession checks and pruning, so it never exceeds the allowance a peer gives it and is never dropped as a flood. - A peer is remembered as speaking pointers only while it is in the routing table, and that set is capped. - A fetched record's signature is verified off the async executor, as ADR-0016 says every pointer signature check is. - A commit for a record this node lost takes the lost state or a newer one, never an older one, so a write verified before the loss cannot roll the node back.
A possession check fetched the record from a peer, and a record that did not verify was penalised by the fetch and then again by the check as a record not held. The fetch now says whether it penalised an invalid record, and the check does not add a second penalty for it.
…swers A possession check looked up the close group once and then asked each peer in turn. Asking waits on that peer's other requests and on the peers before it, so a peer could leave the group before it was asked and still be penalised for not holding the record. Membership and capability are now checked again before a penalty. A record this node could not check, because the verification task failed here, was treated as no record and charged to the peer. It is now its own outcome and charges nobody. The judgement of one answer is a pure function with a unit test, which fails if an invalid record, already charged by the fetch, is charged again as missing.
…nterval Every neighbour-sync request started a hint push before admission, so a peer sending small sync requests, even refused ones, made this node scan every pointer it holds, with a routing lookup per record, as often as it liked. Hints now answer only an admitted request, at most once per peer per shortest sync interval, which an honest peer never syncs faster than, and at most eight answers run at once. A request that finds them busy gets no hints this time; the next sync round sends them anyway. The traffic summary counted the six pointer messages but never logged them. It now does, in a fourth summary line.
…nd never refuse a new one Hint answers ran before the neighbour-sync worker's freshness check, so a request later shed as stale had already cost a scan of every pointer, and any sender could be answered. A full map of answered peers also refused every new peer, an honest bootstrapping one included, for a whole sync interval. Answers now run only for a request that is admitted and still fresh, only for a routing-table peer, and a full map forgets the peer answered longest ago instead of refusing anyone.
…obody for it, and stop serving at shutdown This node's own requests to one peer waited for a permit with no limit and past shutdown, outside the request's timeout, so repeated paid updates to one close peer could pile up possession checks waiting forever and hold shutdown open. A request now waits for its permit no longer than its own timeout and not past shutdown, and one never sent is its own outcome: a possession check judges nobody on it. Serve tasks waiting for a serve permit ignored shutdown, so requests admitted just before it kept reading disk and answering after cancellation. The wait now ends at shutdown, and the work behind it.
The waits added for shutdown picked either branch at random when a permit was free and shutdown had already begun, so a request could still be sent, or a stored record read and served, after cancellation. Shutdown is now checked first and again once a permit is held. A test asks repeatedly with a free permit after shutdown and requires nothing be sent each time.
e7e6ac5 to
5f5b457
Compare
Both change the pointer possession check. #240 judges a peer's answer in judge_possession, after which the peer is charged only if it still owes the record. The final state here adds one outcome to that judgement: a peer serving a different final state from the one offered holds the other side of an owner's fork, as the merge rule told it to, so it is named loudly and not charged.
Hardening for merged pointer replication (#231), from a review of it. Each change brings the code back in line with what ADR-0016 already says it does.
Not changed: a peer asking which state this node holds is answered from the index, not the file. The module does that on purpose, so a vote costs no disk read, and a lost file is caught on its first read. A possession check still penalises a peer that does not answer in time, as ADR-0003 does for chunks under the same admission limits.
Linear issue
Closes V2-1363
Risk tier
Compatibility
Semver impact
Test evidence
mainat56b5770(0.21.0, with main's Rust 1.99 lint updates), with no conflicts: the seven reviewed commits are unchanged. At head5f5b457, locally on macOS with Rust 1.99.0:cargo test --lib --features test-utils1,233 passed;cargo test --lib --no-default-features1,188 passed;pointer_convergence15; thee2epointer, replication and fresh-offer tests, 50, passed over QUIC;cargo clippy --all-targets --all-features -- -D warnings,cargo fmt --check,cargo docwith-D warningsandscripts/adr-governance.pyagainstmainall clean. The full CI matrix runs on this head. feat(pointer): keep the first final state, and look before taking one #239 merges this head, and its combined run is listed there.a_thin_view_of_the_group_cannot_adopt_on_one_vote: one visible peer can no longer decide a repair. It fails when the quorum width goes back to the peers seen.a_commit_after_a_loss_takes_nothing_older_than_what_was_lost:put_bytesof an older state after a loss isStale, the lost state itself is restored. It fails when the commit rule is reverted.outbound_permits_in_use_are_never_dropped, plus a compile-time check that the per-peer serve allowance is at least twice what an honest node asks of one peer.a_possession_check_judges_each_answer_once,a_request_that_cannot_be_sent_in_time_is_given_up_not_queued(including shutdown winning over a free permit, every time) anda_peer_is_answered_with_hints_once_per_sync_interval. Each fails when its fix is reverted, which I checked.pointer_convergence15/15.e7e6ac5:cargo test --lib --features test-utils1,233 passed;e2e110 passed, 3 ignored as onmain;migration_reclaims_disk2,migration_crash_safety5,migration_shared_volume5,storage_scale2,webrtc_direct_devnet3,poc_commitment_audit_attacks19,poc_audit_handler_live16,poc_bootstrap_stall3,poc_shutdown_lmdb_drain1, andpointer_convergence15, all passing.cargo clippy --all-targets --all-features -- -D warnings,cargo fmt --check,cargo docwith--deny=warningspass.New dependency
none
ADR
https://github.com/WithAutonomi/ant-node/blob/main/docs/adr/ADR-0016-pointers-immutable-owner.md
Mitigation / rollback
Revert. Nothing is persisted or changed on the wire, so a node running the previous code behaves as before.