Skip to content

fix(storage): stop counting a peer that disconnects mid-tally as unreported - #228

Open
grumbach wants to merge 3 commits into
WithAutonomi:mainfrom
grumbach:fix/peer-count-disconnect
Open

grumbach wants to merge 3 commits into
WithAutonomi:mainfrom
grumbach:fix/peer-count-disconnect

Conversation

@grumbach

@grumbach grumbach commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Linear issue

Closes V2-1290

Split from V2-1260, whose saorsa-core half (saorsa-core #163) has merged.

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Changes the migration signal's output, in tally_peers only:

  • A peer that disconnected mid-tally is no longer counted, and no peer_state line is written for it.
  • A peer that disconnected and came back between the two reads is classified from its agent read again, not from the stale first read.
  • A peer that is connected but still has no agent stays in the unreported bucket, so the count never drops a peer this node can see.

No wire, format or penalty change.

The third commit is test-only. It fixes migration_reclaims_disk, which fails intermittently on the btrfs CI job on main and on unrelated branches, and which was the one red check here.

Compatibility

  • Wire: none.
  • Storage: none.
  • API: none.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

Rebased onto main at 56b5770. Three commits: the original fix and the reconnect re-read in src/storage/migration_signal.rs (unchanged by the rebase), and a test-only fix in tests/migration_reclaims_disk.rs. No Cargo.toml or Cargo.lock change, and no other src/ change.

The peer-count fix:

  • What it fixes, from the 990-node testnet for refactor(storage)!: remove the LMDB chunk store, and never refuse a start over what it left behind #218: one tick in 21,402 had peers_unreported=1, a client/0.27.3 connection tallied with agent=none. The tally takes the peer list, then reads each agent, and saorsa-core drops the agent when the peer's last channel closes.
  • saorsa-core chore(release): promote rc-2026.6.4 #163 is in saorsa-core 0.28.0, which main already pins. It records and drops a peer's agent under the peer's connection entry, so a connected peer always has one: no agent means the peer was not connected at that read. The code does not depend on this. A connected peer with no agent still counts as unreported.
  • No unit test for the race itself: it needs a peer to disconnect, or disconnect and reconnect, inside the loop, which the store harness cannot stage.

The btrfs reclaim test (retiring_the_legacy_environment_returns_its_bytes_to_the_filesystem):

  • It failed in 8 of the last 56 CI runs, on main and on unrelated branches, always at the recovery check with about 26.3 MB recovered of a 54.1 MB environment. The space does come back. The free space btrfs reports lags its allocations until the transaction commits, and the per-file fsync does not commit it, so the peak read straight after the copy missed part of the file store. CI logs put that between 7 and 28 MB, and the test fails whenever it passes 27 MB. Every free-space reading on Linux now follows syncfs(2) of the volume. Making the readings prompt then exposed the end reading landing mid-delete, so the test also waits for the retirement tombstone to go before measuring.
  • Production code is unchanged. Measured on a Linux 6.17 VM, a fresh 3 GiB loop image per run made as CI makes it, binaries built with Rust 1.99:
    • Unchanged test, idle: 22 of 22 pass, each reading a peak cost of 88,555,520 bytes, as 20 of 32 passing CI runs did. Paired readings show the commit adds exactly 18,092,032 bytes to it.
    • Unchanged test with a disk writer and busy CPUs alongside: 2 of 6 fail at the same assertion as on CI, recovering 25,382,912 and 26,562,560 bytes of the 54,145,024-byte environment.
    • With the fix, btrfs: 86 of 86 pass (50 as CI runs them, 10 with the start delayed 0 to 27 s, 26 with readings logged, 6 of those under the same load), every one with peak cost 106,647,552, end cost 52,428,800 (the payload) and recovered 54,218,752 (the whole environment).
    • With the fix, ext4 and XFS: 13 of 13 each, with the same numbers as CI on ext4 and within 20,480 bytes of them on XFS.
    • Negative control, the fix with the environment's data file held open so its blocks stay allocated: fails on btrfs, ext4 and XFS ("recovered 4096 bytes"), so the commit cannot make space appear.

Gates, on Linux (VM, Rust 1.99, RUSTFLAGS=-D warnings) unless noted:

  • cargo clippy --all-targets --all-features --locked -- -D warnings and cargo fmt --all -- --check: clean on Linux, and on macOS (where the syncfs code is compiled out). RUSTDOCFLAGS=-D warnings cargo doc --all-features --no-deps (macOS): clean. cargo audit (macOS): clean; the lockfile is unchanged from main.
  • cargo test --lib --features test-utils: 1227 passed. migration_signal alone: 15 passed. cargo check --lib --no-default-features --locked: clean. cargo test --lib --no-default-features: 1182 passed.
  • migration_reclaims_disk 2, migration_crash_safety 5 (3 ignored, as on main) and migration_shared_volume 5, each passed on fresh btrfs, ext4 and XFS loop volumes made as the CI job makes them, and on the root filesystem.
  • storage_scale 4 (1 ignored), webrtc_direct_devnet 3 and, with test-utils, 6 (1 ignored), e2e 110 (3 ignored, as on main), poc_commitment_audit_attacks 19, poc_audit_handler_live 16, poc_bootstrap_stall 3, poc_shutdown_lmdb_drain 1. All passed, none failed.
  • One environment note, not from this branch: with TMPDIR on a tmpfs, storage::migration::tests::a_node_with_room_copies_everything_and_removes_the_legacy_store stays in Bridging until its deadline. It does the same on main at 56b5770, and passes in 3 s with TMPDIR on a disk, as on the CI runners. The lib counts above are with TMPDIR on a disk.

New dependency

none

ADR

https://github.com/WithAutonomi/ant-node/blob/main/docs/adr/ADR-0014-file-based-chunk-store-and-lmdb-retirement.md

Mitigation / rollback

Revert the three commits. The first two only change the migration signal's log output; the third only changes a test.

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — reviewed at exact head 674aa0d2eb9dcf794497a69ac6affc15accf47ff together with saorsa-core #163.

The tally fix is conservative in every relevant interleaving:

  • agent=None followed by disconnected means the peer left after the connected_peers snapshot, so skipping it removes the observed false Unreported tick.
  • If the peer reconnects between the two reads, it remains Unreported; that is a possible false outstanding count, not false clearance.
  • If an agent is read before a disconnect, the snapshot's last announced state is counted, matching the existing snapshot semantics.
  • A still-connected peer with no agent remains Unreported.

The change therefore cannot manufacture a Files result or falsely clear the migration gate. It is safe on its own and gains the stronger missing-agent meaning from saorsa-core #163.

Verification:

  • focused migration-signal suite: 15 passed
  • clippy all targets/features: passed
  • no-default-features library check: passed
  • formatting and rustdoc warnings check: passed
  • GitHub build/test/lint/filesystem/platform matrix is green

Non-blocking gap: no deterministic mid-tally disconnect/reconnect test was added. The branch logic is small and the paired transport invariant was checked directly, so I do not consider this a merge blocker.

The failing Security Audit is inherited, not introduced: Cargo.lock is byte-identical to the base and the newly published RUSTSEC-2026-0285 affects that existing rustls version. It should be handled by the release train in a separate dependency bump to rustls >=0.23.45.

Operationally, this is consistent with — and closes — the reported false-positive path in the #218 testnet. I have not independently reclassified the whole run from its Linear evidence here. The peer tally remains observational evidence rather than proof of whole-fleet completion, so the rest of the ADR-0015 gate still applies.

…ported

`tally_peers` lists the connected peers and then reads each one's user agent.
saorsa-core drops a peer's agent when its last channel closes, so a peer that
disconnects between the two reads has no agent, and the tally counted it as a
node running a build from before the migration signal. On a 990-node testnet
this put a departing client in `peers_unreported` on one tick in 21,402, which
is exactly the count the release gate for removing the LMDB store reads.

A peer with no agent that is no longer connected is now skipped: it is not a
peer this node can see. A peer that is still connected with no agent recorded
keeps its place in the unreported bucket, so the count stays conservative with
or without the saorsa-core change that records the agent under the peer's
connection entry.
saorsa-core records a peer's user agent and removes it under the same lock
as the peer's connection entry, so a connected peer always has an agent.
When `tally_peers` reads no agent and then finds the peer connected, the
peer disconnected and came back between the two reads. The tally counted it
as unreported from the first, stale read, even when the peer had announced
a finished store or was a client. That is the same false `unreported`
reading the previous commit removes for a departing peer.

The agent is now read a second time in that case and the peer is classified
from what it announced. If the second read still finds no agent, the peer
stays in the unreported bucket, so a connected peer is never dropped from
the count.
retiring_the_legacy_environment_returns_its_bytes_to_the_filesystem fails
intermittently on the btrfs CI job, on main and on unrelated branches,
always at the recovery check with about 26.3 MB recovered of a 54.1 MB
environment. The space does come back. The peak was read before btrfs had
settled its accounting, and the end could be read while the retired
environment was still being deleted.

The free space btrfs reports can lag what it has allocated and freed until
its transaction commits, every 30 seconds by default, and the fsync after
each chunk file does not commit it. Read straight after the copy, the peak
missed whatever part of the file store was not committed yet. Passing CI
runs read a peak cost of 79.5 to 99.6 MB, 7 to 27 MB short of the
106,647,552 bytes it settles to on a test machine, and the failing runs fit
a 28 MB shortfall: once it passes half the environment (27 MB), the
recovery measured from it comes up short however much space comes back.
Every free-space reading on Linux now follows a syncfs(2) of the volume,
which on btrfs commits the running transaction.

Reading promptly exposed the second problem. Retirement renames the
environment to a tombstone before a detached thread deletes it, so the old
name is gone before anything is deleted, and a reading taken while the
tombstone was still being deleted saw only part of the environment back:
enough to pass the recovery check, with the one-copy check passing by as
little as 0.6 MB. The test now also waits for the tombstone to go, for up
to 80 seconds, which is what the old wait and the space poll after it gave
the deletion between them.

Production code is unchanged. Measured on a fresh 3 GiB loop image per
run, made the way CI makes it:

- Unchanged test, btrfs, idle machine: 22 of 22 passed, every one reading
  a peak cost of 88,555,520 bytes, as 20 of 32 passing CI runs did. In
  paired runs the commit adds exactly 18,092,032 bytes to that peak cost,
  making it 106,647,552.
- Unchanged test, btrfs, with a disk writer and busy CPUs alongside it,
  standing in for a runner that has just finished a build: 2 of 6 failed
  at the same assertion as on CI, recovering 25,382,912 and 26,562,560
  bytes of the 54,145,024-byte environment. With the disk writer alone, 6
  of 6 passed.
- This change, btrfs: 86 of 86 passed (50 as CI runs them, 10 with the
  start delayed by 0 to 27 seconds, 26 with readings logged, 6 of those
  under the same load), each with the same numbers: peak cost 106,647,552,
  end cost 52,428,800 (the payload), recovered 54,218,752 (all of the
  environment). Under load, the reading without the commit was as much as
  29,360,128 bytes short at the peak in those same runs.
- This change, ext4 and XFS: 13 of 13 each, with the same numbers as CI on
  ext4 and within 20,480 bytes of them on XFS. The commit moves no reading
  on ext4 and one by 28,672 bytes on XFS.
- With the environment's data file held open across retirement, so its
  blocks stay allocated, this change still fails on btrfs, ext4 and XFS.
@grumbach
grumbach force-pushed the fix/peer-count-disconnect branch from 019ec6a to 8218e8d Compare October 2, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants