Conversation
dirvine
left a comment
There was a problem hiding this comment.
APPROVE — reviewed at exact head 674aa0d2eb9dcf794497a69ac6affc15accf47ff together with saorsa-core #163.
The tally fix is conservative in every relevant interleaving:
agent=Nonefollowed by disconnected means the peer left after theconnected_peerssnapshot, so skipping it removes the observed falseUnreportedtick.- If the peer reconnects between the two reads, it remains
Unreported; that is a possible false outstanding count, not false clearance. - If an agent is read before a disconnect, the snapshot's last announced state is counted, matching the existing snapshot semantics.
- A still-connected peer with no agent remains
Unreported.
The change therefore cannot manufacture a Files result or falsely clear the migration gate. It is safe on its own and gains the stronger missing-agent meaning from saorsa-core #163.
Verification:
- focused migration-signal suite: 15 passed
- clippy all targets/features: passed
- no-default-features library check: passed
- formatting and rustdoc warnings check: passed
- GitHub build/test/lint/filesystem/platform matrix is green
Non-blocking gap: no deterministic mid-tally disconnect/reconnect test was added. The branch logic is small and the paired transport invariant was checked directly, so I do not consider this a merge blocker.
The failing Security Audit is inherited, not introduced: Cargo.lock is byte-identical to the base and the newly published RUSTSEC-2026-0285 affects that existing rustls version. It should be handled by the release train in a separate dependency bump to rustls >=0.23.45.
Operationally, this is consistent with — and closes — the reported false-positive path in the #218 testnet. I have not independently reclassified the whole run from its Linear evidence here. The peer tally remains observational evidence rather than proof of whole-fleet completion, so the rest of the ADR-0015 gate still applies.
674aa0d to
019ec6a
Compare
…ported `tally_peers` lists the connected peers and then reads each one's user agent. saorsa-core drops a peer's agent when its last channel closes, so a peer that disconnects between the two reads has no agent, and the tally counted it as a node running a build from before the migration signal. On a 990-node testnet this put a departing client in `peers_unreported` on one tick in 21,402, which is exactly the count the release gate for removing the LMDB store reads. A peer with no agent that is no longer connected is now skipped: it is not a peer this node can see. A peer that is still connected with no agent recorded keeps its place in the unreported bucket, so the count stays conservative with or without the saorsa-core change that records the agent under the peer's connection entry.
saorsa-core records a peer's user agent and removes it under the same lock as the peer's connection entry, so a connected peer always has an agent. When `tally_peers` reads no agent and then finds the peer connected, the peer disconnected and came back between the two reads. The tally counted it as unreported from the first, stale read, even when the peer had announced a finished store or was a client. That is the same false `unreported` reading the previous commit removes for a departing peer. The agent is now read a second time in that case and the peer is classified from what it announced. If the second read still finds no agent, the peer stays in the unreported bucket, so a connected peer is never dropped from the count.
retiring_the_legacy_environment_returns_its_bytes_to_the_filesystem fails intermittently on the btrfs CI job, on main and on unrelated branches, always at the recovery check with about 26.3 MB recovered of a 54.1 MB environment. The space does come back. The peak was read before btrfs had settled its accounting, and the end could be read while the retired environment was still being deleted. The free space btrfs reports can lag what it has allocated and freed until its transaction commits, every 30 seconds by default, and the fsync after each chunk file does not commit it. Read straight after the copy, the peak missed whatever part of the file store was not committed yet. Passing CI runs read a peak cost of 79.5 to 99.6 MB, 7 to 27 MB short of the 106,647,552 bytes it settles to on a test machine, and the failing runs fit a 28 MB shortfall: once it passes half the environment (27 MB), the recovery measured from it comes up short however much space comes back. Every free-space reading on Linux now follows a syncfs(2) of the volume, which on btrfs commits the running transaction. Reading promptly exposed the second problem. Retirement renames the environment to a tombstone before a detached thread deletes it, so the old name is gone before anything is deleted, and a reading taken while the tombstone was still being deleted saw only part of the environment back: enough to pass the recovery check, with the one-copy check passing by as little as 0.6 MB. The test now also waits for the tombstone to go, for up to 80 seconds, which is what the old wait and the space poll after it gave the deletion between them. Production code is unchanged. Measured on a fresh 3 GiB loop image per run, made the way CI makes it: - Unchanged test, btrfs, idle machine: 22 of 22 passed, every one reading a peak cost of 88,555,520 bytes, as 20 of 32 passing CI runs did. In paired runs the commit adds exactly 18,092,032 bytes to that peak cost, making it 106,647,552. - Unchanged test, btrfs, with a disk writer and busy CPUs alongside it, standing in for a runner that has just finished a build: 2 of 6 failed at the same assertion as on CI, recovering 25,382,912 and 26,562,560 bytes of the 54,145,024-byte environment. With the disk writer alone, 6 of 6 passed. - This change, btrfs: 86 of 86 passed (50 as CI runs them, 10 with the start delayed by 0 to 27 seconds, 26 with readings logged, 6 of those under the same load), each with the same numbers: peak cost 106,647,552, end cost 52,428,800 (the payload), recovered 54,218,752 (all of the environment). Under load, the reading without the commit was as much as 29,360,128 bytes short at the peak in those same runs. - This change, ext4 and XFS: 13 of 13 each, with the same numbers as CI on ext4 and within 20,480 bytes of them on XFS. The commit moves no reading on ext4 and one by 28,672 bytes on XFS. - With the environment's data file held open across retirement, so its blocks stay allocated, this change still fails on btrfs, ext4 and XFS.
019ec6a to
8218e8d
Compare
Linear issue
Closes V2-1290
Split from V2-1260, whose saorsa-core half (saorsa-core #163) has merged.
Risk tier
Changes the migration signal's output, in
tally_peersonly:peer_stateline is written for it.No wire, format or penalty change.
The third commit is test-only. It fixes
migration_reclaims_disk, which fails intermittently on the btrfs CI job on main and on unrelated branches, and which was the one red check here.Compatibility
Semver impact
Test evidence
Rebased onto main at
56b5770. Three commits: the original fix and the reconnect re-read insrc/storage/migration_signal.rs(unchanged by the rebase), and a test-only fix intests/migration_reclaims_disk.rs. NoCargo.tomlorCargo.lockchange, and no othersrc/change.The peer-count fix:
peers_unreported=1, aclient/0.27.3connection tallied withagent=none. The tally takes the peer list, then reads each agent, and saorsa-core drops the agent when the peer's last channel closes.The btrfs reclaim test (
retiring_the_legacy_environment_returns_its_bytes_to_the_filesystem):syncfs(2)of the volume. Making the readings prompt then exposed the end reading landing mid-delete, so the test also waits for the retirement tombstone to go before measuring.Gates, on Linux (VM, Rust 1.99,
RUSTFLAGS=-D warnings) unless noted:cargo clippy --all-targets --all-features --locked -- -D warningsandcargo fmt --all -- --check: clean on Linux, and on macOS (where thesyncfscode is compiled out).RUSTDOCFLAGS=-D warnings cargo doc --all-features --no-deps(macOS): clean.cargo audit(macOS): clean; the lockfile is unchanged from main.cargo test --lib --features test-utils: 1227 passed.migration_signalalone: 15 passed.cargo check --lib --no-default-features --locked: clean.cargo test --lib --no-default-features: 1182 passed.migration_reclaims_disk2,migration_crash_safety5 (3 ignored, as on main) andmigration_shared_volume5, each passed on fresh btrfs, ext4 and XFS loop volumes made as the CI job makes them, and on the root filesystem.storage_scale4 (1 ignored),webrtc_direct_devnet3 and, withtest-utils, 6 (1 ignored),e2e110 (3 ignored, as on main),poc_commitment_audit_attacks19,poc_audit_handler_live16,poc_bootstrap_stall3,poc_shutdown_lmdb_drain1. All passed, none failed.TMPDIRon a tmpfs,storage::migration::tests::a_node_with_room_copies_everything_and_removes_the_legacy_storestays in Bridging until its deadline. It does the same on main at56b5770, and passes in 3 s withTMPDIRon a disk, as on the CI runners. The lib counts above are withTMPDIRon a disk.New dependency
none
ADR
https://github.com/WithAutonomi/ant-node/blob/main/docs/adr/ADR-0014-file-based-chunk-store-and-lmdb-retirement.md
Mitigation / rollback
Revert the three commits. The first two only change the migration signal's log output; the third only changes a test.