Repository navigation
perf(nn-worker): release the raw handoff's mapped rows once copied - #204
Merged
RobbinBouwmeester merged 8 commits intoOct 17, 2026
Merged
Conversation
…copied The worker memory-maps the raw handoff matrix and copies it into its own float32 matrix. Every mapped page it read stayed in its resident set until the mapping was dropped after the copy, so for a moment both matrices were resident: on the six-run HYE diaPASEF pool (28.5 M PSMs x 424 features, a 48 GB handoff) the experiment's resident peak was that copy, 95 GB, against 51 GB for the training that follows. With the retrace frames held per window group it is the experiment's highest point. fill_raw_matrix now drops each 131,072-row group of the mapping from the resident set once the group is copied (release_rows, MADV_DONTNEED). The pages stay in the page cache, so a read of a released row only faults it back in with the same bytes. Measured on Linux with a 320 MB matrix on disk: RssFile 371 MB after reading it, 58 MB after the release, and a fill that leaves RssFile at 58 MB; every copied value unchanged. A no-op on Windows (no madvise) and for an array that is not memory-mapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of a stack on #196; merge in order: 9 -> 10 -> 11 -> 12 -> 13 -> 14 -> 15 -> 16 -> 17. Measured on the six ProteoBench HYE diaPASEF runs (60 min, timsTOF SCP) on a 128-core host at 120 threads, NN seed 0,
MUMDIA_CACHE_DIR=off; docs/35_diapasef.md section 16 has the details.Summary
The NN worker copies the memory-mapped raw handoff into its own matrix, and the mapped pages it had read stayed resident until the copy ended. It now drops each copied 131,072-row group (
MADV_DONTNEED; a no-op on Windows). Pooled rescore of the six runs: 98,524 peptides either way, maximum RSS 98.3 -> 51.0 GB. Checked on Linux with a file-backed matrix (RssFile 371 -> 58 MB, values unchanged) and a new worker test.Checks
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo test --workspace,ci/gen_config_reference.py --check,ci/check_doc_refs.py,python -m compileall scripts.🤖 Generated with Claude Code