Repository navigation
Land the diaPASEF speed/memory stack (#196, #198-#206) on main - #209
Merged
Merged
Conversation
`rt_im_train.multihead_anchors = auto` now also refits the multi-head calibration on search anchors on every adapting run whose spectra carry 1/K0, whatever its seed anchor count. On diaPASEF the gain does not fall with the count the way it does on Orbitrap data: - ProteoBench HYE diaPASEF, six 60 min runs, 11,144 seed anchors: +5.7% peptides (93,061 -> 98,373), +6.5% ions at k = 3, w_rt 40-45 s -> 24-27 s (one seed); - E. coli 15 min, 4,639 seed anchors: +0.6% over three seeds, every seed up, entrapment FDP 0.42-0.51% -> 0.39-0.46%. `active`, `decide` and `record_auto` take `has_im` (from the input path in `run`, from the converted spectra in `run-experiment`); the INFO line names the reason and `anchors_auto` records `ion_mobility`. 3D runs are unchanged. docs/08 section 4f, docs/35 (arm T2 measured, ProteoBench HYE diaPASEF table, note that the six-run 5 min tables ran without multi-head calibration), config reference and CHANGELOG updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- extract.windows_in_flight defaults to 4 on spectra that carry 1/K0 (16 otherwise). One HYE diaPASEF run: 45.1 GiB at 16 in flight, 21.4 at 4, identical output; six-run experiment 181 -> 94 GiB peak at bit-identical identifications. - retrace sums each MS1 frame of a row once (Ms1Seen) instead of once per grid point that reaches it; values bit-identical. - retrace reader budget: four concurrent calls get eight readers each, instead of the first call taking the whole process-wide budget (the slowest of five concurrent retraces read on one thread for 390 s). - skeleton centroid table: extract writes the rows retrace rebuilds as zeros on their axis, with the same rule retrace classifies by (retrace::row_kind, Skeleton::zeroes), recorded in the table's report; retrace refuses a skeleton it would not rebuild in full and publishes nothing if a skeleton fragment row would be copied. - docs/35 section 16 (memory and speed), config reference regenerated, CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
convert.tdf_ms1_denoise (default false): before centroiding, keep only MS1 points on runs of at least 5 occupied mobility scans (TOF neighbourhood +/- 3, gaps of up to 2 scans bridged, two passes), drop features no diaPASEF window reaches (windows padded by 3 Th and 0.015 1/K0, kept whole on any overlap), and drop the weak m/z halo of bright ions: the method and default MS1 settings of dnoise (Garrett, Diedrich and Yates, bioRxiv 2026, doi:10.64898/2026.08.27.747603, MIT licence), implemented on convert's raw points. Retrace applies the same filter to the MS1 frames it holds, and decodes no MS1 frame under retrace.ms1 = false. MS/MS is never filtered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retrace decoded every frame of the run once and held them all (36-39 GB per HYE diaPASEF run before the MS1 filter). The MS1 frames are still decoded once and held, since every MS1 trace reads them. The MS2 frames are now held window by window: a span's candidates are consecutive in precursor m/z, so before each span retrace decodes the frames whose slots isolate a window reaching that m/z stretch and drops the others (hold_window_frames), then re-picks that contiguous range of candidates on them (repick_rows, with the sidecar and fragment rows loaded once in RepickData). Every candidate is re-picked once, the candidates after the last span for the psms patch. Per-candidate results do not depend on which frames of other windows are held, so every output is unchanged; the report adds ms_window_frames and counts every frame decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re a factor Folds were base_peptide_id % folds, in the engine's classifiers and in the nn_torch worker. Every base_peptide_id of the imported HYE library is even, so two folds put every PSM into fold 0 (the worker stopped with "fold 0 is empty in training or holdout") and four folds used only two. The fold is now key % folds while that gives every fold at least half its share, and otherwise the key's splitmix64 mix modulo folds (rescoring::fold_assignment, nn_rescore_worker.fold_keys_for), with a warning. A key's PSMs, a target and its paired decoy, stay in one fold, and every library the plain modulo already spread keeps its folds and its scores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under parallel_runs = auto with rt_library_scope = first_run_only, the first run's whole chain ran alone so that its peak could size the chains after it. When that run adapts the library with a bootstrap search (rt_im_train.multihead_anchors = search, or auto on 1/K0 spectra), the adaptation already searches the run in this process: extract, features and compete on the seed-anchored library, all of the finish but retrace. Its peak now sizes the chains, and the first run's finish (rt-im-train to compete on the adapted library) is the first item of the parallel queue instead of a step of its own. On the six HYE diaPASEF runs the finish took about 4 min alone while one of the six per-run pools sat idle in the parallel phase after it. The outputs do not change: each stage reads only its own run's inputs, and the first run's finish now runs in a pool of the chain width, as the other runs' do. An adaptation without an in-process search (seed anchors, fine-tune) keeps the previous order, as do an explicit parallel_runs and a grouped run. adapt_rt_library returns an Adapted struct that records whether it searched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_denoise Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s by the chains in flight The features main pass and retrace each take their extra decode threads from a process-wide pool (DecoderLease), sized for a few concurrent calls: four extra feature loaders, and 7 x 4 retrace readers. Under parallel_runs = auto the per-run chains run in pools of their own, so on the six HYE diaPASEF runs the first two feature passes took their two extra loaders each and the other three decoded on one loader: 280-295 s against 95-104 s, and those three passes were the last four minutes of the parallel phase. The chains of an automatic plan are sized on the measured peak of a chain or a bootstrap search of the first run that ran alone with all of its loaders, so while they run both pools now grow to one call's share per chain (sched::BudgetGrowth, features::grow_loaders_for_chains, retrace::grow_readers_for_chains), and shrink back when the phase returns. An explicit parallel_runs, grouped bands and single runs are unchanged. The loaders and readers only decode, and the tables keep their order, so no row changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s the current one A main-pass loader decoded a batch of its chunk (on the codec pool, one job per column group) and then copied it into the chunk, one after the other. gdb samples of a standalone features pass on a HYE diaPASEF run put the loaders at 73% waiting for their own batch to decode and 27% copying, with the codec pool 64% idle: the pass was bound by the loaders, and each loader by that serial alternation. Each loader's chunk stream now takes its batches from a thread of its own that decodes one batch ahead (Batches::prefetch, a channel of one batch), so the decode of batch i + 1 overlaps the copy of batch i. The batches arrive in order, so every chunk, feature value and table byte is unchanged (a_chunk_decoded_ahead_is_the_chunk_decoded_in_line compares every value bit of every chunk of the kernel fixture, with and without 1/K0). Under --threads 1 the stream decodes in line as before, adding no thread the budget did not grant. The other readers of the chromatogram table (the confident-bounds pass, open_at) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a chunk's features were computed in parallel, the pass wrote them into the value columns one row at a time on one thread: per row a peptide length parse, a protein count, two seed-map lookups, the isotope features and one scattered write into each of ~435 columns. On a HYE diaPASEF run that assembly took 20-31 s of one core per pass (assemble_ms) while the pass's pool waited. The rows are independent, so the chunk is now assembled in blocks of 2,048 rows in parallel, each block writing its own rows of every column through disjoint slices. Every value is computed as before and lands in the same place, so the features table is unchanged (the feature-digest and chunk invariance tests pass). ValueMatrix::set remains for the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…copied The worker memory-maps the raw handoff matrix and copies it into its own float32 matrix. Every mapped page it read stayed in its resident set until the mapping was dropped after the copy, so for a moment both matrices were resident: on the six-run HYE diaPASEF pool (28.5 M PSMs x 424 features, a 48 GB handoff) the experiment's resident peak was that copy, 95 GB, against 51 GB for the training that follows. With the retrace frames held per window group it is the experiment's highest point. fill_raw_matrix now drops each 131,072-row group of the mapping from the resident set once the group is copied (release_rows, MADV_DONTNEED). The pages stay in the page cache, so a read of a released row only faults it back in with the same bytes. Measured on Linux with a 320 MB matrix on disk: RssFile 371 MB after reading it, 58 MB after the release, and a fill that leaves RssFile at 58 MB; every copied value unchanged. A no-op on Windows (no madvise) and for an array that is not memory-mapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extract writes three MS1 isotope traces (mono, +1, +2) per candidate, each read on the candidate's grid as the maximum over the nearest MS1 scan and its ms1_scan_halfwidth neighbours. When retrace follows, the skeleton then zeroed every one of those rows, because retrace rebuilds them from the raw events. gdb samples of a standalone extract of a HYE diaPASEF run put about half of extract's busy time in those MS1 reads (ms1_near). The skeleton decision reads only the row's identity (Skeleton::zeroes, the rule retrace classifies by), so it is now taken first, and a row it zeroes is written as zeros without being read. The chromatogram table is the one the reading-then-zeroing wrote, and the apex MS1 reads that feed the features are unchanged. Without retrace, or under retrace.ms1 = false, nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…memory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or reads no environment The config reference scans `let mut name = |param: ...|` closures as possible environment readers; the parallel assembly's setter matched and added 51 false entries. It is now bound as `let put = &mut |...|`, the same closure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w_kind Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s1Denoise Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(extract): a diaPASEF extraction gate (extract.gate_support_min, opt-in)
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PRs #196 and #198-#206 were each merged into the branch below them, so the stack collapsed into
tims/7-docs-bench. That branch had already been merged into main through #194, so none of tims/8 to tims/17 reached main. This PR carriestims/7-docs-bench(at 177e663, the merge of #206) onto main, with main merged in (zip9 from #208, rustls from #197). The merge was clean.Contents, as reviewed in the individual PRs:
.dinput (convert.tdf_ms1_denoise, default on)key % foldsempties a foldextract.gate_support_min, the diaPASEF gate (opt-in)Measured on the six-run HYE diaPASEF benchmark: 41.6 min against 49.9 min with identical identifications (stack arm, tims/13-16).
No new code relative to the merged PRs; CI here is the first run of the whole stack against current main.
🤖 Generated with Claude Code