Repository navigation
Fix/scbutterfly - #81
Merged
Merged
Conversation
Both 2022 competition solutions predict a z-scored profile per cell (the competition scored per-cell Pearson only). Mapping that back with one global slope and intercept gave every predicted cell the same mean and standard deviation, so per-gene correlations, overall correlations and RMSE/MAE scored the missing per-cell level rather than the methods. src/utils/cell_scale.py predicts each cell's target mean and log standard deviation from its input (ridge regression on a truncated SVD of the normalized input plus two sequencing-depth features) and restores them, with the spread shrunk by one least-squares factor fitted on training-cell predictions (out-of-fold for senkin_tmp). Per-cell correlations are unchanged. ss_opm_predict keeps the old global map for model bundles trained before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on the scored scale - chrom_utils.parse_chrom: split peak names on the first ':' or '-'. NeurIPS 2022 names peaks chr1:start-end, so every peak became its own chromosome with its own encoder block: pbmc_multiome was OOM-killed (ATAC -> GEX, 174,978 "chromosomes") or hit the 8 h limit (GEX -> ATAC, 7,002 "chromosomes") on rc7. - Keep scButterfly's HVG selection and peak filter for the input modality only, so every target gene/peak is modelled instead of zero-filled. - target_scale.py: convert GEX predictions exactly from scButterfly's median-library-size log1p to log CP10k; map ATAC predictions onto the dataset's TF-IDF with one least-squares line per peak (slope >= 0), fitted on the held-out validation cells. - Patch Model.__init__ to serve cells as dense rows of the sparse matrix instead of densifying both modalities (92 GB for the 2022 ATAC input). - Model bundles without metadata["model_all_target_features"] keep the old preprocessing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll in DataLoader workers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e labels - target_scale.ScaleCalibration: map both GEX and ATAC predictions with one least-squares line fitted on the held-out validation cells. On the 2021 and 2022 Multiome datasets it matched or beat per-feature lines and, for GEX, the exact log CP10k conversion (which inflates the decoder's conditional means for sparse genes: RMSE 0.553 vs 0.518 on bmmc_multiome); it leaves every correlation as is. - predict_cells: give the direction Model.test does not need a single cell. - read_modality drops unused layers/obsm; _to_counts_X takes the counts layer out of its input; build_butterfly frees intermediate copies. - train: veryhighmem + veryhightime; predict: veryhighmem. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lls show no positive association Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace scbutterfly's own scale map (target_scale.py) with CellScale, which ss_opm and senkin_tmp already use: scbutterfly_train fits its ridge model on the training cells and its spread shrinkage on the held-out validation cells (predicted with the saved weights) and saves cell_scale.npz; scbutterfly_predict applies it. Bundles without the file keep scButterfly's own output. Also reuse library functions in chrom_utils: np.argsort/np.unique for the chromosome order and counts, itertools.groupby for run lengths, pd.Index.get_indexer to scatter predictions by name; and share the held-out split through validation_cells(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rcannood
reviewed
Oct 7, 2026
| - type: nextflow | ||
| directives: | ||
| label: [highmem, hightime, midcpu, highsharedmem, gpu] | ||
| label: [veryhighmem, hightime, midcpu, highsharedmem, gpu] |
Member
There was a problem hiding this comment.
veryhighmem likely won't affect the execution of this component, because the gpu nodes are constrained in memory. you could switch to a different node using biggpu.
Contributor
Author
There was a problem hiding this comment.
Thanks! I'll change it to biggpu. It might actually not be needed after the optimizations, but let's go big to be on the safe side and then roll back if we can
VladimirShitov
commented
Oct 8, 2026
3 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Fix incorrect and suboptimal behavior of scbutterfluy:
common_butterfly.pyto make scbutterfly calculation feasible on 2022 datasetChecklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!