Restore the per-cell scale of senkin_tmp and ss_opm predictions

[VladimirShitov]

Describe your changes

Problem

senkin_tmp and ss_opm are the two 2022 Kaggle solutions, and that competition scored only the mean per-cell Pearson correlation. Both predict a z-scored profile per cell, and the wrappers mapped it back to the target scale with one global slope and intercept. Every predicted cell therefore has the same mean and standard deviation: 0.5955 and ~0.45 on bmmc_cite, where the true per-cell mean ranges from 0.16 to 2.83.

Per-cell correlations are unaffected. Per-gene correlations, overall correlations and RMSE/MAE, however, scored that missing information rather than the methods. On the rc7 leaderboard both methods rank 2nd–3rd on per-cell Pearson for GEX → ADT and 10th–12th on per-protein Pearson.

Change

• src/utils/cell_scale.py (new):
  • Predicts each cell's target mean and log standard deviation from its input profile: a ridge regression (RidgeCV) on a 64-component truncated SVD of the normalized input plus log total counts and log detected features.
  • Gives every z-scored prediction that mean back, together with its standard deviation times one shrinkage factor. Where the predicted profile correlates only weakly with the truth, the spread that minimises squared error is smaller than the cell's standard deviation, e.g. on GEX → ATAC with per-cell r ≈ 0.19.
  • The factor is fitted by least squares on training-cell predictions: the out-of-fold predictions for senkin_tmp, and the same 5,000-cell subsample the old global map used for ss_opm.
  • The fitted model is saved with numpy.savez, not a pickle, so it loads safely in the predict image.
  • Per-cell correlations are unchanged by construction.
• senkin_tmp_train: replaces the global affine map with CellScale.
• ss_opm_train / ss_opm_predict: train fits and saves cell_scale.npz; predict applies it. Model bundles without that file, such as the S3 test resources, still get the old global map.

Validation

1. Offline comparison of estimators. On saved predictions, the per-cell z-scores were rescaled with each candidate estimate and scored against the test solution. Ridge on SVD plus depth features beat kNN and LightGBM on correlations. Restoring the full per-cell standard deviation raised RMSE wherever the per-cell correlation is low (GEX → ATAC: 0.51 → 0.65), hence the shrinkage factor. With it, every correlation and error improved or held, apart from three small losses on the multiome targets: ATAC → GEX 2021 MAE (+0.005 end to end), and GEX → ATAC 2022 per-gene Pearson (−0.002) and RMSE (+0.0005).

2. End-to-end runs of the changed components on the full datasets, against the runs of the current code on the same data, scored like the benchmark's metric components:

method	dataset	per-cell Pearson	per-gene Pearson	per-gene Spearman	RMSE	MAE
senkin_tmp	2021 GEX → ADT	0.821 → 0.821	0.486 → 0.671	0.317 → 0.555	0.779 → 0.659	0.574 → 0.487
ss_opm	2021 GEX → ADT	0.821 → 0.821	0.505 → 0.691	0.353 → 0.583	0.786 → 0.660	0.577 → 0.484
ss_opm	2022 GEX → ADT	0.619 → 0.619	0.320 → 0.468	0.276 → 0.396	0.406 → 0.387	0.293 → 0.292
ss_opm	2021 ADT → GEX	0.740 → 0.740	0.153 → 0.179	0.123 → 0.177	0.351 → 0.350	0.171 → 0.170
ss_opm	2022 ADT → GEX	0.867 → 0.867	0.137 → 0.142	0.119 → 0.136	0.206 → 0.206	0.106 → 0.105
ss_opm	2021 GEX → ATAC	0.190 → 0.190	0.098 → 0.102	0.079 → 0.092	0.509 → 0.510	0.138 → 0.135
ss_opm	2022 GEX → ATAC	0.190 → 0.190	0.060 → 0.058	0.057 → 0.059	0.349 → 0.350	0.091 → 0.082
ss_opm	2021 ATAC → GEX	0.457 → 0.457	0.128 → 0.147	0.101 → 0.134	0.523 → 0.523	0.250 → 0.254
ss_opm	2022 ATAC → GEX	0.631 → 0.631	0.126 → 0.135	0.109 → 0.127	0.357 → 0.356	0.179 → 0.172

senkin_tmp on 2022 GEX → ADT is still running (Slurm job 40930275); code/cell_scale_check/compare_scores.py adds it when it finishes.

viash test (native engine) passes for ss_opm_train, ss_opm_predict (old bundle → fallback path) and senkin_tmp_train. ss_opm_predict was also run on a bundle with cell_scale.npz produced by the train test.

Checklist before requesting a review

• I have performed a self-review of my code

• Check the correct box. Does this PR contain:

  • Breaking changes
  • New functionality
  • Major changes
  • Minor changes
  • Bug fixes

• Proposed changes are described in the CHANGELOG.md

• CI Tests succeed and look good!