Skip to content

CI: Simplify SBML test suite sensitivity check harness - #3577

Draft
dweindl wants to merge 3 commits into
AMICI-dev:mainfrom
dweindl:sbml_suite_harness
Draft

dweindl wants to merge 3 commits into
AMICI-dev:mainfrom
dweindl:sbml_suite_harness

Conversation

@dweindl

@dweindl dweindl commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Simplify the SBML test suite sensitivity check harness.

Logging every sensitivity check attempt over the full suite (temporary instrumentation, CI runs 43764432333 and 43862917864) showed that most of the retry machinery is no longer used:

  • All checks pass at the first tolerance (1x, DAEs 100x), except the adjoint check of 00373 (simulation failure at 1x, all-inconclusive skip after the retry). The up to 6-step tolerance ladder is replaced by a single retry at 100x looser tolerances, for all models.
  • The fallback to looser quadrature tolerances never helped. Quadrature tolerances are now always tight.
  • The loosened FSA/ASA agreement tolerance after a retry was never needed (max. required rtol 2e-11). Always use rtol 1e-6.
  • The forward check no longer checks sigmay sensitivities, which are constant without an observation model. This removes 41% of the checked Jacobian entries for 01395, the slowest case (forward check 197 s -> 145 s). The remaining time is mostly fiddy's per-entry overhead, not integration.

Also: share the setup of the three sensitivity checks, clone the forward solver for the adjoint part of the consistency check, and remove dead code from jax_sensitivity_check (a tolerance case list not covering any checked case, an unreachable re-simulation).

Before/after on the full suite: identical outcome for each of the 2872 sensitivity checks (2821 passed, 51 all-inconclusive adjoint event skips).

For 01395, the lower-bound override now only covers the Hill coefficients with nominal value 1 (v1_h, v3_h, v15_h); the other 12 were never checked before and pass. Also, the states of 01395 are now kept non-negative: slightly negative species values due to integration errors made the non-integer Hill powers NaN, so that simulations failed at isolated parameter values (hit by more thorough finite-difference checks). The exact solution can't become negative within the check bounds.

_agrees_closely stays (it decides 01104, 01107, 01108). The 01148 tolerance special case in apply_settings also stays (the simulation check fails without it).

References #3262.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.99%. Comparing base (7fb8c57) to head (1e878ea).
⚠️ Report is 1 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (7fb8c57) and HEAD (1e878ea). Click for more details.

HEAD has 16 uploads less than BASE
Flag BASE (7fb8c57) HEAD (1e878ea)
petab_sciml_benchmarks 2 1
petab_sciml 2 1
petab 4 2
cpp_python 2 1
python 13 4
cpp 2 0
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             main    #3577       +/-   ##
===========================================
- Coverage   80.71%   49.99%   -30.73%     
===========================================
  Files         331      331               
  Lines       23688    23696        +8     
  Branches     1674     1674               
===========================================
- Hits        19120    11847     -7273     
- Misses       4562    11751     +7189     
- Partials        6       98       +92     
Flag Coverage Δ
cpp ?
cpp_python 34.86% <ø> (-0.02%) ⬇️
petab 47.17% <ø> (+0.03%) ⬆️
petab_sciml 16.50% <ø> (+<0.01%) ⬆️
petab_sciml_benchmarks 15.11% <ø> (+<0.01%) ⬆️
python 38.87% <ø> (-34.11%) ⬇️
sbmlsuite 49.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 219 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Logging every sensitivity check attempt over the full suite showed that
most of the retry machinery is no longer used:

* All checks pass at the first tolerance (1x, DAEs 100x), except the
  adjoint check of 00373 (simulation failure at 1x). Replace the up to
  6-step tolerance ladder by a single retry at 100x looser tolerances,
  for all models.
* The fallback to looser quadrature tolerances never helped. Always use
  tight quadrature tolerances.
* The loosened FSA/ASA agreement tolerance after a retry was never
  needed (max. required rtol 2e-11). Always use rtol 1e-6.
* Don't check `sigmay` sensitivities in the forward check: without an
  observation model, `sigmay` is constant. This removes 41% of the
  checked entries for 01395, the slowest case.

Also share the setup of the three sensitivity checks, clone the forward
solver for the adjoint part of the consistency check, and remove dead code
from `jax_sensitivity_check`.

References AMICI-dev#3262.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dweindl dweindl self-assigned this Nov 16, 2026
The bound override for case 01395 fixed the lower bound of all 15 Hill
coefficients at their nominal value, so their sensitivities were never
checked. Only `v1_h`, `v3_h`, and `v15_h` are 1 and lead to
`0**(negative)` when decreased; the others are 2 or 4. Restrict the
override to those three.

References AMICI-dev#3262.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Slightly negative species values due to integration errors make the
non-integer powers in the Hill terms NaN, so that the simulation fails at
isolated parameter values. Sensitivity checks that evaluate more points,
such as fiddy's per-direction noise-floor probes, hit them.

In the exact solution, no state can become negative within the check
bounds: initial values are non-negative, all outfluxes are proportional
to the respective species, and the influxes are non-negative.

References AMICI-dev#3262.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant