Skip to content

CI: Tighten tolerances in fiddy-based gradient checks - #3588

Draft
dweindl wants to merge 2 commits into
AMICI-dev:mainfrom
dweindl:ci_fiddy_check_tolerances
Draft

dweindl wants to merge 2 commits into
AMICI-dev:mainfrom
dweindl:ci_fiddy_check_tolerances

Conversation

@dweindl

@dweindl dweindl commented Nov 18, 2026

Copy link
Copy Markdown
Member

Tighten the solver tolerances in some gradient checks, so that they no longer rely on fiddy overestimating the solver noise.

ICB-DCM/fiddy#96 makes fiddy's noise-floor probe robust to nearby jumps. In several checks, the estimated noise floor, and with it the check tolerance, becomes smaller and no longer covers the actual solver error:

  • test_fiddy.py: at the default solver tolerances, the solver error of the finite differences exceeds the check tolerance. Now rtol 1e-12, atol 1e-15.
  • Adjoint gradients: the backward quadrature has its own relative tolerance (default 1e-8), which set_relative_tolerance does not change. This limits the accuracy of adjoint gradients to ~1e-8, fiddy's default check rtol (failures: test_presim_event_outputs on Windows, SalazarCavazos_MBoC2020 and Blasi_CellSystems2016 unscaled). The adjoint gradient checks now use a relative quadrature tolerance of 1e-10. In the benchmark gradient checks, this increases the number of backward steps by at most 1.3x; tying the quadrature tolerances to the state tolerances would increase it up to 7x.
  • Blasi_CellSystems2016: rtol 1e-13, as the finite differences at 1e-12 are not accurate enough.

Validated with ICB-DCM/fiddy#96 on a fork branch. Since AMICI installs fiddy from main, this should be merged before ICB-DCM/fiddy#96. The related change for SBML test case 01395 goes to #3577.

Relates to ICB-DCM/fiddy#93.

🤖 Generated with Claude Code

dweindl and others added 2 commits November 18, 2026 10:55
At the default tolerances, the finite differences' own solver error
exceeds the check tolerance. This went unnoticed because fiddy's noise
probe mistook the solver error for noise and widened the tolerance
(ICB-DCM/fiddy#93).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The adjoint gradient is a quadrature with a default relative tolerance
of 1e-8, independent of the state tolerances. This limited the accuracy
of the adjoint gradients in `test_presim_event_outputs` and the
benchmark gradient checks (SalazarCavazos_MBoC2020, Blasi_CellSystems2016)
to ~1e-8, the default `rtol` of the fiddy checks.

A relative quadrature tolerance of 1e-10 suffices; tying it to the state
tolerances (1e-12..1e-15) increased the number of backward steps up to
7-fold for some benchmark models.

Also simulate Blasi_CellSystems2016 at `rtol` 1e-13, as the finite
differences at 1e-12 were not accurate enough for the scaled gradient.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.72%. Comparing base (7cb3c09) to head (c782c57).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3588      +/-   ##
==========================================
- Coverage   80.75%   80.72%   -0.04%     
==========================================
  Files         331      331              
  Lines       23699    23699              
  Branches     1674     1675       +1     
==========================================
- Hits        19139    19131       -8     
- Misses       4554     4562       +8     
  Partials        6        6              
Flag Coverage Δ
cpp 75.28% <ø> (-0.05%) ⬇️
cpp_python 34.86% <ø> (ø)
petab 47.17% <ø> (ø)
petab_sciml 16.50% <ø> (ø)
petab_sciml_benchmarks 15.11% <ø> (ø)
python 72.97% <ø> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant