Repository navigation
Improve the function evaluations at the start params - #727
Conversation
optimagic only caught exceptions in the first evaluation of the objective. With an inf start value, scipy_lbfgsb reported success at a wrong point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The message for a non-finite objective now contains the complete output of the objective function instead of positions in the flattened value, which are not meaningful for pytrees. User provided derivatives (jac and fun_and_jac) are now also checked for finiteness at the start params. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
🚀 New features to boost your workflow:
|
Check finiteness of the objective on its scalar aggregate instead of selecting the aggregation level by function type. Check user provided derivatives only for algorithms that need them, since others never use the derivative. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re used Algorithms that do not need derivatives never call jac or fun_and_jac, so the evaluation at the start params was wasted. Collect the derivative evaluations in one dict that feeds both the converter and the finiteness check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The condition `origin == "jac" or "fun_and_jac"` was always true, so a non-finite numerical derivative was reported as "the derivative provided via numerical". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@hmgaudecker and @timmens, this PR introduces a few behavior changes. What is your opinion? Now would be a good time as the next release is probably 0.6.0. |
There was a problem hiding this comment.
Thanks! This closes a real gap. Starting from a non-finite point is not something any optimizer recovers from, and the origin == "jac" or "fun_and_jac" fix is a nice catch. A few points, the first one I'd like addressed before merging.
-
Size of the error message:
_format_in_fullprints the whole output without a limit. For likelihood problems, the objective output is one value per observation.estimagic.estimate_ml, for example, passesloglikeobs. The user-provided derivative has one entry per parameter. In our application (skillmodels, health-cognition), that is one value per person and thus a gradient with some 14k entries. A single non-finite contribution produces a very large exception message. It ends up in pytask's captured traceback and in the SLURM log, where nobody reads the relevant line.Suggestion: report the paths of the non-finite leaves. Positions in the flat array are not meaningful, but tree paths are. Report the number of non-finite entries too, and then either the full repr below a size threshold or a truncated one above it. The pytree test cases would still pass with paths:
'b',c. -
Derivatives of fixed parameters: The derivative is checked after
converter.derivative_to_internal. So non-finite entries for parameters that are fixed by constraints are ignored, and with transforming constraints they spread to the internal entries they enter. I think this is the right behaviour, and it matches the check during the optimization. But it is a user-visible rule, so it deserves:- a sentence in the docstring
- a test:
nanin the derivative of a fixed parameter does not raise.
-
Question regarding multistart and global optimizers: Raising for these is a deliberate choice according to the description. Should the message say that the start params must be valid even with multistart? A user there may think of them as one sample among many. Optional.
-
Double evaluation of
fun_and_jac: If onlyfun_and_jacis given,funis derived from it viasplit_fun_and_jac. Sofun_and_jacruns once increate_optimization_problemand again here. This is unchanged by the PR, but since the block is rewritten anyway, the derivative from the first call could be kept. For expensive objectives that saves a full evaluation at the start. -
Minor stuff:
- Tests: a
maximizecase would show that the check uses the internal (sign-flipped) value correctly. AUserFunctionRuntimeError→InvalidFunctionErrorcase for a user who catches the old exception would document behaviour change 3. - CHANGES.md: the entry mentions only the error. The behaviour change for derivative-free algorithms (no evaluation of
jac/fun_and_jacat the start params) seems worth its own line.
- Tests: a
Downstream check. We pinned the PR head (c843a18) in health-cognition and ran our pipeline's debug estimation (skillmodels CHS, fides, fun_and_jac plus a likelihood-type fun, one A100) end to end. The run covered AMN start values, two fides iterations, inference and our convergence check. All of it passed, with no new warnings or errors.
Co-authored-by: Hans-Martin von Gaudecker <hmgaudecker@gmail.com>
…lues # Conflicts: # CHANGES.md
- List the paths of non-finite entries in the error messages and abbreviate large outputs instead of printing them in full. - Say in the error message and the docs that the start params have to be valid for global optimizers and multistart, too. - Evaluate fun_and_jac only once at the start params if fun is derived from it. - Add tests for maximize and for the changed exception type. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nd_jac If only fun_and_jac was provided or jac=True, keyword arguments had to be passed via fun_kwargs and fun_and_jac_kwargs. Providing just one of them failed, or silently used default values for the other function. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Thanks for the review, @hmgaudecker. All points are addressed:
|
optimagic evaluates the objective and user provided derivatives at the start params before it optimizes. This PR improves these evaluations in two ways.
Check that the results are finite. So far, only exceptions were caught. If the objective returns
infornanat the start params, the optimizer starts from an invalid point. For example, with aninfstart value,scipy_lbfgsbreports success at a wrong point andscipy_neldermeadsilently moves away. The default penalty used witherror_handling="continue"is also derived from the start value. This PR raises anInvalidFunctionErrorif the objective, or a derivative provided viajacorfun_and_jac, is not finite at the start params. The message lists the paths of the non-finite entries and abbreviates large outputs.Only evaluate derivatives that are used. User provided derivatives were evaluated at the start params even for algorithms that do not use derivatives. They are now only evaluated and checked if the algorithm needs them (
algo_info.needs_jac).Evaluate
fun_and_jaconly once. Iffunis derived fromfun_and_jac, the derivative from the first evaluation is reused instead of evaluatingfun_and_jaca second time.Share kwargs between
funandfun_and_jac. Iffunis derived fromfun_and_jac,fun_kwargsandfun_and_jac_kwargsare used for both if only one of them is given.Numerical derivatives are not evaluated at the start params, because that would cost an extra derivative evaluation. A non-finite numerical derivative still raises at the algorithm's first derivative evaluation, as before.
Behavior changes
inf,-infornan,minimizeandmaximizeraise anInvalidFunctionErrorbefore the optimization starts. This applies to all algorithms, including global optimizers and multistart, where the start params matter less. Before, the optimization ran from the invalid point.infalso raise.jacorfun_and_jacat the start params raises anInvalidFunctionErrorbefore the optimization starts. Before, aUserFunctionRuntimeErrorwas raised at the algorithm's first derivative evaluation. Code that catchesUserFunctionRuntimeErrorno longer catches this case.needs_jac=False,jacis no longer called at the start params, and neither isfun_and_jacunless it is also used to compute the objective. A derivative function that raises, returns non-finite values, or returns output that cannot be aligned with the params and the objective output no longer causes an error with these algorithms.🤖 Generated with Claude Code