Skip to content

Adding pred_measure feature - #363

Open
florence-bockting wants to merge 194 commits into
loo-v3.0.0from
pred_measure
Open

florence-bockting wants to merge 194 commits into
loo-v3.0.0from
pred_measure

Conversation

@florence-bockting

@florence-bockting florence-bockting commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #281
Fixes #223, #213, #201, #135, #220

Summary

This PR adds a predictive performance API. It computes density scores, scoring rules, and point-prediction metrics under four evaluation sources: in-sample, PSIS-LOO, k-fold, and holdout (test data).

elpd(), crps(), scrps(), loo_crps(), loo_scrps(), loo_predictive_metric(), and loo_compare() are deprecated. They still work with their current APIs.

The comment below lists all new features and functions. It marks the parts that come from #380.

How to navigate this PR

Deprecations (implementations retained)

Deprecated Replacement
elpd() measure_elpd() / *_pred_measure()
crps(), scrps() (x, x2) measure_rps(), measure_srps() (ypred)
loo_crps(), loo_scrps() loo_pred_measure(..., measure = "rps" | "srps")
loo_predictive_metric() loo_pred_measure()
loo_compare() (#380) model_compare()

Each deprecated function warns once per session.

Dependencies

Vignettes, test-data generation, and pkgdown CI install brms from GitHub master as k-fold support for categorical models (brms#1890) is not yet on CRAN. Remove the pin after the next brms CRAN release.

Known limitations

  • group_ids grouping is not implemented.
  • The formula derivations article is not finished.
  • The brms GitHub pin remains.

Acknowledgements

This work builds on a prototype by @VisruthSK (VisruthSK/loo-sandbox) under guidance from @jgabry.

@github-actions

github-actions Bot commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if eaebc86 is merged into loo-v3.0.0:

Status Name Current Incoming Interval
✔️ loo_matrix 1.47s 1.47s [-0.62%, +0.94%]
✔️ loo_function 1.33s 1.33s [-0.88%, +0.64%]

Further explanation regarding interpretation and methodology can be found in the documentation.

@florence-bockting

florence-bockting commented Jun 8, 2026 •

Copy link
Copy Markdown
Contributor Author

There is an open issue requesting a subset() method for psis objects #343 and an (unrelated) open PR #110. Might be worth considering whether we can address this as well in this refactoring PR. If yes, we should

Moved this to PR #379

@VisruthSK VisruthSK added this to the v3.0.0 milestone Jun 9, 2026
@florence-bockting florence-bockting mentioned this pull request Jul 1, 2026
6 tasks
@florence-bockting florence-bockting changed the title LOO refactoring and adding pred_measure feature Adding pred_measure feature Jul 1, 2026
@florence-bockting
florence-bockting changed the base branch from master to loo-v3.0.0 July 1, 2026 09:33
@florence-bockting
florence-bockting marked this pull request as ready for review October 19, 2026 09:50
@florence-bockting

Copy link
Copy Markdown
Contributor Author

@jgabry just a quick checkin, whether you received the review notification.
But don't be pressured, I just wanted to ensure that this PR is not forgotten.

@jgabry

jgabry commented Nov 3, 2026

Copy link
Copy Markdown
Member

Thanks for the reminder! I missed the first notification. It looks like there are a couple of merge conflicts, although they're files that don't affect my ability to review it.

@jgabry

jgabry commented Nov 10, 2026

Copy link
Copy Markdown
Member

@VisruthSK Any idea why the pkgdown workflow is failing? Is that related to this PR at all?

@VisruthSK

VisruthSK commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Ah I didn't see this failed. Before it was a pkgdown thing, the function index needed to be updated. Now looks to be a dependency thing:

Reading vignettes/articles-online-only/pred-measure-workflow.Rmd
Error while shutting down parallel: unable to terminate some child processes
Error: 
! in callr subprocess.
Caused by error in `.f(.x[[i]], ...)`:
! Failed to render
  'vignettes/articles-online-only/pred-measure-workflow.Rmd'.
✖ Quitting from pred-measure-workflow.Rmd:110-117 [load-libraries-internal]
Caused by error:
! there is no package called 'devtools'

Just added devtools to the pkgdown build, and should remove it later.

@jgabry

jgabry commented Nov 10, 2026

Copy link
Copy Markdown
Member

Thanks, seems like it's still failing but maybe for a different reason now?

@VisruthSK

Copy link
Copy Markdown
Member

Hopefully is fixed now.

I noticed there was some custom CSS, primarily for callouts it seems like? I think we should move that to https://github.com/stan-dev/pkgdown-config, @florence-bockting do you want to open a PR? I could do it too, and that way styling is consistent and not owned at all by loo.

output:
  rmarkdown::html_vignette:
    css:
      - !expr system.file("rmarkdown/templates/html_vignette/resources/vignette.css", package = "rmarkdown")
      - styles.css

@florence-bockting

Copy link
Copy Markdown
Contributor Author

Hopefully is fixed now.

I noticed there was some custom CSS, primarily for callouts it seems like? I think we should move that to https://github.com/stan-dev/pkgdown-config, @florence-bockting do you want to open a PR? I could do it too, and that way styling is consistent and not owned at all by loo.

output:
  rmarkdown::html_vignette:
    css:
      - !expr system.file("rmarkdown/templates/html_vignette/resources/vignette.css", package = "rmarkdown")
      - styles.css

Sorry, for causing problems due to this custom css.
Do you think its worth a PR or shall I rather remove this custom css?
I don't know what the history of the vignette styles is - is it welcome to deviate from this or is this somehow part of a corporate design strategy of Stan?

@jgabry

jgabry commented Nov 11, 2026

Copy link
Copy Markdown
Member

I don't know what the history of the vignette styles is - is it welcome to deviate from this or is this somehow part of a corporate design strategy of Stan?

I don't think it's part of a broader design strategy. I'm ok with deviating for individual vignettes or we could change the style for all the vignettes if there's something better than what we have now.

@VisruthSK

Copy link
Copy Markdown
Member

I think the specific styles were for callout, I don't remember off the top of my head if standard pkgdown already supports callout stylings, and assumed it didn't (I think Quarto doesn't with certain outputs?).

I need to update the pkgdown workflow here, but the site is up at: https://mc-stan.org/loo/pr/363/index.html

@jgabry

jgabry commented Nov 12, 2026

Copy link
Copy Markdown
Member

Sorry I've been slow on this. I'm working on my review. This is a huge PR and it's pretty complicated so it's taking me a while. I should have a first round of comments soon. Just a heads up that it's going to be a big review with a lot of comments. Overall I think this is really great, but I did notice a lot of things that I'm putting into general review comments and then also some more specific comments on the code.

@jgabry jgabry left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I've been slow on this. This is a really big PR with important changes so I've been taking my time. After you read my review and make any changes you agree with, I should review again (not just the changes but the whole thing) to make sure I didn't miss anything. I'm very confident I'm missing things with a PR this big. I should also do a more careful reading of the documentation, I focused mostly on the code here but I may have more doc comments.

Most of my review comments on specific lines of code are about edge cases and API decisions. I'm putting the bigger comments here and there are a lot of them. Sorry if some of these I should have thought of sooner! I organized the comments into into categories so they're easier to go through. There are some things related to model_compare that I definitely could have noticed in the previous PR but I didn't until now. I think I also made a few of the same comments in multiple places (e.g. here and on a specific line).

Overall this is really awesome and I hope all these comments don't make it seem otherwise!

I did a review pass on my own and then I did a pass together with Claude to supplement it. It found a bunch of edge cases I missed! In my comments on particular lines of code I tried to be transparent about which ones were coming directly from Claude. Not sure if those deserve more scrutiny or less scrutiny than my own comments ;)


Main computations

I checked (with AI assistance) the main computations numerically, and they seem correct:

  • elpd_loo, p_loo and looic match loo(), and the k-fold, holdout and in-sample elpd match too.
  • mae, mse, rmse and acc match loo_predictive_metric().
  • model_compare() reproduces loo_compare() exactly when it should
  • The pred_measure differences and SEs match calculations I asked claude and codex to do.

Deprecation strategy

Maybe we should release the new API first and then in a follow up release do the deprecations, or at least the loo_compare deprecation. brms and rstanarm call loo_compare() so releasing this will cause those users to get warnings that they can't do anything about. We could start by doing a soft deprecation either in the docs only or using lifecycle::deprecate_soft() and then coordinate releases of brms and rstanarm before we do the "hard" deprecation. What do you think?

Simplifications

Now that I'm looking at this as one big PR I think there is room for a decent amount of simplication. Here's an initial list, but there may be more opportunities.

  • The pointwise argument on the non-density measure_*() functions isn't used anywhere, and I think it's broken (I put some comments on the specific code lines).
  • control and measure_rps(scaled = ) just duplicate measure_srps(), and I think they're the only reason for the special-case renaming in do_pred_measure().
  • Some arguments don't seem to do anything: group_ids always errors, save_psis is ignored by insample_pred_measure(), and the built in measures don't use ylp in test_pred_measure() and kfold_pred_measure().
  • measure_info is saved as an attribute on the estimates matrix, then it's removed and then it's merged back in later. I think you could just build a plain list in the loop and it would be a lot simpler.
  • The five user facing functions each pass every argument on to do_pred_measure(), including a bunch that are hard coded NULL. Then do_pred_measure() uses source to figure out which function called it and checks that those NULLs are actually NULL. It could be simpler for each wrapper to just handle its own specific part (e.g., getting the PSIS weights, reading the kfold results, etc.) and then call a shared function that only accepts arguments they all have in common.
  • In model_compare(), the PSIS diagnostics get turned into strings and then parsed with a regex in the print method. It would be simpler to just printthe flagged rows directly.
  • The custom SE handling in model_compare() is spread across .resolve_custom_se_diffs(), inform_missing_custom_se_diff() and part of .pair_measure_stats(). I think it could be one step in the main loop.
  • A few of the model_compare() helpers have fallbacks for when measure_info is missing (e.g. in .measure_is_loss(), .measure_pointwise_diff_method(), .measure_se_diff_fun()). But .compare_metadata_check() already has an error when it's missing, and .measure_info() always sets loss and diff_method, so I don't think those fallbacks can actually ever run. Maybe just remove them? And then I thin k.builtin_loss_measures() can also be removed.
  • The "classic" loo versions of model_compare.default() and compare_pred_measure() compute a lot of the same things separately (e.g., elpd diffs, p_worse, diagnostics, ordering), and that means some helpers have to have two modes because (model_compare_order(), model_order_stat_check(), model_compare_checks()). But isn't comparing plain loo objects basically the same as the pred_measure case with only elpd? I think that means we could we convert them to the same internal format and then we only need one version/path. And we could add a test checking that the output is still exactly the same as loo_compare(). What do you think?
  • It doesn't look like subset_measures(), is.loo_pred_measure() and .builtin_loss_measures() are used anywhere. Can they be removed?
  • There are two different ways deprecations are handled. Most of them use .Deprecated(), which warns every time it's called . But loo_compare() uses the .deprecate_once(). Can we use the same thing for all? The PR description says they all warn once per session.
  • .resolve_digits(), .measure_digits(), and .se_digits() could be one function. Also the extra args to .se_digits() aren't used anywhere.
  • test_compare.R is really long and complicated and a lot of that is from calling internal functions directly, which I don't think we need to do in general. There's also a lot of repeating the same code to build pred_measure objects. I think we should test by calling model_compare instead of internals where possible (i.e. test that what the user interacts with always works properly and that means the internals that are actually relevant to that will be tested indirectly), and add a small helper function for creating the test objects. That should make the file a lot shorter and the tests won't need rewriting if the internals are simplified.

Memory

Right now it looks like every LOO result saves the full S x N log_weights matrix, even when save_psis = FALSE. I had Claude check and that means that e.g. for S=400 and N=2000 I get a 61 MB object from loo_pred_measure but only a 0.1 MB object for regular loo(). Then if save_psis=TRUE the weights are saved twice and the object is 122 MB. I think all this could work if the weights are only saved when asked and we can keep them in just one place. But I might be overlooking something, so I could be wrong.

API questions/decisions

  • What about using log_lik instead of ylp? Isn't that the same quantity that loo_predictive_metric() and loo_crps() already call log_lik? Or is the change intentional (if so that's fine)? We could also maybe use yrep and epred instead of ypred and mupred to be consistent with brms and rstanarm, but not a big deal. I don't feel super strongly about that. I could also be misinterpreting the names.
  • I think in pred_measure() it would be more natural for the object that is being updated to be the first argument. And in we do that then we could just call it x or object instead of predperf, which is pretty standard in R when a function is updating its input object.
  • When we compute PSIS weights from ylp, r_eff is hard-coded to 1 and there's a message about it every time. What about adding an r_eff = 1 argument to loo_pred_measure() like we have for loo() has (and what about cores too?), and then get rid of the message. We actually got rid of the r_eff warning in loo() a while ago because it was annoying.
  • What if we stop inheriting the loo, psis_loo and kfold classes? Right now the classes a user gets depend on how the result was created, so some loo methods work on some results but not others. The measure_*() functions also classes c("measure", "loo"), which I don't think we need/want.
  • measure_brier() has a ypred argument, but acc and bacc have mupred. But I think all three of them are taking P(y = 1), right? If so they could have the same argument name unless there's a reason I'm missing.
  • If there's not a strong reason for exporting it I think we should keep ptw_log_pred_density() internal because it takes normalized weights, unlike all the measure_*() functions.
  • loo_pred_measure(loo = ) needs a saved psis_object even for the default elpd. But brms has save_psis = FALSE by default (I think), so this won't work by default with brms loo objects. Not sure how we want to handle that or if we just accept that.
  • I think there may be too many informational messages, e.g. "Using psis_object for LOO CV from predperf". Do we need these? I think we'd only want them if we're doing something the user wouldn't necessarily expect. If we're doing what they'd expect we don't need a message.
  • Since these days people have autocomplete (if they even write their own code anymore) simplifying pointwise to ptw seems more confusing to me than helpful. Previously it would be nice to type fewer characters, but now I think clarity matters more. What do you think? I don't feel super strongly about this so fine either way.

model_compare() output

Some thing I didn't think of when reviewing the previous PR but that occur to me now, maybe because I'm seeing it in context, or maybe just because it's a second look after not looking for a while.

model_compare() returns one wide data frame that gets sorted by elpd. But the columns for each measure are relative to the best model for that measure, and the reference model is just in an attribute. I think that's going to be confusing, especially for anyone using the output in code. Here are some other possibilities:

  • A long format data frame with columns like model, measure, reference, diff, se_diff and sign_flipped
  • A list with of data frames, one per measure
  • The same as it is now but with the reference model and whether the sign was flipped in columns instead of attributes

Also, I forget if I commented on this in the other PR or not. When elpd isn't included, the ranking just uses the measure that happens to be first alphabetically instead of the order the user listed the measures in. I think users would expect the latter.

Also I don't think model_compare.psis_loo_ss_list() needs to be an exported S3 method.

Input validation checks

  • There's no check that n is the same so e.g. ylp could be passed as its transpose and not error.
  • In-sample and holdout results (and LOO results computed from ylp or psis_object) don't save a yhash, so model_compare() won't warn if the models were fit to different y. Do we want that?
  • Custom measure names don't get validated (see my comment on custom_measure()).
  • There's no check that psis_object is actually a psis object.

Repo contents / cleanup

  • I think we can remove the notes/ directory now, or move anything that's still useful somewhere else.
  • There's some stuff in .Rbuildignore and .gitignore that could go in your global gitignore instead, although I think we can keep the agents and claude stuff, that's useful for everyone
  • test_data_generation.R installs packages. I think we should move the data generation script to data-raw/ and remove the install code
  • The tarball size doubled (2.09 MB to 4.14 MB). We're still under CRAN's limit, so I guess it's ok. But it means we don't have much room after this to add things that increase the size.
  • The pkgdown workflow installs brms from GitHub master, so the development version. Do we want to use that version or the CRAN version of brms? Not sure if @florence-bockting or @VisruthSK changed that. If there's a reason then fine to keep it.

Doc and NEWS

  • There are still a bunch of TODOs in the online articles and some spots in the doc still say to set attributes directly instead of using custom_measure.

  • NEWS is missing subsampled loo_compare() returning a data frame instead of a matrix

  • In the rest of the package we use N for the number of observations and the model_compare() docs and vignette in the PR also use N. But the pred_measure docs and the two pred_measure vignettes use n. Can you switch those to N? There's also one message in the code that uses n (see my inline comment).

Tests

  • See comment above in the simplifications section.
  • When I run the tests there are a lot of warnings that aren't caught that could be caught so the test output is cleaner.
  • Some new functions that should give the same answer as deprecated ones don't have a test checking that.

Comment thread R/pred_measure.R
mupred = NULL,
ylp = NULL,
measures = NULL,
group_ids = NULL,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

group_ids is in all five function signatures but I think it always errors right now. Should we remove it until grouping is implemented? We won't break anything by adding it later.

Comment thread R/pred_measure.R
ylp = NULL,
measures = NULL,
loo = NULL,
group_ids = NULL,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above

Comment thread R/pred_measure.R
ylp = NULL,
measures = NULL,
kfold = NULL,
group_ids = NULL,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above

Comment thread R/pred_measure.R
ylp = NULL,
ylp_test = NULL,
measures = NULL,
group_ids = NULL,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Comment thread R/pred_measure.R
ylp = NULL,
measures = NULL,
predperf,
group_ids = NULL,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

cor(old$pointwise, -new$pointwise[, "rps_loo"]),
0.98
)
expect_false(isTRUE(all.equal(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only checks that the results are not equal, but not that the values are what we expect

cor(old$pointwise, new$pointwise[, "srps_loo"]),
0.98
)
expect_false(isTRUE(all.equal(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only checks that the results are not equal, but not that the values are what we expect

expect_equal(attr(comp_loo, "compare_reference"), c(elpd = comp_loo$model[[1L]]))
})

test_that("model_compare warns when predictive measures differ across models", {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes even though the estimate columns are wrong (see my comment on .model_compare_estimates_table()). I think we should check the actual values.

}
if (!requireNamespace("brms", quietly = TRUE)) {
remotes::install_github("paul-buerkner/brms", ref = "master", upgrade = "never")
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually installs packages (and also uses dplyr, which isn't in DESCRIPTION). This whole script can go in data-raw/, which is in .Rbuildignore

Comment thread R/pred_measure-builtin.R
#' [kfold_pred_measure()], [test_pred_measure()], and [pred_measure()].
#'
#' @export
supported_measures_list <- names(.measure_spec)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is called a "list" but it's not actually an R list, just a character vector. Not a big deal but could be confusing.

I think it would be better as a function called supported_measures(). The migration guide vignette already actually uses it as a function (incorrectly, see comment on that vignette).

It's also passed as function argument in various places, even though it's an argument with only one possible value because it's a constant defined by the package. The tests also get it with getFromNamespace() just to pass it back in (e.g. in tests/testthat/test_pred_measure_helpers.R). So I think this could all be simplified.

@jgabry jgabry left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I've been slow on this. This is a really big PR with important changes so I've been taking my time. After you read my review and make any changes you agree with, I should review again (not just the changes but the whole thing) to make sure I didn't miss anything. I'm very confident I'm missing things with a PR this big. I should also do a more careful reading of the documentation, I focused mostly on the code here but I may have more doc comments.

Most of my review comments on specific lines of code are about edge cases and API decisions. I'm putting the bigger comments here and there are a lot of them. Sorry if some of these I should have thought of sooner! I organized the comments into into categories so they're easier to go through. There are some things related to model_compare that I definitely could have noticed in the previous PR but I didn't until now. I think I also made a few of the same comments in multiple places (e.g. here and on a specific line).

Overall this is really awesome and I hope all these comments don't make it seem otherwise!

I did a review pass on my own and then I did a pass together with Claude to supplement it. It found a bunch of edge cases I missed! In my comments on particular lines of code I tried to be transparent about which ones were coming directly from Claude. Not sure if those deserve more scrutiny or less scrutiny than my own comments ;)


Main computations

I checked (with AI assistance) the main computations numerically, and they seem correct:

  • elpd_loo, p_loo and looic match loo(), and the k-fold, holdout and in-sample elpd match too.
  • mae, mse, rmse and acc match loo_predictive_metric().
  • model_compare() reproduces loo_compare() exactly when it should
  • The pred_measure differences and SEs match calculations I asked claude and codex to do.

Deprecation strategy

Maybe we should release the new API first and then in a follow up release do the deprecations, or at least the loo_compare deprecation. brms and rstanarm call loo_compare() so releasing this will cause those users to get warnings that they can't do anything about. We could start by doing a soft deprecation either in the docs only or using lifecycle::deprecate_soft() and then coordinate releases of brms and rstanarm before we do the "hard" deprecation. What do you think?

Simplifications

Now that I'm looking at this as one big PR I think there is room for a decent amount of simplication. Here's an initial list, but there may be more opportunities.

  • The pointwise argument on the non-density measure_*() functions isn't used anywhere, and I think it's broken (I put some comments on the specific code lines).
  • control and measure_rps(scaled = ) just duplicate measure_srps(), and I think they're the only reason for the special-case renaming in do_pred_measure().
  • Some arguments don't seem to do anything: group_ids always errors, save_psis is ignored by insample_pred_measure(), and the built in measures don't use ylp in test_pred_measure() and kfold_pred_measure().
  • measure_info is saved as an attribute on the estimates matrix, then it's removed and then it's merged back in later. I think you could just build a plain list in the loop and it would be a lot simpler.
  • The five user facing functions each pass every argument on to do_pred_measure(), including a bunch that are hard coded NULL. Then do_pred_measure() uses source to figure out which function called it and checks that those NULLs are actually NULL. It could be simpler for each wrapper to just handle its own specific part (e.g., getting the PSIS weights, reading the kfold results, etc.) and then call a shared function that only accepts arguments they all have in common.
  • In model_compare(), the PSIS diagnostics get turned into strings and then parsed with a regex in the print method. It would be simpler to just printthe flagged rows directly.
  • The custom SE handling in model_compare() is spread across .resolve_custom_se_diffs(), inform_missing_custom_se_diff() and part of .pair_measure_stats(). I think it could be one step in the main loop.
  • A few of the model_compare() helpers have fallbacks for when measure_info is missing (e.g. in .measure_is_loss(), .measure_pointwise_diff_method(), .measure_se_diff_fun()). But .compare_metadata_check() already has an error when it's missing, and .measure_info() always sets loss and diff_method, so I don't think those fallbacks can actually ever run. Maybe just remove them? And then I thin k.builtin_loss_measures() can also be removed.
  • The "classic" loo versions of model_compare.default() and compare_pred_measure() compute a lot of the same things separately (e.g., elpd diffs, p_worse, diagnostics, ordering), and that means some helpers have to have two modes because (model_compare_order(), model_order_stat_check(), model_compare_checks()). But isn't comparing plain loo objects basically the same as the pred_measure case with only elpd? I think that means we could we convert them to the same internal format and then we only need one version/path. And we could add a test checking that the output is still exactly the same as loo_compare(). What do you think?
  • It doesn't look like subset_measures(), is.loo_pred_measure() and .builtin_loss_measures() are used anywhere. Can they be removed?
  • There are two different ways deprecations are handled. Most of them use .Deprecated(), which warns every time it's called . But loo_compare() uses the .deprecate_once(). Can we use the same thing for all? The PR description says they all warn once per session.
  • .resolve_digits(), .measure_digits(), and .se_digits() could be one function. Also the extra args to .se_digits() aren't used anywhere.
  • test_compare.R is really long and complicated and a lot of that is from calling internal functions directly, which I don't think we need to do in general. There's also a lot of repeating the same code to build pred_measure objects. I think we should test by calling model_compare instead of internals where possible (i.e. test that what the user interacts with always works properly and that means the internals that are actually relevant to that will be tested indirectly), and add a small helper function for creating the test objects. That should make the file a lot shorter and the tests won't need rewriting if the internals are simplified.

Memory

Right now it looks like every LOO result saves the full S x N log_weights matrix, even when save_psis = FALSE. I had Claude check and that means that e.g. for S=400 and N=2000 I get a 61 MB object from loo_pred_measure but only a 0.1 MB object for regular loo(). Then if save_psis=TRUE the weights are saved twice and the object is 122 MB. I think all this could work if the weights are only saved when asked and we can keep them in just one place. But I might be overlooking something, so I could be wrong.

API questions/decisions

  • What about using log_lik instead of ylp? Isn't that the same quantity that loo_predictive_metric() and loo_crps() already call log_lik? Or is the change intentional (if so that's fine)? We could also maybe use yrep and epred instead of ypred and mupred to be consistent with brms and rstanarm, but not a big deal. I don't feel super strongly about that. I could also be misinterpreting the names.
  • I think in pred_measure() it would be more natural for the object that is being updated to be the first argument. And in we do that then we could just call it x or object instead of predperf, which is pretty standard in R when a function is updating its input object.
  • When we compute PSIS weights from ylp, r_eff is hard-coded to 1 and there's a message about it every time. What about adding an r_eff = 1 argument to loo_pred_measure() like we have for loo() has (and what about cores too?), and then get rid of the message. We actually got rid of the r_eff warning in loo() a while ago because it was annoying.
  • What if we stop inheriting the loo, psis_loo and kfold classes? Right now the classes a user gets depend on how the result was created, so some loo methods work on some results but not others. The measure_*() functions also classes c("measure", "loo"), which I don't think we need/want.
  • measure_brier() has a ypred argument, but acc and bacc have mupred. But I think all three of them are taking P(y = 1), right? If so they could have the same argument name unless there's a reason I'm missing.
  • If there's not a strong reason for exporting it I think we should keep ptw_log_pred_density() internal because it takes normalized weights, unlike all the measure_*() functions.
  • loo_pred_measure(loo = ) needs a saved psis_object even for the default elpd. But brms has save_psis = FALSE by default (I think), so this won't work by default with brms loo objects. Not sure how we want to handle that or if we just accept that.
  • I think there may be too many informational messages, e.g. "Using psis_object for LOO CV from predperf". Do we need these? I think we'd only want them if we're doing something the user wouldn't necessarily expect. If we're doing what they'd expect we don't need a message.
  • Since these days people have autocomplete (if they even write their own code anymore) simplifying pointwise to ptw seems more confusing to me than helpful. Previously it would be nice to type fewer characters, but now I think clarity matters more. What do you think? I don't feel super strongly about this so fine either way.

model_compare() output

Some thing I didn't think of when reviewing the previous PR but that occur to me now, maybe because I'm seeing it in context, or maybe just because it's a second look after not looking for a while.

model_compare() returns one wide data frame that gets sorted by elpd. But the columns for each measure are relative to the best model for that measure, and the reference model is just in an attribute. I think that's going to be confusing, especially for anyone using the output in code. Here are some other possibilities:

  • A long format data frame with columns like model, measure, reference, diff, se_diff and sign_flipped
  • A list with of data frames, one per measure
  • The same as it is now but with the reference model and whether the sign was flipped in columns instead of attributes

Also, I forget if I commented on this in the other PR or not. When elpd isn't included, the ranking just uses the measure that happens to be first alphabetically instead of the order the user listed the measures in. I think users would expect the latter.

Also I don't think model_compare.psis_loo_ss_list() needs to be an exported S3 method.

Input validation checks

  • There's no check that n is the same so e.g. ylp could be passed as its transpose and not error.
  • In-sample and holdout results (and LOO results computed from ylp or psis_object) don't save a yhash, so model_compare() won't warn if the models were fit to different y. Do we want that?
  • Custom measure names don't get validated (see my comment on custom_measure()).
  • There's no check that psis_object is actually a psis object.

Repo contents / cleanup

  • I think we can remove the notes/ directory now, or move anything that's still useful somewhere else.
  • There's some stuff in .Rbuildignore and .gitignore that could go in your global gitignore instead, although I think we can keep the agents and claude stuff, that's useful for everyone
  • test_data_generation.R installs packages. I think we should move the data generation script to data-raw/ and remove the install code
  • The tarball size doubled (2.09 MB to 4.14 MB). We're still under CRAN's limit, so I guess it's ok. But it means we don't have much room after this to add things that increase the size.
  • The pkgdown workflow installs brms from GitHub master, so the development version. Do we want to use that version or the CRAN version of brms? Not sure if @florence-bockting or @VisruthSK changed that. If there's a reason then fine to keep it.

Doc and NEWS

  • There are still a bunch of TODOs in the online articles and some spots in the doc still say to set attributes directly instead of using custom_measure.

  • NEWS is missing subsampled loo_compare() returning a data frame instead of a matrix

  • In the rest of the package we use N for the number of observations and the model_compare() docs and vignette in the PR also use N. But the pred_measure docs and the two pred_measure vignettes use n. Can you switch those to N? There's also one message in the code that uses n (see my inline comment).

Tests

  • See comment above in the simplifications section.
  • When I run the tests there are a lot of warnings that aren't caught that could be caught so the test output is cleaner.
  • Some new functions that should give the same answer as deprecated ones don't have a test checking that.

@jgabry

jgabry commented Nov 12, 2026

Copy link
Copy Markdown
Member

Not sure why but it looks like my long main comment was posted twice. Also a lot of the comments on individual lines of code are hidden because there are so many of them

@VisruthSK

Copy link
Copy Markdown
Member

I think brms being installed from master was because of this line in the PR description. Though maybe that's out of date now?

Vignettes, test-data generation, and pkgdown CI install brms from GitHub master as k-fold support for categorical models (paul-buerkner/brms#1890) is not yet on CRAN. Remove the pin after the next brms CRAN release.

@jgabry

jgabry commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

I think brms being installed from master was because of this line in the PR description. Though maybe that's out of date now?

Vignettes, test-data generation, and pkgdown CI install brms from GitHub master as k-fold support for categorical models (paul-buerkner/brms#1890) is not yet on CRAN. Remove the pin after the next brms CRAN release.

Thanks, that makes sense. We should probably put a comment in that file next to where brms is installed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

4 participants