Repository navigation
Adding pred_measure feature - #363
florence-bockting wants to merge 194 commits into
Conversation
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if eaebc86 is merged into loo-v3.0.0:
Further explanation regarding interpretation and methodology can be found in the documentation. |
|
There is an open issue requesting a subset() method for psis objects #343 and an (unrelated) open PR #110. Might be worth considering whether we can address this as well in this refactoring PR. If yes, we should
Moved this to PR #379 |
pred_measure feature
|
@jgabry just a quick checkin, whether you received the review notification. |
|
Thanks for the reminder! I missed the first notification. It looks like there are a couple of merge conflicts, although they're files that don't affect my ability to review it. |
|
@VisruthSK Any idea why the pkgdown workflow is failing? Is that related to this PR at all? |
|
Ah I didn't see this failed. Before it was a Just added |
|
Thanks, seems like it's still failing but maybe for a different reason now? |
|
Hopefully is fixed now. I noticed there was some custom CSS, primarily for callouts it seems like? I think we should move that to https://github.com/stan-dev/pkgdown-config, @florence-bockting do you want to open a PR? I could do it too, and that way styling is consistent and not owned at all by output:
rmarkdown::html_vignette:
css:
- !expr system.file("rmarkdown/templates/html_vignette/resources/vignette.css", package = "rmarkdown")
- styles.css |
Sorry, for causing problems due to this custom css. |
I don't think it's part of a broader design strategy. I'm ok with deviating for individual vignettes or we could change the style for all the vignettes if there's something better than what we have now. |
|
I think the specific styles were for callout, I don't remember off the top of my head if standard pkgdown already supports callout stylings, and assumed it didn't (I think Quarto doesn't with certain outputs?). I need to update the pkgdown workflow here, but the site is up at: https://mc-stan.org/loo/pr/363/index.html |
|
Sorry I've been slow on this. I'm working on my review. This is a huge PR and it's pretty complicated so it's taking me a while. I should have a first round of comments soon. Just a heads up that it's going to be a big review with a lot of comments. Overall I think this is really great, but I did notice a lot of things that I'm putting into general review comments and then also some more specific comments on the code. |
There was a problem hiding this comment.
Sorry I've been slow on this. This is a really big PR with important changes so I've been taking my time. After you read my review and make any changes you agree with, I should review again (not just the changes but the whole thing) to make sure I didn't miss anything. I'm very confident I'm missing things with a PR this big. I should also do a more careful reading of the documentation, I focused mostly on the code here but I may have more doc comments.
Most of my review comments on specific lines of code are about edge cases and API decisions. I'm putting the bigger comments here and there are a lot of them. Sorry if some of these I should have thought of sooner! I organized the comments into into categories so they're easier to go through. There are some things related to model_compare that I definitely could have noticed in the previous PR but I didn't until now. I think I also made a few of the same comments in multiple places (e.g. here and on a specific line).
Overall this is really awesome and I hope all these comments don't make it seem otherwise!
I did a review pass on my own and then I did a pass together with Claude to supplement it. It found a bunch of edge cases I missed! In my comments on particular lines of code I tried to be transparent about which ones were coming directly from Claude. Not sure if those deserve more scrutiny or less scrutiny than my own comments ;)
Main computations
I checked (with AI assistance) the main computations numerically, and they seem correct:
- elpd_loo, p_loo and looic match loo(), and the k-fold, holdout and in-sample elpd match too.
- mae, mse, rmse and acc match loo_predictive_metric().
- model_compare() reproduces loo_compare() exactly when it should
- The pred_measure differences and SEs match calculations I asked claude and codex to do.
Deprecation strategy
Maybe we should release the new API first and then in a follow up release do the deprecations, or at least the loo_compare deprecation. brms and rstanarm call loo_compare() so releasing this will cause those users to get warnings that they can't do anything about. We could start by doing a soft deprecation either in the docs only or using lifecycle::deprecate_soft() and then coordinate releases of brms and rstanarm before we do the "hard" deprecation. What do you think?
Simplifications
Now that I'm looking at this as one big PR I think there is room for a decent amount of simplication. Here's an initial list, but there may be more opportunities.
- The
pointwiseargument on the non-densitymeasure_*()functions isn't used anywhere, and I think it's broken (I put some comments on the specific code lines). controlandmeasure_rps(scaled = )just duplicatemeasure_srps(), and I think they're the only reason for the special-case renaming indo_pred_measure().- Some arguments don't seem to do anything:
group_idsalways errors,save_psisis ignored byinsample_pred_measure(), and the built in measures don't useylpintest_pred_measure()andkfold_pred_measure(). measure_infois saved as an attribute on the estimates matrix, then it's removed and then it's merged back in later. I think you could just build a plain list in the loop and it would be a lot simpler.- The five user facing functions each pass every argument on to
do_pred_measure(), including a bunch that are hard codedNULL. Thendo_pred_measure()usessourceto figure out which function called it and checks that thoseNULLs are actuallyNULL. It could be simpler for each wrapper to just handle its own specific part (e.g., getting the PSIS weights, reading the kfold results, etc.) and then call a shared function that only accepts arguments they all have in common. - In
model_compare(), the PSIS diagnostics get turned into strings and then parsed with a regex in the print method. It would be simpler to just printthe flagged rows directly. - The custom SE handling in
model_compare()is spread across.resolve_custom_se_diffs(),inform_missing_custom_se_diff()and part of.pair_measure_stats(). I think it could be one step in the main loop. - A few of the
model_compare()helpers have fallbacks for whenmeasure_infois missing (e.g. in.measure_is_loss(),.measure_pointwise_diff_method(),.measure_se_diff_fun()). But.compare_metadata_check()already has an error when it's missing, and.measure_info()always setslossanddiff_method, so I don't think those fallbacks can actually ever run. Maybe just remove them? And then I thin k.builtin_loss_measures()can also be removed. - The "classic"
looversions ofmodel_compare.default()andcompare_pred_measure()compute a lot of the same things separately (e.g., elpd diffs,p_worse, diagnostics, ordering), and that means some helpers have to have two modes because (model_compare_order(),model_order_stat_check(),model_compare_checks()). But isn't comparing plainlooobjects basically the same as thepred_measurecase with only elpd? I think that means we could we convert them to the same internal format and then we only need one version/path. And we could add a test checking that the output is still exactly the same asloo_compare(). What do you think? - It doesn't look like
subset_measures(),is.loo_pred_measure()and.builtin_loss_measures()are used anywhere. Can they be removed? - There are two different ways deprecations are handled. Most of them use
.Deprecated(), which warns every time it's called . Butloo_compare()uses the.deprecate_once(). Can we use the same thing for all? The PR description says they all warn once per session. .resolve_digits(),.measure_digits(), and.se_digits()could be one function. Also the extra args to.se_digits()aren't used anywhere.test_compare.Ris really long and complicated and a lot of that is from calling internal functions directly, which I don't think we need to do in general. There's also a lot of repeating the same code to buildpred_measureobjects. I think we should test by callingmodel_compareinstead of internals where possible (i.e. test that what the user interacts with always works properly and that means the internals that are actually relevant to that will be tested indirectly), and add a small helper function for creating the test objects. That should make the file a lot shorter and the tests won't need rewriting if the internals are simplified.
Memory
Right now it looks like every LOO result saves the full S x N log_weights matrix, even when save_psis = FALSE. I had Claude check and that means that e.g. for S=400 and N=2000 I get a 61 MB object from loo_pred_measure but only a 0.1 MB object for regular loo(). Then if save_psis=TRUE the weights are saved twice and the object is 122 MB. I think all this could work if the weights are only saved when asked and we can keep them in just one place. But I might be overlooking something, so I could be wrong.
API questions/decisions
- What about using
log_likinstead ofylp? Isn't that the same quantity thatloo_predictive_metric()andloo_crps()already calllog_lik? Or is the change intentional (if so that's fine)? We could also maybe useyrepandepredinstead ofypredandmupredto be consistent with brms and rstanarm, but not a big deal. I don't feel super strongly about that. I could also be misinterpreting the names. - I think in
pred_measure()it would be more natural for the object that is being updated to be the first argument. And in we do that then we could just call itxorobjectinstead ofpredperf, which is pretty standard in R when a function is updating its input object. - When we compute PSIS weights from
ylp,r_effis hard-coded to 1 and there's a message about it every time. What about adding anr_eff = 1argument toloo_pred_measure()like we have forloo()has (and what aboutcorestoo?), and then get rid of the message. We actually got rid of ther_effwarning inloo()a while ago because it was annoying. - What if we stop inheriting the
loo,psis_looandkfoldclasses? Right now the classes a user gets depend on how the result was created, so some loo methods work on some results but not others. Themeasure_*()functions also classesc("measure", "loo"), which I don't think we need/want. measure_brier()has aypredargument, butaccandbacchavemupred. But I think all three of them are taking P(y = 1), right? If so they could have the same argument name unless there's a reason I'm missing.- If there's not a strong reason for exporting it I think we should keep
ptw_log_pred_density()internal because it takes normalized weights, unlike all themeasure_*()functions. loo_pred_measure(loo = )needs a savedpsis_objecteven for the default elpd. But brms hassave_psis = FALSEby default (I think), so this won't work by default with brmslooobjects. Not sure how we want to handle that or if we just accept that.- I think there may be too many informational messages, e.g. "Using psis_object for LOO CV from
predperf". Do we need these? I think we'd only want them if we're doing something the user wouldn't necessarily expect. If we're doing what they'd expect we don't need a message. - Since these days people have autocomplete (if they even write their own code anymore) simplifying
pointwisetoptwseems more confusing to me than helpful. Previously it would be nice to type fewer characters, but now I think clarity matters more. What do you think? I don't feel super strongly about this so fine either way.
model_compare() output
Some thing I didn't think of when reviewing the previous PR but that occur to me now, maybe because I'm seeing it in context, or maybe just because it's a second look after not looking for a while.
model_compare() returns one wide data frame that gets sorted by elpd. But the columns for each measure are relative to the best model for that measure, and the reference model is just in an attribute. I think that's going to be confusing, especially for anyone using the output in code. Here are some other possibilities:
- A long format data frame with columns like
model,measure,reference,diff,se_diffandsign_flipped - A list with of data frames, one per measure
- The same as it is now but with the reference model and whether the sign was flipped in columns instead of attributes
Also, I forget if I commented on this in the other PR or not. When elpd isn't included, the ranking just uses the measure that happens to be first alphabetically instead of the order the user listed the measures in. I think users would expect the latter.
Also I don't think model_compare.psis_loo_ss_list() needs to be an exported S3 method.
Input validation checks
- There's no check that
nis the same so e.g.ylpcould be passed as its transpose and not error. - In-sample and holdout results (and LOO results computed from
ylporpsis_object) don't save ayhash, somodel_compare()won't warn if the models were fit to differenty. Do we want that? - Custom measure names don't get validated (see my comment on
custom_measure()). - There's no check that
psis_objectis actually a psis object.
Repo contents / cleanup
- I think we can remove the
notes/directory now, or move anything that's still useful somewhere else. - There's some stuff in
.Rbuildignoreand.gitignorethat could go in your global gitignore instead, although I think we can keep the agents and claude stuff, that's useful for everyone test_data_generation.Rinstalls packages. I think we should move the data generation script todata-raw/and remove the install code- The tarball size doubled (2.09 MB to 4.14 MB). We're still under CRAN's limit, so I guess it's ok. But it means we don't have much room after this to add things that increase the size.
- The pkgdown workflow installs brms from GitHub master, so the development version. Do we want to use that version or the CRAN version of brms? Not sure if @florence-bockting or @VisruthSK changed that. If there's a reason then fine to keep it.
Doc and NEWS
-
There are still a bunch of TODOs in the online articles and some spots in the doc still say to set attributes directly instead of using
custom_measure. -
NEWS is missing subsampled
loo_compare()returning a data frame instead of a matrix -
In the rest of the package we use
Nfor the number of observations and themodel_compare()docs and vignette in the PR also useN. But thepred_measuredocs and the twopred_measurevignettes usen. Can you switch those toN? There's also one message in the code that usesn(see my inline comment).
Tests
- See comment above in the simplifications section.
- When I run the tests there are a lot of warnings that aren't caught that could be caught so the test output is cleaner.
- Some new functions that should give the same answer as deprecated ones don't have a test checking that.
| mupred = NULL, | ||
| ylp = NULL, | ||
| measures = NULL, | ||
| group_ids = NULL, |
There was a problem hiding this comment.
group_ids is in all five function signatures but I think it always errors right now. Should we remove it until grouping is implemented? We won't break anything by adding it later.
| ylp = NULL, | ||
| measures = NULL, | ||
| loo = NULL, | ||
| group_ids = NULL, |
| ylp = NULL, | ||
| measures = NULL, | ||
| kfold = NULL, | ||
| group_ids = NULL, |
| ylp = NULL, | ||
| ylp_test = NULL, | ||
| measures = NULL, | ||
| group_ids = NULL, |
| ylp = NULL, | ||
| measures = NULL, | ||
| predperf, | ||
| group_ids = NULL, |
| cor(old$pointwise, -new$pointwise[, "rps_loo"]), | ||
| 0.98 | ||
| ) | ||
| expect_false(isTRUE(all.equal( |
There was a problem hiding this comment.
This only checks that the results are not equal, but not that the values are what we expect
| cor(old$pointwise, new$pointwise[, "srps_loo"]), | ||
| 0.98 | ||
| ) | ||
| expect_false(isTRUE(all.equal( |
There was a problem hiding this comment.
This only checks that the results are not equal, but not that the values are what we expect
| expect_equal(attr(comp_loo, "compare_reference"), c(elpd = comp_loo$model[[1L]])) | ||
| }) | ||
|
|
||
| test_that("model_compare warns when predictive measures differ across models", { |
There was a problem hiding this comment.
This test passes even though the estimate columns are wrong (see my comment on .model_compare_estimates_table()). I think we should check the actual values.
| } | ||
| if (!requireNamespace("brms", quietly = TRUE)) { | ||
| remotes::install_github("paul-buerkner/brms", ref = "master", upgrade = "never") | ||
| } |
There was a problem hiding this comment.
This actually installs packages (and also uses dplyr, which isn't in DESCRIPTION). This whole script can go in data-raw/, which is in .Rbuildignore
| #' [kfold_pred_measure()], [test_pred_measure()], and [pred_measure()]. | ||
| #' | ||
| #' @export | ||
| supported_measures_list <- names(.measure_spec) |
There was a problem hiding this comment.
This is called a "list" but it's not actually an R list, just a character vector. Not a big deal but could be confusing.
I think it would be better as a function called supported_measures(). The migration guide vignette already actually uses it as a function (incorrectly, see comment on that vignette).
It's also passed as function argument in various places, even though it's an argument with only one possible value because it's a constant defined by the package. The tests also get it with getFromNamespace() just to pass it back in (e.g. in tests/testthat/test_pred_measure_helpers.R). So I think this could all be simplified.
There was a problem hiding this comment.
Sorry I've been slow on this. This is a really big PR with important changes so I've been taking my time. After you read my review and make any changes you agree with, I should review again (not just the changes but the whole thing) to make sure I didn't miss anything. I'm very confident I'm missing things with a PR this big. I should also do a more careful reading of the documentation, I focused mostly on the code here but I may have more doc comments.
Most of my review comments on specific lines of code are about edge cases and API decisions. I'm putting the bigger comments here and there are a lot of them. Sorry if some of these I should have thought of sooner! I organized the comments into into categories so they're easier to go through. There are some things related to model_compare that I definitely could have noticed in the previous PR but I didn't until now. I think I also made a few of the same comments in multiple places (e.g. here and on a specific line).
Overall this is really awesome and I hope all these comments don't make it seem otherwise!
I did a review pass on my own and then I did a pass together with Claude to supplement it. It found a bunch of edge cases I missed! In my comments on particular lines of code I tried to be transparent about which ones were coming directly from Claude. Not sure if those deserve more scrutiny or less scrutiny than my own comments ;)
Main computations
I checked (with AI assistance) the main computations numerically, and they seem correct:
- elpd_loo, p_loo and looic match loo(), and the k-fold, holdout and in-sample elpd match too.
- mae, mse, rmse and acc match loo_predictive_metric().
- model_compare() reproduces loo_compare() exactly when it should
- The pred_measure differences and SEs match calculations I asked claude and codex to do.
Deprecation strategy
Maybe we should release the new API first and then in a follow up release do the deprecations, or at least the loo_compare deprecation. brms and rstanarm call loo_compare() so releasing this will cause those users to get warnings that they can't do anything about. We could start by doing a soft deprecation either in the docs only or using lifecycle::deprecate_soft() and then coordinate releases of brms and rstanarm before we do the "hard" deprecation. What do you think?
Simplifications
Now that I'm looking at this as one big PR I think there is room for a decent amount of simplication. Here's an initial list, but there may be more opportunities.
- The
pointwiseargument on the non-densitymeasure_*()functions isn't used anywhere, and I think it's broken (I put some comments on the specific code lines). controlandmeasure_rps(scaled = )just duplicatemeasure_srps(), and I think they're the only reason for the special-case renaming indo_pred_measure().- Some arguments don't seem to do anything:
group_idsalways errors,save_psisis ignored byinsample_pred_measure(), and the built in measures don't useylpintest_pred_measure()andkfold_pred_measure(). measure_infois saved as an attribute on the estimates matrix, then it's removed and then it's merged back in later. I think you could just build a plain list in the loop and it would be a lot simpler.- The five user facing functions each pass every argument on to
do_pred_measure(), including a bunch that are hard codedNULL. Thendo_pred_measure()usessourceto figure out which function called it and checks that thoseNULLs are actuallyNULL. It could be simpler for each wrapper to just handle its own specific part (e.g., getting the PSIS weights, reading the kfold results, etc.) and then call a shared function that only accepts arguments they all have in common. - In
model_compare(), the PSIS diagnostics get turned into strings and then parsed with a regex in the print method. It would be simpler to just printthe flagged rows directly. - The custom SE handling in
model_compare()is spread across.resolve_custom_se_diffs(),inform_missing_custom_se_diff()and part of.pair_measure_stats(). I think it could be one step in the main loop. - A few of the
model_compare()helpers have fallbacks for whenmeasure_infois missing (e.g. in.measure_is_loss(),.measure_pointwise_diff_method(),.measure_se_diff_fun()). But.compare_metadata_check()already has an error when it's missing, and.measure_info()always setslossanddiff_method, so I don't think those fallbacks can actually ever run. Maybe just remove them? And then I thin k.builtin_loss_measures()can also be removed. - The "classic"
looversions ofmodel_compare.default()andcompare_pred_measure()compute a lot of the same things separately (e.g., elpd diffs,p_worse, diagnostics, ordering), and that means some helpers have to have two modes because (model_compare_order(),model_order_stat_check(),model_compare_checks()). But isn't comparing plainlooobjects basically the same as thepred_measurecase with only elpd? I think that means we could we convert them to the same internal format and then we only need one version/path. And we could add a test checking that the output is still exactly the same asloo_compare(). What do you think? - It doesn't look like
subset_measures(),is.loo_pred_measure()and.builtin_loss_measures()are used anywhere. Can they be removed? - There are two different ways deprecations are handled. Most of them use
.Deprecated(), which warns every time it's called . Butloo_compare()uses the.deprecate_once(). Can we use the same thing for all? The PR description says they all warn once per session. .resolve_digits(),.measure_digits(), and.se_digits()could be one function. Also the extra args to.se_digits()aren't used anywhere.test_compare.Ris really long and complicated and a lot of that is from calling internal functions directly, which I don't think we need to do in general. There's also a lot of repeating the same code to buildpred_measureobjects. I think we should test by callingmodel_compareinstead of internals where possible (i.e. test that what the user interacts with always works properly and that means the internals that are actually relevant to that will be tested indirectly), and add a small helper function for creating the test objects. That should make the file a lot shorter and the tests won't need rewriting if the internals are simplified.
Memory
Right now it looks like every LOO result saves the full S x N log_weights matrix, even when save_psis = FALSE. I had Claude check and that means that e.g. for S=400 and N=2000 I get a 61 MB object from loo_pred_measure but only a 0.1 MB object for regular loo(). Then if save_psis=TRUE the weights are saved twice and the object is 122 MB. I think all this could work if the weights are only saved when asked and we can keep them in just one place. But I might be overlooking something, so I could be wrong.
API questions/decisions
- What about using
log_likinstead ofylp? Isn't that the same quantity thatloo_predictive_metric()andloo_crps()already calllog_lik? Or is the change intentional (if so that's fine)? We could also maybe useyrepandepredinstead ofypredandmupredto be consistent with brms and rstanarm, but not a big deal. I don't feel super strongly about that. I could also be misinterpreting the names. - I think in
pred_measure()it would be more natural for the object that is being updated to be the first argument. And in we do that then we could just call itxorobjectinstead ofpredperf, which is pretty standard in R when a function is updating its input object. - When we compute PSIS weights from
ylp,r_effis hard-coded to 1 and there's a message about it every time. What about adding anr_eff = 1argument toloo_pred_measure()like we have forloo()has (and what aboutcorestoo?), and then get rid of the message. We actually got rid of ther_effwarning inloo()a while ago because it was annoying. - What if we stop inheriting the
loo,psis_looandkfoldclasses? Right now the classes a user gets depend on how the result was created, so some loo methods work on some results but not others. Themeasure_*()functions also classesc("measure", "loo"), which I don't think we need/want. measure_brier()has aypredargument, butaccandbacchavemupred. But I think all three of them are taking P(y = 1), right? If so they could have the same argument name unless there's a reason I'm missing.- If there's not a strong reason for exporting it I think we should keep
ptw_log_pred_density()internal because it takes normalized weights, unlike all themeasure_*()functions. loo_pred_measure(loo = )needs a savedpsis_objecteven for the default elpd. But brms hassave_psis = FALSEby default (I think), so this won't work by default with brmslooobjects. Not sure how we want to handle that or if we just accept that.- I think there may be too many informational messages, e.g. "Using psis_object for LOO CV from
predperf". Do we need these? I think we'd only want them if we're doing something the user wouldn't necessarily expect. If we're doing what they'd expect we don't need a message. - Since these days people have autocomplete (if they even write their own code anymore) simplifying
pointwisetoptwseems more confusing to me than helpful. Previously it would be nice to type fewer characters, but now I think clarity matters more. What do you think? I don't feel super strongly about this so fine either way.
model_compare() output
Some thing I didn't think of when reviewing the previous PR but that occur to me now, maybe because I'm seeing it in context, or maybe just because it's a second look after not looking for a while.
model_compare() returns one wide data frame that gets sorted by elpd. But the columns for each measure are relative to the best model for that measure, and the reference model is just in an attribute. I think that's going to be confusing, especially for anyone using the output in code. Here are some other possibilities:
- A long format data frame with columns like
model,measure,reference,diff,se_diffandsign_flipped - A list with of data frames, one per measure
- The same as it is now but with the reference model and whether the sign was flipped in columns instead of attributes
Also, I forget if I commented on this in the other PR or not. When elpd isn't included, the ranking just uses the measure that happens to be first alphabetically instead of the order the user listed the measures in. I think users would expect the latter.
Also I don't think model_compare.psis_loo_ss_list() needs to be an exported S3 method.
Input validation checks
- There's no check that
nis the same so e.g.ylpcould be passed as its transpose and not error. - In-sample and holdout results (and LOO results computed from
ylporpsis_object) don't save ayhash, somodel_compare()won't warn if the models were fit to differenty. Do we want that? - Custom measure names don't get validated (see my comment on
custom_measure()). - There's no check that
psis_objectis actually a psis object.
Repo contents / cleanup
- I think we can remove the
notes/directory now, or move anything that's still useful somewhere else. - There's some stuff in
.Rbuildignoreand.gitignorethat could go in your global gitignore instead, although I think we can keep the agents and claude stuff, that's useful for everyone test_data_generation.Rinstalls packages. I think we should move the data generation script todata-raw/and remove the install code- The tarball size doubled (2.09 MB to 4.14 MB). We're still under CRAN's limit, so I guess it's ok. But it means we don't have much room after this to add things that increase the size.
- The pkgdown workflow installs brms from GitHub master, so the development version. Do we want to use that version or the CRAN version of brms? Not sure if @florence-bockting or @VisruthSK changed that. If there's a reason then fine to keep it.
Doc and NEWS
-
There are still a bunch of TODOs in the online articles and some spots in the doc still say to set attributes directly instead of using
custom_measure. -
NEWS is missing subsampled
loo_compare()returning a data frame instead of a matrix -
In the rest of the package we use
Nfor the number of observations and themodel_compare()docs and vignette in the PR also useN. But thepred_measuredocs and the twopred_measurevignettes usen. Can you switch those toN? There's also one message in the code that usesn(see my inline comment).
Tests
- See comment above in the simplifications section.
- When I run the tests there are a lot of warnings that aren't caught that could be caught so the test output is cleaner.
- Some new functions that should give the same answer as deprecated ones don't have a test checking that.
|
Not sure why but it looks like my long main comment was posted twice. Also a lot of the comments on individual lines of code are hidden because there are so many of them |
|
I think brms being installed from master was because of this line in the PR description. Though maybe that's out of date now?
|
Thanks, that makes sense. We should probably put a comment in that file next to where brms is installed. |
Fixes #281
Fixes #223, #213, #201, #135, #220
Summary
This PR adds a predictive performance API. It computes density scores, scoring rules, and point-prediction metrics under four evaluation sources: in-sample, PSIS-LOO, k-fold, and holdout (test data).
insample_pred_measure(),loo_pred_measure(),kfold_pred_measure(),test_pred_measure(), andpred_measure().measure_*().supported_measures_list()lists them.measure_rps()andmeasure_srps()use oneypreddraw matrix and a PWM estimator ((loo_)(s)crps could ask for only one argument with predictions #223).loo_pred_measure()accepts a pre-computedpsis_object(loo_predictive_metric and loo_crps could accept psis objects to reduce computation #213).model_compare()compares models on all measures (loo_compare for crps and loo_crps #220). It comes from Introducemodel_compare()for model comparison with support of new predictive measures #380, which is merged into this branch.elpd(),crps(),scrps(),loo_crps(),loo_scrps(),loo_predictive_metric(), andloo_compare()are deprecated. They still work with their current APIs.The comment below lists all new features and functions. It marks the parts that come from #380.
How to navigate this PR
notes/developer-notes.md: status, design decisions, numerical comparisonsnotes/design-discussions/model_compare.md: review decisions formodel_compare()(Introducemodel_compare()for model comparison with support of new predictive measures #380)vignettes/articles-online-only/overview-measures.Rmd: supported measures and their definitionsvignettes/articles-online-only/pred-measure-workflow.Rmd: end-to-end workflowvignettes/articles-online-only/model-comparison.Rmd: Model comparison: Explanation ofmodel_compare()(Introducemodel_compare()for model comparison with support of new predictive measures #380)vignettes/migration-guide.Rmd: deprecated functions and their replacementsDeprecations (implementations retained)
elpd()measure_elpd()/*_pred_measure()crps(),scrps()(x,x2)measure_rps(),measure_srps()(ypred)loo_crps(),loo_scrps()loo_pred_measure(..., measure = "rps" | "srps")loo_predictive_metric()loo_pred_measure()loo_compare()(#380)model_compare()Each deprecated function warns once per session.
Dependencies
Vignettes, test-data generation, and pkgdown CI install brms from GitHub
masteras k-fold support for categorical models (brms#1890) is not yet on CRAN. Remove the pin after the next brms CRAN release.Known limitations
group_idsgrouping is not implemented.Acknowledgements
This work builds on a prototype by @VisruthSK (VisruthSK/loo-sandbox) under guidance from @jgabry.