Repository navigation
Adding pred_measure feature
#363
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: loo-v3.0.0
Are you sure you want to change the base?
Changes from all commits
1c5d871
bad8062
6d404e1
f5854c2
c42c803
7750200
1926429
02ecbbb
a24fee1
3a670cf
439867e
883dda0
755984d
b260a32
082d7be
e394c30
e9f1a3a
cf0c79b
5cb9f10
562b141
023add0
8346344
e99aeac
884df6e
b924ac5
89d5e9a
7ce2d50
7394ba3
494d24f
9611656
a3019e3
f64cf6c
441f0ea
b0015c7
076e2c7
3e1bf93
06d6281
b46ff58
f1fcaf0
674746e
e17e46d
abc74b5
faa27c5
3e078aa
a1e66e5
0d1e072
8897e1f
1eb66de
166dab6
c0c1fc0
c594818
aa30e54
b1f0b32
63c16a8
d874044
9e930e3
4fe4b0d
88890b6
51b3c9c
81ae8eb
fed3433
585b77c
3f63357
5307f23
b0f94cb
7184cae
91f251a
757021e
f08afcc
806600b
0c85ba8
d0b52fa
1b1d050
8c46b00
e74df04
b3a7db7
12133b8
bd67f6a
ef914d8
f79c4f8
4882727
ffa1786
fa95ff7
29072e6
294f87a
8a9a8e2
6db2531
f0715be
7ffedf6
4dd743c
0a3ecbb
919b29b
b5952b7
795cd4e
69346f3
e52ab8e
d55d9bf
0f830e3
769b37a
b80c946
ca8da41
47af38e
99d71a2
0e07f63
4317ce7
0c1de0c
c66e1e2
5f6bcb9
6dbd05d
91bfdba
6621d2d
0944a95
0140dc1
67f4d12
e339f48
4c5a1a8
791ed25
dc49cd1
cb93257
22d1422
c1d75f8
0e7d193
aa7f286
6971e13
8815998
eec2140
dcf5fde
ba29be1
c5af96b
9ffff9c
dd2ac81
e372858
36795c2
999cfa7
de54993
cd1085d
a18b16e
954bdb1
6341145
390eeb1
2a77502
fbe6e88
71b23c2
ee7b99b
fd427d9
f7a7b4d
a509447
102a76b
d03261f
18fd34b
7573b3d
07d68c8
d958069
2af1cd1
1ac844b
fa986da
706bc0f
88afda5
0bd95d6
c9f72c6
2041008
f26eb02
1637ede
c3c940e
3328f5c
a70f40d
3764847
438f54f
1ada0a3
b427aa8
7002815
959c057
ffae11e
988d798
db2279c
8c581a5
749d421
5481b19
fea24ad
e28acd6
e1dca92
3fc1580
d627dfd
e05a938
de81c46
bbbdc7e
86f47b3
b6c4cb1
10e406a
f719344
e6bad2a
4597a83
12aa948
eaebc86
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -36,6 +36,9 @@ jobs: | |
| - uses: r-lib/actions/setup-r@v2 | ||
| with: | ||
| use-public-rspm: true | ||
| extra-repositories: | | ||
| https://community.r-multiverse.org | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @VisruthSK why do we need r-multiverse here? I think we only need the stan-dev r-universe for cmdstanr. |
||
| https://stan-dev.r-universe.dev | ||
|
|
||
| - uses: r-lib/actions/setup-r-dependencies@v2 | ||
| with: | ||
|
|
@@ -45,7 +48,8 @@ jobs: | |
| local::. | ||
| any::bayesplot | ||
| any::BH | ||
| any::brms | ||
| paul-buerkner/brms | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This just installs whatever the latest development version of brms is, I'm not sure that's what we want. Or is it? |
||
| any::cmdstanr | ||
| any::ggplot2 | ||
| any::rmarkdown | ||
| any::Rcpp | ||
|
|
@@ -58,6 +62,7 @@ jobs: | |
| any::StanHeaders | ||
| any::knitr | ||
| any::withr | ||
| any::lme4 | ||
| stan-dev/pkgdown-config | ||
|
|
||
| - name: Build site | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,11 +19,17 @@ vignettes/loo2-non-factorizable_cache/* | |
| vignettes/*.html | ||
| vignettes/*.pdf | ||
| inst/doc | ||
| *.html | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we need to ignore the html files? |
||
|
|
||
| revdep/* | ||
| tests/testthat/Rplots.pdf | ||
|
|
||
| cran-comments.md | ||
| CRAN-RELEASE | ||
| release-prep.R | ||
|
|
||
| # personal maintainer scratch (not shared) | ||
| internal-notes/ | ||
| notes/loo_se.pdf | ||
| notes/loo-compare-se-diff.md | ||
|
Comment on lines
+31
to
+34
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See my general comments about cleaning up the gitignore and rbuildignore file |
||
| CRAN-SUBMISSION | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,15 @@ | ||
| #' Continuously ranked probability score | ||
| #' Continuously ranked probability score (deprecated) | ||
| #' | ||
| #' As of loo 3.0.0, `crps()`, `scrps()`, `loo_crps()`, and `loo_scrps()` are | ||
| #' **deprecated**. Please use [measure_rps()] and [measure_srps()] instead, | ||
| #' or [loo_pred_measure()] for leave-one-out predictive performance. | ||
| #' See `vignette("migration-guide", package = "loo")` for a full mapping table. | ||
| #' | ||
| #' @details | ||
| #' The new functions use a single matrix of posterior predictive draws | ||
| #' (`ypred`) and a PWM estimator. The deprecated functions require two | ||
| #' independent draw matrices (`x` and `x2`) and use a permutation-based | ||
| #' estimator. See [measure_rps()] for migration details. | ||
| #' | ||
| #' The `crps()` and `scrps()` functions and their `loo_*()` counterparts can be | ||
| #' used to compute the continuously ranked probability score (CRPS) and scaled | ||
|
|
@@ -34,16 +45,19 @@ | |
| #' The former reports estimator and standard error and latter the pointwise | ||
| #' values. Following Bolin & Wallin (2023), a larger value is better. | ||
| #' | ||
| #' @seealso [measure_rps()], [measure_srps()], [loo_pred_measure()] | ||
| #' | ||
| #' @examples | ||
| #' \dontrun{ | ||
| #' # An example using rstanarm | ||
| #' # Deprecated: | ||
| #' library(rstanarm) | ||
| #' data("kidiq") | ||
| #' fit <- stan_glm(kid_score ~ mom_hs + mom_iq, data = kidiq) | ||
| #' ypred1 <- posterior_predict(fit) | ||
| #' ypred2 <- posterior_predict(fit) | ||
| #' crps(ypred1, ypred2, y = fit$y) | ||
| #' loo_crps(ypred1, ypred2, y = fit$y, log_lik = log_lik(fit)) | ||
| #' # -> | ||
| #' measure_rps(y = fit$y, ypred = ypred1) | ||
| #' } | ||
| #' | ||
| #' @references | ||
|
|
@@ -80,6 +94,7 @@ loo_scrps <- function(x, ...) { | |
| #' @rdname crps | ||
| #' @export | ||
| crps.matrix <- function(x, x2, y, ..., permutations = 1) { | ||
| .Deprecated("measure_rps") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See my general comment/question about |
||
| validate_crps_input(x, x2, y) | ||
| repeats <- replicate(permutations, EXX_compute(x, x2), simplify = F) | ||
| EXX <- Reduce(`+`, repeats) / permutations | ||
|
|
@@ -114,6 +129,7 @@ loo_crps.matrix <- | |
| permutations = 1, | ||
| r_eff = 1, | ||
| cores = getOption("mc.cores", 1)) { | ||
| .Deprecated("loo_pred_measure") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See my general comment/question about |
||
| validate_crps_input(x, x2, y, log_lik) | ||
| repeats <- replicate(permutations, | ||
| EXX_loo_compute(x, x2, log_lik, r_eff = r_eff, ...), | ||
|
|
@@ -128,6 +144,7 @@ loo_crps.matrix <- | |
| #' @rdname crps | ||
| #' @export | ||
| scrps.matrix <- function(x, x2, y, ..., permutations = 1) { | ||
| .Deprecated("measure_srps") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See my general comment/question about |
||
| validate_crps_input(x, x2, y) | ||
| repeats <- replicate(permutations, EXX_compute(x, x2), simplify = F) | ||
| EXX <- Reduce(`+`, repeats) / permutations | ||
|
|
@@ -156,6 +173,7 @@ loo_scrps.matrix <- | |
| permutations = 1, | ||
| r_eff = 1, | ||
| cores = getOption("mc.cores", 1)) { | ||
| .Deprecated("loo_pred_measure") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See my general comment/question about |
||
| validate_crps_input(x, x2, y, log_lik) | ||
| repeats <- replicate(permutations, | ||
| EXX_loo_compute(x, x2, log_lik, r_eff = r_eff, ...), | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my general comments about cleaning up the gitignore and rbuildignore file