[Jodielidada]

Changes from our October 27 meeting.

Vignette

• The title is "LEARNER: Separate Penalties, Multiple Sources, and Diagnostics".
• The sections are 1. Base Case: D-LEARNER and LEARNER with One Source, 2. Separate U- and V-Space Penalties, 3. Multiple Sources and Source Weights, and 4. Diagnostics and Sensitivity Checks. Source weights are part of the multiple-source section. Separate penalties come before multiple sources.
• Removed comparisons of estimation accuracy. CV scores are retained to explain parameter selection and diagnostics.
• The dat_asymmetric help page example now fits once and prints the dimensions and the stopping code, and the error comparison is removed.
• Diagnostics is the last section. It checks whether the penalties chosen by CV are on the boundary of their grids, plots the objective history, and runs a sensitivity check.
• In the multiple-source example both chosen penalties are at the upper end of their grids, and the text says so.
• Reduced the sensitivity check to three settings and included its output in the vignette. Users can expand these settings for their own data.

cv.learner() and plot_cv()

• cv.learner() now returns the candidate grids (lambda_1_all and lambda_2_all, or lambda_1_row_all, lambda_1_col_all and lambda_2_all for separate penalties) and n_observed.
• plot_cv(cv) uses them, so no extra arguments are needed. The old arguments still work and take priority, so results from earlier versions can still be plotted.
• The README and the example scripts no longer pass the grids.
• The heatmap goes from pale blue (low error) to dark blue (high error). It used the magma scale before.

Stopping criteria

control now takes stopping_rule, either "absolute" (the default, unchanged) or "relative".

• With "relative", a change in the objective is abs(L_t - L_{t-1}) / abs(L_{t-1}), and the fit stops when the last window changes are all below threshold.
• window is k and threshold is epsilon. k and epsilon also work as names. Giving both names for one setting is an error.
• For the relative rule they default to 5 and 1e-6, and I did not tune them further. For the absolute rule they default to 1 and 0.001, which is the old behavior. window works with both rules, so window = 1 compares only adjacent values.
• max_iter stays separate. The check for an objective that grows too much (code 3) is kept. With the relative rule it runs before the convergence test, so a large threshold cannot hide a diverging fit. A threshold of 0 disables convergence stopping; the fit continues to max_iter unless a failure safeguard triggers code 3.
• The option is described in ?learner and NEWS.md. The vignette does not use it. I called the rule "relative", and that name is easy to change.
• Calls that do not use the new settings give the same results as before. I compared 22 single-source fits with the released 1.0.0 and 181 fits with the build before this PR, and all of them are identical.
• On dat_asymmetric with lambda_1 = 0 I multiplied the data by 0.1, 1 and 10. The relative rule stops after 784 updates each time. The absolute rule stops after 656 and 770 updates, and reaches the limit of 5000 at the largest scale.

Also changed

• The part of the diagnostics section that refits the chosen penalties on the CV folds and shows their stopping codes now runs when the vignette is built, so its output is on the page. It was not run before.
• The multiple-source example says that 4 of the 18 internal CV fits reach the iteration limit.
• The version is now 1.1.0.9000, so a development build can be told apart from 1.1.0.
• The Installation section and the first check in the vignette now point to devtools::install_github("stmcg/learner"), and so does the header of the script. This only works once these changes are in main. To try this branch before that, use devtools::install_github("stmcg/learner@stopping-rule-option").
• Each code block in the vignette has a Copy button (vignettes/copy-code.html), and inst/examples/advanced-learner.R has the same code as the blocks. The figures have alt text.
• test-cv-multisource.R and test-reference-regression.R are adjusted for the new fields in the CV result. The helper used by the stopping tests moved to tests/testthat/helper-stopping.R. There are new tests for the stopping option and for the stored CV grids.

Questions

• I kept the old rule as the default and added the relative rule as an option. Would you like the relative rule to become the default later? I am happy to change it if you think it should.
• The default max_iter is still 100. That is small for both rules on the example data, and the help page says so. Would you like me to raise the default, or is it better to leave it as it is?

Checks

R CMD check --no-manual gives Status OK on macOS (R 4.6.1), all 494 expectations pass, and the vignette builds in about 12 seconds. I have not checked the PDF manual or other platforms. GitHub Actions should cover those. I also looked through the examples on the other help pages, and none of them compares performance.

Please merge with a regular merge commit.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.16%. Comparing base (87c386c) to head (b21bfe2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #8      +/-   ##
==========================================
+ Coverage   99.11%   99.16%   +0.05%     
==========================================
  Files           7        7              
  Lines         562      601      +39     
==========================================
+ Hits          557      596      +39     
  Misses          5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.