Repository navigation
Conversation
… clearing it
With n_bootstrap > 0, CallawaySantAnna cleared event_study_vcov at fit time
and in the post-fit aggregate("event_study") replay, so HonestDiD fell back
to diag(se**2) and its Delta^SD intervals under-covered (#836). The clearing
kept the analytical VCV away from percentile-bootstrap SEs, but the
multiplier bootstrap already holds a consistent covariance: the event-study
SEs are np.std(draws, ddof=1) of es_boot_matrix, so the covariance of the
same draws has se**2 on its diagonal by construction.
CSBootstrapResults now carries that covariance over the horizons with a
finite bootstrap SE, and fit() and the replay (CallawaySantAnna and DMLDiD)
hand it on. It is not stored when n_bootstrap does not exceed the number of
horizons (B draws give rank <= B - 1), when a draw is non-finite, or with
fewer than two PSUs; consumers then keep the diagonal fallback. PreTrendsPower
and the DiagnosticReport parallel-trends check now read the same matrix.
With fewer PSUs than event-study horizons the bootstrap covariance is singular, as the analytical one is, so HonestDiD's container route now raises its singular-covariance ValueError where it used to fall back to the diagonal. State it under Behavioral Changes and in the CallawaySantAnna REGISTRY note.
…e the container route
HonestDiD.fit(results) on a CallawaySantAnnaResults object assigned the
stored event_study_vcov unvalidated, while the aggregate('event_study')
container route ran _validate_vcov_subblock(..., allow_singular=False).
With fewer PSUs than horizons the stored matrix (analytical or bootstrap)
is singular, so the native route returned finite bounds where the
container route raised. Run the same check on the native route.
- tests: singular few-PSU regression on both routes (analytical and
bootstrap); the survey-df test moves from 4 PSUs (df=2, rank-2 VCV
over 6 horizons - singular) to 10 nested PSUs (df=8).
- docs: REGISTRY CS/HonestDiD notes, changelog fragment, and the
practitioner guide line that still described the diagonal fallback on
bootstrapped CS containers.
…ce, not vcov=None
Overall assessment✅ Looks good Executive summary
Methodology
The implementation is internally consistent: covariance and SE use the same draws and Code QualityNo findings. Fit-time and replay paths share the same bootstrap result fields and override helpers. PerformanceNo findings. The added covariance calculation is small relative to the existing multiplier-bootstrap draw generation. MaintainabilityNo findings. Covariance provenance and indexing are explicit, and replay propagation is centralized at Tech DebtNo untracked issues introduced. The documented low-draw and unavailable-covariance fallbacks remain explicit. SecurityNo findings or apparent secrets/PII. Documentation/TestsNo findings. Tests cover base-period and anticipation variants, insufficient draws, native/container parity, singular few-PSU designs, DML replay, and DiagnosticReport routing. A Test execution was unavailable because |
2 participants
Summary
maintainerCanModifyis on) and fix(staggered): carry the bootstrap event-study covariance instead of clearing it #839 merges; this PR then closes without merging. Endgame check: tree-hash equality between the two heads.70afb12,63aad7d): bootstrappedCallawaySantAnnafits (andDMLDiDreplays) carry the covariance (ddof=1) of the event-study multiplier-bootstrap draws asevent_study_vcovinstead of clearing it. Its diagonal is the bootstrapse**2by construction (the SEs arenp.std(draws, ddof=1)of the same matrix), so HonestDiD, PreTrendsPower and DiagnosticReport read a full covariance consistent with the reported SEs instead ofdiag(se**2). Not stored (diagonal fallback kept) whenn_bootstrap <= n_horizons(rank-deficient by construction), a draw is non-finite, or fewer than 2 PSUs.52238be: the nativeHonestDiD.fit(results)route assignedevent_study_vcovunvalidated while theaggregate('event_study')container route ran_validate_vcov_subblock(..., allow_singular=False); with fewer PSUs than horizons the native route returned finite bounds from a singular matrix where the container raised. Pre-existing on analytical fits; now both routes validate. Also fixes the stalellms-practitioner.txtline.6482556:docs/v4-deprecations.yamlM-020 note still describedvcov=Noneon bootstrap replay.Problem (#836)
Issue reproducer and an independent Monte Carlo (Rust backend, 100 units, 500 panels, Delta^SD(M=0) interval for the average post-period effect, nominal 95%):
n_bootstrap=0n_bootstrap=199Bootstrap vs analytical event-study correlations agree to 0.06 at 999 draws.
Methodology references (required if estimator / math changes)
honest_did.AGGTEobjbuilds Sigma from the influence-function matrix even whendidreports bootstrap SEs; here Sigma on a bootstrapped fit is the covariance of the bootstrap draws, so its diagonal matches the reported SEs. Documented in REGISTRY (new CallawaySantAnna**Note:**, updated HonestDiD**Note (deviation from R):**, PreTrendsPower Sigma_22 note, post-fitaggregate()note).Validation
tests/test_staggered.py(draw-covariance contract across varying/universal/anticipation bases;n_bootstrap <= n_horizonsnot stored),tests/test_honest_did.py(bootstrap vcov carried; native/container parity; below-draw-limit diagonal fallback; NEW singular few-PSU case raises on both routes for analytical and bootstrap; survey-df test moved from 4 to 10 nested PSUs because its old design was intrinsically singular - survey ES vcov rank <= df = 2 over 6 horizons),tests/test_aggregate_contract.py(replay vcov parity with fit time; consumers read the full covariance),tests/test_diagnostic_report.py(bootstrapped CS now takesjoint_wald_event_study),tests/test_dml_did.py(replay carries the covariance).premerge_scan.py0 findings.Security / privacy
Changelog
changelog.d/20261005-cs-bootstrap-event-study-vcov.md(Fixed + Behavioral Changes).