Skip to content

Add analytic SE and CI for IPW estimator via influence function#1641

Open
genrichez wants to merge 3 commits into
py-why:mainfrom
genrichez:feature/ipw-analytic-se
Open

Add analytic SE and CI for IPW estimator via influence function#1641
genrichez wants to merge 3 commits into
py-why:mainfrom
genrichez:feature/ipw-analytic-se

Conversation

@genrichez

Copy link
Copy Markdown

Adds analytic standard errors and confidence intervals for PropensityScoreWeightingEstimator using the influence function approach from Lunceford & Davidian (2004).

Problem
The IPW estimator has no _estimate_std_error or _estimate_confidence_intervals. When a user asks for CIs, the base class silently falls back to a bootstrap. This is slow and the user never asked for it.

Solution
The estimator computes a Hajek (ratio) weighted mean in each treatment arm. The influence function for this is:
IF_i = (T_i/e_i) * (Y_i - mu1) / (sum(T/e)/n) - ((1-T_i)/(1-e_i)) * (Y_i - mu0) / (sum((1-T)/(1-e))/n)

Then SE = sqrt(Var(IF) / n) and CI = estimate +/- z * SE.
I added:
_compute_influence_function() - computes the per-unit IF for ATE, ATT, and ATC
_estimate_std_error() - returns sqrt(Var(IF)/n)
_estimate_confidence_intervals() - returns estimate +/- z * SE
The stored arrays (_ipw_T, _ipw_Y, _ipw_ps) are saved during estimate_effect so the IF can be computed later without re-fitting.
Important detail: Hajek vs Horvitz-Thompson
All weighting schemes here are actually ratio estimators because the code divides by sum_dy_weights:

est = data["d_y"].sum() / sum_dy_weights - data["dbar_y"].sum() / sum_dbary_weights

This is sum(T*Y/e)/sum(T/e) - ..., which is the Hajek form. Using the simpler Horvitz-Thompson IF (which assumes (1/n)*sum(...) without normalization) gives SE about 3x too large. The implementation uses the correct ratio-linearized IF for all schemes.
Tests
Added 13 new tests covering:

  1. SE is positive and finite
  2. CI covers the true ATE
  3. CI width shrinks with sample size
  4. SE scales as 1/sqrt(n)
  5. Analytic SE is close to bootstrap SE
  6. All 3 target units (ATE, ATT, ATC)
  7. All 3 weighting schemes
  8. Influence function has mean near zero
  9. Heterogeneous effects: ATT differs from ATE with positive selection
  10. Existing 2 tests pass unchanged.

Known behavior
The IF-based SE is slightly conservative (about 30% wider than the true sampling variability) because it treats propensity scores as known. This is standard - same behavior as R's WeightIt and Stata's teffects ipw. It means the CI has over-coverage (safe) rather than under-coverage.
Reference
Lunceford, J.K. & Davidian, M. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statistics in Medicine, 23(19), 2937-2960.
Closes #1640

@github-actions

This comment has been minimized.

@genrichez

Copy link
Copy Markdown
Author

Thanks for checking it out, fixed in the latest commit along with the other suggestions

genrichez added 2 commits July 5, 2026 11:19
The PropensityScoreWeightingEstimator had no _estimate_std_error or
_estimate_confidence_intervals, so CI requests silently fell back to
bootstrap. This adds the closed-form influence-function-based variance
from Lunceford & Davidian (2004).

All weighting schemes in this estimator are ratio (Hajek) estimators
because estimate_effect divides by sum of weights. The IF accounts for
this ratio structure. Covers ATE, ATT, and ATC targets.

Includes 13 new tests validating against bootstrap, checking sqrt(n)
scaling, coverage, and all scheme/target combinations.

Signed-off-by: genrichez <22434764+genrichez@users.noreply.github.com>
- Fix ATT control-arm IF normalizer: use mean((1-T)*e/(1-e)) instead
  of mean(T). The control counterfactual is a ratio with its own
  denominator, not the treatment proportion.
- Apply the same fix to ATC treated-arm IF for consistency.
- Add hasattr guard in _estimate_std_error so calling it before
  estimate_effect gives a clear RuntimeError instead of AttributeError.
- Add comment noting that DataFrame target_units uses the population
  ATE influence function (per-unit custom IF not yet supported).
- Add comment explaining why we use sum(IF^2)/n^2 (asymptotically
  equivalent to var(IF)/n since IF has mean zero by construction).

Signed-off-by: genrichez <meltsefon1@gmail.com>
Signed-off-by: genrichez <22434764+genrichez@users.noreply.github.com>
@genrichez
genrichez force-pushed the feature/ipw-analytic-se branch from fd37f48 to 17bf69e Compare July 5, 2026 11:19
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Note

The pull request was not created — a fallback review issue was created instead due to protected file changes: #1646

🤖 This is an automated response from Repo Assist.

Thanks for the swift fixes, @genrichez! I've reviewed the updated commit and everything looks correct:

  • ATT control-arm normalizer ✅ — now correctly uses sum_w0_att / n instead of p, matching the ratio-linearization derivation
  • ATC treated-arm normalizer ✅ — symmetrically uses sum_w1_atc / n
  • hasattr guard ✅ — clear error message when estimate_effect hasn't been called first
  • DataFrame target_units comment ✅ — clearly documents the population-ATE fallback

The math is solid across all three target units and the tests are comprehensive. This PR is in great shape! 🎉

Generated by 🌈 Repo Assist, see workflow run. Learn more.

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repo-assist.md@11c9a2c442e519ff2b427bf58679f5a525353f76

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PropensityScoreWeightingEstimator: no analytic standard errors (silent bootstrap fallback)

1 participant