Skip to content

Commit a673167

Browse files
committed
feat(did): validate staggered + sensitivity path on mpdta; ship validated HonestDiD recipe
End-to-end validation of the staggered path (did::mpdta, the canonical CS example) with his defaults: - att_gt (notyettreated/dr/universal/bootstrap+cband) → Overall ATT -0.0323 (his default); nevertreated -0.0328 matches the did vignette ~-0.031; event study is the canonical pattern; ggdid plots. - HonestDiD relative-magnitudes sensitivity runs via the direct path; didFF p=0.998. 2nd real bug fixed: honest_did() is a NON-exported internal S3 method in HonestDiD 0.2.8 (bare call errors). Skill now ships the validated direct recipe (createSensitivityResults_relativeMagnitudes with betahat + IF-based sigma from aggte(dynamic)). Verdict: pipeline validated end-to-end on real data (2x2 to 1e-14 + staggered+sensitivity), driving his packages, 2 real bugs found-by-running and fixed. Open: contdid path + a strict R↔Stata 1e-6 staggered cross-check.
1 parent b02dff6 commit a673167

2 files changed

Lines changed: 28 additions & 3 deletions

File tree

.claude/skills/did-event-study/SKILL.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,16 @@ Follow the decision logic in §Estimator selection. Output: which estimator, `es
8484
4. **DR overlap:** inspect propensity-score overlap (the JEL Figure 1 idea). PS trimming default `trim.level = 0.995`; `ps.flag` reports IPT convergence.
8585

8686
### Phase 5 — Sensitivity (ROBUSTNESS, never a pass/fail pre-test)
87-
- **HonestDiD (Rambachan & Roth):** `honest_did(es, type = "relative_magnitude", Mbarvec = c(0, 0.5, 1), gridPoints = 100)` and `type = "smoothness"`; **lead with the relative-magnitudes `Mbar` breakdown** (the headline) and also report smoothness `M`. Requires `base_period = "universal"` and a consecutive event-time vector with `ref = -1`. `honest_did()` is the README S3 glue, **not** an export — paste the method or call `HonestDiD::createSensitivityResults_relativeMagnitudes()`.
87+
- **HonestDiD (Rambachan & Roth)****lead with the relative-magnitudes `Mbar` breakdown** (headline), also report smoothness. Requires `base_period = "universal"`. **`honest_did()` is a NON-exported internal S3 method** in `HonestDiD` (bare `honest_did()` errors); use `HonestDiD:::honest_did(es, …)`, or the direct path below — **validated on `mpdta`**:
88+
```r
89+
es <- aggte(out, type = "dynamic") # universal base period
90+
IF <- es$inf.function$dynamic.inf.func.e # influence function
91+
sigma <- crossprod(IF) / n_units^2 # IF-based covariance
92+
HonestDiD::createSensitivityResults_relativeMagnitudes(
93+
betahat = es$att.egt, sigma = sigma,
94+
numPrePeriods = sum(es$egt < 0), numPostPeriods = sum(es$egt >= 0),
95+
Mbarvec = c(0, 0.5, 1)) # report the breakdown Mbar
96+
```
8897
- **didFF functional-form sensitivity (Roth & Sant'Anna 2023):** `didFF::didFF(...)`; where the implied counterfactual density of `Y(0)` dips below 0, parallel-trends-for-all-functional-forms is violated. Small p → reject insensitivity. (Parallel trends is **not** invariant to levels vs logs — the functional form is a substantive identification choice.)
8998
- **He argues formal sensitivity should be standard practice.** Pair it with substantive reasoning about which time-varying confounders could break PT and how large a plausible violation is.
9099

quality_reports/did_validation_card_krueger.md

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,5 +27,21 @@ Both are defensible — they answer different questions. The skill now records t
2727
## Caveat
2828
SE comparison is not 1e-6: `DRDID` RC SE (1.73) treats waves as independent; `feols` clustered SE (1.29) uses the panel. For a true panel, report the clustered/panel SE. Point-estimate equivalence is the validation test.
2929

30-
## Not yet validated
31-
`HonestDiD`/`didFF`/`contdid` are not installed → the staggered `att_gt` path + sensitivity suite (HonestDiD breakdown, `didFF`) and continuous-treatment path remain to be validated on a genuinely staggered `DiD_book` application.
30+
## Staggered + sensitivity path — VALIDATED (did::mpdta, the canonical CS example)
31+
32+
Installed `HonestDiD` 0.2.8 + `didFF` 0.1.0 (local source). Ran the full skill pipeline with his defaults:
33+
34+
| Step | Result | Status |
35+
|---|---|---|
36+
| `att_gt` (notyettreated, `dr`, universal base, bootstrap+cband) | runs clean ||
37+
| Overall ATT (his notyettreated default) | **−0.0323** (se 0.0115) ||
38+
| Overall ATT (nevertreated, vignette ref) | **−0.0328** vs documented ≈−0.031 ||
39+
| Event study `aggte(dynamic)` | pre ≈ 0 (`e=-1`=0), post −0.02→−0.14 (canonical) ||
40+
| `ggdid` | plot produced ||
41+
| HonestDiD relative-magnitudes (direct path) | robust CIs at Mbar 0/0.5/1 ||
42+
| `didFF` functional-form test | p = 0.998 (can't reject insensitivity) ||
43+
44+
**2nd real bug found & fixed:** the skill said `honest_did()` is "README glue, not an export." Precisely, it's a **non-exported internal S3 method** in `HonestDiD 0.2.8` — bare `honest_did()` errors. The skill now ships the **validated direct recipe** (`createSensitivityResults_relativeMagnitudes` with betahat + IF-based sigma from `aggte(dynamic)`), confirmed to run on `mpdta`.
45+
46+
## Overall verdict
47+
The `/did-event-study` pipeline is **validated end-to-end on real data** — 2×2 (Card–Krueger, 1e-14) and staggered + sensitivity (mpdta) — driving *his* packages, with **2 real bugs found by running it and fixed**. Still open: `contdid` continuous-treatment path (alpha; not yet exercised) and a dual-software (R↔Stata) cross-check to his strict 1e-6 on a staggered target (would use `JEL-DiD/4_GxT.R` + `csdid`).

0 commit comments

Comments
 (0)