Docs: Add Synthetic Control sensitivity checks walkthrough#871
Open
drbenvincent wants to merge 4 commits into
Open
Docs: Add Synthetic Control sensitivity checks walkthrough#871drbenvincent wants to merge 4 commits into
drbenvincent wants to merge 4 commits into
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
drbenvincent
added
documentation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #871 +/- ##
=======================================
Coverage 94.59% 94.59%
=======================================
Files 80 80
Lines 12764 12764
Branches 770 770
=======================================
Hits 12074 12074
Misses 486 486
Partials 204 204 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This was referenced Apr 30, 2026
2 tasks
Collaborator
Author
|
The failing It's an upstream regression in Once #878 lands I'll rebase this branch onto main so |
Adds a sensitivity-analysis section to sc_pymc.ipynb that walks through the pipeline API (EstimateEffect -> SensitivityAnalysis -> GenerateReport) with PlaceboInTime as the SC default check, and documents the other SC-applicable checks (ConvexHullCheck, LeaveOneOut, PlaceboInSpace, PriorSensitivity) with interpretation guidance. Mirrors the structure introduced for Staggered DiD in #846 and links back to the central guide from #818. Includes a runnable pipeline cell and HTML report iframe so the walkthrough demonstrates an end-to-end placebo-in-time check with concrete pass/fail narrative and a five-step "if this check fails" troubleshooting block. Closes #789 Made-with: Cursor
Manually rerun so the Synthetic Control sensitivity walkthrough renders correctly in the built docs. Made-with: Cursor
- Promote PlaceboInTime so its Walkthrough and "If this check fails"
subsections sit at #### beneath it, instead of competing with it at
the same heading level.
- Rewrite the pass/fail interpretation to explain why this specific
dataset produces a NOT SUPPORTED verdict (fold 1 has only 12
pre-treatment observations to fit 7 donors, blowing up the
hierarchical null's tau).
- Add a :::{note}::: callout describing the placebo-fold-too-early
failure mode and linking to issue #875 for the upstream fix.
- Seed PlaceboInTime's hierarchical-null sampler via sample_kwargs so
the printed P value is reproducible across runs.
Refs #789, #875
Made-with: Cursor
The PlaceboInTime verdict on this dataset is borderline rather than a clean fail, and small MCMC sampling noise (issue #876) can flip it either side of the 0.95 threshold between runs. Update the docs to treat it as such instead of asserting a specific verdict: - Rewrite "Interpreting the result on this dataset" to describe the outcome as borderline and explicitly point at the reproducibility note, instead of asserting NOT SUPPORTED. - Drop the over-specific sd numbers (sd≈30, sd≈9) so the narrative stays consistent if MCMC noise shifts them slightly. - Expand the :::note::: callout into "Known limitations being tracked" with two bullets: #875 (placebo windows that land too early) and the newly-filed #876 (unseeded pm.sample_posterior_predictive in the hierarchical-null fit). No code or executed-output changes; the rendered cell still prints P(actual outside null) = 0.923 and NOT SUPPORTED, which the new text treats as one borderline draw rather than the headline verdict. Refs #789, #875, #876 Made-with: Cursor
drbenvincent
force-pushed
the
docs/789-sc-sensitivity-walkthrough
branch
from
86c4e7e to
d15eb3f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a focused sensitivity-check walkthrough for Synthetic Control to the
sc_pymc.ipynbnotebook, addressing #789 (follow-up to #749 / #785).This continues the docs work started in #818 (central guide) and #846 (Staggered DiD walkthrough), keeping the same structure:
EstimateEffect→SensitivityAnalysis→GenerateReport) and linking back to the central guide andpipeline_workflow.PlaceboInTime,ConvexHullCheck,LeaveOneOut,PlaceboInSpace,PriorSensitivity) with one-line interpretations and noting which checks belong to other estimators.PlaceboInTimedeep-dive with how-it-works narrative grounded in citations (abadie2010synthetic,abadie2021using,reichardt2019quasi).PlaceboInTime(n_folds=2, random_seed=seed)on the existing SC dataset together withGenerateReport(include_plots=True).staggered_did_pymc.ipynb.Notes
NOT SUPPORTEDplacebo-in-time result for the demo dataset (P=0.905, just below the 0.95 default threshold). This ties directly into the "If this check fails" troubleshooting section so readers see the failure path and the next steps.Closes #789
Test plan
prek run --files docs/source/notebooks/sc_pymc.ipynbpassespython scripts/validate_notebooks.py docs/source/notebooks/sc_pymc.ipynbpassesjupyter nbconvert --to notebook --execute --inplace ...so all outputs are populatedMade with Cursor