Add Bayesian experimentation triptych

[NathanielF]

Experimentation Workflow and Cycle: Assurance, Sensitivity and Meta-Analysis.

PyMC examples covers a lot of ground in hierarchical modelling and causal inference, but the experimentation workflow is underspecified. We have scattered A/B testing notebooks but nothing that treats planning, interpretation, and synthesis as a connected sequence — the kind of end-to-end picture a practitioner actually needs when running repeated experiments across markets, time periods, or populations. This PR is an attempt to fill that gap.

The addition is a triptych of notebooks under causal_inference/:

• Assurance planning — Bayesian power / assurance curves, design vs. analysis prior separation, prior effective sample size
• Sensitivity to unmeasured confounding — bias parameter modelling, tipping-point analysis, sensitivity surface over prior width and effect threshold
• Meta-analysis and hierarchical borrowing — no/complete/partial pooling, shrinkage, τ as the quantity of substantive interest, predictive distributions for the next experiment

The notebooks cross-reference each other and are meant to be read as a loop: each notebook's posterior becomes the next one's prior. The framing is product experimentation but the machinery maps directly to clinical trials — close enough that the FDA's January 2026 draft guidance on Bayesian methodology names the same tools: assurance as "Bayesian power averaged over a design prior," hierarchical models as the recommended mechanism for subgroup borrowing and cross-trial synthesis, sensitivity analysis as building uncertainty about assumptions into the prior itself.

Related to #884

• Notebook follows style guide https://docs.pymc.io/en/latest/contributing/jupyter_style.html
• [] PR description contains a link to the relevant issue:
  • a tracker one for existing notebooks (tracker issues have the "tracker id" label)
  • or a proposal one for new notebooks
• Check the notebook is not excluded from any pre-commit check: https://github.com/pymc-devs/pymc-examples/blob/main/.pre-commit-config.yaml

Helpful links

• https://github.com/pymc-devs/pymc-examples/blob/main/CONTRIBUTING.md

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[aloctavodia]

Update the examples to run with PyMC > 6 and ArviZ > 1

[NathanielF]

Thanks @aloctavodia will do. Want to rework the writing a bit. Will work on it over the weekend and update the environment too.

[aloctavodia]

make it clear these are just examples. Maybe something like

The engine is the same whether the response is continuous (like revenue per visitor) or binary (like conversion);

Or expand the sentence.

Reply via ReviewNB

[aloctavodia]

az.style.use("arviz-variat")

seaborn is not used

Reply via ReviewNB

[aloctavodia]

pc = az.plot_dist(idata_gauss_demo,

var_names=["delta"],

visuals={"title": {"text": r"Posterior on $\delta$ at $N=200$ (one simulated dataset)"}}

)

az.add_lines(pc,

values=0.0,

visuals={"ref_line": {"color": "C1", "label": "ref = 0.0"}},

);

pc.get_viz("plot").legend()

Reply via ReviewNB

[aloctavodia]

This also works (just wanted to mention, no need to change it)

mcmc_mu = az.mean(idata_gauss_demo)["delta"].item()

mcmc_sd = az.std(idata_gauss_demo)["delta"].item()

Reply via ReviewNB

[aloctavodia]

similar previous post

Reply via ReviewNB

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

aloctavodia commented on 2026-06-09T10:24:51Z
----------------------------------------------------------------

similar to my comment for the first notebook

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

aloctavodia commented on 2026-06-09T10:24:52Z
----------------------------------------------------------------

you can do something like this

dt = az.from_dict(

{"posterior": {

"theta_new": theta_new_samples,

"d_hat_new": d_hat_new_samples,

}},

sample_dims=["sample"],

)

pc = az.plot_dist(

dt,

kind="hist",

sample_dims=["sample"],

cols=[],

aes={"color": ["__variable__"]},

visuals={"point_estimate":False,

"point_estimate_text":False,

"credible_interval":False,

},

)

pc.add_legend("__variable__")

[NathanielF]

Thanks for these comments @aloctavodia i'll try and address these later this week!

[NathanielF]

@aloctavodia I think this should be ready now.