The goal of alphaN is to help the user set their significance level as a
function of the sample size. The function alphaN allows users to set
the significance level as function of the sample size based on the
evidence and the prior features they desire. The function JABt and
JABp converts test statistics and JAB_plot plots the Bayes factor as a function
of the alphaN_plot plots the alpha level as a function
of sample size for a given Bayes factor.
Calculations are based on Wulff & Taylor
(2024). If you enjoy the
package, please consider citing the paper (see citation("alphaN")).
As of version 0.2.0, alphaN() can also calibrate the alpha level to
the effect-size and moment Bayes factors of Klauer, Meyer-Grant &
Kellen (2024), which center
the alternative hypothesis on an effect size of your choosing
(method = "ES" and method = "moment").
If you’re not an R user, you may also be interested in the associated Shiny app.
To install the latest release version from CRAN use:
install.packages("alphaN")You can install the development version of alphaN from GitHub with:
# install.packages("devtools")
devtools::install_github("jespernwulff/alphaN")Here is an example: We are planning to run a linear regression model
with 1000 observations. We thus set n = 1000. The default BF is 1
meaning that we want to avoid Lindley’s paradox, i.e., we just want the
null and the alternative to be at least equally likely when we reject
the null.
library(alphaN)
alpha <- alphaN(n = 1000, BF = 1)
alpha
#> [1] 0.008582267Therefore, to obtain evidence of at least 1, we should set our alpha to 0.0086.
Raising BF asks for more evidence before you are allowed to reject,
which lowers alpha. The method argument selects the prior behind
Jeffreys’ approximate Bayes factor:
# Moderate (BF = 3) and strong (BF = 10) evidence at n = 1000
alphaN(n = 1000, BF = 3)
#> [1] 0.002549145
alphaN(n = 1000, BF = 10)
#> [1] 0.0006911392
# Balancing Type I and Type II error rates instead of the default prior
alphaN(n = 1000, BF = 3, method = "balanced")
#> [1] 0.024221The methods "ES" and "moment" (new in 0.2.0) answer the question:
which alpha do I need so that a significant result corresponds to a
Bayes factor of at least BF against an alternative centered on the
effect size de I actually care about?
# Moderate evidence, targeting a medium-sized effect (Cohen's d = 0.5)
alphaN(1000, BF = 3, method = "ES", de = 0.5)
#> [1] 0.002189564
alphaN(1000, BF = 3, method = "moment", de = 0.5)
#> [1] 0.0004913521Because the moment prior treats effects near zero as a priori implausible, the alpha it implies falls much faster with the sample size than under JAB:
ns <- c(100, 1000, 10000)
tab <- rbind(JAB = alphaN(ns, BF = 3),
ES = alphaN(ns, BF = 3, method = "ES"),
moment = alphaN(ns, BF = 3, method = "moment"))
colnames(tab) <- paste0("n = ", ns)
round(tab, 5)
#> n = 100 n = 1000 n = 10000
#> JAB 0.00910 0.00255 0.00073
#> ES 0.01185 0.00219 0.00058
#> moment 0.00788 0.00049 0.00002JAB() computes Jeffreys’ approximate Bayes factor for a coefficient
directly from a fitted lm() or glm() object; JABt() and JABp()
do the same from a t-statistic or a p-value if that is all you have
(e.g., from a published paper):
set.seed(1)
d <- data.frame(x = rnorm(200), z = rnorm(200))
d$y <- 0.2 * d$x + rnorm(200)
m <- lm(y ~ x + z, data = d)
JAB(m, covariate = "x")
#> [1] 22.50664
# From summary statistics alone
JABt(n = 200, t = 2.8)
#> [1] 3.56385
JABp(n = 200, p = 0.005)
#> [1] 3.634824alphaN_plot() compares alpha as a function of sample size across the
four JAB-type priors:
alphaN_plot(BF = 3)
JAB_plot() shows how the Bayes factor maps onto the p-value for a
given sample size, marking the alpha levels needed for evidence
thresholds of 1, 3, and 10:
JAB_plot(n = 1000, BF = 3)