← Every exhibit
The A/B test experimental architecture
Examine statistical power, significance thresholds, sample ratio mismatch, and MDE boundaries.
| Experimental Component | Statistical Function | Standard Operational Threshold | Primary Risk If Ignored |
|---|---|---|---|
| Statistical Power ($1 - \beta$) | Probability of detecting a true effect if one exists | 80% to 90% ($\beta = 0.10$ to $0.20$) | Type II error: discarding real growth innovations |
| Significance Level ($\alpha$) | Probability of rejecting null hypothesis when it is true | 5% ($\alpha = 0.05$, two-tailed, $Z \ge 1.96$) | Type I error: shipping ineffective or harmful code |
| Sample Ratio Mismatch (SRM) | Goodness-of-fit test for allocation integrity | $\chi^2$ test $p$-value $\ge 10^{-3}$ | Experimental invalidation due to tracking or CDN bias |
| Minimum Detectable Effect | Smallest relative lift the test is sized to detect | Derived from baseline variance and sample size | Running underpowered tests with uninterpretable noise |
Swipe or scroll horizontally if the table is wider than your screen.
Reference & Evidence
Source: Author's experimental framework grounded in causal inference and split-testing literature from Kohavi et al. (2013), Kohavi et al. (2009), and Lewis and Rao (2015).
Related exhibits
-
Decision Errors and Statistical Power
From the essay What is an A/B test? randomized controlled trials, statistical power, and governance
-
Taxonomy of Experimentation Architectures
From the essay What is an A/B test? randomized controlled trials, statistical power, and governance
-
The Metric Hierarchy in Controlled Experiments
From the essay What is an A/B test? randomized controlled trials, statistical power, and governance