← Every exhibit
Decision Errors and Statistical Power
| Decision | Null hypothesis true | Treatment effect real (lift) |
|---|---|---|
| Reject the null (declare a winner) | Type I error (alpha): a false positive, about 5% of the time | Correct decision (power, one minus beta): a true discovery, 80% to 90% of the time |
| Fail to reject (retain the control) | Correct decision (one minus alpha): a true negative, about 95% of the time | Type II error (beta): a false negative, 10% to 20% of the time |
Swipe or scroll horizontally if the table is wider than your screen.
Reference & Evidence
Source: Table from this essay. Sources and interpretation are given in the article.
Related exhibits
-
The A/B test experimental architecture
From the essay What is an A/B test? randomized controlled trials, statistical power, and governance
-
Taxonomy of Experimentation Architectures
From the essay What is an A/B test? randomized controlled trials, statistical power, and governance
-
The Metric Hierarchy in Controlled Experiments
From the essay What is an A/B test? randomized controlled trials, statistical power, and governance