← Every exhibit
Which operational miscalculations undermine incrementality testing?
| Miscalculation | Root cause | Experimental failure | Corrective protocol |
|---|---|---|---|
| Running tests without sufficient sample size | Underestimating sales variance relative to small ad lift | Yields wide confidence intervals that span zero; inconclusive tests | Perform pre-test power calculations following Lewis and Rao (2015) |
| Treating pre-post changes as causal | Turning off ads nationally and measuring before vs after | Conflates seasonal demand swings and macro trends with ad lift | Always use concurrent control groups or synthetic geo-controls |
| Ignoring spillover and interference | Treatment exposure leaks into control regions via word-of-mouth | Violates SUTVA (Stable Unit Treatment Value Assumption) | Establish geographical buffer zones around test territories |
| Evaluating incrementality on short windows | Measuring conversions only during the active test period | Misses delayed conversions and ad-stock decay effects | Extend measurement windows post-intervention to track decay |
| Applying uniform incrementality across channels | Assuming paid social and brand search have identical lift | Over-funds brand search while under-funding true prospecting | Measure incrementality independently across discrete channel tiers |
Swipe or scroll horizontally if the table is wider than your screen.
Reference & Evidence
Source: Table from this essay. Sources and interpretation are given in the article.
Related exhibits
-
The incrementality causal measurement hierarchy
From the essay What is incrementality?
-
Why do observational attribution dashboards report phantom returns?
From the essay What is incrementality?
-
Which operational miscalculations undermine marketing mix modeling?
From the essay What is marketing mix modeling?