Holdout test
A holdout test withholds ads from a randomly-selected control group, then compares performance vs. the test group to measure incremental lift.
What does Holdout test actually mean?
A holdout test is the gold-standard incrementality methodology. A randomly-selected portion of your audience (typically 5-20%) is excluded from advertising. After 2-4 weeks, conversion rates between the holdout group and the exposed group are compared. The difference is the incremental contribution of advertising: the conversions that wouldn't have happened without ads.
Meta provides a built-in ‘Conversion Lift Study’ tool for holdout testing, but most brands run their own via geo-holdouts (some markets get ads, others don't) because Meta's tool requires high spend levels and limits campaign types. The geo-holdout approach uses similar markets (e.g., Tampa vs. Orlando) as natural control/test pairs.
Holdout tests cost real money (you give up 5-20% of attributed performance during the test window), but they produce the most defensible measurement of true ad lift available. Most performance teams run them quarterly to calibrate platform-reported attribution.
Example
A DTC brand pauses Meta ads in 5 mid-sized markets for 4 weeks. Compared to 5 control markets where ads ran, the paused markets show 18% lower conversions. Meta's reported ROAS = 4×; incremental ROAS = 4× × 0.55 = ~2.2×.
When it matters
Holdout tests are the most rigorous way to verify ad incrementality. Quarterly cadence keeps platform-reported metrics calibrated to reality.
Related terms
Incrementality
Incrementality measures the additional revenue ads generate vs. what would have happened without them, usually measured via holdout tests or geo-experiments.
Geo-experiment
A geo-experiment uses geographic regions as test/control groups, running ads in some markets and pausing in others to measure incremental lift.
MMM
MMM is a top-down statistical method that estimates each marketing channel's contribution to revenue using time-series data. It's privacy-friendly and attribution-window-agnostic.
Attribution
Attribution determines which ad touchpoint receives credit for a conversion, using last-click, first-click, time-decay or data-driven models.
Find out which ads actually win.
advrt builds ads around your real product photos, tests them live on Meta and judges each one against your own account. Free plan, no credit card.