Meta's A/B test or advrt: which should test your ad creative?

Meta's help center sets out the rules. You can start a test from the Ads Manager toolbar using an existing campaign or ad set as a template, from the Experiments tool, or by duplicating a campaign, ad set or ad (A/B test types). In Experiments you can compare up to 5 campaigns (Create an A/B test in Experiments). Variables are creative, audience, placement or a custom mix (Choosing a variable).

On timing, Meta says "we recommend a minimum of 7-day tests. A/B tests can only be run for a maximum of 30 days, but tests shorter than 7 days may produce inconclusive results" (A/B testing best practices). Meta's marketing page goes further and suggests at least 2 weeks (A/B testing on Meta).

Meta gives no minimum dollar budget. It says a test "should have a budget that will produce enough results to confidently determine a winning strategy," and suggests aiming for an estimated power of 80% or higher, which Ads Manager shows during setup (About confidence). If you duplicate a campaign for the test, the budget is duplicated too.

Meta compares the versions with each other. It simulates possible outcomes "tens of thousands of times" and reports the version with the lowest cost per result, with a confidence percentage: the chance you'd get a similar result if you ran the test again (How winning campaigns are determined). For A/B tests, "a 65 percent or higher confidence percentage represents a winning result."

advrt compares each ad with your account. Before launch it records the ad account's last 30 days. If the account had at least 10 purchases, each ad is tested on purchases with a Poisson model: an ad that spent S should get S ÷ CPA purchases at your normal rate. A winner needs at least 3 purchases, a CPA at least 15% better and a one-sided probability of 10% or less under the normal rate. With less purchase data, advrt uses click-through rate with a one-sided z-test, at least 1,000 impressions and a gap of at least 20%.

The two approaches answer different questions. Meta's tells you which of these versions did best. advrt's tells you which of these ads is clearly better or worse than what your account usually gets. Meta's confidence percentage and advrt's 10% significance level are different statistics, so don't compare the numbers directly.

Meta also has a creative testing feature inside existing campaigns. It can "compare up to 7 creative variants with delivery provided to new test ads," runs inside the current campaign so learnings carry over, and is "supported in campaigns using the Highest volume bid strategy only" (Creative testing, checked October 2026).

Meta suggests spending no more than 20% of the existing budget on the test ads. Two lines on that page matter for small advertisers: "A confidence level is not included," and "The test does not make any automatic changes based on the results." You read the comparison metric and decide.

This tool fits well when you already have a stable campaign and want to slot new ads in without disturbing it. advrt fits better when you're testing new concepts from scratch in a separate campaign and want a stated rule for when an ad has won or lost.

Meta's split is more rigorous. Because audiences are randomly divided so "nobody sees both," differences between versions can be put down to the variable you changed. advrt's ad sets share the same country and age targeting and Meta's delivery optimises each separately, so two ad sets may reach somewhat different people.

advrt's baseline is also a comparison with the past. If your last 30 days included a sale, a slow season or a big influencer spike, every ad in the test is measured against that. Meta's A/B test avoids this by running all versions at the same time against each other.

Use Meta's tool when you're testing an audience, a placement or a bid strategy, when you have budget for 7 days or more, or when the decision is large enough that you want the cleanest possible answer. advrt doesn't test any variable except the creative.

Usually not, and that applies to both tools. Meta says ad sets exit learning "after about 50 results in the week after the ad set's last significant edit" (About the learning phase). At a $40 CPA that's about $2,000 per ad set per week, or $10,000 a week for a 5-creative test with one ad set each.

Most small-brand creative tests spend a fraction of that, so every ad set stays in learning. That's acceptable for a comparison as long as all ad sets are in the same state with the same budget, which is why advrt gives each creative an equal daily budget and launches them together.

We computed this from advrt's verdict code (lib/loop/judge.ts) for an account with a $40 CPA over its last 30 days. Each row shows the most an ad can spend with that many purchases and still be called a winner, and what that means for its CPA. Meta doesn't publish an equivalent table for its A/B tests; its winner depends on a simulation of the versions against each other.

The pattern is the price of small samples. With 3 purchases an ad must be about 63% cheaper per purchase than normal to be called; with 10 purchases, about 38%. An ad that's 20% better than normal will usually run to the 5× CPA stop and come back inconclusive. Meta's A/B test has the same limit in a different form: its help pages warn that versions that are too similar may not produce a confident winner.

One more honest caveat: advrt checks verdicts every day, and looking repeatedly raises the real chance of a false call above the nominal 10%. The verdict code accepts that on purpose, because pausing a loser or iterating on a winner is cheap to reverse. If you need a decision you can defend in a board meeting, run Meta's A/B test for the full window.