A/B Testing in Marketing 2026: Significance, Sample Size, and the Mistakes Almost Everyone Makes

A/B testing in marketing promises something seductive: you no longer have to argue about who is right in the meeting — you let users vote. That is fundamentally correct, and it is why a method from agricultural statistics became the most important tool in conversion optimization.

The catch rarely makes it into the tutorial: an A/B test that is analyzed incorrectly is not neutral. It is worse than no test at all, because it dresses a gut decision in the clothing of science. You make the same arbitrary choice as before, but now with a dashboard writing “95% confidence” underneath it.

So this article does three things. First the mechanics: how a clean test is built. Second the statistics in the form you actually need — no wall of formulas, but no comfortable half-truths either. Third, the mistakes that make results not merely imprecise but worthless.

We did not copy the numbers for those mistakes from anywhere. We simulated them ourselves: millions of artificial users where we knew in advance what was true — and then measured how often common analysis practice claims the opposite.

The short version

• “Significant” does not mean “correct.” A p-value below 0.05 says: if there were no difference, data like this would be rare. It says nothing about how likely it is that your variant is better.
• Peeking destroys exactly that statement. In our simulation with no real difference at all: analyze once → 5.1% false alarms. Peek 10 times and stop at the first green result → 22.0%. Peek 50 times → 32.0%.
• Sample size is not paperwork, it is the actual question. At a 5% baseline conversion rate, detecting a 5% relative improvement needs roughly 244,000 visitors. Detecting a 20% improvement needs 16,316.
• Undersized tests don’t just lie more often, they lie upward. With a true effect of +5%, the significant runs in our simulation reported an average of +22.6% (n = 5,000 per variant). That’s the winner’s curse — and the reason test wins so rarely show up in the P&L.
• Check the split before you check the result. If variant A received 51.4% and B 48.6% of traffic, the test is broken no matter how nice the curve looks (sample ratio mismatch).
• Most ideas don’t work. At Microsoft, only about one third of tested ideas improved the target metric. That’s not bad news — it’s the entire point of the method.
• For small sites, A/B testing is often the wrong tool. At 200 visitors a day, a clean test for a 10% improvement takes over two years. What to do instead is further down.

Kohavi’s CXL talk walks through real Microsoft experiments where trustworthy-looking numbers turned out to be artifacts — the practical counterpart to the simulations below.

What an A/B test is — and what it isn’t

An A/B test splits your traffic randomly into two groups. Group A sees the existing version (the control), group B sees a variant with exactly one thing changed. After a predetermined number of visitors, you compare a predetermined metric.

The three phrases carrying all the weight are randomly, exactly one thing, and predetermined. Drop any of them and you still get numbers, but they no longer carry a conclusion.

• Randomly means assignment must have nothing to do with the user. If variant B only ships to desktop users, you are measuring the device, not your idea.
• Exactly one thing means: change the headline and the button and the image, and afterwards you know something worked — but not what. Testing many changes at once is what multivariate tests are for, and they need substantially more traffic.
• Predetermined is where most tests fail, and it earns its own section below.

How this differs from a before/after comparison

The most common substitute for a real test is: “We switched the page on Monday, and this week conversion is up 12%.” That is not a test, it is a story. That week the weather was different, a campaign launched, it was the start of the month, a competitor had an outage. An A/B test doesn’t remove these confounders by measuring them — it removes them by having both groups experience them simultaneously.

That’s why running in parallel is non-negotiable. An “A/B test” where variant A runs in March and variant B in April is a before/after comparison with a nicer name.

What A/B tests cannot do

• They explain nothing. A test tells you B converts better, never why. The why comes from user interviews, session recordings, and surveys — and those belong before the test, because they produce the hypothesis.
• They measure the short term. A more aggressive pop-up lifts newsletter signups and costs you returning visitors over months. The test runs for two weeks and sees none of it.
• They are useless for rare events. If you make twelve sales a month, A/B testing cannot measure anything. More on that below.

Statistical significance: what it actually says

Almost every dashboard shows something like “96% confidence” or “p = 0.04”. The most common translation is: “We are 96% sure B is better.” That translation is wrong, and most bad decisions in this field follow from this single misunderstanding.

What the p-value actually answers is a hypothetical question:

Assuming A and B were in truth exactly equal — how often would we see a difference at least as large as the one we measured?

A p-value of 0.04 means: in 4% of cases. Nothing more. It is a statement about the data under an assumption, not about the truth of the assumption. You cannot read the probability that your variant is better out of it — for that you would also need to know how plausible the idea was beforehand. That’s precisely what Bayesian analysis provides, which is why many modern tools now offer it.

For practice, a stricter but more honest rule of thumb:

Significance is a filter against randomness, not proof of effect. It says: “This is probably not just noise.” It does not say: “This will show up in your revenue.”

The two error types, and why only one gets discussed

The significance level (usually 5%) limits the first error type. The second is governed by statistical power, usually set to 80%. That means: if the effect you wanted to detect really exists, you find it in 80% of cases — and miss it in 20%.

Almost nobody talks about the second type, even though it’s the more expensive one in marketing. A missed winner produces no incident, no crisis meeting, no note. It quietly becomes “didn’t do anything”, and the idea never comes back to the table. You eventually notice type I errors. You never notice type II.

Sample size is the real question

The most important number is fixed before the test starts: how large does the difference have to be before you care? This value is the MDE (minimum detectable effect), and it determines everything downstream.

The relationship is unforgiving: halve the effect you want to detect and you quadruple the traffic you need. Here are the numbers, computed for a 5% significance level (two-sided) and 80% power:

Our own calculation, two-sided two-proportion test, α = 0.05, power = 0.80. Reproducible with the standard sample size formula for two proportions; “relative improvement” means 5% → 5.25% counts as 5%.

Translated into calendar days, the number becomes genuinely uncomfortable. At 5% baseline conversion, aiming to detect a 10% improvement:

• 5,000 visitors/day: 12 days — doable.
• 1,000 visitors/day: 62 days — hard, but possible.
• 200 visitors/day: 312 days. In that time your product, your audience, and your competition all change. The result would be stale before it was finished.

The practical consequence is rarely said out loud: if your calculation returns 300 days, that is not a reason to start the test anyway and look after two weeks. It’s the answer that A/B testing is the wrong tool for this question. What helps instead is in “When your traffic isn’t enough” below.

Mistake 1: Peeking — the most expensive error nobody considers an error

The pattern is always the same. The test is running, the dashboard is open, on day three variant B shows “97% confidence”. The test is stopped, B is rolled out, everyone is happy.

This procedure destroys the very basis on which that 97% was computed. The significance calculation assumes the sample size was fixed in advance. If instead you keep looking until the result pleases you, you are no longer testing the hypothesis — you are testing whether noise eventually gets big enough by chance. It will.

Evan Miller worked this out in 2010 in “How Not To Run an A/B Test” and arrives, for the extreme case (evaluating after every single observation, stopping at 5% significance), at an actual error rate of 26.1% instead of the assumed 5%.

Because that sounds theoretical, we reproduced it ourselves — in a situation where we know for certain there is no difference. Both variants convert at exactly 5%. We simulate 4,000 complete test runs of 10,000 visitors per variant and count how often a “winner” is declared:

Our own simulation, 6 August 2026. Two identical variants at 5% conversion, α = 0.05 two-sided, stopping at the first significant interim reading. 4,000 runs per row, Python 3, fixed random seed.

Read the last row again: roughly one in three tests finds a winner that does not exist. Not because your idea is bad — because you kept looking until the numbers happened to line up.

And here’s the part that really hurts in practice: these false alarms are not evenly distributed. They appear preferentially early, when the curves swing the most. The test that “wins” after two days is precisely the one you should trust least. In practice, it is the one that generates the most excitement.

The countermeasures, in order:

1. Compute the sample size beforehand and write it down. Not in your head — in the ticket describing the test. A target only you know gets quietly adjusted the moment results look good.
2. Run full weeks, at least one, preferably two. Tuesday users behave differently from Sunday users. Stopping after 9 days double-counts a weekday.
3. Looking is allowed, stopping is not. Interim readings are useful for catching technical faults (see sample ratio mismatch). They are not a basis for decisions.
4. If you genuinely need the ability to stop early, use methods built for it — sequential testing, alpha spending, always-valid inference. Several modern tools support this. What doesn’t work is standard analysis plus early stopping.

Mistake 2: Sample ratio mismatch — the test that never happened

You planned a 50/50 split. At the end, variant A has 51,400 visitors and B has 48,600. Sounds like ordinary variation, but it isn’t: at around 100,000 assignments the deviation should be well under one percentage point. A 51.4/48.6 split is so unlikely that there is almost certainly a technical cause.

This is called sample ratio mismatch (SRM). Microsoft Research uses a deliberately strict threshold for it: a χ² test on the split with p < 0.0005 raises the alarm. In practice, genuine SRM cases land far below that.

The causes are almost always mundane and almost always biasing:

• The variant’s script loads more slowly, and some users bail before assignment — specifically those on the worst connections.
• A bot filter behaves differently on one variant.
• A redirect loses users along the way.
• The test was restarted mid-flight and some users had already seen a variant.

Why this is worse than an imprecise result: if users drop out of one group systematically, randomization is gone. The remaining groups are no longer comparable, and no statistical method can repair that. A test with SRM does not give a weak result. It gives none.

This is why the check belongs at the beginning of analysis, not the end. Split first, metric second. Anyone who looks at the result first will find reasons not to take the SRM too seriously.

Mistake 3: The winner’s curse — why test wins vanish from the P&L

This one is the subtlest, and it explains an observation almost every marketing team knows: twelve tests were “won” over the year, each with a double-digit improvement — yet overall conversion sits roughly where it started.

The cause is selection bias. An undersized test cannot detect a real, small effect at all. It becomes significant only when noise happens to point in the same direction as the real effect. That makes the winners systematically the overstated runs.

We simulated this too. We build in a true effect of +5% (5% versus 5.25% conversion) and look at what the tests that reach significance report:

Our own simulation, 6 August 2026. True effect in every row: +5% relative. α = 0.05 two-sided; 1,500 / 800 / 300 runs. At n = 5,000, an additional 5 runs reached significance pointing in the wrong direction.

The first row is the normal case for small and mid-sized shops. The true effect is 5%, the dashboard reports 22.6%, and the slide says “+22% conversion”. What gets rolled out is a change that really delivers a twentieth of that. Nobody is lying — the number genuinely appears in the tool.

Note the middle column as well: at n = 5,000 the test finds the real effect in 8.7% of runs. In over 90% of runs, nothing happens. A team testing this way experiences A/B testing as “usually does nothing, but when it works it’s huge”. Both halves of that sentence are artifacts of an undersized sample.

Practical consequences:

• Plan with the lower bound of the confidence interval, not the point estimate. If the interval runs from +2% to +40%, budget for +2%.
• Replicate big winners. An effect confirmed by a second, independent test is real. The winner’s curse does not survive replication.
• Measure the sum of your tests against your top-line metric. If twelve winners at 15% each produce no visible movement over the year, those 15% were never real.

Mistake 4: Too many metrics at once

Test a variant against 20 metrics — conversion, average order value, bounce rate, time on page, clicks on five buttons, newsletter, scroll depth — and at a 5% significance level one of them will turn significant by chance on average, even if the variant does nothing. The probability of getting at least one false alarm across 20 independent metrics is roughly 64%.

This rarely happens on purpose. It happens like this: the test shows nothing on the primary metric. You look at the secondary ones, find something green, and retell the test. “B improves conversion” becomes “B improves time on page, which pays off long term.”

The countermeasure is organizational, not statistical: define one success metric in advance (Kohavi calls this the Overall Evaluation Criterion). Everything else is observation feeding the next hypothesis — never the decision basis for this test. If you genuinely need to evaluate several metrics as equals, you must correct the significance level (Bonferroni and relatives) and budget more traffic accordingly.

There’s a necessary counterpart: guardrail metrics. These are numbers that must not get worse — load time, error rate, returns, unsubscribes. They don’t decide the winner, but they can disqualify it.

Kohavi built the experimentation platforms at Amazon, Microsoft, and Airbnb; here he puts statistical power, the winner’s curse, and the one-third rule in context firsthand.

What the big players know: most ideas don’t work

Perhaps the most important number in the entire field comes from Microsoft’s experimentation platform: only about one third of tested ideas improve the target metric. One third does nothing, one third makes things worse. Google is reported to hit an even stricter figure — around 10% of controlled experiments lead to a change being shipped.

This number is usually read as depressing. It’s the opposite. It means two thirds of the changes an experienced team believes in are not improvements. Without testing, all of them would ship. So the value of testing lies less in the winners than in the prevented regressions — and nobody sees those, because they never go live.

From this follows an honest expectation: if your team claims to win 80% of its tests, it is either measuring the obvious (broken checkout versus working checkout) — or analyzing incorrectly. The third possibility is rare.

When your traffic isn’t enough

For many websites the honest output of the sample size calculation is: A/B testing is not the right tool here. At 200 visitors a day and 2% conversion, that’s four sales daily; a test for a 20% improvement would need a good seven months.

That’s not a reason to give up — it’s a reason to change method:

1. Test higher in the funnel. Clicks on “add to cart” are ten to fifty times more frequent than completed purchases. The metric is a weaker proxy, but it’s measurable at all. Careful: a win up there is only real once it arrives at the bottom — keep completed orders as a guardrail.
2. Test big changes instead of small ones. A completely new page concept can move 30%; a button color might move 2%. The table above shows that this is exactly the difference between 3 days and 300. The price: you won’t know which of the twenty changes did the work.
3. Use qualitative methods. Five user tests surface the coarse usability problems. No statistics required, and the problems you find there are usually larger than anything an A/B test could ever resolve.
4. Fix the measurable things without testing first. Load time, broken mobile forms, missing shipping cost information, forced registration before purchase. Those aren’t hypotheses — those are defects.
5. Use before/after deliberately and with caveats. When a clean test is impossible, a documented before/after comparison across several weeks beats nothing — as long as you don’t sell it as proof.

The key sentence of this section: a test too small to find the truth will still find something. And that something is noise with a confidence score next to it. No test is better than a test whose result you cannot interpret.

The process that makes mistakes unlikely

1. Write a hypothesis, not an idea. A hypothesis has the form: “Because [observation], we expect [change] to improve [metric] by [magnitude].” The observation is the crucial part — it comes from analytics, user interviews, or support tickets. Without it you are testing opinions.

2. Fix the metric and MDE, compute the sample. One success metric, one to three guardrails, one minimum effect. The sample size follows, and from it the runtime. If the runtime is unrealistic, the test is cancelled here, not in week three.

3. Implement cleanly and verify first. An A/A test (both groups see the same thing) run for a few days catches most technical faults: if it produces a “winner”, your setup is broken, not your idea. Also check: is tracking double-counting? Is assignment stable across sessions? Does the page flicker because the variant is applied after render?

4. Let it run and don’t touch it. Full weeks. No stopping on a green interim reading. Interim looks exclusively for SRM and error checks.

5. Analyze in a fixed order. Split first (SRM), then the primary metric, then guardrails, and only then everything else — with everything else counting only as input for the next hypothesis.

6. Document everything, losers included. A test archive holding hypothesis, result, and confidence interval prevents the same idea from being re-tested every eight months. And it is the only way to later check whether your winners actually show up in the top-line number.

Tools: the landscape after Google Optimize

Google Optimize was shut down on 30 September 2023, and to this day there is no free successor from Google. GA4 itself cannot run A/B tests — it can analyze results from other tools if you send the variant along as a parameter.

Roughly sorted by approach:

• Client-side tools (VWO, AB Tasty, Convert, Optimizely Web) alter the page via JavaScript in the browser. Fast to set up, usable without developers — but they bring the flicker effect and cost load time. For sites where Core Web Vitals matter, that’s a real drawback.
• Server-side tools and feature flags (GrowthBook, Unleash, Optimizely Feature Experimentation, LaunchDarkly) decide the variant before the page is served. No flicker, no added load time, works for APIs and apps too — but it requires engineering work.
• Open source and self-hosted: GrowthBook can be run yourself. That matters when assignment data must not leave the building — a privacy argument that has been tipping more decisions since Optimize was discontinued.
• Platform-native testing: Shopify, Shopware, and the major shop systems have apps and plugins for it. Convenient, but check which statistical method sits underneath and whether interim readings tempt you into stopping early.

The selection question that matters more than the feature list: does the tool only show “96% confidence”, or does it also show the confidence interval and the sample size it assumes? Tools that display only a green number train you to peek. Tools that show the target and your progress toward it train you to wait.

Frequently asked questions

How long should an A/B test run?

At least one full week, preferably two, and always until the pre-computed sample size is reached — whichever comes later. Full weeks matter because behavior differs between weekdays and weekends. A test ending after 9 or 11 days double-weights individual weekdays.

How many visitors do I need for an A/B test?

It depends on your baseline conversion rate and the effect you want to detect. At 5% conversion and a desired 10% relative improvement, roughly 62,500 visitors in total; for a 20% improvement only 16,300, but for a 5% improvement 244,000. Rule of thumb: half the effect, four times the traffic.

What does 95% statistical significance actually mean?

It means: if there were truly no difference between the variants, you would see a result like the observed one in at most 5% of cases. It does not mean the variant is better with 95% probability — that statement cannot be derived from a p-value.

Can I stop an A/B test early once it’s significant?

No, not with standard analysis. Our simulation with two identical variants shows: with ten interim looks and stopping at the first green reading, the test declares a non-existent winner in 22% of cases instead of 5%. If you need the ability to stop early, use methods built for it, such as sequential testing or always-valid inference.

What is a sample ratio mismatch?

A deviation between the planned and actual traffic split, for example 51.4% versus 48.6% instead of 50/50. It points to a technical fault in assignment. Because users drop out of one group systematically, randomization is destroyed and the result is no longer analyzable. Microsoft Research checks for it with a strict threshold of p < 0.0005.

Can I run A/B tests with GA4?

Not directly. GA4 has no built-in experimentation feature since Google Optimize was retired on 30 September 2023. You need a separate tool for serving variants; GA4 can then analyze results if the variant is sent as an event parameter.

Why don’t my winning tests show up in revenue?

That’s typical of the winner’s curse. Undersized tests only reach significance when random noise amplifies the real effect, so winners are systematically overstated. In our simulation, significant tests reported an average of +22.6% when the true effect was +5% (5,000 visitors per variant). The remedies: plan with the lower bound of the confidence interval, and confirm big winners in a second test.

How many metrics may I evaluate?

One as the decision basis, plus a few guardrail metrics that must not get worse. Evaluate 20 metrics as equals and the probability of at least one chance hit is around 64% — even if your variant does nothing.

Is A/B testing worth it for small websites?

Usually not in its classic form. At 200 visitors a day, a test for a 10% improvement takes over 300 days. Better options are larger changes instead of small details, metrics higher in the funnel, qualitative user testing, and fixing clear technical defects.

Conclusion

A/B testing in marketing works — but not as a dashboard you stare at until it turns green. It works as a procedure with one inconvenient property: the most important decisions are made before the test starts. Which metric counts, how large the effect must be, how many visitors are required, when it ends. After that, the test is just execution.

The three numbers from our own simulations worth remembering: ten interim looks turn a 5% false alarm rate into 22%. A true effect of 5% appears as +22.6% when the sample is too small. And at the big companies only about one third of ideas work — which means the two prevented regressions are worth as much as the one winner.

If you take away one thing, make it this: compute the sample size before you test. If it yields a number you can reach in reasonable time, you have a tool. If it yields 300 days, you have an answer — namely that this question needs to be answered differently.

If you want to keep reading from here: load time is the one factor that almost always pays off and needs no test — that’s what our Core Web Vitals Test is for. How to get reliable numbers on your visibility in the first place is covered in SISTRIX vs Ahrefs. And why study averages often say nothing about your specific site is laid out in more detail in Google AI SEO — same problem, different topic.