What is an A/B test? Definition, steps, and an example

What is an A/B test?

An A/B test is an experiment: some visitors see the page as it is (version A, the original), the others see a modified version (B, the variant), and you compare what the two groups do. It’s also called a split test. Because visitors are assigned at random, the two groups are alike in every way except the page they see, so a clear difference between them comes from the change, not from the season, a campaign, or a sale day.

That’s what sets a test apart from a before-and-after comparison. Shipping the new page and comparing next month with last month mixes the page’s effect with everything else that changed in between. In an A/B test, both versions run at the same time, on the same traffic.

The test is judged on a primary metric chosen before launch: a conversion rate, revenue per visitor, a margin, a qualified lead. Guardrails watch what the variant must not damage, such as page speed or the return rate. The field’s terms are defined in the glossary.

Why assign visitors at random?

Because that’s what makes the two groups comparable. If you showed the variant to morning visitors and the original to evening visitors, or to mobile on one side and desktop on the other, the measured difference would mostly reflect the difference between those audiences.

The assignment also has to be stable: a returning visitor must see the same version again, otherwise they count in both groups. Probant assigns each visitor to one of 10,000 buckets based on their identifier, which stays the same for the same visitor, and each bucket belongs to one version.

One last precaution: check that the actual split matches the planned one. In the shipping bar test, 30,650 visitors saw the original and 30,590 the variant, or 50.0% and 50.0%. A gap larger than chance allows, a sample ratio mismatch (SRM), almost always points to a technical bug: fix it and start again rather than interpret the result.

How does an A/B test work?

In six steps, half of which happen before the first visitor arrives.

1. A hypothesis. What you change, and why it should help: “showing how much is left to spend before free shipping will grow cart sizes.”
2. A primary metric and guardrails, set before you see a single number.
3. A test plan: the effect you want to be able to detect, the number of visitors needed, the duration in days, and the checkpoint dates.
4. The variant, built and checked on the real page, with a visual editor or in code.
5. The test itself, which runs until the planned checkpoint while guardrails are monitored.
6. The decision: ship the variant, remove it, or acknowledge that the test can’t settle the question.

How many visitors, and for how long?

As many as the plan requires, and that plan is calculated. It depends on the traffic of the tested pages, the current value of the metric, and the effect you want to be able to see. The smaller that effect, the more visitors you need to tell it apart from chance.

In the shipping bar test, the original’s revenue per visitor was €24.58. To detect a +5.5% effect, with a 5% risk of a false conclusion and 80% power, the engine planned 58,900 visitors, or 18 days of traffic, and three checkpoints: day 6, day 12, and day 18.

The duration calculator runs the same calculation on your numbers, no account needed. If the resulting duration is unreasonable, aim for a larger effect, a more frequent metric, or a busier page rather than launch a test that can’t conclude.

How do you read A/B test results?

With a confidence interval, not a single number. On day 18, the variant showed +6.0% on revenue per visitor, with a 95% confidence interval from +2.2% to +10.0%. The whole interval sits on the right side of zero: the effect is positive, and its true size likely falls within that interval. Probant’s report puts it in one line: “Conclusive.” (proven gain).

The other metrics add nuance. Margin per visitor rises less (+3.0%, from -0.7% to +6.7%) and its interval overlaps zero: free shipping costs part of what it brings in. That’s the whole subject of “Measure margin, not just revenue.”

Finally, the measured gain of a winning variant is too good on average. Corrected using the store’s past tests, the shipping bar’s gain goes from +6.0% to +4.9%, and that’s the number to promise. The explanation is in the article on the winner’s curse.

Which mistakes skew an A/B test?

Four mistakes come up often, and each one can produce winners that aren’t real.

• Checking every day and stopping at the first “significant” difference. Across 1,000 simulated A/A tests of 18 days, where both versions are identical, this habit declares a winner in 21.8% of cases; read at the planned checkpoints, the same tests declare only 5.6%. The details are in “Checking a test every day manufactures winners.”
• Switching metrics after the fact. If you pick, at the end, whichever tracked metric moved the most, you’ll almost always find a “winner.” The primary metric is set before launch; the others are read with a correction for how many there are.
• Ignoring a sample ratio mismatch. A visitor imbalance between A and B signals a measurement bug; the result that comes with it is worthless, whatever it says.
• Judging on revenue alone when the variant touches prices, discounts, or shipping fees: margin may not follow.

How Probant guards against these, with planned checkpoints, guardrails, and blocked verdicts, is described on the “How Probant decides” page; guardrails stop a test on their own if it damages something.

A/B test, A/A test, multivariate test: what’s the difference?

An A/B test compares one variant with the original. When you compare several variants at once, it’s called an A/B/n test: the principle is the same, but you need more visitors, and the risk of error is shared across the comparisons.

An A/A test pits a page against itself. It isn’t looking for a gain: it checks the setup and measures the real risk of a given way of reading results. The A/A simulator shows it in a few seconds.

A multivariate test combines several changes at once (a headline, an image, a button) to measure the effect of each and how they interact. It answers more questions, at the cost of much more traffic.

Which tool should you use for A/B testing?

An A/B testing tool does four things: split visitors, show the right version without making the page flicker, measure conversions, and calculate the result. Tools differ mostly on statistical method, how they handle peeking, guardrails, script weight, data hosting, consent, and price. Our comparisons set Probant against the main ones, based on their public documentation, with sources.

Probant locks the plan before launch, concludes at the planned checkpoints, expresses the result in euros of margin, and corrects for the winner’s curse. The demo shows this guide’s test and a few others, no sign-up required.

Frequently asked questions