A/B testing terms, defined simply.

A test in which both versions are identical, so any winner it declares is false. Simulating A/A tests measures the real risk of a given way of reading results.

The peeking problem · A/A simulator

An experiment that shows the current version of a page (A) to some visitors and a modified version (B) to others, assigned at random, then compares the two on a primary metric set before launch. The lift is read with its confidence interval at the planned checkpoints.

What is an A/B test? · How Probant decides

The probability of declaring a winner when the two versions perform the same. Probant holds it at 5% over the whole test, checkpoints included. Reading a test every day at the usual threshold pushes it far higher.

The peeking problem

A range you can reread at any time and stop on at the first crossing without inflating the risk (Johari, Pekelis, and Walsh, 2017). It’s wider than a classic confidence interval. Probant uses it for rollout guardrails.

Rollouts

A moment set in the test plan when Probant reads the test to decide. Between checkpoints, the daily range is shown, but it isn’t enough to conclude. A test plan includes three by default, evenly spaced.

Planned checkpoints

The range of true lifts compatible with the data, at the 95% level. Probant always shows the lift with its range: if the range is entirely above zero, the variant wins; if it contains zero, nothing is proven.

Verdict rules

The share of visitors who place an order, or who take the action the test measures.

Duration calculator

Variance reduction: the same metric measured before the test explains part of the differences between visitors, and removing it narrows the confidence interval without biasing the effect. Probant applies it to your test reports, using each visitor’s history before their first exposure.

The statistical engine

A metric that must not get worse (return rate, page speed, errors), with a limit set in the test plan. A guardrail stops the test only if its whole range exceeds the limit; a range that straddles the limit doesn’t prove harm.

Verdict rules · Guardrails

An adjustment to p-values when you compare several things at once, so the risk of being wrong at least once stays at the stated level. Probant applies it to differences between segments.

The statistical engine

What each exposed visitor leaves once product costs, discounts, and shipping are subtracted. A variant can raise revenue and cut into margin, for example by paying for more free deliveries.

Measuring margin

The smallest lift the test has a good chance of detecting if it exists, with 80% power and a 5% risk. The smaller it is, the more visitors you need: aiming for an effect half as large takes about four times as many visitors.

Duration calculator

The threshold the z-score must exceed at a checkpoint to conclude. Very high at the first checkpoint, it falls toward the usual threshold at the last one, so the overall risk stays at 5%. Probant computes it for 1 to 5 checkpoints.

Planned checkpoints

The probability of seeing a lift at least this large if the two versions performed the same. Below 5%, the lift is called significant; in Probant, that’s the same as a 95% confidence interval that excludes zero.

The statistical engine

The number of visitors set by the test plan before launch, based on the target effect and the area’s traffic. Until it’s reached, only a boundary crossed at a checkpoint allows a decision.

Duration calculator

What past tests say about the typical size of real effects, before you look at a new test. Probant estimates it from the store’s completed tests and uses it to correct for the winner’s curse.

The winner’s curse

The difference between the variant and the original, relative to the original: B / A − 1. A positive effect means the variant does better, in proportion to the starting value.

The statistical engine

Revenue divided by the number of visitors exposed to the test, whether they buy or not: a visitor with no order counts as €0. It combines conversion rate and average order value, and tells you what a change brings in.

Automatic withdrawal of a version being rolled out as soon as a guardrail is crossed, that is, as soon as its whole always-valid range exceeds the limit. All traffic goes back to the original.

Rollouts

The share of traffic that sees the new version during a gradual rollout. Probant moves to the next step once the minimum duration has passed and no guardrail is crossed; to go to all traffic, every guardrail must sit entirely on the right side of its limit.

Rollouts

A subset of visitors (mobile, desktop, new, returning customers) on which you reread the result. A difference between segments is presented as real only after a test of that difference, corrected with the Holm method.

The statistical engine

A correction that pulls the measured lift toward the average of past tests, more so when the test is small. This is how Probant corrects for the winner’s curse.

The winner’s curse

Visitors aren’t split between versions as planned, a sign of an assignment problem. Probant checks every test with a chi-square test: below a p-value of 0.001, the verdict is blocked, whatever the lift.

Verdict rules

The probability of detecting the target effect if it really exists. Probant plans at 80% power: a real effect of the target size goes unnoticed 20% of the time.

Duration calculator

The modified version of the page, compared with the original, which is the current version. The original is arm A, the variant arm B.

The tendency of a winning test’s reported gain to overstate the real gain. Among many tests, the ones that clear the threshold are partly the ones chance favored, so their measured lift is too good on average.

The winner’s curse

The lift divided by its standard error: how many times the lift exceeds the expected noise. At each checkpoint, Probant compares the z-score with the O’Brien-Fleming boundary.

Planned checkpoints