Published methods, checked against their reference values.

Normal and chi-square distributions

Every other method relies on these two distributions. The cumulative distribution function goes through the same incomplete gamma function as the chi-square, which keeps precision far into the tails, where small p-values are decided.

Relative lift: difference test and Fieller interval

The p-value comes from the test of the difference between the two variants, which errs as often in one direction as in the other. The lift is still shown as a percentage, and its interval gathers every lift that this same test doesn’t reject: “significant” and “interval that excludes zero” always say the same thing. A variant with zero conversions is never read as a certain loss, and when the original is too close to zero, the percentage interval is reported as unbounded rather than made up. Revenue per visitor counts visitors who didn’t buy as €0.

Differences between segments

Each segment has its own lift and standard error. Cochran’s Q test compares these lifts with their precision-weighted mean against a chi-square distribution. With few segments, the test lacks power: a non-significant result doesn’t prove the effect is the same everywhere.

Interactions between concurrent tests

Two overlapping tests form a factorial design: each shared visitor sees one of four combinations. If the first test’s effect is the same whatever the second test’s variant, the difference in differences is zero. Interactions are rare; measuring them lets tests run together without splitting traffic.

Multivariate test: main effects and interactions

Each combination is served as a variant. All traffic serves every section: a level’s mean pools every combination that contains it. An interaction compares a level’s effect depending on the level of another section; for the same size, it takes far more visitors than a main effect. Without a conclusive interaction, the recommendation follows the main effects.

Power and sample size

The test plan starts from the effect you want to be able to detect and the area’s traffic, and derives the number of visitors and the duration. Conversely, a number of visitors gives the minimum detectable effect. When a test ends without a decision, the same relationship tells you how many days it would take to narrow the interval.

Concurrent tests: traffic share left by priority

Each layer draws its bucket with its own salt: two layers are independent, and slices within one layer don’t overlap. A visitor enters the test if they’re in none of the slices given up: within a layer, the chance of being in one equals the total width given up; across layers, the remainders multiply. The plan’s duration is divided by this share.

Sample ratio mismatch (SRM) check

A test planned at 50/50 that gets clearly more visitors on one side signals an assignment problem: a redirect, a cache, a bot. The results aren’t reliable then, whatever the lift shown. The threshold is strict because the check is rerun every time the report is viewed.

Diagnosing a sample ratio mismatch

An imbalance most often comes from exposures lost on one side only, in a browser where the variant fails. Each segment is checked, the Holm correction avoids flagging a segment just because of the number of comparisons, and if the other segments together are balanced, the gap is concentrated in the ones flagged. Missing visitors are counted at the pace of the other variants.

Comparing survey responses

The test says whether the gap between distributions exceeds what chance explains, not which answer makes the difference. Shares are read among visitors who responded: a variant can change who responds, so the response rate itself is compared across all exposed visitors. With too few expected responses per cell, the report says the test isn’t reliable.

CUPED variance reduction

The same metric, measured before the test, explains part of the differences between visitors. Removing it doesn’t bias the effect, since it doesn’t depend on the version seen, and it narrows the interval more as the correlation gets stronger.

Without pre-exposure history (a new site, new visitors), the lift stays unadjusted, and the report says so.

Always-valid confidence intervals (mSPRT)

A classic confidence interval is valid only once, on the planned date. An always-valid interval can be reread as often as you like, and you can stop at the first crossing without inflating the risk. It’s wider: that’s the price of being free to look.

O’Brien-Fleming boundaries

Very strict at the first checkpoint and close to the usual threshold at the last, they let you stop a test with a clear result early while keeping the overall risk at 5%. The engine rejects plans it can’t compute exactly rather than extrapolating.

Alpha spending (Lan and DeMets)

The 5% risk is spent checkpoint by checkpoint: almost nothing at the first, almost everything at the last. Checkpoints don’t need to be evenly spaced, and each threshold accounts for those already passed: looking on the planned dates doesn’t create winners.

Futility via conditional power

At a checkpoint, if the target effect were the true lift, what chance would remain of concluding at the next checkpoints? Below one in ten, the test can stop: the target effect is no longer reachable. It’s a recommendation, never an imposed stop, and the decision thresholds don’t move: the plan’s 5% risk stays the same.

Holm correction

Comparing several segments multiplies the chances of finding a gap by chance. The Holm correction adjusts p-values so the risk of being wrong at least once stays at the stated level.

Bonferroni simultaneous intervals

Showing several intervals at once multiplies the chances that one of them misses the true value. Each is computed at a 5% risk divided by the number of intervals, so together they keep the stated risk.

Retained differences and false-discovery-adjusted intervals

A map compares dozens of elements and depths, so chance alone would make a few stand out. The Benjamini-Hochberg selection keeps only the differences that hold up, keeping the average share of false differences among them under 5%, and each interval widens accordingly: it excludes zero exactly for a retained difference.

Inverse-variance pooling

Several tests of the same idea each give an estimate. Each one weighs according to its precision: a test with half the standard error weighs four times as much. The model assumes the tests target the same effect.

Bayesian shrinkage

The store’s completed tests give the typical size of real effects. A new winner’s lift is pulled toward that average, more so when the test is small. The result is the gain you can reasonably expect.

In your workspace, at least two completed, measured tests are needed to estimate the prior; before that, the report shows the raw lift and says so.

Bayesian reading with a skeptical prior

The prior is centered on zero, and its width is that of the effects actually measured by the store’s past tests, with noise removed. A flat prior only restates the p-value and inflates small lucky tests. The decision remains the one in the sealed plan; when the two readings disagree, the report explains why.

Off by default. At least 10 stopped, measured tests are needed to estimate the prior; before that, there’s no reading, because no published, verifiable width applies to every store.

False positive risk of a winner

A p-value under 5% doesn’t give the probability that the winner is false: that also depends on power and on how rare good ideas are. When 10% of ideas work, a winner has a 22% risk of being a false positive, even at 80% power.

With no past tests, the default prior assumes 10% of ideas are good, the low end of the published range.

Difference between two effects and slope over time

Two estimates from different visitors are compared through the z of their difference. A series of daily estimates is summarized by a line weighted by their precision: a slope clearly opposite to the effect says it’s fading.

Display breaks for a change

A day’s share varies more than a single random draw would: the Pearson chi-square of the reference days, divided by its degrees of freedom, measures that dispersion, weekdays and weekends included. Counts become effective sample sizes, and each share gets its exact Clopper-Pearson interval. A break is flagged when the day’s upper bound falls below the lower bound of the previous days, at the three-sigma level.

A day’s drop or spike against its rolling baseline

Yesterday’s rate is compared with the pooled rate of the baseline days through a two-proportion z, whose variance is multiplied by those days’ Pearson dispersion: a slow weekend or a regular campaign doesn’t trigger an alert. Traffic follows the quasi-Poisson model used in surveillance systems, at the low threshold for a drop and the high threshold for a spike. The chosen risk is split across the tracked series (Bonferroni), and nothing is judged below five expected conversions.

Missing display from day one

An ordinary interval reread with each batch eventually raises false alerts. The mixture likelihood ratio of a share against a fixed value keeps its risk however often you reread it, even when the share varies from hour to hour: the alert fires as soon as the evidence is in, at the three-sigma level, and resets when the share is back to normal, proven the same way.

Interval for a share of tests

With few tests, a share is very uncertain: 3 tests out of 4 means between 30% and 95% at 95%. The Wilson interval stays within [0, 1] and is accurate on small samples, where the usual approximation goes wrong; under ten tests, Probant shows no percentage at all.

Median

The middle value holds up better than a mean against one exceptional test.

Capping extreme values

One outlier order weighs more than hundreds of others. The percentile is computed on visitors who have a value, across all variants: the interval narrows without the cap favoring a variant.

Ratio metrics (delta method)

Average order value divides amounts by orders, but it’s the visitor who is randomized, so a visitor’s orders aren’t independent. The delta method gives the variance of the ratio while keeping the visitor as the unit.

Per-pageview percentile (speed)

Load times are highly skewed: a few slow pages pull the mean up. The 75th percentile of pageviews resists that tail, and its interval is read between two ranks computed while accounting for pages from the same visitor, without estimating a density.

Dynamic allocation (Thompson sampling)

Each variant gets a share equal to the probability that it’s the best, mixed with equal shares so none drops below the floor. Draws use a fixed seed, so the same reallocation replays identically. This test maximizes conversions while it runs; it doesn’t prove a variant is better.

Dynamic allocation by device (one bandit per context)

A variant can suit mobile but not desktop. With few contexts, the simplest and safest approach is an independent bandit per context: each device has its own counts and draws, without borrowing other devices’ results. A low-traffic device stays at equal shares longer. The device is read in the browser, and a visitor keeps it from one visit to the next, so they keep their variant.

Dynamic allocation on an amount (revenue per visitor)

Most visitors buy nothing, and a few carts are very large: revenue per visitor breaks down into the probability of buying and the buyers’ average order value. The first follows a rate’s beta distribution; the second, a Student’s t distribution centered on the observed average order value, wider when buyers are few. Each draw multiplies the two, and the variant that wins the most draws gets the most traffic. A variant with no buyers takes the pooled order value, so only its probability of buying sets it apart.

Epoch-stratified reading

When shares change, comparing rates across all periods mixes different visitors (Simpson’s paradox). The lift is measured within each epoch, where the split is fixed, then combined by the epoch’s weight, so the comparison is made on equal populations. For a contextual test, each device in an epoch forms its own stratum, with its own shares. For an amount, each stratum compares mean amounts per visitor, with their variance.

Non-inferiority and equivalence (TOST)

The report’s 95% confidence interval is enough: “not worse” when its lower bound is above −m (one-sided test at 2.5%), “equivalent” when it fits entirely within ± m (two one-sided tests, Schuirmann). The sample size assumes a true difference of zero; it grows as 1/m². There’s never a “winner”: the conclusion is about the margin.

Student’s t distribution

With a few dozen zones instead of thousands of visitors, the normal distribution would give intervals that are too narrow. Student’s t widens them more when the units are few.

Correlation between visitors in the same zone

Visitors in the same zone resemble each other: a thousand visitors in ten zones tell you less than a thousand visitors drawn one by one. The design effect measures this loss, and a geo test’s plan accounts for it.

Difference between zones

The unit of analysis is the zone, not the visitor, because the zone is what was randomized. Degrees of freedom are those of the zones, and the robust variance is corrected because it underestimates uncertainty when zones are few.

Matched zones

Two zones that looked alike before the test are randomized against each other, so what separates them afterward owes little to the luck of the draw. The noise of this estimator, measured on history, is used to set the duration.

Permutation test

The random assignment is replayed: if the observed lift stays exceptional among all possible assignments, it owes nothing to a distributional assumption. All assignments are enumerated when there are few enough; beyond that, a fixed-seed sample is used.

Number of zones needed

Power depends mostly on the number of zones and little on each zone’s traffic: adding visitors to a zone doesn’t replace one more zone.

Switchback over time

Two neighboring windows resemble each other, and the variance accounts for it. The start of each window is set aside until the previous version stops having an effect, and the carryover from one window to the next is estimated separately.

Interval for a ratio (Fieller)

A percentage lift is a ratio of two uncertain means. Fieller’s interval adds no approximation to the test’s: it excludes zero exactly when the difference test concludes. If the denominator can’t be told apart from zero, the interval is unbounded, and Probant says so.

Real difference or mix effect (Kitagawa)

An overall rate is the average of each group’s rate, weighted by its share. Two populations can therefore differ only because they don’t mix the groups the same way: more phones on one side, and phones convert less. The decomposition splits the observed gap exactly in two: what remains at equal composition, and what comes from the mix. When, in every group with enough traffic, the gap goes the other way, that’s Simpson’s paradox, and Probant says so. It’s an accounting identity, not a test: it explains a gap, it doesn’t tell you whether it holds beyond chance.

Verdict rules

The traffic split comes first, then guardrails, then the primary metric’s interval. All blocking causes are listed together so they can be fixed in one go. A classic interval is valid only for a reading at the end: before the planned sample size, it gives neither a winner nor a loser, even if it already excludes zero. Only an interval valid at every look can conclude earlier.