Checking a test every day manufactures winners

The 5% threshold only holds for a single look

A test starts on a Monday. On Tuesday, variant B is ahead. On Wednesday, the gap narrows. On Thursday, the tool reports a “significant” difference. The team stops the test and ships B. It’s a common scene in testing programs, and an expensive one.

The 5% threshold of a classic test has a precise meaning. If the two versions perform the same, a test read once, on the planned date, wrongly declares a difference in 5% of cases. That figure assumes a single look. Every extra look is another chance to land on a difference caused by chance alone. Stopping at the first “significant” difference amounts to keeping the worst of those chances.

Armitage, McPherson, and Rowe described the effect back in 1969: when accumulating data is tested at the same threshold after each new look, the real risk of a false conclusion grows with the number of looks. It never comes back down to the stated risk.

An A/A test makes the error visible

The simplest way to measure this cost is to compare a page with itself. In an A/A test, versions A and B are identical: any winner is a false winner, and the share of false winners directly measures the real risk.

We simulated 1,000 A/A tests of 18 days, with the same number of visitors every day. For each test and each day, we compute the cumulative difference between A and B divided by its uncertainty: the z-score. Then we read the same tests in two ways.

• Daily reading. Check every day and stop the test as soon as the difference crosses the classic 5% threshold, meaning an absolute z-score above 1.96.
• Planned checkpoints. Look only on days 6, 12, and 18, with a threshold for each checkpoint set before launch.

The draws are pseudo-random with a fixed seed, so the simulation returns the same figures every time this page loads.

Reading daily raises the risk from 5% to 21.8%

After 18 days, daily reading had declared a winner in 218 of 1,000 tests, or 21.8%. Read at the planned checkpoints, the same tests declare only 56, or 5.6%.

The daily-reading curve climbs from the start: 4.3% of tests cross the threshold on day 1, and the stated 5% risk is exceeded by day 2. It never comes back down, since a stopped test stays stopped. The first days are the most misleading: with few visitors, the observed difference swings a lot from one day to the next and crosses the threshold easily.

The planned-checkpoint curve stays flat between checkpoints and ends at 5.6%. Within ±1.4 points over 1,000 runs: consistent with the planned 5% risk. The remaining gap comes from the random draws: over 1,000 tests, this share varies by about 1.4 percentage points around its expected value.

A team that reads its tests every day and stops at the first significant difference turns roughly one no-effect test in 5 into a winner. Those false winners get shipped, their announced gain never shows up in revenue, and the next tests start from wrong conclusions.

O’Brien-Fleming boundaries let you look without distorting the risk

Banning looks isn’t realistic. A test that sinks sales has to be able to stop early, and a clearly better variant shouldn’t wait for no reason. The classic method sets the number of checkpoints in advance and gives each one a stricter threshold, calculated so that the overall risk stays at 5%.

The boundaries proposed by O’Brien and Fleming in 1979, tabulated by Jennison and Turnbull (2000, chapter 2), are very strict at first and close to the classic threshold at the end. For 3 equally spaced checkpoints, Probant’s statistical engine gives:

At the first checkpoint, only a huge difference stops the test: it takes a p-value below 0.05%, a threshold 96 times stricter than the classic one. At the last checkpoint, the boundary (2.00) is barely higher than the classic threshold (1.96). The test’s power is almost intact: to keep it at 80%, the planned sample size grows by only 1.7%.

The “Free shipping progress bar” test in the demo store was planned with boundaries of this kind. Its z-score at each checkpoint:

• Day 6: z-score of 2.30, against a boundary of 3.47. A daily reading would already have concluded; the test kept running.
• Day 12: z-score of 2.32, against a boundary of 2.45. A daily reading would already have concluded; the test kept running.
• Day 18: z-score of 3.09, against a boundary of 2.00. The boundary is crossed, so the test can conclude.

Probant sets the checkpoints before the first visitor

In Probant, a test plan is locked at launch: primary metric, target effect, duration, number of checkpoints, and boundaries. The report shows the difference and its range every day so you can follow the test, but only a planned checkpoint can conclude. Between two checkpoints, the test stays “running,” whatever the day’s difference.

• The duration and checkpoint days are calculated before launch, from the page’s traffic and the target effect (see Test plan and duration).
• The boundaries come from the statistical engine, checked against the values published by Jennison and Turnbull.
• Guardrails follow their own rule, with ranges designed to be read continuously: a test that hurts the return rate or page speed can stop at any time.

To measure the cost of peeking yourself, try the A/A simulator.

Sources

• Armitage, P., McPherson, C. K., and Rowe, B. C. (1969). “Repeated significance tests on accumulating data.” Journal of the Royal Statistical Society, Series A, 132(2), 235–244.
• O’Brien, P. C., and Fleming, T. R. (1979). “A multiple testing procedure for clinical trials.” Biometrics, 35(3), 549–556.
• Jennison, C., and Turnbull, B. W. (2000). Group Sequential Methods with Applications to Clinical Trials. Chapman & Hall/CRC, chapter 2.