The peeking problem: run it yourself.
Two identical versions, 1,000 tests simulated in your browser. Read every day at the 5% threshold, 21.8% of them (give or take ±2.6 points) declare a winner that doesn’t exist. Read at Probant’s three checkpoints: 5.6%, give or take ±1.4 points.
Choose how often you look.
Each square is a 18-day A/A test: A and B are the same page, so any winner is false. A square lights up on the day the reading stops on a lift that crosses the threshold.
Impatient reading
Every day, 18 looks in total, stopping at the first lift that crosses the threshold.
Probant’s reading
Three checkpoints set before launch, with O’Brien-Fleming boundaries.
Cumulative false winners, day by day
Impatient reading: 218 false winners out of 1,000, or 21.8%. Probant’s reading: 56 out of 1,000, or 5.6%.
Every extra look adds risk.
A 5% threshold promises one false winner in 20 when you look only once. Each extra look is another chance for noise to cross the threshold, and the impatient reader stops at the first one.
Across 10,000 tests: one look at the end, 4.9%; one look a day, 24.7% (give or take ±0.8 points); 36 looks, 29.7%. The experiment above, over 1,000 tests, finds 21.8% for one look a day, give or take ±2.6 points: two draws, two estimates of the same risk, with overlapping intervals. Armitage, McPherson, and Rowe calculated this risk back in 1969: it keeps growing with the number of looks.
| Looks | False winners |
|---|---|
| 1 lookat the end | 4.9% |
| 2 looksevery 9 days | 8.3% |
| 3 looksevery 6 days | 11.1% |
| 6 looksevery 3 days | 15.9% |
| 9 looksevery 2 days | 19.0% |
| 18 looksevery day | 24.7% |
| 36 lookstwice a day | 29.7% |
A lift crosses the threshold by chance, then falls back.
With no difference between A and B, the z-score, the lift divided by its uncertainty, drifts with each new visitor. It goes beyond ±1.96 for a day or two, then comes back down.
A reader who stops that day keeps a false winner. O’Brien-Fleming boundaries are high early on, when the z-score moves most, and only come down to the usual threshold at the last checkpoint, so a brief excursion rarely crosses them. In this simulation, 5.6% of tests cross a boundary. Within ±1.4 points over 1,000 runs: consistent with the planned 5% risk.
Looking is fine. Deciding on a glance isn’t.
Set the duration before you launch.
It follows from the effect you want to be able to detect and the tested area’s traffic.
Calculate a durationDecide only at planned checkpoints.
Probant shows the daily confidence interval so you can follow the test, but only a boundary crossed at a checkpoint allows a decision. The test plan includes three checkpoints by default, and you can choose from 1 to 5.
Planned checkpointsTo monitor continuously, switch intervals.
An always-valid interval stays accurate even when reread at any moment. It’s wider, so Probant reserves it for rollout guardrails.
RolloutsMeasure your own way of reading.
Duration, number of tests, checkpoints: the A/A simulator replays the experiment with your settings.
Open the simulator
References
- Armitage, McPherson, and Rowe, “Repeated significance tests on accumulating data,” Journal of the Royal Statistical Society, Series A, vol. 132, no. 2, 1969.
- O’Brien and Fleming, “A multiple testing procedure for clinical trials,” Biometrics, vol. 35, no. 3, 1979.
- Jennison and Turnbull, “Group Sequential Methods with Applications to Clinical Trials,” Chapman & Hall/CRC, 2000.
- Johari, Koomen, Pekelis, and Walsh, “Peeking at A/B Tests: Why it matters, and what to do about it,” KDD, 2017.
The step-by-step story, with the boundary table and the references, is in the blog post Checking a test every day manufactures winners.
The method, point by point.
Conclusive Or not yet…
Create your workspace, or explore the demo first with data from a fictional store.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway