Simulate A/A tests and count the false winners.
In an A/A test, both versions are identical, so any winner is false. Set the number of tests, their length, and the planned checkpoints, then compare daily reading at the 5% threshold with Probant’s reading.
Daily reading
You check all 18 days and stop as soon as the lift crosses the 5% threshold.
Probant’s reading
Three checkpoints set before launch, with O’Brien-Fleming boundaries.
Cumulative false winners, day by day
Daily reading: 218 false winners out of 1,000, or 21.8%. Probant’s reading: 56 out of 1,000, or 5.6%.
Simulation result
Read every day, 21.8% of tests declare a false winner: 4.4 times the planned 5% risk. Read at Probant’s checkpoints: 5.6%. Within ±1.4 points over 1,000 runs: consistent with the planned 5% risk.
| Reading | False winners | Share | Simulation margin |
|---|---|---|---|
| Daily reading | 218 of 1,000 | 21.8% | ± 2.6 points |
| Probant’s reading | 56 of 1,000 | 5.6% | ± 1.4 points |
Planned risk: 5%. The margin is the 95% interval around a share measured over 1,000 tests: the more tests, the closer the simulated share gets to the true risk.
What the tool simulates, and what it leaves out.
Each test gets the same number of visitors every day. Its cumulative standardized lift, the z-score, moves at random, since A and B don’t differ.
Daily reading.
The z-score is read every evening; the test stops on the first day the lift exceeds the usual 5% threshold. That’s how a dashboard checked every day gets read.
Probant’s reading.
The z-score is read only at the planned checkpoints, placed as in a Probant test plan, and compared with the engine’s O’Brien-Fleming boundaries, for 1 to 5 checkpoints.
What isn’t simulated.
Day-to-day traffic swings, weekly patterns, and returning visitors. The simulation isolates one thing: the cost of repeated reading.
The calculation runs in your browser, and no data is sent. The full explanation is on the The peeking trap page and in the article “Checking a test every day manufactures winners.”
Plan this test
A test plan calculates the same duration from your store’s measured traffic, sets its checkpoints, and seals them before the first visitor.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway