Is your A/B test really a winner?
Enter each version’s visitors and conversions, or their revenue. Probant’s engine gives the confidence interval of the difference, checks the visitor split, and accounts for how many times you looked at the result.
Prefilled with the “Free shipping progress bar” test from the demo store (Maison Lune, fictional), on day 18.
Verdict
Conclusive. Proven gain
Conversion rate is up +3.9%, interval from +0.7% to +7.2%.
- Conversion rate
- 20.0% → 20.8%
- p-value
- 0.016z-score 2.40, threshold 1.96
- Split (SRM)
- normalp-value 0.808, alert below 0.001
Even when proven, a winner can be false: with 10% of ideas that work (published default) and 80% power, 22% of winners like this one have no real effect (Kohavi, Deng, and Vermeer, 2022). The announced gain is also too good on average: Probant corrects it using your past tests.
Why the announced gain is too goodThe same result, read once or every day.
Conversion rate in the “Free shipping progress bar” test: 20.0% for the original, 20.8% for the variant, or +3.9% (z-score 2.40). Read once, at the planned end, the result crosses the 1.96 threshold: the variant wins.
Read every day for 18 days, the same figure proves nothing: at that pace, 23.7% of tests with no real effect declare a winner, instead of 5%. That’s why Probant sets its checkpoints before launch and reads them with O’Brien-Fleming boundaries.
Four checks, all run by the engine.
The difference and its interval.
Relative difference B / A − 1, tested on the difference in means; its 95% interval is Fieller’s, which never contradicts the p-value. See the confidence interval and the p-value.
The visitor split.
Chi-squared test against the planned split; below 0.001, the verdict is blocked. See SRM.
Repeated reads.
For one read, the usual threshold; up to 5 reads, the last O’Brien-Fleming boundary (Jennison and Turnbull, 2000); beyond that, no known threshold. The real risk comes from a simulation of A/A tests, as in the A/A simulator.
The false-positive risk.
Among winners declared under the same conditions, the share with no real effect (Kohavi, Deng, and Vermeer, “A/B Testing Intuition Busters,” KDD 2022). See also the winner’s curse.
Before the test, the duration calculator tells you how many visitors you need. The calculation runs in your browser, and no data is sent.
Frequently asked questions.
My test is significant: can I stop it?
Only if you looked at it once, at the planned end. Every look where you could have stopped it raises the risk of a false winner: read daily for 18 days at the 5% threshold, a test with no real effect declares a winner 23.7% of the time. Enter the number of looks: the tool applies the threshold that keeps the stated risk.
What is the SRM check?
It checks that visitors split between the versions as planned, with a chi-squared test. Below a p-value of 0.001, an assignment or tracking problem is almost certain, and the observed difference means nothing, however clear it looks.
Why can a proven winner be false?
Because most ideas tested change nothing. With 10% of ideas that work, 80% power, and a 5% threshold, 22% of declared winners have no real effect (Kohavi, Deng, and Vermeer, 2022). Enter your own share of winners for a figure that fits you.
Why doesn’t the tool correct the gain for the winner’s curse?
Because it takes your team’s past tests: they show what effect size is plausible for you. No published value is reliable enough to replace them. Probant makes this correction in every report, using your test library.
Is my data sent anywhere?
No. The calculation runs in your browser; nothing is sent or stored.
Plan this test
A test plan calculates the same duration from your store’s measured traffic, sets its checkpoints, and seals them before the first visitor.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway