Shipping
4 tests · 3 winners
+2.5% pooled effect, +1.3% to +3.6%
High confidence. Together, the tests point the same way: the pooled range excludes zero.
A testing program that doesn’t contradict itself.
Checkpoints set before launch, gains corrected for the winner’s curse, ideas ranked from past tests: the same rule applies from one test to the next.
Program summary
65%of tests conclusive: proven gain, loss, or “no effect”
Across 1,000 simulated A/A tests, with no real difference:
22% false winners when reading every day6% at the planned checkpoints
Maison Lune, a fictional demo store · library computed by the engine
A testing program loses credibility one test at a time.
Rarely through one big failure. More often through winners that never show up in the numbers and tests that contradict each other.
- Daily peekingReading a test every day and stopping at the first clear lift multiplies false winners.
- The winner’s curseA winner’s measured effect is too good on average, so the announced gain never shows up in the quarter.
- Overlapping testsTwo tests on the same page see the same visitors, and one muddies the reading of the other.
- A backlog sorted on gut feelHomegrown scores mix hoped-for impact and estimated effort, with no link to what past tests delivered.
- A program with no memoryResults live in slide decks, so teams retest what already failed.
The rule is set before the test, and it’s the same for everyone.
Each answer rests on a published method and on the library of your own tests.
Planned checkpointsO’Brien-Fleming boundaries set at launch: you can look at the daily range, but only a planned checkpoint decides.Across 1,000 simulated A/A tests: 22% false winners when reading daily, 6% at the planned checkpoints.Learn more A corrected gainThe measured effect is pulled toward what your past tests deliver on average, before it’s converted into euros.“Free shipping progress bar”: +6.0% measured, +4.9% reported after correction.Learn more A ranking backed by numbersExpected effect from tests on the same theme, duration from the area’s traffic, gain per day of effort.Top of the list: “Free shipping from €50”, +2.5% based on 4 shipping tests.Learn more A library that remembersCompleted tests on one scale, grouped into lessons by theme, each with a confidence level.“Shipping” lesson: 4 tests · 3 winners, pooled effect +2.5%.Learn more Visible checksTraffic split, sample reached, boundary crossed: listed next to the verdict for anyone who wants to check.Planned sample size: 104% · A/B split: 50.0 / 50.0 · Sample ratio mismatch (SRM) test: p = 0.81Learn more Tests that don’t mixWithin one layer, tests take separate slices of traffic, so a visitor sees only one of them.A slice that overlaps another is rejected when you create it.Learn more
Yesterday’s lessons size tomorrow’s ideas.
In the demo store: lessons from the most-tested themes, and the ideas at the top of the ranking. An idea’s expected effect comes from its theme’s lesson.
Lessons from the most-tested themes
Mobile browsing
5 tests · 2 winners
+0.4% pooled effect, -0.3% to +1.2%
Medium confidence. Enough tests, but the pooled effect is still compatible with no effect.
Promotions
3 tests · 1 loser
-0.9% pooled effect, -2.1% to +0.3%
Medium confidence. Enough tests, but the pooled effect is still compatible with no effect.
Top of the ranking
- Rank 1Free shipping from €50
- Expected effect
- +2.5%based on 4 shipping tests
- Time to conclude
- 14 daysmeasuring checkout startsinstead of 62 days on orders
- Estimated gain
- +€62,900per month
- Rank 2Free returns noted in the cart
- Expected effect
- +1.7%prior2 reassurance tests, too few
- Time to conclude
- 29 daysmeasuring checkout startsinstead of 132 days on orders
- Estimated gain
- +€43,000per month
- Rank 3Secure payment methods shown in the cart
- Expected effect
- +1.7%prior2 reassurance tests, too few
- Time to conclude
- 29 daysmeasuring checkout startsinstead of 132 days on orders
- Estimated gain
- +€43,000per month
A program week, with no arguments over the numbers.
How a typical week goes, and what the demo store shows at each step.
- Review
Review the idea ranking.
Ideas are ranked by expected gain per day of effort. Ideas whose theme lowered the metric are flagged “Avoid.”
Demo storeTop of the list: “Free shipping from €50”, “Free returns noted in the cart”, and “Secure payment methods shown in the cart”. - Launch
Launch with a locked plan.
Target effect, visitors, checkpoint days, and guardrails are set at launch and visible to the whole team.
Demo storeFree shipping progress bar: target effect 5.5%, 58,900 visitors, checkpoints on days 6, 12, and 18. - Checkpoints
Decide only at planned checkpoints.
Between checkpoints, you can look at the daily range without deciding anything. The extra cost of planned looks is built into the test length.
Demo storeExtra cost of the three checkpoints: 1.7% more visitors. - Decision
Decide on the corrected gain.
The report gives the verdict, the gain corrected for the winner’s curse in revenue and margin, and the checks alongside.
Demo store+€125,700 per month, corrected from +6.0% to +4.9%. - Build on it
Let the library learn.
The completed test joins its theme’s lesson, winner or not, and the expected effect of related ideas updates.
Demo store“Shipping” lesson: 4 tests · 3 winners, high confidence.
Frequently asked questions.
Can we look at a test while it runs?
Yes, the daily range stays visible. But only a planned checkpoint allows a decision: across 1,000 simulated A/A tests (two identical versions), reading the test every day declares a false winner 22% of the time, versus 6% at the planned checkpoints.
How are ideas ranked?
Each idea gets an expected effect drawn from past tests on the same theme (or from the whole learnings library when the theme has too few tests), a duration based on its area’s traffic, and a monthly gain. The ranking divides that gain by the estimated effort.
Can two tests run on the same page?
Yes, in two different layers. Within one layer, tests take separate slices of traffic: a visitor sees only one of them, and Probant rejects a slice that overlaps another.
What happens to a test with no detectable effect?
It goes into the learnings library with its range, like a winner, and counts toward its theme’s lessons and the expected effect of related ideas.
Is the reported gain the one the test measured?
No. It’s corrected for the winner’s curse before being converted into euros: the measured effect is pulled toward what your past tests deliver on average, because a winner overestimates its effect on average.
Other use cases
The same engine, for other site areas and other teams. The demo opens on the screen this page describes.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway