One range per day
Each bar is the day’s difference and its 95% range. It narrows as visitors come in.
Scroll to move through the test.
Probant decides at moments set before launch.
The test plan sets the metric, the target effect, the duration, and three checkpoints before the first visitor. At each checkpoint, a boundary says whether you may conclude; the verdict follows rules applied in the same order every time.
Test plan · Free shipping progress bar
- Primary metric
- Revenue per visitor
- Minimum detectable effect
- 5.5%
- Planned sample size
- 58,900
- Duration
- 18 days, 3 checkpoints
- Guardrails
- Return rate ≤ +25.0% · Mobile speed ≤ +4.0%
The test plan is locked at launch.
Changing the metric or the duration after seeing the first numbers means choosing the question based on the answer. Probant sets everything before the first visitor, and the report judges the test against that plan.
- One primary metric
- The one that says what the change brings in. Secondary metrics are read alongside but can’t replace it after the fact.
- The target effect
- The smallest lift the test must be able to detect. It sets the number of visitors needed.
- The duration
- Calculated from the target effect and the traffic of the tested area, including checkpoint overhead: it’s the same calculation as the duration calculator.
- The checkpoints
- Their days and boundaries are known before launch. A promising lift doesn’t let you add one.
- Risk and power
- A 5% risk of declaring a false winner, and a 80% chance of detecting the target effect if it exists.
- Guardrails
- Each has its own limit. A test that crosses one is blocked, whatever the gain on the primary metric.
Three checkpoints, with strict boundaries early on.
At each checkpoint, Probant measures whether the primary metric’s lift is clear enough to conclude: that’s the z-score, the lift divided by its uncertainty, compared with an O’Brien-Fleming boundary. Until the boundary is crossed, the test continues.
1.96
z 2.30
z 2.32
z 3.09
- Day 6, checkpoint 1z = 2.30, above the usual threshold but below the 3.47 boundary: the test continues.
- Day 12, checkpoint 2z = 2.32, above the usual threshold but below the 2.45 boundary: the test continues.
- Day 18, checkpoint 3z = 3.09, beyond the 2.00 boundary: the test can conclude.
On day 6, the lift had already crossed the usual 1.96 threshold. A tool that reads every day would have declared a winner; Probant waited until day 18. Why daily reading creates false winners.
The boundary at checkpoint k of K is C·√(K/k): very high at the first look, close to the usual threshold at the last. The overall risk stays at 5%, for only +1.7% more visitors.
Boundaries by number of checkpoints, for a 5% risk and 80% power
| Checkpoints | Boundary at each checkpoint | Extra visitors |
|---|---|---|
| 1 | 1.96 | none |
| 2 | 2.80 · 1.98 | +0.8% |
| 3 (default) | 3.47 · 2.45 · 2.00 | +1.7% |
| 4 | 4.05 · 2.86 · 2.34 · 2.02 | +2.4% |
| 5 | 4.56 · 3.23 · 2.63 · 2.28 · 2.04 | +2.8% |
Sources: O’Brien and Fleming, “A multiple testing procedure for clinical trials,” Biometrics, 1979; constants and inflation factors tabulated by Jennison and Turnbull, “Group Sequential Methods with Applications to Clinical Trials,” 2000, chapter 2.
The demo test, day by day.
18 days, 3 planned checkpoints. Free shipping progress bar, demo store.
Day 6
The day’s range moves above zero. A tool that reads results every day would already call a winner.
The difference isn’t clear enough to conclude yet (Checkpoint 1: z = 2.30, below the boundary of 3.47). Probant waits for the next checkpoint. Z-score and boundary, in the glossary
Day 12
The day’s range stays above zero. The boundary gets closer to the usual threshold, but isn’t crossed.
The difference isn’t clear enough to conclude yet (Checkpoint 2: z = 2.32, below the boundary of 2.45). Probant waits for the next checkpoint. Z-score and boundary, in the glossary
Day 18
Conclusive.
Showing progress toward free shipping: revenue per visitor +6.0% measured, +4.9% expected once corrected for the winner’s curse. The effect on margin isn’t established yet: +3.0% measured, range from -0.7% to +6.7%.
The difference is clear enough to conclude at this checkpoint (Checkpoint 3: z = 3.09, above the boundary of 2.00). Z-score and boundary, in the glossary
The verdict follows six rules, in this order.
The engine applies the same rules to every test. The first two block the result: they make it doubtful or rule out going live. The rest look only at the primary metric’s confidence interval.
- 1
Sample ratio mismatch
A chi-square test compares the visitors each version received with the planned split. Below a p-value of 0.001, the result is blocked: the assignment is suspect, whatever the lift shown.
See this case in the demoSample ratio mismatch. - 2
Stopped by a guardrail
Each guardrail has a limit set in the test plan. It blocks the test if its whole confidence interval exceeds the limit; an interval that straddles the limit doesn’t prove harm. All blocking causes are listed together so they can be fixed in one go.
See this case in the demoStopped by a guardrail. - 3
Conclusive, proven gain
The primary metric’s confidence interval is entirely above zero once the planned sample size is reached. Before that, only an always-valid interval can conclude.
See this case in the demoConclusive. Proven gain - 4
Conclusive, proven loss
The interval is entirely below zero: the variant does worse than the original. Same word, “Conclusive.”, because the proof is the same; the orange dot and the text give the direction.
Conclusive. Proven loss - 5
No detectable effect
The planned sample size is reached and the whole interval sits between −1% and +1%: if there is an effect, it’s negligible. The rule applies to the whole interval, not to the minimum effect targeted in the plan.
See this case in the demoNo detectable effect. - 6
Not conclusive yet
In every other case. Before the planned sample size, the test continues to the next checkpoint. After it, the interval contains zero but extends beyond the margin: the report says how many days it would take to decide.
See this case in the demoNot conclusive yet…
Before the planned sample size, a winner or loser is accepted only if the decision is sequential: that’s what checkpoint boundaries are for. Otherwise, the engine notes that the sample size hasn’t been reached.
The demo test: “Positive effect: the range goes from +2.2% to +10.0%, entirely above 0.”
During a rollout, guardrails are reread continuously.
Once the winner is chosen, the new version goes out in steps to a growing share of traffic. At that point you need to be able to stop at any time: Probant reads guardrails with always-valid confidence intervals, which stay accurate even when reread at every check.
A wider interval you can reread without distorting the risk.
A classic confidence interval is accurate only once, on the planned date. The always-valid interval (mSPRT; Johari, Pekelis, and Walsh, 2017) can be reread as often as you like, and you can stop at the first crossing without inflating the risk. In exchange, it’s wider.
To decide a test, three checkpoints cost less (+1.7% more visitors). To monitor a rollout, where you want to be able to stop at any time, the always-valid interval is the right tool.
Return rate · “You may also like” recommendations
50/50 test, demo store. Measured lift: +77.9%.
Classic confidence interval+45.8% to +109.9%
Always-valid interval+25.4% to +130.3%
The whole always-valid interval exceeds the limit: automatic rollback, and all traffic goes back to the original.
The method, point by point.
The peeking trapWhy checking a test every day produces false winners.Read the page The winner’s curseWhy the reported gain is too good, on average.Read the page The statistics engineIts methods, their sources, and how each is verified.Read the page GlossaryConfidence interval, SRM, guardrail, MDE…Read the page Verify a reportRecompute a sealed plan’s fingerprint yourself.Read the page
Conclusive Or not yet…
Create your workspace, or explore the demo first with data from a fictional store.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway