Know the duration before the first visitor.
Probant starts from your traffic to tell you what effect the test can detect, how many visitors it needs, how many days it will run, and when it will be read. Then the plan stays locked.
Test plan · Free shipping progress bar
- Primary metric
- Revenue per visitor
- Minimum detectable effect
- 5.5%
- Planned sample size
- 58,900
- Duration
- 18 days, 3 checkpoints
- Guardrails
- Return rate ≤ +25.0% · Mobile speed ≤ +4.0%
Read the transcript
How long until you know? Probant starts from your traffic: you know the duration before the first visitor, and the rule stays fixed after that.
You pick a target effect; Probant gives the duration. On “Add to cart” (8.2%), 8% can be detected in 30 days at the zone’s traffic; 2% would take 466 days.
Watched daily, decided at checkpoints. “Free shipping progress bar”: on day 6, z = 2.30 cleared the classic threshold of 1.96 but not the boundary of 3.47. The test continued until day 18.
At launch, the plan is sealed. Target effect, 58,900 visitors, three checkpoints, and guardrails: a fingerprint lets you verify that nothing has changed.
Work out the duration of your next test in the demo, no sign-up needed.
A test plan computed, then locked.
The visuals come from two demo store tests: “Free shipping progress bar”, launched on Oct 23, and “Estimated delivery date”, in preparation.
You choose the target effect; Probant works out the duration.
For each metric, Probant computes the visitors and days needed to detect a given effect, based on the page’s traffic and only on visitors who accepted measurement. It recommends the smallest effect measurable within a month.
- On “Add to cart”, 8% can be detected in 30 days
- A more sensitive metric shortens the test
- Black Friday and Christmas fall during the test: these periods are analyzed separately, as set in the test plan
You can look at the test every day, but it is decided only at checkpoints.
The test plan sets checkpoints in advance with O’Brien-Fleming boundaries: very strict at first, close to the classic threshold at the end. On day 6, z was 2.30, not enough against the boundary of 3.47, so the test continued. The boundary was crossed on day 18.
- 1 to 5 checkpoints, 3 by default
- The extra sample for checkpoints is already included in the duration (+1.7% visitors here)
- The false winner risk stays the one set in the plan
| Checkpoint | Boundary | Observed z |
|---|---|---|
| Day 6 | 3.47 | 2.30 |
| Day 12 | 2.45 | 2.32 |
| Day 18 | 2.00 | 3.09 |
At launch, the plan is locked, guardrails included.
Primary metric, target effect, sample size, duration, checkpoints, and guardrail limits are set before the first visitor. The report is read against this plan, because changing the rule mid-test is the surest way to produce a false winner.
A multivariate test tells you in advance whether it can conclude.
Each visitor counts toward every section: a section with L levels needs L times the sample size of an A/B test, with the risk shared across the reported comparisons. On “Orbe product page: title, photo, and badge” (12 combinations), detecting 10% on each section takes 36 days at the store’s traffic.
- Reporting interactions between sections too: 169 days.
- When traffic is short, the plan suggests removing a level or a section, each time with its new duration.
- Up to 16 combinations: beyond that, each would get too few visitors to be read.
- Computed on your workspace’s measured traffic, and sealed at launch like an A/B test plan.
What can’t be split visitor by visitor is tested by region or by time slot.
A price, shipping fees, or stock can’t differ between two neighbors. Probant then randomizes whole French departments, or switches the entire site from one time slot to the next, and analyzes regions or time slots rather than individual visitors.
- By region: departments are paired on their pre-test rate, then randomized within each pair
- The department comes from a postal code header set by your CDN, or from a MaxMind-format geolocation database; the IP address is neither kept nor logged. Without one of these sources, the department level is unavailable, and the screen says so
- By time slot: the whole site serves one version during a slot, then the other
- Read on regions or time slots: “Conclusive.” only if both the Student’s t-test and the permutation test conclude
- The plan tells you in advance what too few regions won’t be able to prove
In detail.
- Effects offered
- 2%, 5%, 8%, 10%, 15%, 20% relative, from finest to coarsest.
- Duration
- In days, based on the daily traffic of the tested pages (in your workspace, the figure you enter from your analytics tool), split between the original and the variants. The recommendation aims for 30 days at most.
- Traffic share
- 100%, 75%, 50%, 25%, 10% of the page’s traffic, excluding the holdout.
- Checkpoints
- 1 to 5 checkpoints, with tabulated O’Brien-Fleming boundaries (Jennison and Turnbull).
- Risk and power
- False winner risk of 5%; the target effect is detected 80% of the time.
- Consent
- Only visitors who accepted measurement count toward available visitors.
- End
- “Stop at the end of the planned duration” is checked by default: on that date, the script stops applying the test and the stop is recorded in the audit log. An end date you enter takes precedence; if unchecked, the test runs until you stop it.
- Expected effect
- For a test based on an idea, the plan shows the effect the learnings library suggests, and warns when the target effect exceeds it or when the test covers only some devices.
- Before launch
- Script seen on the target pages, orders received from the back office, running tests actually displayed, script cost measured in the lab, overlap with other tests on the page, excluded pages. A check without data stays “not verified”, and nothing blocks the launch.
- Metrics
- Rates (order, add to cart…) or per-visitor averages (revenue per visitor). An average also needs its standard deviation.
Frequently asked questions.
What if my traffic is low?
The test plan tells you before launch: the duration gets longer, and beyond a month the result holds less well for the rest of the year. Probant then suggests a larger effect or a more frequent metric, such as add to cart rather than order.
Can I look at results before the end?
Yes: the range is shown every day. But only a planned checkpoint can conclude. On day 6 of “Free shipping progress bar”, z = 2.30 was above the classic threshold of 1.96; the plan’s boundary was 3.47, and the test continued.
Can I change the plan once the test is live?
No. Metric, target effect, duration, and checkpoints are locked at launch, which is what makes the verdict reliable. You can stop a test, but not change how it is read.
Next: Ideas
See this feature in the demo, on the screen this page describes, or move on to the next one.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway