Roll out the winner in steps, with guardrails.
A test proves that a variant earns more; it doesn’t prove that nothing breaks at scale. A rollout exposes it gradually, checks guardrails at every step, and rolls back automatically at the first proven harm.
Read the transcript
Ship the winner to everyone, right away? The share of visitors goes up in steps, and drops to zero at the first proven harm.
Each step has a minimum duration. 25%, 50%, then 100%. “Free shipping from €60” is at 50%.
Guardrails are rechecked at every checkpoint. The ranges stay valid whatever day you read them, so checking them at every checkpoint doesn’t multiply false alarms.
At the first proven harm, roll back. Same rule for a test: “You may also like” recommendations, return rate from 1.6% to 2.9%. All visitors went back to the original on Nov 10 at 05:12.
This rollout and this test are in the demo, no sign-up needed.
Scale up without needless risk.
The visuals come from the demo store’s rollouts, replayed checkpoint by checkpoint by the engine: steps of 25%, 50%, 100%.
Each step has a minimum duration.
The new version is shown to a growing share of traffic and compared, at each step, with visitors still on the original. The next step is allowed once the minimum duration has passed and no guardrail is breached: you decide when, or, with automatic advance turned on, Probant moves to the next step at the following checkpoint. “Free shipping from €60” uses automatic advance: its first step advanced on its own. Moving to 100% waits until every range is settled, and so does automatic advance.
- Intermediate steps cap the risk
- Going to 100%: every range must be on the safe side of its limit, because afterward there is no original left to compare with; automatic advance waits for the same condition
- Automatic advance: “Move to the next step automatically after the minimum duration if no guardrail worsens”, an option off by default, which you can set at launch or later
- 25%passedOct 5 → Oct 73 days · minimum 3 days
- 50%in progressOct 8 → Nov 1033 days · minimum 3 days
- 100%upcoming—
- JavaScript errorscan’t be predicted
Guardrails are read at every checkpoint.
Guardrail ranges are always valid: you can check them at every checkpoint and stop at the first breach without inflating the error risk, which a classic confidence interval does not allow.
- Guardrails are the workspace’s metrics with the “guardrail” role, each with its limit: in the demo, return rate, page speed, and JavaScript errors
- Speed (LCP, INP, CLS) read at the 75th percentile per pageview of each group, with an interval that accounts for pages from the same visitor
- State of each guardrail: within limit, not settled yet, or limit exceeded
Proven harm triggers a rollback.
As soon as a guardrail range is entirely beyond its limit, all visitors go back to the original: rollback is always automatic, whether or not advance is. The same rule stops a test: in the demo, the return rate of “You may also like” recommendations exceeded its +25.0% limit on Nov 10 at 05:12, and the test stopped.
Nov 10 at 05:12, after 13 checkpoints. All visitors see the original again.
The history shows what advanced each step.
Each step is logged with its dates and what ended it: automatic advance, a team member’s decision, or a guardrail. Nothing to change on your site: the script and the server helper read each rollout’s share from the configuration, and an exposed visitor stays exposed at the following steps.
- Free returns highlighted: served to everyone since Sep 28
- Simplified filters on mobile: served to everyone since Aug 31
- Shipping costs shown on the product page: served to everyone since Jul 27
- Guest checkout: served to everyone since Jun 30
- Simplified order summary: served to everyone since Mar 26
| Rollout | Steps | Advance |
|---|---|---|
| Free returns highlighted | 25% (advanced automatically) → 50% (decided) → 100% | automatic |
| Simplified filters on mobile | 10% (decided) → 25% (decided) → 50% (decided) → 100% | manual |
| Shipping costs shown on the product page | 10% (advanced automatically) → 25% (advanced automatically) → 50% (decided) → 100% | automatic |
| Guest checkout | 25% (decided) → 50% (decided) → 100% | manual |
| Simplified order summary | 25% (decided) → 50% (decided) → 100% | manual |
In detail.
- Suggested steps
- Standard: 25% → 50% → 100%; Cautious: 10% → 25% → 50% → 100%. Up to six steps, the last one at 100%.
- Minimum duration
- 3 days in full per intermediate step, to cover different days of the week.
- Advance
- Manual by default: you decide when. As an option, per rollout: “Move to the next step automatically after the minimum duration if no guardrail worsens”. Monitoring applies it at most every 5 minutes, logs it in the audit log, and announces it in alerts.
- Speed
- LCP, INP, and CLS read at the 75th percentile per pageview, step by step, exposed vs. original.
- Rollback
- Automatic as soon as a guardrail is breached; the rule is checked again before every step up.
- Who sees the variant
- Each visitor is drawn once for the rollout; the share only goes up, so an exposed visitor stays exposed.
- Holdout
- The workspace’s holdout stays out: it measures the cumulative impact of winners, rolled-out ones included.
- Feature flags
- A feature flag serves a value to your server code, by rules or by percentage, and can be turned off in one action. It doesn’t drive a rollout; it can follow a test, and then that test’s rollout.
Frequently asked questions.
Why roll out a proven winner in steps?
The test measured the primary metric on part of the traffic. The rollout checks, at a larger scale, that the workspace’s guardrails don’t get worse: returns, speed, errors.
What does a visitor see during a rollback?
The original, from the next configuration on. The rollout disappears from the public configuration; an older script also shows the original, not an unintended variant.
When do checks happen?
In the demo, checkpoints are replayed day by day or week by week. In your workspace, the rule is reread as events arrive, and before any step up.
Can Probant advance steps on its own?
Yes, if you turn on automatic advance for the rollout, at launch or later. Probant then moves to the next step as soon as the minimum duration has passed with no guardrail breached, and to 100% only when every range is within its limit. It logs this in the audit log and announces it in alerts. Without guardrails, or if there is a sample ratio mismatch, it waits.
Next: Personalization
See this feature in the demo, on the screen this page describes, or move on to the next one.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway