A test that harms something stops on its own.
A variant can raise revenue while pushing up returns, slowing down mobile, or breaking a page. Guardrails watch what must not get worse, and stop the test as soon as the harm is proven.
Return rate: 1.6% → 2.9% of orders. An increase of +78%, beyond the +25% tolerance.
Read the transcript
What if the variant breaks something? Each guardrail watches a metric that must not get worse, and stops the test as soon as the harm is proven.
Each guardrail has a limit, set in the plan. It is breached only if its whole range is past the limit; otherwise it stays under watch.
A proven increase, but still under the limit. On “You may also like” recommendations, the increase in return rate is proven from Nov 4, but not yet past +25%, so the test continues.
Proven harm stops the test. On Nov 10 at 05:12, the whole range is past +25%: return rate from 1.6% to 2.9%. Visitors go back to the original.
A sample ratio mismatch blocks the test. On “One-page checkout”, B got only 47.2% of visitors: results are hidden, and the diagnosis points to Safari iOS.
The stopped test and its guardrails are in the demo, no sign-up needed.
Limits set in advance, read at every checkpoint.
Three situations from the demo store: a winner under watch, a test blocked by an assignment failure, and a test stopped by a guardrail.
Each guardrail has a limit, set in the test plan.
A guardrail is breached only when its whole range is beyond the limit: the harm is then proven. A range that overlaps the limit stays on “watch”. That is the case for javascript errors in “Free shipping progress bar”: -37.8% to +61.4%, against a limit of null.
- During the test, a breached guardrail blocks the verdict: no winner is possible
- Limits are visible in the test plan, before the first visitor
- Within limit
- The whole range stays below the limit.
- Watch
- The range overlaps the limit.
- Breached
- The whole range is beyond it: the harm is proven.
A sample ratio mismatch blocks the test.
If B gets fewer visitors than planned, the two groups are no longer comparable: B has lost the visitors for whom the page failed. On “One-page checkout”, B received only 47.2% of visitors. The browser-by-browser diagnosis points to Safari iOS: B is missing 1,108 visitors there, while the other browsers are balanced.
- Chi-square test at every reading, at a 0.001 threshold to avoid false alarms
- Results hidden until diagnosed
- Data discarded on restart, because a skewed split can’t be corrected afterward
| Browser | A | B | Share of B | p |
|---|---|---|---|---|
| Safari iOS | 2,960 | 1,852 | 38.5% | < 0.001 |
| Chrome Android | 3,380 | 3,379 | 50.0% | 1.00 |
| Chrome desktop | 2,640 | 2,641 | 50.0% | 1.00 |
| Safari macOS | 1,474 | 1,474 | 50.0% | 1.00 |
Proven harm switches everyone back to the original.
Guardrails are read at every checkpoint with an always-valid range, so you can look as often as you like without multiplying false alarms. On “You may also like” recommendations, the return rate went from 1.6% to 2.9% of orders, an increase of +78% beyond the +25% tolerance. Switched back to the original on Nov 10 at 05:12.
In detail.
- Guardrails
- Return rate, Page speed (LCP), JavaScript errors in the demo; each with its limit and direction (increase or decrease to avoid).
- Rule
- Breached when the whole range is beyond the limit; watch when it overlaps; within limit otherwise.
- During the test
- 95% range for each guardrail in the report; a breached guardrail blocks the verdict.
- During a rollout
- Always-valid ranges (mSPRT, Johari, Pekelis, and Walsh, 2017), read at every checkpoint of the step; automatic rollback on the first breach.
- Traffic split
- Chi-square on each version’s visitors, at a 0.001 threshold, then browser by browser to find the cause.
- In your workspace
- Each guardrail has its own limit, set on the metric (5% suggested by default). During a test as during a rollout, the rule is reread as events arrive, with always-valid ranges; a breach stops the test or switches visitors back to the original.
Frequently asked questions.
Can a guardrail be triggered by chance?
Rarely: the whole range has to be beyond the limit. Always-valid ranges keep that risk under control even when they are reread at every checkpoint, which a classic confidence interval does not allow.
Why wasn’t “You may also like” recommendations stopped earlier?
From Nov 4, the increase in returns was demonstrated, but not yet beyond +25%. So the test continued until Nov 10 at 05:12. To stop earlier, lower the tolerance in the test plan.
When do checks happen?
In the demo, at each daily checkpoint. In your workspace, the rule is reread as events arrive, and before any step up.
Next: Rollouts
See this feature in the demo, on the screen this page describes, or move on to the next one.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway