A metric that must not get worse (return rate, page speed, errors), with a limit set in the test plan. A guardrail stops the test only if its whole range exceeds the limit; a range that straddles the limit doesn’t prove harm.
Le plan fixe au taux de retour une limite de +25%: une hausse prouvée au-delà arrêterait le test, quel que soit le gain de chiffre d’affaires. Mesuré: +2.6%, avec une fourchette from -9.2% to +15.9%.
Toute la fourchette reste sous la limite: le garde-fou tient. L’écart mesuré seul ne suffirait pas à arrêter le test: seule une fourchette entièrement au-delà de la limite prouve un dégât.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
A range you can reread at any time and stop on at the first crossing without inflating the risk (Johari, Pekelis, and Walsh, 2017). It’s wider than a classic confidence interval. Probant uses it for rollout guardrails.
Automatic withdrawal of a version being rolled out as soon as a guardrail is crossed, that is, as soon as its whole always-valid range exceeds the limit. All traffic goes back to the original.
The share of traffic that sees the new version during a gradual rollout. Probant moves to the next step once the minimum duration has passed and no guardrail is crossed; to go to all traffic, every guardrail must sit entirely on the right side of its limit.
Create your workspace, or explore the demo first with data from a fictional store.