Variance reduction: the same metric measured before the test explains part of the differences between visitors, and removing it narrows the confidence interval without biasing the effect. Probant applies it to your test reports, using each visitor’s history before their first exposure.
Autre nom : Controlled-experiment Using Pre-Experiment Data
Le CA par visiteur des 14 jours précédant le test est corrélé à 0.16 avec celui du test. CUPED retire cette part prévisible: 2.7% de la variance de l’écart disparaît, et l’effet mesuré ne change pas (+6.0%).
Sans CUPED, la fourchette va from +2.17% to +10.02%; avec, from +2.22% to +9.97%: elle se resserre de 1.3%. À précision égale, il aurait fallu 62,925 visiteurs sans CUPED, au lieu de 61,240. Le gain dépend de la corrélation: plus le comportement passé prédit le comportement pendant le test, plus il est grand.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
The range of true lifts compatible with the data, at the 95% level. Probant always shows the lift with its range: if the range is entirely above zero, the variant wins; if it contains zero, nothing is proven.
The number of visitors set by the test plan before launch, based on the target effect and the area’s traffic. Until it’s reached, only a boundary crossed at a checkpoint allows a decision.
The smallest lift the test has a good chance of detecting if it exists, with 80% power and a 5% risk. The smaller it is, the more visitors you need: aiming for an effect half as large takes about four times as many visitors.
Create your workspace, or explore the demo first with data from a fictional store.