Visitors aren’t split between versions as planned, a sign of an assignment problem. Probant checks every test with a chi-square test: below a p-value of 0.001, the verdict is blocked, whatever the lift.
Autre nom : sample ratio mismatch
Prévu à parts égales, le test a reçu 30,650 visiteurs sur l’originale et 30,590 sur la variante. Test du khi-deux: valeur p 0.808, loin du seuil de 0.001. La répartition est normale.
Sur le même total, un écart de 815 visiteurs entre les deux versions, soit 1.3% du trafic, suffirait à déclencher l’alerte: le verdict serait bloqué, quel que soit l’écart de chiffre d’affaires.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
A test in which both versions are identical, so any winner it declares is false. Simulating A/A tests measures the real risk of a given way of reading results.
The modified version of the page, compared with the original, which is the current version. The original is arm A, the variant arm B.
The probability of seeing a lift at least this large if the two versions performed the same. Below 5%, the lift is called significant; in Probant, that’s the same as a 95% confidence interval that excludes zero.
Create your workspace, or explore the demo first with data from a fictional store.