A test in which both versions are identical, so any winner it declares is false. Simulating A/A tests measures the real risk of a given way of reading results.
1,000 tests A/A de 18 jours, entre deux versions identiques: tout gagnant déclaré est faux. Lus chaque jour au seuil de 5%, 21.8% d’entre eux en déclarent un (±2.6 points, la précision d’une simulation de cette taille).
Lus aux three contrôles prévus par Probant, 5.6% (±1.4 points). La simulation ne dit rien d’un test en particulier: elle mesure le risque réel d’une façon de lire les résultats.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
The probability of declaring a winner when the two versions perform the same. Probant holds it at 5% over the whole test, checkpoints included. Reading a test every day at the usual threshold pushes it far higher.
Visitors aren’t split between versions as planned, a sign of an assignment problem. Probant checks every test with a chi-square test: below a p-value of 0.001, the verdict is blocked, whatever the lift.
A moment set in the test plan when Probant reads the test to decide. Between checkpoints, the daily range is shown, but it isn’t enough to conclude. A test plan includes three by default, evenly spaced.
Create your workspace, or explore the demo first with data from a fictional store.