The probability of declaring a winner when the two versions perform the same. Probant holds it at 5% over the whole test, checkpoints included. Reading a test every day at the usual threshold pushes it far higher.
Autre nom : type I error rate, significance level
Sur 10,000 tests A/A simulés de 18 jours, entre deux versions identiques, une lecture unique au seuil de 5% déclare un faux gagnant dans 4.9% des cas (±0.4 points): le risque annoncé.
Lus chaque jour au même seuil, avec arrêt au premier écart “significatif”, ces tests déclarent un faux gagnant dans 24.7% des cas (±0.8 points). Les points de contrôle de Probant gardent le risque à 5% sur l’ensemble du test.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
A test in which both versions are identical, so any winner it declares is false. Simulating A/A tests measures the real risk of a given way of reading results.
A moment set in the test plan when Probant reads the test to decide. Between checkpoints, the daily range is shown, but it isn’t enough to conclude. A test plan includes three by default, evenly spaced.
The probability of seeing a lift at least this large if the two versions performed the same. Below 5%, the lift is called significant; in Probant, that’s the same as a 95% confidence interval that excludes zero.
Create your workspace, or explore the demo first with data from a fictional store.