An adjustment to p-values when you compare several things at once, so the risk of being wrong at least once stays at the stated level. Probant applies it to differences between segments.
Autre nom : Holm–Bonferroni method
Le test se relit par appareil (mobile +9.5%, desktop +2.7%) et par ancienneté (new +7.4%, returning +4.4%). Les écarts entre segments ont des valeurs p de 0.081 et 0.454.
Corrigées par la méthode de Holm pour ces deux comparaisons, elles deviennent 0.162 et 0.454. Aucune ne passe sous 5%: le rapport ne présente pas l’écart entre segments comme réel.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
A subset of visitors (mobile, desktop, new, returning customers) on which you reread the result. A difference between segments is presented as real only after a test of that difference, corrected with the Holm method.
The probability of seeing a lift at least this large if the two versions performed the same. Below 5%, the lift is called significant; in Probant, that’s the same as a 95% confidence interval that excludes zero.
The probability of declaring a winner when the two versions perform the same. Probant holds it at 5% over the whole test, checkpoints included. Reading a test every day at the usual threshold pushes it far higher.
Create your workspace, or explore the demo first with data from a fictional store.