A subset of visitors (mobile, desktop, new, returning customers) on which you reread the result. A difference between segments is presented as real only after a test of that difference, corrected with the Holm method.
Relu par segment, le CA par visiteur donne: mobile +9.5% (from +4.4% to +15.0%); desktop +2.7% (from -3.1% to +8.7%); new +7.4% (from +2.6% to +12.5%); returning +4.4% (from -1.8% to +11.1%).
Les écarts paraissent nets, mais après le test de différence et la correction de Holm, aucun n’est établi: chaque segment a moins de visiteurs que le test entier, et sa fourchette est plus large. Le rapport s’en tient à l’effet global.
Chiffres calculés par le moteur de Probant sur la boutique de démonstration, Maison Lune, fictive : ouvrir la démo.
An adjustment to p-values when you compare several things at once, so the risk of being wrong at least once stays at the stated level. Probant applies it to differences between segments.
Visitors aren’t split between versions as planned, a sign of an assignment problem. Probant checks every test with a chi-square test: below a p-value of 0.001, the verdict is blocked, whatever the lift.
The difference between the variant and the original, relative to the original: B / A − 1. A positive effect means the variant does better, in proportion to the starting value.
Create your workspace, or explore the demo first with data from a fictional store.