A winner’s reported gain is too good, on average.
Among many tests, the winners are partly the ones chance helped, so their measured lift overstates the real gain on average. Probant corrects it with the store’s completed tests before announcing an amount.
Revenue per visitor · Free shipping progress bar
Demo store
+€125,700in expected revenue a month, after correction
A winner owes its win partly to chance.
Every measurement mixes the real effect with noise. Among a store’s tests, the ones that clear the threshold include those with a good real effect and those that noise pushed up. Keeping only the winners means keeping those lucky pushes, so their average lift is too good.
Each measurement, on its own, is right on average. Selecting the winners is what creates the bias, and it grows as the test gets smaller, and therefore noisier.
The store’s completed tests
Demo store, relative effects on their primary metric
- Completed tests
- 48
- Average effect
- +1.7%
- Spread of measured lifts
- 4.0% (standard deviation)
- Share due to measurement noise
- 28% of the variance
- Spread of true effects
- 3.4% (standard deviation)
Measurement noise explains 28% of the variance in measured lifts: true effects are more tightly clustered around their average than the lifts suggest.
The demo test, before and after correction.
“Free shipping progress bar” measured +6.0% on Revenue per visitor across 61,240 visitors. Compared with the store’s 48 completed tests, the expected gain is +4.9%.
48 completed testsaverage +1.7%, true effects from -4.9% to +8.3%
Measured lift +6.0%interval from +2.2% to +10.0%
After correction +4.9%interval from +1.6% to +8.2%
The correction is a weighted average.
The corrected gain falls between the measured lift and the average of past tests. The more precise the test, the more it pulls toward its own measurement. This is Bayesian shrinkage: the prior comes from past tests, the likelihood from the test.
Past tests give the prior.
Average of measured effects: +1.7%. Spread of true effects: 3.4%, that is, the variance of measured lifts minus their average measurement noise (method of moments). It takes at least two completed tests; without them, there’s nothing to correct with.
The test is weighted by its precision.
The test’s standard error is 1.95%. Its weight is τ² / (τ² + se²), where τ is the spread of true effects and se the standard error: 75% for the test’s measurement, 25% for the average of past tests.
The corrected gain is the average of the two.
75% × +6.0% + 25% × +1.7% = +4.9%. Its confidence interval runs from +1.6% to +8.2%, narrower than the measurement’s alone (from +2.2% to +10.0%), since it combines two sources.
In euros, the correction removes about €27,600 a month.
The announced amount is the lift per visitor multiplied by the area’s visitors over a month, rounded to the nearest hundred. It’s proportional to the lift used.
a month, roughly: the figure a tool without correction would report.
in expected revenue a month, range +€41,600 to +€209,800.
a month, range -€5,400 to +€60,900. With no margin history of its own, it gets the same relative correction as revenue.
The method, point by point.
Conclusive Or not yet…
Create your workspace, or explore the demo first with data from a fictional store.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway