The winner’s curse: why the announced gain falls short

Keeping only the winners means keeping the lucky tests

Every difference measured in a test adds up two things: the variant’s real effect, and noise, meaning the chance that split visitors between A and B. That noise pushes one way as often as the other. Across tests picked at random, it cancels out on average: the estimate is unbiased.

The problem comes from sorting. Only the tests that won get shipped, which means those with a large measured difference. And a difference is more often large when noise happened to push in the right direction. Among winners, noise is no longer neutral: it inflates the announced effect. This is the winner’s curse.

The consequence is concrete. The gain announced in the report doesn’t show up in revenue over the following months. The team starts doubting the tool or the method, when the problem was the announcement, too good from the start. The gap is wider when tests lack visitors, because noise weighs more.

Winners announce +6.3% on average for a real effect of +5.1%

To measure this gap, we simulated 10,000 tests whose true effects look like those of the demo store: an average effect of +1.7%, with a spread of 3.4% around that average, estimated from its 48 completed tests. Each test is measured with the same precision as the “Free shipping progress bar” test, a standard error of 1.9%. A test is declared a winner when its difference is positive and significant at the 5% level.

Tests declared winners

3,000 of 10,00030.0%

Effect announced by these winners, on average

+6.3%

Real effect of the same tests, on average

+5.1%

Overstatement

+22.6%

None of these tests was measured badly. Taken one by one, each gives an unbiased estimate. Keeping only the winners is what creates the gap: on average, they announce 22.6% more than they actually deliver.

The correction draws on tests you’ve already completed

An isolated, noisy result should be read in light of what tests usually produce. If past tests rarely exceed a few percent, a large difference measured with a wide interval is probably inflated in part by noise.

Probant’s statistical engine works in two steps.

1. Estimate the distribution of true effects from completed tests. Across the 48 tests of the demo store, the average effect is +1.7%. The spread of measured effects mixes the spread of true effects with each test’s noise. Removing the noise, the mean of the squared standard errors, leaves the spread of true effects alone: 3.4%. This is the method of moments.
2. Combine that distribution with the test result, each weighted by its precision, the inverse of its variance. The more precise the test relative to the usual spread, the less the estimate moves. For the “Free shipping progress bar” test, the correction covers 25% of the distance between the measured difference and the average effect of past tests.

This is the most classic Bayesian calculation, a normal prior combined with a normal likelihood, also known as shrinkage. From the running test it uses only the estimate and its standard error, and from the history only the effects and standard errors of completed tests. The engine’s other methods and how they are checked are described on the “The statistics engine” page.

On the shipping bar, the estimate goes from +6.0% to +4.9%

The test measures a revenue-per-visitor difference of +6.0%, with a 95% confidence interval from +2.2% to +10.0%. After correction, the estimate is +4.9%, within an interval from +1.6% to +8.2%. The corrected interval is also narrower, because it benefits from the information past tests provide.

The verdict doesn’t change: even corrected, the interval stays above zero. Only the announced gain goes down. Converted to a month of cart traffic, the expected gain after correction is +€125,700 in revenue, within a range of +€41,600 to +€209,800. Since the amount is proportional to the difference, the raw estimate would have announced a gain 1.2 times higher.

The correction has known limits

• It needs a history. It takes at least two completed tests to estimate a spread, and the estimate gets sharper with every completed test.
• It assumes past tests resemble the running one. A price or shipping-fee change doesn’t have the same magnitude as a wording change; mixing the two widens the spread and weakens the correction.
• It always leaves room for the data. When all the spread of past tests is explained by noise, the spread of true effects is set to a small floor rather than zero; otherwise, later tests would be ignored entirely.
• It doesn’t replace a plan. A test stopped at the first significant difference is still a test stopped too early (see “Checking a test every day manufactures winners”).

Probant shows the measured difference and quantifies the corrected one

In the demo store’s report, the measured difference and its range stay visible, and only the corrected estimate is used to calculate amounts. The “Winner’s curse” line of the reliability checks reads “corrected, +4.9%.”

Monthly amounts start from the corrected estimate and are rounded to the nearest hundred euros: showing an estimated gain down to the euro would suggest a precision it doesn’t have. In your workspace, the same library of completed tests is used to estimate the expected effect of your ideas before you test them.

Sources

• Thaler, R. H. (1988). “Anomalies: The Winner’s Curse.” Journal of Economic Perspectives, 2(1), 191–202.
• Efron, B., and Morris, C. (1975). “Data Analysis Using Stein’s Estimator and Its Generalizations.” Journal of the American Statistical Association, 70(350), 311–319.
• Gelman, A., and Carlin, J. (2014). “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science, 9(6), 641–651.