9 metrics · 3 guardrails

What your metrics can detect.

The smallest difference a 4-week test could tell apart from chance on each metric, based on traffic from the last 7 days. A typical Maison Lune winner is +3.7%: 2 of 6 primary metrics can detect it in that time.

Typical winner: +3.7%; median of 8 winners
Variance reduced by CUPED: 6%; at best, on add to cart: the same metric over the previous 14 days explains part of the differences between visitors, which the report removes.

The most sensitive, Product click, detects from 1.8%. The least sensitive, Add to cart, from 5.5%.

What each metric detects in 4 weekswith CUPEDwithout CUPEDtypical winner, +3.7%stop threshold

Primary metrics · minimum detectable effect

Guardrails · drift that triggers the automatic stop

Catalog9

“Detects from”: the smallest difference a 4-week A/B test would tell apart from chance, not a result. For a guardrail, the drift that would trigger the automatic stop within 4 weeks. The reading compares a primary metric with a typical winner (+3.7%), and a guardrail with its stop threshold.

Primaryscriptmeasured in warehouse

Revenue per visitor

Revenue brought in by each visitor exposed to the test, whether they buy or not. It’s the metric that best shows what a change earns.

Formula

Order amounts from exposed visitors, divided by the number of exposed visitors. A visitor with no order counts as €0.

Baseline: €23.89
Standard deviation: €59.89; per visitor
Population: 3,402 visitors / day; Cart visitors
CUPED: variance reduced by 3%; ρ = 0.16, based on each visitor’s history

Minimum detectable effect

Duration	without CUPED	with CUPED
2 weeks	6.4%	6.3%
4 weeks	4.6%	4.5%

A smaller difference may exist, but the A/B test wouldn’t tell it apart from chance in that time.

Slow

It takes 43 days to see an effect the size of a typical winner (+3.7%), 42 days with CUPED. Aim for clear-cut changes or a busier zone.

Source: “Order” event sent by the site with its amount, reconciled with back-office orders
Usage: 7 finished tests · 5 running

Method

As in a real workspace: visitors from the last 7 days, the same traffic assumed over 2 and 4 weeks, two versions with equal shares, a 5% risk of a false conclusion, and 80% power. Visitors who don’t buy count as €0. CUPED, applied in a test’s report to its primary metric and per-visitor secondary metrics, uses each visitor’s value over the 14 days before their first exposure: ρ is the correlation between the two periods, and variance drops by ρ². Ratios, percentiles, and guardrails are read without CUPED. A guardrail’s automatic stop waits until its entire always-valid range (mSPRT, τ = 25%) crosses the threshold. A drift equal to the threshold never gets there, hence the delay given for 2 times the threshold. Page speed is read at the 75th percentile of page views. Maison Lune’s visitors are randomly drawn from the store’s rates, and everything else is calculated from them.

What your metrics can detect.

Typical winner

+3.7%

median of 8 winners

Variance reduced by CUPED

at best, on add to cart: the same metric over the previous 14 days explains part of the differences between visitors, which the report removes.

The most sensitive, Product click, detects from 1.8%. The least sensitive, Add to cart, from 5.5%.

without CUPED

with CUPED

2 weeks

6.4%

6.3%

4 weeks

4.6%

4.5%

Method

Measure

The winner’s curse

Measure

The winner’s curse

What your metrics can detect.

Catalog9

Revenue per visitor

Method

Measure

The winner’s curse

What your metrics can detect.

Catalog9

Revenue per visitor

Method