Reassuring shoppers about shipping pays off.
Showing shipping costs early, highlighting free returns, and lowering the free-shipping threshold beat the original by +2.1% to +5.8%. Pooled effect: +2.5%, between +1.3% and +3.6%.
Loading…
Every statement links to the tests behind it.
Showing shipping costs early, highlighting free returns, and lowering the free-shipping threshold beat the original by +2.1% to +5.8%. Pooled effect: +2.5%, between +1.3% and +3.6%.
Cutting the filters and options shown on mobile helped in 2 of 5 tests, and never hurt. Pooled effect: +0.4%, between -0.3% and +1.2%.
Red strikethrough prices drew clicks but lowered average order value, and no promotion highlight produced a gain. Pooled effect: -0.9%, between -2.1% and +0.3%.
A lesson only states a direction if at least 3 tests on the theme point that way together, with the pooled effect’s 95% range entirely on one side of zero. Otherwise it stays “Not conclusive yet…”
What the stopped tests at Maison Lune learned, and whether each one followed the rules that make its verdict reliable. Each share is read on its own tests, as a number of tests, with the list one click away. No overall score, because it would hide which practice is missing.
Public benchmark: at Spotify, about 12% of tests win, but 64% learn something, whether a gain, a detected regression, or a conclusive neutral result (M. Bellato, M. Schultzberg, and S. Ankargren, “Beyond Winning: Spotify’s Experiments with Learning Framework,” Spotify Confidence, 2025). Here, a neutral result is conclusive when its 95% range excludes the minimum effect set in the plan, in both directions.
Hypothesis, metrics, target effect, sample size, and traffic split locked at launch, with their fingerprint, so the rules can’t change midway.
Visitors split between variants as planned. Otherwise, the measurement is skewed.
Stopped at the plan’s sample size, or by a stop the plan allows (boundary crossed at a checkpoint, guardrail). A test with no planned sample size doesn’t meet it.
Winner rolled out, loser discarded, nothing rolled out without proof. A deviation is shown with its justification and isn’t counted as matching.
For each winner, the probability that it actually has no effect, based on the power reached and the workspace’s success rate. Without a target effect in the plan, it can’t be calculated and counts as not met.
For each rolled-out variant, the difference measured again during the rollout steps, on other visitors, confirms the gain.
For a test tracked in Subscriptions, exposed visitors are read again after the stop at day 90, 180, and 365 (MRR, retention, net revenue) to confirm the decision. Only readings closed for the whole cohort count, with simultaneous ranges.
A test counts if it has a valid verdict and meets every criterion that applies to it. No weighting, no score. The share is read on the same tests as learning.
| Quarter ended | Tests | Learned | Winners | No reservations |
|---|---|---|---|---|
| Q4 2024few tests | 3 | 1 of 3 | 1 of 3 | 0 of 3 |
| Q1 2025few tests | 2 | 1 of 2 | 1 of 2 | 0 of 2 |
| Q2 2025few tests | 3 | 1 of 3 | 1 of 3 | 0 of 3 |
| Q3 2025few tests | 2 | 0 of 2 | 0 of 2 | 0 of 2 |
| Q4 2025few tests | 4 | 3 of 4 | 0 of 4 | 0 of 4 |
| Q1 2026 | 11 | 73% (8 of 11) | 18% (2 of 11) | 27% (3 of 11) |
| Q2 2026 | 11 | 91% (10 of 11) | 18% (2 of 11) | 82% (9 of 11) |
| Q3 2026 | 11 | 55% (6 of 11) | 27% (3 of 11) | 36% (4 of 11) |
| Q4 2026few tests | 2 | 2 of 2 | 1 of 2 | 1 of 2 |
Below 10 tests, a share is shown as a number of tests, because a percentage would be misleading on so few. Above that, the 95% range is given with each share.
When you plan a test, this distribution tells you whether the target effect is realistic for your store. It also corrects for the winner’s curse: “Free shipping from €60” measures +5.8%, and +4.8% is expected given the 47 other tests.
| Test | Ended | Result | Decision | Takeaway |
|---|---|---|---|---|
| Related products under the buy button | Oct 12 | Undecided-1.1%on “Add to cart” | Draws some attention away from the button, with no net loss | |
| Free shipping from €60 | Oct 2 | Winner+5.8%on “Revenue per visitor” | Customers add an item to reach the threshold | |
| Search suggestions | Sep 28 | Undecided+1.1%on “Product click” | Possible gain, too small to detect at this traffic | |
| Fewer form fields | Sep 28 | Undecided+1.0%on “Conversion” | Shorter form, effect too small to detect at this traffic | |
| Free returns highlighted | Sep 11 | Winner+3.4%on “Revenue per visitor” | Clearing up doubts about returns reassures new customers | |
| Strikethrough prices in red | Aug 28 | Loser-3.4%on “Revenue per visitor” | More clicks, but smaller orders | |
| Delivery time shown in the cart | Aug 24 | Undecided+1.2%on “Conversion” | Possible effect, too small to detect at this traffic | |
| Secure payment badges | Aug 17 | Undecided+0.6%on “Conversion” | No effect, so trust in payment wasn’t the obstacle |
Relative effect of the variant compared with the original on the test’s primary metric, with a 95% range in percentage points. Only visitors exposed to a single variant count, from their first exposure until the test stopped. An imported test is recalculated from its tool’s raw data. A switchback or geo test is read on its randomized pairs of time slots or zones, at the checkpoint that concludes, or else the last one read. Its range is that checkpoint’s, valid across all the plan’s checkpoints.
A complement to the verdicts, turned on in Settings → Statistics. Decisions followed each test’s plan.
9 of 11 test plan winners keep a credible interval above zero with a skeptical prior; their average effect goes from +5.8% measured to +3.8% posterior.
Each test’s prior is centered on zero and estimated from the other stopped, measured tests in Maison Lune’s library (standard deviation from 3.4% to 3.8%). The first figure comes from a flat (uninformative) prior, which treats a huge effect as just as plausible as a small one.