Cart · all devices · launched Oct 23 by Camille · day 18

Free shipping progress bar

ConclusiveProven gainRunning · day 18 of 18

B beats the original.

Measured: +6.0%; +2.2% → +10.0%
Expected, corrected for winner’s curse: +4.9%; +1.6% → +8.2%

The measured difference in revenue per visitor is +6.0%. Winners overstate their effect on average. Corrected for the winner’s curse, the expected effect is +4.9%, and that’s what the amounts below use. The effect on “Margin per visitor” is not established yet: +3.0% measured, but the range (from -0.7% to +6.7%) includes zero.

Expected revenue / month: +€125,700; +41,600 → +209,800; corrected for the winner’s curse · 103,555 visitors per month
Expected margin / month: +€27,700; -5,400 → +60,900; corrected for the winner’s curse · 103,555 visitors per month

The rollout goes in steps, with an automatic return to the original if a guardrail degrades along the way.

Reliability

1 needs a look, 1 to watch

Planned sample size104% (passed)
A/B split50.0 / 50.0 (passed)
Sample ratio mismatch (SRM) testp = 0.81 (passed)
Saw both versions0.3% (needs a look)
Guardrails breachednone, 1 to watch (to watch)
Boundary crossed at checkpointday 18 · z = 3.1 (passed)
Orders reconciled98.6% (information)
Known costs (margin)96.7% (information)
Winner’s cursecorrected, +4.9% (passed)

Not known yet

“Return rate”: 30-day value, complete on Dec 9, 2026 at the earliest if the test stopped today.

Risk that this winner is false: 11%

Out of 100 winners obtained under these conditions, about 11 would actually have no effect.

The threshold5% risk, half of which can produce a false winner (the other half, a false loser).
Power reached81% chance of detecting the target effect (+6%) with the actual sample. The lower it is, the more real winners go unnoticed.
Good ideas are rare11 winners out of 48 finished tests in Maison Lune’s library: 21% used, tempered by the 10% prior.

Bayesian reading

A complement to the verdict, turned on in Settings → Statistics. The decision follows the sealed test plan.

Both readings agree: even pulled toward zero by the prior, the estimate for B stays above zero.
Posterior effect
+4.7%
between +1.3% and +8.1% (95% credible interval) · measured +6.0%
Probability that B is better
99.7%
with a flat (uninformative) prior: over 99.9%
Expected loss
Roll out B: €0.00039 per visitor
Keep A: €1.2 per visitor

Prior. True effect assumed centered on zero, with a standard deviation of 3.7%: 95% of effects between -7.3% and +7.3%. Width estimated from 48 stopped, measured tests in Maison Lune’s library, excluding this one, winners and losers alike.

Expected loss. What you lose on average if the choice turns out to be wrong, in the metric’s unit.

Not the same as the expected effect after rollout. The winner’s curse correction moves the difference toward the average of past tests to estimate the gain; this reading moves it toward zero, so it assumes nothing about the idea being tested.

All metrics

Metric	A	B	Relative difference · range	Reading
Revenue per visitorprimary	€24.58	€26.06	Winner +6.0% +2.2 → +10.0%	Winner
Margin per visitorafter product costs, discounts, and shipping	€10.03	€10.33	Undecided +3.0% -0.7 → +6.7%	Undecided
Conversion rate	20.0%	20.8%	Winner +3.9% +0.7 → +7.2%	Winner
Average order value	€122.90	€125.40	Winner +2.0% 0.0 → +4.1%	Winner
Orders with free shippingexplains the gap between revenue and margin	41.0%	57.0%	Expected+16 points	Expected
Guardrails
Return rateprovisional, 30-day value on Dec 9	1.6%	1.7%	Within limit +2.6% -9.2 → +15.9%	Within limit
Mobile speed (LCP)	2.41 s	2.42 s	Within limit +0.4% -0.2 → +1.0%	Within limit
JavaScript errorsshare of visitors with an error · limit +25%	0.12%	0.12%	To watch +0.2% -37.8 → +61.4%	To watch

D’où vient l’écart de B ?

Ce que compte le chiffre

« CA par visiteur » est une valeur par visiteur exposé : la somme de ce qu’il envoie après l’exposition, 0 s’il n’a rien envoyé. Chaque visiteur compte dans la variante de sa première exposition.

Les comptes

A · Originale : €753,377.00 au total sur 30,650 visiteurs, soit €24.58 par visiteur.
B · Free shipping progress bar : €797,167.80 au total sur 30,590 visiteurs, soit €26.06 par visiteur.

Le calcul

Écart relatif = valeur par visiteur de B ÷ valeur par visiteur de A − 1 = €26.06 ÷ €24.58 − 1 = +6.02%. Méthode
Test de la différence entre les deux variantes : z = 3.09, p = 0.002 (bilatéral).
Fourchette à 95.0% : from +2.17% to +10.0%. Ce sont tous les écarts que ce même test ne rejette pas (méthode de Fieller) : « significatif » et « fourchette qui exclut zéro » disent toujours la même chose. Méthode

Ce qui limite la lecture

Répartition contrôlée : les visiteurs se répartissent comme prévu, aux variations du hasard près (p = 0.808). Méthode
Une borne de contrôle du plan est franchie : le plan autorise à conclure avant l’échantillon prévu. Méthode

Segments

Difference of B vs. A in each segment, 95% range. A visitor counts in the segment of their first exposure.

Device

differences consistent · adjusted p 0.16

Mobile37,969 · 62%+9.53%+4.4 → +15.0%
Desktop23,271 · 38%+2.67%-3.1 → +8.7%

New or returning

differences consistent · adjusted p 0.45

New43,481 · 71%+7.43%+2.6 → +12.5%
Returning17,760 · 29%+4.41%-1.8 → +11.1%

Traffic source

Only one segment received, so there is nothing to compare.

No visitors with this context yet.

After correcting for 2 compared dimensions, these differences between segments are not significant. Don’t draw conclusions from them.

A segment is measured once each variant has 100 visitors and 10 visitors with a value.

Surveys1

Share of each answer among visitors who responded (their latest response, one per visitor), 95% ranges. A variant can change who responds, so to decide, compare the response rate across all exposed visitors.

How did your order go?

Responses sent under enquete-commande-detail, then under “enquete-commande-detail--q2” and so on for the following questions (4 questions, conditional logic). A proportion metric on this name measures the response rate.

Measurement survey: asked identically in every branch, including the original, so it can be compared across variants.

Completed the survey

A · Original640 out of 785 finished: 81.5% (between 78.7% and 84.1%)

B · Shipping bar662 out of 788 finished: 84.0% (between 81.3% and 86.4%)

Each question after the first is only asked to respondents whose answers lead to it, so its response rate and shares are read among them. Those who close the survey partway count where they stopped.

Question 1How did your order go?

AOriginal785 answers out of 30,650 exposed visitors: 2.6% (between 2.4% and 2.7%)

Very well46.9% · between 43.4% and 50.4% · 368
Well34.4% · between 31.2% and 37.8% · 270
So-so14.1% · between 11.9% and 16.8% · 111
Badly4.6% · between 3.3% and 6.3% · 36

BShipping bar788 answers out of 30,590 exposed visitors: 2.6% (between 2.4% and 2.8%)

Very well50.5% · between 47.0% and 54.0% · 398
Well36.2% · between 32.9% and 39.6% · 285
So-so10.0% · between 8.1% and 12.3% · 79
Badly3.3% · between 2.3% and 4.8% · 26

Answers distributed differently between Original and Shipping bar, beyond chance at a 5% risk (chi-square, p = 0.035, corrected for 3 compared questions (Holm)).

Question 2What convinced you to order from us?

AOriginal555 answers out of 638 respondents who were asked: 87.0% (between 84.2% and 89.4%)

The price36.2% · between 32.3% and 40.3% · 201
The selection30.1% · between 26.4% and 34.0% · 167
Delivery16.2% · between 13.4% and 19.5% · 90
A recommendation8.6% · between 6.6% and 11.3% · 48
An offer8.8% · between 6.7% and 11.5% · 49

BShipping bar605 answers out of 683 respondents who were asked: 88.6% (between 86.0% and 90.8%)

The price30.7% · between 27.2% and 34.5% · 186
The selection26.6% · between 23.2% and 30.3% · 161
Delivery27.1% · between 23.7% and 30.8% · 164
A recommendation7.8% · between 5.9% and 10.2% · 47
An offer7.8% · between 5.9% and 10.2% · 47

Answers distributed differently between Original and Shipping bar, beyond chance at a 5% risk (chi-square, p = 0.001, corrected for 3 compared questions (Holm)).

Question 3What went wrong?

AOriginal124 answers out of 147 respondents who were asked: 84.4% (between 77.6% and 89.3%)

Payment7.3% · between 3.9% and 13.2% · 9
The stated lead time23.4% · between 16.8% and 31.6% · 29
The fees55.6% · between 46.9% and 64.1% · 69
The website8.9% · between 5.0% and 15.2% · 11
Something else4.8% · between 2.2% and 10.2% · 6

BShipping bar84 answers out of 105 respondents who were asked: 80.0% (between 71.4% and 86.5%)

Payment11.9% · between 6.6% and 20.5% · 10
The stated lead time35.7% · between 26.3% and 46.4% · 30
The fees31.0% · between 22.1% and 41.5% · 26
The website14.3% · between 8.4% and 23.3% · 12
Something else7.1% · between 3.3% and 14.7% · 6

Answers distributed differently between Original and Shipping bar, beyond chance at a 5% risk (chi-square, p = 0.030, corrected for 3 compared questions (Holm)).

Question 4Tell us more, if you don’t mind.

AOriginal85 answers out of 124 respondents who were asked: 68.5% (between 59.9% and 76.1%)

BShipping bar57 answers out of 84 respondents who were asked: 67.9% (between 57.3% and 76.9%)

Funnels

Purchase

Shipping bar does better from “Add to cart”.

1Product pageA · 70.1%21,480baselineB · 70.0%21,410-0.1%-1.4 → +1.2
2Add to cartA · 36.0%11,020 · 51% of the previous stepbaselineB · 37.2%11,390 · 53% of the previous step+3.6%+0.8 → +6.4
3Checkout startedA · 25.6%7,840 · 71% of the previous stepbaselineB · 26.7%8,170 · 72% of the previous step+4.4%+0.9 → +8.0
4CommandeA · 20.0%6,130 · 78% of the previous stepbaselineB · 20.8%6,357 · 78% of the previous step+3.9%-0.2 → +8.1

Share of exposed visitors who reach each step in order, each in the variant of their first exposure. Simultaneous ranges: 4 comparisons share a 5% risk. Step-to-step rates describe the funnel without being compared, since who reaches the previous step already depends on the variant. The verdict is still the primary metric’s.

Difference in revenue per visitor, day by day

95% range each day. Decisions are made only at the 3 planned checkpoints, to avoid piling up false winners.

Day 18

Estimated difference: +6.0%
Range: +2.2 → +10.0
Visitors: 61,200

checkpoint 1 · z 2.3 < 3.47checkpoint 2 · z 2.3 < 2.45checkpoint 3 · z 3.1 ≥ 2.00+30%+20%+10%0-10%

J6 · Oct 28, 2026 · “New collection” email campaign sent to subscribers, linking to both variants.
J11 · Nov 2, 2026 · Payment incident: PayPal unavailable from 2:00 to 4:30 p.m.
J15 · Nov 6, 2026 · 15% off lighting, shown in both variants.

Notes

What happened during the test (campaign, incident, promotion), marked on the chart so you can read a difference in context. Notes are not used in any calculation.

“New collection” email campaign sent to subscribers, linking to both variants.Oct 28, 2026 · Camille Roux
Payment incident: PayPal unavailable from 2:00 to 4:30 p.m.Nov 2, 2026 · Camille Roux
15% off lighting, shown in both variants.Nov 6, 2026 · Camille Roux

Does the effect hold over time?

Each visitor counts for what they do in the 24 hours after their first visit to the test, so visitors from the first days and later ones are followed for the same length of time.

No sign that the effect is fading.+5.4% for visitors who arrived in the first 3 days, +5.6% for later ones (adjusted p 1.000); slope of -0.01% per cohort day over 17 days (adjusted p 1.000): nothing beyond chance.

Arrived in the first 3 days10,200 visitors+5.4%-5.4 → +17.4
Arrived later (days 4 to 17)47,600 visitors+5.6%+0.5 → +11.0
New visitors41,038 visitors+5.5%+0.1 → +11.4
Returning visitors16,762 visitors+5.5%-3.0 → +14.8

Periods analyzed separately

An additional reading: the verdict is still the plan’s, on all visitors. For each period, visitors whose first visit to the test falls within it are compared with the others, and all are followed until the end of the test. 95% ranges. The difference between the two effects is tested with the Holm correction across all comparisons in this section.

Maison Lune private sale · Oct 30 to Nov 2, 2026
Set in the sealed plan
- Arrived during the period14,270 visitors+2.1%-5.0 → +9.8
- Arrived outside the period46,970 visitors+7.3%+2.8 → +12.0
Nothing shows the effect changed during the “Maison Lune private sale” period: +2.1% during, +7.3% outside the period, a 5.2 points difference (adjusted p 0.234). A non-significant difference doesn’t prove the effect was the same.

The test plan, set at launch

Metric, target effect, duration, and checkpoints were set before the first visitor. The verdict is read against this plan, because changing the rules midway is the surest way to produce a false winner.

Primary metric: Revenue per visitor; baseline €24.58 per visitor
Target effect: +5.5%; detected 80% of the time
Sample size: 58,900; visitors, 29,450 per version
Planned duration: 18 days; last checkpoint on day 18

Checkpoints

Checkpoint 1 · 19,634 visitors33% of the sample, reached on day 6 · boundary not crossed, the test continues|z| ≥ 3.47
Checkpoint 2 · 39,267 visitors67% of the sample, reached on day 12 · boundary not crossed, the test continues|z| ≥ 2.45
Checkpoint 3 · 58,900 visitors100% of the sample, reached on day 18 · boundary crossed: the plan allows a conclusion|z| ≥ 2.00

Very strict thresholds at first, close to the standard threshold at the end. Looking 3 times costs 1.7% more visitors and keeps the false-winner risk at 5%.

Decision rules

Winner if, at a checkpoint, the difference in Revenue per visitor crosses the boundary and no guardrail has degraded.
Return rate: no more than +25%, range included. Beyond that, the test stops and traffic returns to the original.
Mobile speed (LCP): no more than +4%, range included. Beyond that, the test stops and traffic returns to the original.
Traffic split: 50/50, checked at every reading. An imbalance hides the results until it’s diagnosed.

Sealed plan

Locked at launch on Oct 23, 10:47 AM by Camille Roux · fingerprint 6b984159…77933d

No deviations. The test follows its sealed plan.

What was sealed · show

Hypothesis: If the cart shows how much is left to qualify for free shipping, customers will add an item rather than pay for shipping, and revenue per visitor will go up.
Primary metric: Revenue per visitor
Secondary metrics: Margin per visitor, Conversion rate
Guardrails: Return rate (stop on an increase of more than 25.0%), Mobile speed (LCP) (stop on an increase of more than 4.0%)
Target effect: +6%, risk 5%, power 80%
Checkpoints: 3
Sample size: 29,450 visitors per variant
Planned duration: 18 days
Traffic split: A / B: 50% / 50%
Targeting: zone “Cart”, 100% of its traffic · workspace holdout group of 5.0% excluded · pages /panier
Separately analyzed periods: Maison Lune private sale (Oct 30 to Nov 2, 2026)
Entered traffic: 3,272 visitors per day, mean 24.58 (standard deviation 58.28)

Full fingerprint: 6b98415912596ed972d1076b4520ba76f957fa180060ddd899cd85518f77933d. It’s the SHA‑256 of the text below, so anyone can recompute it and check that the plan wasn’t altered.

{"allocation":[{"changes":"4f53cda18c2baa0c","key":"a","name":"Original","weight":1},{"changes":"0d9a08f73a38dae2","key":"b","name":"Progress bar","weight":1}],"declared":{"baseline":{"kind":"mean","mean":24.58,"sd":58.28126285522647},"dailyVisitors":3272},"hypothesis":"If the cart shows how much is left to qualify for free shipping, customers will add an item rather than pay for shipping, and revenue per visitor will go up.","metrics":[{"aggregation":"visitor","capPercentile":null,"direction":null,"eventName":"commande","key":"ca","kind":"mean","limit":null,"name":"Revenue per visitor","role":"primary"},{"aggregation":"visitor","capPercentile":null,"direction":null,"eventName":"marge","key":"marge","kind":"mean","limit":null,"name":"Margin per visitor","role":"secondary"},{"aggregation":"visitor","capPercentile":null,"direction":null,"eventName":"commande","key":"conversion","kind":"proportion","limit":null,"name":"Conversion rate","role":"secondary"},{"aggregation":"visitor","capPercentile":null,"direction":"increase-is-bad","eventName":"retour","key":"retours","kind":"proportion","limit":2500,"name":"Return rate","role":"guardrail"},{"aggregation":"visitor","capPercentile":null,"direction":"increase-is-bad","eventName":"lcp","key":"lcp","kind":"mean","limit":400,"name":"Mobile speed (LCP)","role":"guardrail"}],"periods":[{"end":"2026-11-02","key":"saisie-2026-10-30-maison-lune-private-sale","label":"Maison Lune private sale","source":"manual","start":"2026-10-30"}],"plan":{"alpha":0.05,"armSizes":null,"looks":3,"mde":0.05520567661771653,"perArm":29450,"plannedDays":18,"power":0.8,"totalVisitors":58900},"targeting":{"endsAt":null,"holdoutShare":0.05,"rules":{"urlPattern":"/panier"},"startsAt":null,"trafficShare":1,"zone":"Cart"},"version":1}

Once rolled out, the winning variant can leave the script. Its changes become code that stays on your site.

Write-up

draft written by Probant

Result. B beats the original. On “Revenue per visitor”, B: +6.0% measured, from +2.2% to +10.0%.

Expected effect. Corrected for the winner’s curse, the expected effect once rolled out is +4.9% (from +1.6% to +8.2%), vs. +6.0% measured. Expected gain: +€125,700 per month (+41,600 → +209,800).

Margin per visitor. The effect on “Margin per visitor” is not established yet: +3.0% measured, but the range (from -0.7% to +6.7%) includes zero.

Segments. After correcting for multiple comparisons, the differences between segments cannot be told apart from chance, so there is nothing to conclude.

Guardrails. “JavaScript errors”: to watch, the range still overlaps the limit. “Return rate”, “Mobile speed (LCP)”: within the limit.

Test plan. The test followed the plan sealed at launch.

What’s next

Backlog ideas to test next, measurable in under 60 days. Expected effect, duration, and gain come from the engine, as on the Ideas screen.

Cart · shipping
Free shipping from €50
No. 1 in the backlog, Cart. +2.5% expected, based on 4 shipping tests; measurable in 14 days.
+€62,900expected per monthCreate the test
Cart · reassurance
Free returns noted in the cart
No. 2 in the backlog, Cart. +1.7% expected, prior; measurable in 29 days.
+€43,000expected per monthCreate the test
Cart · reassurance
Secure payment methods shown in the cart
No. 3 in the backlog, Cart. +1.7% expected, prior; measurable in 29 days.
+€43,000expected per monthCreate the test

Sessions to watch

All 12

A and B take turns, each from highest to lowest score, so every variant has sessions at the top of the list.

How these sessions are chosen

Each session from a test visitor gets a score, the sum of:

rage-click burst (3 clicks within 700 ms): +3, up to 3 bursts
dead click (nothing changes within a second): +2, up to 3
JavaScript error: +2, up to 3
saw the tested element, then left without converting: +5
converted after seeing the tested element: +4

Sessions with a score of zero aren’t suggested. On equal scores, the most recent comes first; the list then alternates the variants (A, B, A, B…) so each one has its sessions near the top. A session ends after 30 minutes of inactivity.

Summaries built from the recorded events, without an AI model. No input is recorded. Form fields and areas marked data-probant-mask are masked.

Click and scroll maps

Where people click and how far they read, variant by variant, from session recordings.

Open

/panier12,201 visitors · changed by the test

Element or depthABDifference · range
Reach 90% of the pagescrollA50.7%3,090 / 6,096B55.1%3,361 / 6,105+4.4 pts+2.3 → +6.4 · the difference holds
Reach 100% of the pagescrollA40.2%2,449 / 6,096B44.4%2,712 / 6,105+4.2 pts+2.2 → +6.3 · the difference holds
Reach 80% of the pagescrollA64.3%3,922 / 6,096B68.2%4,163 / 6,105+3.9 pts+1.9 → +5.8 · the difference holds
Ambre candle · €14.90[data-cross-sell="bougie-ambre"]A4.8%293 / 6,096B8.4%510 / 6,105+3.5 pts+2.5 → +4.6 · the difference holds
Filament bulb · €9.90[data-cross-sell="ampoule-filament"]A3.4%205 / 6,096B6.0%365 / 6,105+2.6 pts+1.8 → +3.5 · the difference holds
Shipping calculated at the next step[data-cart-total] pA8.3%508 / 6,096B5.9%361 / 6,105-2.4 pts-3.5 → -1.4 · the difference holds
Reach 70% of the pagescrollA80.6%4,915 / 6,096B82.9%5,059 / 6,105+2.2 pts+0.7 → +3.8 · the difference holds
Increase quantity[data-qty-plus]A3.9%237 / 6,096B5.7%350 / 6,105+1.8 pt+1.0 → +2.7 · the difference holds
Shipping and returns[data-footer-help]A5.3%321 / 6,096B3.6%218 / 6,105-1.7 pt-2.5 → -0.9 · the difference holds

10 other comparisons consistent with no difference

Element or depthABDifference · range
Reach 60% of the pagescrollA89.6%5,459 / 6,096B90.7%5,540 / 6,105+1.2 pt0.0 → +2.4 · consistent with no difference
Check out[data-checkout]A48.6%2,963 / 6,096B50.4%3,077 / 6,105+1.8 pt-0.3 → +3.8 · consistent with no difference
Lighting[data-nav-luminaires]A1.1%70 / 6,096B1.5%92 / 6,105+0.4 pt-0.1 → +0.8 · consistent with no difference
Have a question?[data-chat-launcher]A1.3%79 / 6,096B1.2%72 / 6,105-0.1 pt-0.6 → +0.3 · consistent with no difference
Free shipping from €60[data-announcement]A3.7%225 / 6,096B3.5%214 / 6,105-0.2 pt-0.9 → +0.6 · consistent with no difference
Remove[data-remove]A2.5%153 / 6,096B2.4%145 / 6,105-0.1 pt-0.8 → +0.5 · consistent with no difference
Apply[data-promo-apply]A5.7%347 / 6,096B5.5%336 / 6,105-0.2 pt-1.1 → +0.8 · consistent with no difference
Masked field#promo-codeA8.5%516 / 6,096B8.3%504 / 6,105-0.2 pt-1.3 → +0.9 · consistent with no difference
Opaline lamp[data-cart-item-title]A2.1%125 / 6,096B2.1%131 / 6,105+0.1 pt-0.5 → +0.7 · consistent with no difference
Menu[data-menu-toggle]A2.9%179 / 6,096B2.8%173 / 6,105-0.1 pt-0.8 → +0.6 · consistent with no difference

Share of the page’s visitors who click the element at least once, or whose screen reaches that depth, Original vs. Progress bar. 19 comparisons (14 elements, 5 depths): differences that hold are selected so that, on average, no more than 5% of the flagged ones are false, and each range, at 97.6%, is widened accordingly, so it excludes zero exactly when a difference holds. A difference in clicks describes behavior. It doesn’t say which variant is better: only the test’s metric decides that.

Forms

Where people hesitate, correct, and drop off, field by field and variant by variant. Never what they type.

Open

/commande/livraison · [data-checkout-form] · 2,309 visitors started it

The test is conclusive. Progress bar completes it more often than Original (+7.8 pts, range +4.5 → +11.1). The difference comes down to “Delivery method” (-6.5 pts in drop-offs for Progress bar). The form shows where the difference happens; only the test’s metric decides.

FormA · completeB · completeDifference · range
Completion rateA75.2%857 / 1,140B83.0%970 / 1,169+7.8 pts+4.5 → +11.1 · clear difference

Original vs. Progress bar, among visitors who started the form. Ranges at 95.0%: the 5% risk is split across 1 comparison.

FieldA · leave hereB · leave hereDifference · range
Delivery method[name="livraison"]A10.5%120 / 1,140B4.0%47 / 1,169-6.5 pts-9.4 → -3.6 · clear difference

Original vs. Progress bar, among visitors who started the form. Ranges at 99.38%: the 5% risk is split across 8 comparisons. 2 fields with too few exits to compare.

B beats the original.

Measured

+6.0%

+2.2% → +10.0%

Expected, corrected for winner’s curse

+4.9%

+1.6% → +8.2%

Expected revenue / month

+€125,700

+41,600 → +209,800

corrected for the winner’s curse · 103,555 visitors per month

Expected margin / month

+€27,700

-5,400 → +60,900

corrected for the winner’s curse · 103,555 visitors per month

The rollout goes in steps, with an automatic return to the original if a guardrail degrades along the way.

Metric

Relative difference · range

Reading

Revenue per visitorprimary

€24.58

€26.06

Winner

+6.0% +2.2 → +10.0%

Winner

Margin per visitorafter product costs, discounts, and shipping

€10.03

€10.33

Undecided

+3.0% -0.7 → +6.7%

Undecided

Conversion rate

20.0%

20.8%

Winner

+3.9% +0.7 → +7.2%

Winner

Average order value

€122.90

€125.40

Winner

+2.0% 0.0 → +4.1%

Winner

Orders with free shippingexplains the gap between revenue and margin

41.0%

57.0%

Expected+16 points

Expected

Guardrails

Return rateprovisional, 30-day value on Dec 9

1.6%

1.7%

Within limit

+2.6% -9.2 → +15.9%

Within limit

Mobile speed (LCP)

2.41 s

2.42 s

Within limit

+0.4% -0.2 → +1.0%

Within limit

JavaScript errorsshare of visitors with an error · limit +25%

0.12%

To watch

+0.2% -37.8 → +61.4%

To watch

How did your order go?

Measurement survey: asked identically in every branch, including the original, so it can be compared across variants.

Completed the survey

A · Original640 out of 785 finished: 81.5% (between 78.7% and 84.1%)

B · Shipping bar662 out of 788 finished: 84.0% (between 81.3% and 86.4%)

Each question after the first is only asked to respondents whose answers lead to it, so its response rate and shares are read among them. Those who close the survey partway count where they stopped.

Question 1How did your order go?

AOriginal785 answers out of 30,650 exposed visitors: 2.6% (between 2.4% and 2.7%)

Very well46.9% · between 43.4% and 50.4% · 368
Well34.4% · between 31.2% and 37.8% · 270
So-so14.1% · between 11.9% and 16.8% · 111
Badly4.6% · between 3.3% and 6.3% · 36

BShipping bar788 answers out of 30,590 exposed visitors: 2.6% (between 2.4% and 2.8%)

Very well50.5% · between 47.0% and 54.0% · 398
Well36.2% · between 32.9% and 39.6% · 285
So-so10.0% · between 8.1% and 12.3% · 79
Badly3.3% · between 2.3% and 4.8% · 26

Answers distributed differently between Original and Shipping bar, beyond chance at a 5% risk (chi-square, p = 0.035, corrected for 3 compared questions (Holm)).

Question 2What convinced you to order from us?

AOriginal555 answers out of 638 respondents who were asked: 87.0% (between 84.2% and 89.4%)

The price36.2% · between 32.3% and 40.3% · 201
The selection30.1% · between 26.4% and 34.0% · 167
Delivery16.2% · between 13.4% and 19.5% · 90
A recommendation8.6% · between 6.6% and 11.3% · 48
An offer8.8% · between 6.7% and 11.5% · 49

BShipping bar605 answers out of 683 respondents who were asked: 88.6% (between 86.0% and 90.8%)

The price30.7% · between 27.2% and 34.5% · 186
The selection26.6% · between 23.2% and 30.3% · 161
Delivery27.1% · between 23.7% and 30.8% · 164
A recommendation7.8% · between 5.9% and 10.2% · 47
An offer7.8% · between 5.9% and 10.2% · 47

Answers distributed differently between Original and Shipping bar, beyond chance at a 5% risk (chi-square, p = 0.001, corrected for 3 compared questions (Holm)).

Question 3What went wrong?

AOriginal124 answers out of 147 respondents who were asked: 84.4% (between 77.6% and 89.3%)

Payment7.3% · between 3.9% and 13.2% · 9
The stated lead time23.4% · between 16.8% and 31.6% · 29
The fees55.6% · between 46.9% and 64.1% · 69
The website8.9% · between 5.0% and 15.2% · 11
Something else4.8% · between 2.2% and 10.2% · 6

BShipping bar84 answers out of 105 respondents who were asked: 80.0% (between 71.4% and 86.5%)

Payment11.9% · between 6.6% and 20.5% · 10
The stated lead time35.7% · between 26.3% and 46.4% · 30
The fees31.0% · between 22.1% and 41.5% · 26
The website14.3% · between 8.4% and 23.3% · 12
Something else7.1% · between 3.3% and 14.7% · 6

Answers distributed differently between Original and Shipping bar, beyond chance at a 5% risk (chi-square, p = 0.030, corrected for 3 compared questions (Holm)).

Question 4Tell us more, if you don’t mind.

AOriginal85 answers out of 124 respondents who were asked: 68.5% (between 59.9% and 76.1%)

BShipping bar57 answers out of 84 respondents who were asked: 67.9% (between 57.3% and 76.9%)

Funnels

Purchase

Shipping bar does better from “Add to cart”.

1Product pageA · 70.1%21,480baselineB · 70.0%21,410-0.1%-1.4 → +1.2
2Add to cartA · 36.0%11,020 · 51% of the previous stepbaselineB · 37.2%11,390 · 53% of the previous step+3.6%+0.8 → +6.4
3Checkout startedA · 25.6%7,840 · 71% of the previous stepbaselineB · 26.7%8,170 · 72% of the previous step+4.4%+0.9 → +8.0
4CommandeA · 20.0%6,130 · 78% of the previous stepbaselineB · 20.8%6,357 · 78% of the previous step+3.9%-0.2 → +8.1

Periods analyzed separately

Checkpoints

Checkpoint 1 · 19,634 visitors33% of the sample, reached on day 6 · boundary not crossed, the test continues|z| ≥ 3.47

Checkpoint 2 · 39,267 visitors67% of the sample, reached on day 12 · boundary not crossed, the test continues|z| ≥ 2.45

Checkpoint 3 · 58,900 visitors100% of the sample, reached on day 18 · boundary crossed: the plan allows a conclusion|z| ≥ 2.00

Very strict thresholds at first, close to the standard threshold at the end. Looking 3 times costs 1.7% more visitors and keeps the false-winner risk at 5%.

Decision rules

Winner if, at a checkpoint, the difference in Revenue per visitor crosses the boundary and no guardrail has degraded.

Return rate: no more than +25%, range included. Beyond that, the test stops and traffic returns to the original.

Mobile speed (LCP): no more than +4%, range included. Beyond that, the test stops and traffic returns to the original.

Traffic split: 50/50, checked at every reading. An imbalance hides the results until it’s diagnosed.

{"allocation":[{"changes":"4f53cda18c2baa0c","key":"a","name":"Original","weight":1},{"changes":"0d9a08f73a38dae2","key":"b","name":"Progress bar","weight":1}],"declared":{"baseline":{"kind":"mean","mean":24.58,"sd":58.28126285522647},"dailyVisitors":3272},"hypothesis":"If the cart shows how much is left to qualify for free shipping, customers will add an item rather than pay for shipping, and revenue per visitor will go up.","metrics":[{"aggregation":"visitor","capPercentile":null,"direction":null,"eventName":"commande","key":"ca","kind":"mean","limit":null,"name":"Revenue per visitor","role":"primary"},{"aggregation":"visitor","capPercentile":null,"direction":null,"eventName":"marge","key":"marge","kind":"mean","limit":null,"name":"Margin per visitor","role":"secondary"},{"aggregation":"visitor","capPercentile":null,"direction":null,"eventName":"commande","key":"conversion","kind":"proportion","limit":null,"name":"Conversion rate","role":"secondary"},{"aggregation":"visitor","capPercentile":null,"direction":"increase-is-bad","eventName":"retour","key":"retours","kind":"proportion","limit":2500,"name":"Return rate","role":"guardrail"},{"aggregation":"visitor","capPercentile":null,"direction":"increase-is-bad","eventName":"lcp","key":"lcp","kind":"mean","limit":400,"name":"Mobile speed (LCP)","role":"guardrail"}],"periods":[{"end":"2026-11-02","key":"saisie-2026-10-30-maison-lune-private-sale","label":"Maison Lune private sale","source":"manual","start":"2026-10-30"}],"plan":{"alpha":0.05,"armSizes":null,"looks":3,"mde":0.05520567661771653,"perArm":29450,"plannedDays":18,"power":0.8,"totalVisitors":58900},"targeting":{"endsAt":null,"holdoutShare":0.05,"rules":{"urlPattern":"/panier"},"startsAt":null,"trafficShare":1,"zone":"Cart"},"version":1}

Free shipping progress bar

B beats the original.

Reliability

Risk that this winner is false: 11%

Bayesian reading

All metrics

Ce que compte le chiffre

Les comptes

Le calcul

Ce qui limite la lecture

Segments

Device

New or returning

Traffic source

Surveys1

Question 1How did your order go?

Question 2What convinced you to order from us?

Question 3What went wrong?

Question 4Tell us more, if you don’t mind.

Funnels

Difference in revenue per visitor, day by day

Notes

Does the effect hold over time?

Periods analyzed separately

Maison Lune private sale · Oct 30 to Nov 2, 2026

The test plan, set at launch

Checkpoints

Decision rules

Sealed plan

Write-up

What’s next

Free shipping from €50

Free returns noted in the cart

Secure payment methods shown in the cart

Sessions to watch

Click and scroll maps

Forms

Free shipping progress bar

B beats the original.

Reliability

Risk that this winner is false: 11%

Bayesian reading

All metrics

Ce que compte le chiffre

Les comptes

Le calcul

Ce qui limite la lecture

Segments

Device

New or returning

Traffic source

Surveys1

Question 1How did your order go?

Question 2What convinced you to order from us?

Question 3What went wrong?

Question 4Tell us more, if you don’t mind.

Funnels

Difference in revenue per visitor, day by day

Notes

Does the effect hold over time?

Periods analyzed separately

Maison Lune private sale · Oct 30 to Nov 2, 2026

The test plan, set at launch

Checkpoints

Decision rules

Sealed plan

Write-up

What’s next

Free shipping from €50

Free returns noted in the cart

Secure payment methods shown in the cart

Sessions to watch

Click and scroll maps

Forms