Mistake 1
Calling a winner as soon as a difference looks significant.
Checked every day 21.8%
Checkpoints set by Probant 5.6%
Share of false winners in 1,000 simulated A/A tests, where both versions are identical.
A/B tests thattell the truth.
Probant measures each test on the metric that matters for your business. Pick yours:
Ecommerce · Shopify, WooCommerce, PrestaShop
Every test is measured on margin per visitor.
A variant that sells more but lowers your margin is flagged before rollout.
| Test | Status | Effect and range |
|---|---|---|
| Free shipping progress barCart | Not conclusive yet61,240 visitors | +6.0% |
| Simplified mobile menuNavigation | Not conclusive yet…day 9 of 13 | +1.9% |
| Customer reviews at the top of the pageProduct page | No detectable effect.64,100 visitors | -0.4% |
| “You may also like” recommendationsProduct page | Stopped by a guardrail.stopped 3 h ago | Returns +1.2 pts |
| One-page checkoutCheckout | Sample ratio mismatch.19,800 visitors | Hidden until diagnosed |
Free shipping progress bar
The test is starting.
Expected revenue, corrected for the winner’s curse+€125,700 per month
Stopped by a guardrail“You may also like” recommendationsGuardrail: returns +1.2 pts
Teams testing with Probant
Connects to
- Shopify
- WooCommerce
- PrestaShop
- Stripe
- HubSpot
- Pipedrive
- BigQuery
- Snowflake
- Slack
Probant at a glance
An A/B testing tool that doesn’t sell you false winners.
Probant declares a winner only when it can prove it. It concludes on dates set before launch, corrects the gain before reporting it, and puts it in euros on the metric that actually makes you money.
Winners on one number, losers on the one that counts.
Store
“Free returns highlighted”
A winning test… on revenue
Revenue per visitor+3.4%
Margin per visitor-8.9%
A tool that reads only revenue keeps this variant. Probant shows margin alongside it and flags the variant.
Quote form
“Free quote in 2 minutes”
Shortened form, read on day 60
More forms submitted, fewer customers. Probant decides on the leads your CRM qualifies.
Subscription
“First month at half price”
A subscription offer, followed after the test
+87.9%Paid conversion
-31.8%Subscribers at day 90
-39.1%Subscribers at day 180
The promotion wins at first, then drives subscribers away. Probant follows the cohort after the test.
- No false winnersProbant concludes only at checkpoints set before launch. Across 1,000 simulated tests with no real effect, checking every day finds 21.8% “winners”; Probant finds 5.6%.
- A gain that isn’t inflatedThe reported gain is corrected for the winner’s curse, with its range, before it’s converted to euros.
- Measured on what paysMargin, the leads your CRM qualifies, or subscribers still there at day 90, rather than conversion rate alone.
- A conclusion everyone understandsA sentence in plain English and an amount in euros, instead of a “confidence level.”
One demo test, from start to finish.
Read the transcript
Does this free-shipping bar really pay off? One test, from the first change to the decision.
The variant is built on the real page. One click under the cart, and variant B says: “Only €11.00 away from free shipping.”
Each visitor always sees the same version. 18 days, three checkpoints, and a target effect of 5.5%: the plan is set before the first visitor.
Watch every day, decide at checkpoints. On day 6, z was 2.30, below the boundary of 3.47, so the test went on. The boundary is crossed on day 18.
A verdict in one sentence, in euros. +6.0% measured, +4.9% reported after correcting for the winner’s curse: +€125,700 in revenue per month.
The winner rolls out in steps. 25%, 50%, then 100% of visitors, with guardrails rechecked at each step. At the first proven harm, everyone goes back to the original.
The test joins the library. The store’s 48 completed tests sit on the same scale. Most effects are small (median +0.9%), which is why gains are corrected.
A/B tests that tell the truth. The demo is open, no sign-up needed.
Three common mistakes when reading a test.
Run the simulation
Mistake 2
Judging a test on revenue alone.
Revenue per visitor+3.4%
Margin per visitor-8.9%
“Free returns highlighted” raised revenue and lowered margin. Probant shows margin next to revenue.
See metricsMistake 3
Letting a variant that hurts the site keep running.
Stopped by a guardrail“You may also like” recommendationsGuardrail: returns +1.2 pts
Probant monitors return rate, mobile speed, and errors. If one gets worse, the test stops and visitors go back to the original.
See guardrailsWhat Probant measures in your store.
Readings from “Free shipping progress bar”: original → variant, effect, and range.
Decision
The metric the test concludes on.
- Revenue per visitor+6.0%€24.58 → €26.06 · range +2.2% to +10.0%
Monitored
Shown alongside, to catch an opposite effect.
- Margin per visitor+3.0%€10.03 → €10.33 · range -0.7% to +6.7%
- Expected margin per month+€27,700range -€5,400 to +€60,900
Guardrails
If one of them gets measurably worse, the test stops.
- Return rate+2.6%1.6% → 1.7% · within the +25.0% limit
- Mobile speed+0.4%2.41 s → 2.42 s · within the +4.0% limit
- JavaScript errors+0.2%0.12% → 0.12% · to watch
The five steps of a test.
1
Install the script.
Paste two lines on your site, or use the Shopify app, WooCommerce plugin, or PrestaShop module.
With your store’s plugin, product costs are read automatically to calculate margin.
Install
<script async src="https://getprobant.com/sdk/v1.js" data-key="pk_…"></script> probant.track("commande", montant);
Getting started
- Snippet added to the site
- First events received
- First test launched
Each step checks itself off as soon as Probant detects it.
2
Build the variant.
Edit your page in the editor, or describe the variant to the assistant in one sentence.
- Preview on your live page
- Fragile selectors flagged, with a stable anchor suggested
- Contrast checked before launch
Assistant
“Add a bar under the cart that shows how much is left to qualify for free shipping.”
Changes
Progress bar addedCart
Automatic checks
Text contrastAA
Selectorstable
3
Calculate the duration.
Probant calculates the duration from your traffic before launch. After that, the plan stays fixed.
Test plan · Free shipping progress bar
- Primary metric
- Revenue per visitor
- Minimum detectable effect
- 5.5%
- Planned sample size
- 58,900
- Duration
- 18 days, 3 checkpoints
- Guardrails
- Return rate ≤ +25.0% · Mobile speed ≤ +4.0%
4
Read the report.
The conclusion in one sentence, then the calculations in detail.
+€125,700 per month
in expected revenue per month, range +€41,600 to +€209,800
Free shipping progress bar
Conclusive.
Result. Showing progress toward free shipping: revenue per visitor +6.0% measured, +4.9% expected once corrected for the winner’s curse. The effect on margin isn’t established yet: +3.0% measured, range from -0.7% to +6.7%.
Why. More orders reach the threshold: customers add an item rather than pay for shipping.
Next. Expected revenue: +€125,700 per month after correcting for the winner’s curse. Then test a €50 threshold, measured on margin.
Metrics
| Metric | Original | Variant | Difference | 95% range |
|---|---|---|---|---|
| Revenue per visitorprimaryOriginal €24.58 · Variant €26.06 | €24.58 | €26.06 | +6.0% | +2.2% to +10.0% |
| Margin per visitorafter product costs, discounts, and shippingOriginal €10.03 · Variant €10.33 | €10.03 | €10.33 | +3.0% | -0.7% to +6.7% |
| Conversion rateOriginal 20.0% · Variant 20.8% | 20.0% | 20.8% | +3.9% | +0.7% to +7.2% |
| Average order valueOriginal €122.90 · Variant €125.40 | €122.90 | €125.40 | +2.0% | 0.0% to +4.1% |
| Orders with free shippingexplains the gap between revenue and marginOriginal 41.0% · Variant 57.0% | 41.0% | 57.0% | +16 points | Expected effect of the variant |
Guardrails
| Guardrail | Original | Variant | Difference | 95% range |
|---|---|---|---|---|
| Return rateprovisional, 30-day value on Dec 9Original 1.6% · Variant 1.7% | 1.6% | 1.7% | +2.6% | Within limit, limit +25.0% |
| Mobile speed (LCP)Original 2.41 s · Variant 2.42 s | 2.41 s | 2.42 s | +0.4% | Within limit, limit +4.0% |
| JavaScript errorsshare of visitors with an error · limit +25%Original 0.12% · Variant 0.12% | 0.12% | 0.12% | +0.2% | Watch |
Amounts
Expected revenue / month+€125,700+€41,600 to +€209,800
Expected margin / month+€27,700-€5,400 to +€60,900
Free shipping-€23,800per month, already deducted from margin
Checks
Planned sample size104%
A/B split50.0 / 50.0
Sample ratio mismatch (SRM) testp = 0.81
Saw both versions0.3%
Guardrails breachednone
Boundary crossed at checkpointday 18 · z = 3.1
Orders reconciled98.6%
Known costs (margin)96.7%
Winner’s cursecorrected, +4.9%
5
Roll out step by step.
The winner rolls out step by step. If a guardrail gets worse, all visitors go back to the original.
Free shipping from €6050% step in progress
25%
50%
100%
Free returns highlightedrolled out to 100%
25%
50%
100%
Everything else in the plan.
Cumulative impactAfter 14 days.Feature flags60 sat most for an emergency kill switch to reach your servers, with the default settingTeam and governance25 permissionsnamed, to build your own roles, on top of the four fixed rolesIntegrations and API
The total gain from rolled-out tests.
+€511,400in revenue in October 2026, measured against a holdout of 5% of visitors who see no tests.Sample ratio mismatchA check on how visitors are split.
2,960 / 1,852visitors in A and B on Safari iOS, for “One-page checkout”. The result stays hidden until the cause is found.Guardrails1.6% → 2.9%return rate: “You may also like” recommendations switched back to the originalLearnings library37 tests, 8 winners, 6 losers, on one scale.
Ideas+2.5%expected for “Free shipping from €50”, based on 4 shipping testsPersonalization+12.6%on “Add to cart”: “Express delivery for loyal mobile shoppers”, against its 10% holdoutSession recordings and click maps12 sessionsto watch first, out of 14 recorded for “Free shipping progress bar”Bandit
Traffic shifts toward the variant that converts best.
Current banner 10.0%
“Up to 40% off” 10.1%
“Free shipping during the sale” 79.9%
- Slack alerts and signed webhooks
- CSV export and share links
- Management API and MCP server
How long a test takes on your traffic.
Enter your traffic and the effect you want to detect.
Compared with the usual way of reading tests.
| Question | Usual reading | With Probant |
|---|---|---|
| When to conclude? | As soon as the difference looks significant. | At checkpoints set before launch. |
| What do you read? | A percentage and a confidence level. | A written conclusion, the range, and the amount in euros. |
| Which metric? | Conversion rate. | Revenue and margin per visitor. |
| The reported gain? | The observed difference. | The difference corrected for the winner’s curse. |
| The traffic split? | Up to you to check. | Checked continuously; the test is blocked if it drifts. |
| Going live? | All traffic at once. | Gradual steps and automatic rollback. |
Already testing with another tool?
Import your past tests.
Import your history from AB Tasty, VWO, or Kameleoon. Probant recalculates every verdict from visitors and conversions.
Out of 12 imported tests:3 of 7 reported winners still hold after recalculation.Average reported gain +9.5% · corrected +4.6%
Import my historyYour site and your data.
Security and GDPR- Under 7 KB
- An asynchronous script that doesn’t hide the page while it loads.
- No flicker
- Assignment can happen on the server.
- Consent first
- By default, no cookies and nothing sent until the visitor agrees.
- Hosted in Europe
- In Amsterdam, with Scaleway.
- 13 months
- How long events are kept.
A fixed plan, never an overage charge.
Creating a workspace is free: it starts on the Free plan. Paid plans cost €590 to €1,990 per month depending on the number of tested visitors, or €5,900 to €19,900 per year, which is 2 months free. Your invoice never exceeds the plan price.
- Free10,000 tested visitors per monthTo install Probant, check your tracking with an A/A test, and try the editor.Create your workspace
- Essential€590 per monthor €5,900 per year (2 months free)250,000 tested visitors per monthFor a store that tests continuously, down to revenue per visitor.
- Growth€1,990 per monthor €19,900 per year (2 months free)2,000,000 tested visitors per monthTo test on all the traffic of an established store, several areas at once, and detect smaller effects.
Beyond 2,000,000 tested visitors per month, or for several sites: Custom plan, quoted on request, on an annual contract. Plan details
Frequently asked questions.
Another question? Write to us.
Does it work with Shopify?
Yes. Probant reads your orders and product costs from Shopify to calculate margin on the server.
How is margin calculated?
From the product cost of each order, minus any free shipping. The report shows what share of orders has a known cost.
Will the script slow down my site?
It weighs less than 7 KB and loads without blocking the page. To avoid any flicker, assignment can happen on the server.
Do I need to know statistics?
No. Probant calculates the plan, the checkpoints, and the conclusion, and writes it in plain English.
What about GDPR?
By default, the script stores nothing and sends nothing before the visitor consents. Your test data is stored in the European Union, in Amsterdam, with Scaleway. Events are kept for 13 months.
Conclusive Or not yet…
Create your workspace, or explore the demo first with data from a fictional store.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway