Probant vs. Optimizely
Both tools compared point by point, based on Optimizely’s public documentation, accessed November 7, 2026. Every statement about Optimizely links to its source.
What Optimizely offers.
Optimizely presents itself as “the AI platform for marketing”: CMS, experimentation, commerce, personalization, and analytics.1 Its Web Experimentation product offers a visual editor for testing quickly, or server-side control.2 Its press releases are datelined New York.3
Probant is for sites that want A/B test verdicts that hold up, with a plan set before launch and guardrails monitored through rollout. It measures what each conversion is worth: the margin on an order, a qualified or signed lead, a paid subscription.
The differences, point by point.
On the left, what Optimizely’s public documentation says, each sentence with its source. On the right, what Probant does. “Not mentioned” means nothing was found in that documentation, which does not prove a feature is missing.
| Criterion | Optimizely | Probant |
|---|---|---|
| Statistical method | Stats Engine: sequential experimentation, with no fixed horizon.4 Feature Experimentation also offers fixed-horizon frequentist and Bayesian methods, with non-informative priors.56 | Frequentist, with a sequential plan set before launch: target effect, duration, and checkpoints. The verdict fits in one sentence, with a 95% range. Where to check |
| Peeking | Results can be monitored continuously and stay “always valid,” thanks to false discovery rate control.7 | Decisions are only made at planned checkpoints, with O’Brien-Fleming boundaries: very strict at first, close to the usual threshold at the end. During a rollout, ranges are always valid (mSPRT), so you can check them at any time. Where to check |
| Traffic split (SRM) | Automatic sequential detection (SSRM) for Stats Engine A/B tests.8 It has limits, including a maximum duration of 45 days.9 When SRM occurs, the documentation recommends pausing the experiment immediately.10 | The traffic split is checked with a chi-square test every time the report is read. If the split is off, the test moves to “blocked” status and no winner is declared. Where to check |
| Guardrails | Alerts on guardrail metrics, by email or Slack.11 Optional automatic pause, off by default, applied at the next scheduled check, not in real time.1213 | Guardrail metrics are tracked with their range; a guardrail is only breached if the whole range crosses the limit. A breached guardrail blocks the test’s verdict, stops a running test on its own, and during a rollout moves visitors back to the original automatically. Where to check |
| Gradual rollouts | Feature Rollouts: gradual rollout (10%, 25%, 50%, 100%), with measurement at each step.14 The feature is in beta.15 A guide describes ramping up traffic gradually.16 | Steps of 25% → 50% → 100% by default, 10% → 25% → 50% → 100% in cautious mode. Automatic rollback when a guardrail is breached; an exposed visitor stays exposed in the following steps. Where to check |
| Multi-armed bandit | “Multi-armed bandit” allocation: according to their documentation, it does not produce statistical significance and pushes traffic toward the best-performing variants.17 | Optimization test: Thompson sampling, reallocated at most once a day, with a minimum share per variant. It maximizes a conversion rate, or an amount per visitor such as revenue: purchase probability and average order value drawn from their distributions, with values capped at a percentile as in an A/B test. Optionally, shares per device (mobile, tablet, desktop): one draw per device, each with its own learning and floor. A visitor always keeps their variant: a new split only applies to visitors who arrive after it. The screen says so: it maximizes conversions (or the amount per visitor) during the test, without proving that a variant is better. The difference is read period by period (and device by device), with no declared winner; the CSV export gives the same reading. Where to check |
| Revenue and margin per visitor | Revenue per visitor and per paying visitor in the results.18 Capping or winsorization of extreme values.19 Margin per visitor: not mentioned in their public documentation (as of November 7, 2026). | Revenue and margin per visitor, each with its range; the gain is stated in euros. Margin is calculated on the server from reconciled orders, the cost of each product (Shopify or CSV), and store fees; no margin verdict below 90% of revenue covered by known costs. Where to check |
| Winner’s curse | Not mentioned in their public documentation (as of November 7, 2026). | The reported gain is corrected for the winner’s curse by Bayesian shrinkage toward the effects of the store’s past tests. Where to check |
| Bayesian mode | Bayesian available in Feature Experimentation, alongside fixed-horizon frequentist.5 Non-informative priors.6 | Optional, off by default: a Bayesian reading next to the verdict, with the probability that the variant does better, the credible interval, and the expected loss of each choice. The prior is centered on zero and as wide as the effects measured by the store’s completed tests; while there are too few of them to estimate it, the Bayesian reading is not shown. The decision always follows the sealed plan; when the two readings disagree, the report explains why. Where to check |
| Multiple comparisons | False discovery rate control through a tiered version of Benjamini-Hochberg, across metrics and variants.20 This control no longer holds when results are segmented.21 | Segment comparisons are corrected with the Holm method. With several variants, the error risk is split across comparisons (Bonferroni). Where to check |
| Multivariate testing | Full factorial design: all combinations are tested.22 It shows how elements interact.23 Beyond 25 combinations, even a high-traffic site struggles to reach a conclusion.24 | Full factorial design generated by Probant: two to four sections, each with two to four levels, up to sixteen combinations, each served as a variant. Main effects of each section across all traffic and interactions between two sections, corrected with the Holm method on the comparisons stated in the plan. Based on the workspace’s measured traffic, the plan says whether the test can reach a conclusion and suggests removing a level or a section; no best combination is declared without a check. Where to check |
| Variance reduction (CUPED) | CUPED available in Web Experimentation, for numeric metrics only.2526 By default, on the first three decision metrics.27 | CUPED applied to live test reports, using each visitor’s history before their first exposure; without history, the difference stays unadjusted and the report says so. Where to check |
| Visual editor | The new visual editor appears as an overlay on the page, with no iframe.28 A browser extension, Web Companion, opens it from the browser.29 | Two ways to edit, producing the same changes: a page preview loaded by Probant, without its scripts, or the live page on the site, scripts included. On the site, the editor is loaded by the script already installed, through a short-lived signed link for one draft and one variant, with no browser extension. It only sends the chosen changes to Probant. Where to check |
| Widgets | Ready-made experiment templates: countdown, top or bottom banner, pop-up with image.303132 | Banner, countdown, pop-up, badge, reassurance, action bar, delivery date, social proof, promo code, free shipping, and survey, set up without code in the editor and saved in the workspace library for other tests. A widget is just a set of ordinary changes: escaped text, links over https or to the site, and styles applied without a style attribute, so a strict CSP does not need loosening. The countdown follows the site’s time zone and disappears at the deadline; the pop-in opens on arrival, on exit intent, or after scrolling, closes with the keyboard, and does not reappear during the visit if consent allows. Their small modules only load for visitors in the variant. The delivery date is calculated in the visitor’s browser (cutoff time, business days, public holidays); social proof shows the exact number of purchases received over a rolling week, and only above a threshold. The survey asks up to 8 questions (closed answers, recommendation scale, open question) with conditional logic, on the page or through a link outside the site (email, QR code), based on 70 templates (recommendation, satisfaction, effort, checkout blockers, cancellation, concept test, multi-question surveys…); open answers are redacted before sending (email addresses, phone numbers, cards, IBANs), kept for 180 days, and grouped by theme; each question is read by variant, with its range and response rate, and comparisons across all questions are corrected together; a response through a link is attached to the variant the visitor saw, recalculated by Probant, or stays outside the test. Where to check |
| Script and flicker | Synchronous snippet to avoid flicker.33 It contains running experiments, audiences, goals, and jQuery; each experiment is limited to 1,048,572 bytes.3435 Performance Edge handles assignment at the network edge, with a “microsnippet.”36 Reference snippet size: not mentioned in their public documentation (as of November 7, 2026). | Script under 7 KB compressed, loaded asynchronously, that does not hide the page while it loads. Server-side assignment available, for a display with no flicker at all. Where to check |
| Granular permissions | Feature Experimentation: roles per project, refined by environment, flag, and audience.37 Web Experimentation: most collaborator roles apply project by project.38 | Four fixed roles, plus custom roles the owner builds from a catalog of named permissions: prepare, launch and stop, roll out, approve a launch, write variant JavaScript, manage flags, metrics, margin, members, and keys. No privilege escalation: billing, roles, transfer, and deletion of the workspace stay with the owner; no one can grant more than their own permissions, and an API key cannot exceed its author’s. Where to check |
| Data hosting | EU hosting as an option, “where applicable.”39 Hosting subprocessors for Web and Feature Experimentation: AWS (United States) and Google Cloud (United States, Europe).4041 Hosting in France is not mentioned in their public documentation (as of November 7, 2026). | Database and compute in Amsterdam, with Scaleway, in the European Union. Scaleway, which hosts the database and compute, is a French company. Published subprocessor list: Scaleway (Amsterdam, Netherlands (nl-ams region), in the European Union), Brevo (European Union: servers in France and Germany, encrypted backups in Belgium (Google Cloud), according to Brevo), Mistral AI (European Union by default (European API endpoint), according to Mistral), ClickHouse (option not used), Stripe (Contract with Stripe Payments Europe, Ltd. (Ireland); data transferred to Stripe, LLC (United States), according to Stripe), Sentry (European Union: Frankfurt (Germany), including backups, according to Sentry). Depending on the feature used, Mistral may temporarily transfer data outside the European Union to the subprocessors it publishes; its data processing agreement limits these transfers to countries the EU recognizes as adequate, or covers them with standard contractual clauses. Stripe transfers payment data to Stripe, LLC, in the United States, under the Data Privacy Framework and, failing that, standard contractual clauses. Whatever region is chosen, Sentry may keep the data of Probant’s Sentry account (members, settings, audit log, project descriptions) in the United States; reported errors stay in Frankfurt. Where to check |
| Consent by default | By default, the snippet sets its cookies: you need to call the optOut API before it loads to prevent this.4243 Another option: load the snippet through a tag manager, after consent.44 | Consent required by default: no cookie, no assignment, and nothing sent before the visitor agrees. Consent given mid-page: the page does not change in front of the visitor; their variant starts on the next page. First-party cookie, lasting 390 days at most, never extended. Where to check |
| Pricing model | No public pricing: custom quote.45 Usage-based billing, based on monthly active users (MAU).46 The documentation covers MAU overages.47 | Creating a Probant workspace is free; paid plans cost €590 to €1,990 per month depending on the number of tested visitors (2 months free with annual billing), with no overage charges. Agency plan: 1,000,000 tested visitors per month, shared across client workspaces, from 3 workspaces, for €1,490 per month or €14,900 per year. Where to check |
What Optimizely also does.
Probant shares these points with Optimizely.
Sequential statistics
Stepped rollouts
10%, 25%, 50%, 100%, the same steps as Probant’s cautious mode, in beta.1415
Automatic SRM detection
Sequential detection for Stats Engine A/B tests.8
Automatic pause on a guardrail
As an option, per guardrail metric.12
Multiple comparisons control
Tiered Benjamini-Hochberg, across metrics and variants.20
Server-side assignment
Server-side control, or edge assignment with Performance Edge.236
When to choose Optimizely over Probant.
Based on Optimizely’s public documentation.
What sets Probant apart.
These points are not in Optimizely’s public documentation, accessed November 7, 2026. A missing mention does not prove a feature is missing; if these points matter to you, ask Optimizely.
Margin per visitor
Probant measures revenue and margin per visitor, each with its range.
OptimizelyMargin per visitor is not mentioned in their public documentation (as of November 7, 2026).
Learn moreWinner’s curse correction
The reported gain is corrected by Bayesian shrinkage toward the effects of the store’s past tests.
OptimizelyA winner’s curse correction is not mentioned in their public documentation (as of November 7, 2026).
Learn moreSRM that blocks the verdict
A traffic split imbalance moves the test to “blocked” status, and no winner is declared.
OptimizelyOptimizely’s documentation recommends pausing immediately, which you do yourself.10 Automatic blocking is not mentioned there (as of November 7, 2026).
A published script size
Probant’s script weighs less than 7 KB compressed and does not hide the page while it loads.
OptimizelyA reference snippet size is not mentioned in their public documentation (as of November 7, 2026).
Learn more
Frequently asked questions.
Does Optimizely correct for peeking?
Yes. Its Stats Engine is sequential and its results stay valid under continuous monitoring.47 False discovery control no longer holds when results are segmented.21 Probant corrects segment comparisons with the Holm method.
Does Optimizely offer gradual rollouts?
Yes, in beta: 10%, 25%, 50%, 100%, with measurement at each step.1415 In Probant, guardrails monitor each step, with automatic rollback to the original.
Sources.
Official pages from Optimizely, accessed November 7, 2026. Quotes are reproduced in their original language.
- Source 1: www.optimizely.com/company/press/free-to-grow
Optimizely is the AI platform for marketing, built for the experience-makers who shape how brands show up in the world.
- Source 2: www.optimizely.com/products/web-experimentation
Run experiments wherever your customers are. Use a visual editor for fast web testing, or go deeper with server-side control.
- Source 4: docs.optimizely.com/experimentation-strategy/docs/how-and-why-statistical-significance-changes-over-time-in-optimizely-experimentation
Optimizely's Stats Engine uses sequential experimentation, not the fixed-horizon experiments you see on other platforms.
- Source 5: docs.optimizely.com/experimentation-strategy/docs/statistical-analysis-methods-overview
Optimizely Feature Experimentation offers the following statistical analysis methods for running A/B tests:
- Source 6: docs.optimizely.com/experimentation-strategy/docs/bayesian-statistics
the current version of Bayesian statistics uses uninformed priors
- Source 7: docs.optimizely.com/experimentation-strategy/docs/statistical-analysis-methods-overview
Monitor results continuously during the experiment. The results are always valid because Optimizely controls the false discovery rate.
- Source 8: docs.optimizely.com/experimentation-strategy/docs/optimizely-s-automatic-sample-ratio-mismatch-detection
Optimizely Experimentation's automatic SRM detection uses a statistical method called sequential sample ratio mismatch (SSRM).
- Source 9: docs.optimizely.com/experimentation-strategy/docs/optimizely-s-automatic-sample-ratio-mismatch-detection
That are running for 45 days or less.
- Source 10: docs.optimizely.com/experimentation-strategy/docs/take-action-now-srm-detected-experiment-health
The experiment should be paused immediately to investigate the issue.
- Source 11: docs.optimizely.com/web-experimentation/docs/guardrail-alerts
Guardrail alerts let you configure metric alerts that protect business-important metrics during an experiment.
- Source 12: docs.optimizely.com/web-experimentation/docs/guardrail-alerts
This setting applies per guardrail metric and is off by default.
- Source 13: docs.optimizely.com/web-experimentation/docs/guardrail-alerts
The pause happens at the next scheduled check after the breach, not in real time.
- Source 14: docs.optimizely.com/feature-experimentation/docs/run-feature-rollouts-in-feature-experimentation
Roll out a feature progressively (10%, 25%, 50%, then 100%) and measure impact at each stage.
- Source 15: docs.optimizely.com/feature-experimentation/docs/run-feature-rollouts-in-feature-experimentation
This feature is currently in beta.
- Source 16: docs.optimizely.com/experimentation-strategy/docs/controlled-traffic-allocation-ramping
Ramping is when you gradually expose traffic to new test variations.
- Source 17: docs.optimizely.com/experimentation-strategy/docs/experimentation-distribution-methods
MAB optimizations do not generate statistical significance. Instead, they push traffic to the better-performing variations.
- Source 18: docs.optimizely.com/web-experimentation/docs/configure-revenue-tracking
you can see the revenue earned per visitor, revenue per paying visitor, purchases, and the total revenue earned per variation on the Optimizely Experimentation Results page.
- Source 19: docs.optimizely.com/web-experimentation/docs/outlier-management
Optimizely lets you choose between two outlier management methods: metric capping (capping at a specified metric value) and Winsorization (capping at a specified percentile).
- Source 20: docs.optimizely.com/experimentation-strategy/docs/false-discovery-rate-control
Optimizely's Stats Engine uses a tiered version of the Benjamini-Hochberg procedure for false discovery rate control to correct statistical significance across multiple metrics and variations.
- Source 21: docs.optimizely.com/experimentation-strategy/docs/false-discovery-rate-control
But, when you segment results, the false discovery rate control is not maintained.
- Source 22: docs.optimizely.com/web-experimentation/docs/experiment-types-in-web-experimentation
Testing all possible combinations of a multivariate test is also known as full factorial testing.
- Source 23: docs.optimizely.com/web-experimentation/docs/experiment-types-in-web-experimentation
reveals information about how they interact with one another
- Source 24: docs.optimizely.com/web-experimentation/docs/experiment-types-in-web-experimentation
Even a site with reasonably high traffic can have trouble completing a test with more than 25 combinations.
- Source 25: docs.optimizely.com/web-experimentation/docs/uvn0-cuped-controlled-experiment-using-pre-experiment-data
CUPED (Controlled-experiment Using Pre-Experiment Data) is a statistical method that reduces variance in A/B tests
- Source 26: docs.optimizely.com/web-experimentation/docs/uvn0-cuped-controlled-experiment-using-pre-experiment-data
This feature is available only for numeric metrics: it is mathematically inapplicable to conversion metrics.
- Source 27: docs.optimizely.com/web-experimentation/docs/uvn0-cuped-controlled-experiment-using-pre-experiment-data
By default, CUPED applies only to the first three decision-making metrics in the experiment.
- Source 28: support.optimizely.com/hc/en-us/articles/37424389168013-New-Visual-Editor
The new Visual Editor is a WYSIWYG (What You See Is What You Get) editor that uses an overlay instead of an iframe.
- Source 29: support.optimizely.com/hc/en-us/articles/37424389168013-New-Visual-Editor
Optimizely Web Companion is a browser extension that opens the Visual Editor for Web Experimentation directly from your browser.
- Source 30: docs.optimizely.com/web-experimentation/docs/get-started-with-experience-templates
Countdown timer – Displays a countdown timer. For example, use this to count down to the start of an event.
- Source 31: docs.optimizely.com/web-experimentation/docs/get-started-with-experience-templates
Displays information at the top or bottom of the page.
- Source 32: docs.optimizely.com/web-experimentation/docs/get-started-with-experience-templates
Displays a pop-up with an image.
- Source 33: docs.optimizely.com/web-experimentation/docs/load-snippet-synchronously-and-asynchronously
Optimizely Web Experimentation uses a synchronous snippet to prevent flickering and a balanced content delivery network (CDN) system to make sure the impact on the page is minimized.
- Source 34: docs.optimizely.com/web-experimentation/docs/page-load-speed
The snippet contains the information of any running experiments, all audience and goal definitions, and the jQuery library.
- Source 35: docs.optimizely.com/web-experimentation/docs/HnnD-site-performance-best-practices
Each experiment is limited to a maximum of 1,048,572 bytes.
- Source 36: docs.optimizely.com/performance-edge/docs/overview
Optimizely Performance Edge is designed to improve page performance by performing targeting and variation assignment tasks at the Edge instead of in the browser.
- Source 37: docs.optimizely.com/feature-experimentation/docs/granular-roles-and-permissions-in-feature-experimentation
Granular roles and permissions let you control access to environments, flags, and audiences with precision.
- Source 38: docs.optimizely.com/web-experimentation/docs/5nV4-manage-collaborators
Most collaborator roles are project-specific, giving you different collaborator statuses across different projects.
- Source 39: www.optimizely.com/trust-center/optimizely-eu-data-privacy
If you choose to select the EU Data Hosting (where applicable), your data will be stored in EU data centers ensuring that you have full control over data location.
- Source 40: www.optimizely.com/legal/sub-processors/third-party
Web & Feature Experimentation Amazon Web Services Inc. Cloud hosting and infrastructure provider United States
- Source 41: www.optimizely.com/legal/sub-processors/third-party
Web & Feature Experimentation Google Cloud Cloud Service Provider United States, Europe
- Source 42: docs.optimizely.com/experimentation-strategy/docs/configure-cookie-and-local-storage-consent
Since a cookie opt-in option is not required in all regions or for all cookies, Optimizely Web Experimentation does not set the optimizelyOptOut cookie by default.
- Source 43: docs.optimizely.com/experimentation-strategy/docs/configure-cookie-and-local-storage-consent
Add the code above the Optimizely Web Experimentation snippet on your page. Otherwise, the JavaScript snippet will run and set the tracking cookie(s) and storage items on your visitor.
- Source 44: docs.optimizely.com/experimentation-strategy/docs/configure-cookie-and-local-storage-consent
If you use a tag manager on your site, you can use conditional logic in the tag manager to load the Optimizely Web Experimentation JavaScript snippet only if a visitor has provided consent.
- Source 45: www.optimizely.com/plans
Get pricing tailored to your business. Tell us about your goals and requirements, and our team will put together a personalized Optimizely quote that fits your needs.
- Source 46: docs.optimizely.com/experimentation-strategy/docs/review-impression-usage
Starting in September 2020, Optimizely Experimentation introduced a simplified usage billing component: Monthly Active Users (MAUs).
- Source 47: docs.optimizely.com/experimentation-strategy/docs/troubleshoot-mau-overages
If your Monthly Usage Information reports more Monthly Active Users (MAUs) than expected, work through this article to identify the cause.
Migrate from Optimizely
Your Optimizely history can be imported and is reread by Probant’s engine: every verdict is recalculated from visitors and conversions.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway