Probant compared with other A/B testing tools.
17 A/B testing and experimentation tools, compared using only their official public documentation, accessed November 7 and 11, 2026. Every statement links to its page and quote.
Established CRO platforms
Experimentation platforms long used by brands and agencies.
Enterprise suites
A/B testing as a module of a personalization suite for large enterprises.
Web CRO with public pricing
Web testing tools with published prices, for marketing and CRO teams.
Shopify natives
Testing apps for Shopify stores only.
Product and data warehouse
Platforms built for product and data teams, often connected to a warehouse.
How these comparisons are made.
Verifiable, dated facts, with shared features stated as such.
- Sources
- Official sources only: the vendors’ own pages (documentation, pricing, subprocessor lists, legal notices, and app listings) and, for the Dynamic Yield acquisition, a Mastercard filing with the SEC. No reviews or third-party articles.
- Date
- AB Tasty, Kameleoon, VWO, and Optimizely: pages accessed November 7, 2026. Adobe Target, Dynamic Yield, Convert, Varify.io, Omniconvert, Intelligems, Shoplift, GrowthBook, PostHog, Statsig, Eppo, LaunchDarkly, and Amplitude: pages accessed November 11, 2026. Documentation changes, so a point may have changed since.
- Quotes
- Every statement about a competitor links to its page, with the quote in its original language.
- “Not mentioned”
- Nothing was found in the vendor’s public documentation, which does not prove a feature is missing.
- Shared features
- When a competitor does the same as Probant (sequential testing, stepped rollouts, hosting in Europe…), the page says so.
- Wingify
- AB Tasty and VWO merged under the Wingify brand (announced January 20, 2026), which states that it keeps running both platforms. Each has its own page.
- Acquisitions
- Statsig was acquired by Amplitude, and Eppo by Datadog, which sells it as Datadog Experiments. Their pages say so at the top, with sources.
Pages cited
- AB Tasty
- 27 official pages: wingify.com, abtasty.com, docs.abtasty.com.
- Kameleoon
- 22 official pages: kameleoon.com, docs.kameleoon.com, storage.googleapis.com.
- VWO
- 24 official pages: wingify.com, vwo.com, help.wingify.com.
- Optimizely
- 30 official pages: optimizely.com, docs.optimizely.com, support.optimizely.com.
- Adobe Target
- 26 official pages: experienceleague.adobe.com, partners.adobe.com.
- Dynamic Yield
- 28 official pages: support.dynamicyield.com, sec.gov, dy.dev, apps.shopify.com.
- Convert
- 23 official pages: convert.com, support.convert.com, docs.developers.convert.com.
- Varify.io
- 15 official pages: varify.io, docs.varify.io.
- Omniconvert
- 20 official pages: omniconvert.com, help.omniconvert.com.
- Intelligems
- 24 official pages: intelligems.io, apps.shopify.com, docs.intelligems.io.
- Shoplift
- 20 official pages: shoplift.ai, docs.shoplift.ai, apps.shopify.com.
- GrowthBook
- 33 official pages: growthbook.io, docs.growthbook.io, github.com.
- PostHog
- 26 official pages: posthog.com.
- Statsig
- 30 official pages: docs.statsig.com, statsig.com, amplitude.com.
- Eppo
- 36 official pages: geteppo.com, datadoghq.com, docs.datadoghq.com, docs.geteppo.com.
- LaunchDarkly
- 27 official pages: launchdarkly.com.
- Amplitude
- 27 official pages: amplitude.com.
Probant and each category, criterion by criterion.
Each competitor cell links to the matching row of its comparison, where every sentence has its source. “Not mentioned”: nothing was found in its public documentation, accessed November 7 and 11, 2026.
Established CRO platforms
| Criterion | Probant | AB Tasty | Kameleoon | VWO | Optimizely |
|---|---|---|---|---|---|
| Statistical method | Sequential frequentist, plan set at launchWhere to check | Bayesian by default; frequentist on request2 sources | Frequentist by default; Bayesian or sequential as options2 sources | Bayesian by default; frequentist as an option1 source | Sequential (Stats Engine); other methods in Feature Experimentation2 sources |
| Peeking | O’Brien-Fleming checkpointsWhere to check | Sequential alerts to spot losing variants1 source | “Always valid” sequential mode2 sources | Alpha-spending in sequential mode2 sources | “Always valid” results1 source |
| Traffic split (SRM) | Continuous check; the verdict is blockedWhere to check | Sequential alert, in early access2 sources | Alert; pause it yourself2 sources | Notification, from the Pro plan2 sources | Automatic detection; pause it yourself2 sources |
| Guardrails | Test stops on its own; automatic rollback in rolloutsWhere to check | Automatic stop on the primary goal1 source | Automatic pause when a goal crosses a threshold1 source | Alert or automatic disabling of the variant1 source | Alerts; optional automatic pause2 sources |
| Gradual rollouts | 25% → 50% → 100%; cautious: 10% → 25% → 50% → 100%Where to check | Steps and rollback, documented in a deprecated section2 sources | Steps or linear ramp; rollback on KPIs2 sources | Scheduled steps; automatic return to 0%2 sources | 10%, 25%, 50%, 100%, in beta2 sources |
| Multi-armed bandit | Thompson on a rate or revenue, per device; read by periodWhere to check | Dynamic allocation (multi-armed bandit)1 source | Multi-armed bandit and contextual bandit2 sources | “Multi-Arm Bandit” campaigns1 source | Bandit; no statistical significance, according to Optimizely1 source |
| Revenue and margin per visitor | Revenue and margin per visitorWhere to check | Revenue per visitor (RevenueIQ); margin not mentioned1 source | Revenue per visitor; margin not mentioned1 source | Revenue per visitor; margin not mentioned1 source | Revenue per visitor; margin not mentioned1 source |
| Winner’s curse | Corrected by Bayesian shrinkageWhere to check | Not mentionedSee details | Not mentionedSee details | Not mentionedSee details | Not mentionedSee details |
| Multivariate testing | Full factorial; effects and interactions corrected by HolmWhere to check | All combinations, shown at random1 source | All combinations, created automatically1 source | Best combination; for high-traffic sites2 sources | Full factorial; interactions between elements2 sources |
| Bayesian mode | Optional; prior drawn from your past tests, the plan decidesWhere to check | Bayesian by default, with “chance to win”1 source | Your choice; non-informative priors by default2 sources | Default; decision through ROPE2 sources | In Feature Experimentation; non-informative priors2 sources |
| Visual editor | Preview, or live site through the script, no extensionWhere to check | Page in an iframe; Chrome extension without the tag2 sources | On top of the site, with the script installed2 sources | Proxy without SmartCode; Chrome extension as fallback2 sources | Overlay with no iframe; Web Companion extension2 sources |
| Widgets | 11 templates (delivery date, survey…); not in the base scriptWhere to check | Library and presets; countdown in a pop-in or banner2 sources | Widget Studio: banner, pop-in, countdown1 source | No-code widgets, display frequency, reusable3 sources | Templates: banner, countdown, pop-up3 sources |
| Script and flicker | Under 7 KB, no page hiding while loadingWhere to check | 35 kB tag when empty; synchronous recommended2 sources | 29 kB; page hidden for up to 750 ms3 sources | Page hidden by default in async mode; synchronous recommended2 sources | Synchronous snippet, with all active experiments2 sources |
| Granular permissions | Custom roles by action, no privilege escalationWhere to check | Fixed roles; per folder in Feature Experimentation2 sources | Predefined roles per project, combinable1 source | Five roles with predefined permissions1 source | Roles per project, per flag, per environment2 sources |
| Data hosting | Amsterdam (database, computing); safeguarded transfers outside the EUWhere to check | Subprocessors in the EU, United States, and India2 sources | French customers hosted in France2 sources | EU (Belgium) as an option; account data in the US2 sources | EU as an option; France not mentioned1 source |
| Consent by default | Required: nothing before agreementWhere to check | Your choice, including a no-consent mode1 source | Your choice, including “consent required”2 sources | Applied to experimentation as an option1 source | Cookies set unless opted out2 sources |
| Pricing model | Free, then €590 to €1,990 per monthWhere to check | On quote, based on MAU2 sources | From $495 per month; Enterprise on quote2 sources | On quote, based on MAU2 sources | On quote, usage-based (MAU)2 sources |
Enterprise suites
| Criterion | Probant | Adobe Target | Dynamic Yield |
|---|---|---|---|
| Statistical method | Sequential frequentist, plan set at launchWhere to check | Welch’s t-test; Bayesian as an option3 sources | Bayesian (Probability to Be Best)2 sources |
| Peeking | O’Brien-Fleming checkpointsWhere to check | Fixed horizon; continuous monitoring in Bayesian3 sources | 14-day minimum duration + expected loss4 sources |
| Traffic split (SRM) | Continuous check; the verdict is blockedWhere to check | Not mentionedSee details | Manual check recommended2 sources |
| Guardrails | Test stops on its own; automatic rollback in rolloutsWhere to check | Additional metrics; automatic stop not mentioned1 source | Secondary metrics; automatic stop not mentioned2 sources |
| Gradual rollouts | 25% → 50% → 100%; cautious: 10% → 25% → 50% → 100%Where to check | Not mentionedSee details | Gradual steps, rollback without redeploying2 sources |
| Multi-armed bandit | Thompson on a rate or revenue, per device; read by periodWhere to check | Auto-Allocate (80/20 bandit)3 sources | Dynamic allocation (90/10 bandit)3 sources |
| Revenue and margin per visitor | Revenue and margin per visitorWhere to check | RPV with 95% confidence interval; margin not mentioned3 sources | Revenue/user with credible interval; margin not mentioned3 sources |
| Winner’s curse | Corrected by Bayesian shrinkageWhere to check | Not mentionedSee details | Not mentionedSee details |
| Multivariate testing | Full factorial; effects and interactions corrected by HolmWhere to check | Full factorial, main effects and interactions3 sources | Not mentionedSee details |
| Bayesian mode | Optional; prior drawn from your past tests, the plan decidesWhere to check | Yes, conversion goals only3 sources | P2BB, P2BC, credible intervals4 sources |
| Visual editor | Preview, or live site through the script, no extensionWhere to check | VEC (WYSIWYG) and form-based composer4 sources | Visual editor + CSS/JS2 sources |
| Widgets | 11 templates (delivery date, survey…); not in the base scriptWhere to check | Recommendation templates; widgets not mentioned1 source | Hundreds of ready-made templates2 sources |
| Script and flicker | Under 7 KB, no page hiding while loadingWhere to check | at.js hides the body; 3000 ms timeout4 sources | Two synchronous scripts in the head3 sources |
| Granular permissions | Custom roles by action, no privilege escalationWhere to check | 4 fixed roles; per property on Premium3 sources | Access levels + custom roles3 sources |
| Data hosting | Amsterdam (database, computing); safeguarded transfers outside the EUWhere to check | AWS; Dublin central cluster possible3 sources | Separate US and EU data centers2 sources |
| Consent by default | Required: nothing before agreementWhere to check | Opt-in off by default5 sources | Active consent mode as an option4 sources |
| Pricing model | Free, then €590 to €1,990 per monthWhere to check | Standard or Premium licenses; public pricing not mentioned2 sources | Paid, on contact; public pricing not mentioned1 source |
Web CRO with public pricing
| Criterion | Probant | Convert | Varify.io | Omniconvert |
|---|---|---|---|---|
| Statistical method | Sequential frequentist, plan set at launchWhere to check | Frequentist, Bayesian, and sequential3 sources | One-sided frequentist by default; Bayesian (Pro)2 sources | Bayesian by default, frequentist as an option2 sources |
| Peeking | O’Brien-Fleming checkpointsWhere to check | Sequential (confidence sequences), Pro plan3 sources | Advice to wait; sequential “SOON”2 sources | Recommended stopping rules; sequential not mentioned2 sources |
| Traffic split (SRM) | Continuous check; the verdict is blockedWhere to check | Chi-square test at 99%, label in the report2 sources | Not mentionedSee details | Automatic detection and notification1 source |
| Guardrails | Test stops on its own; automatic rollback in rolloutsWhere to check | Secondary goals as “guardrails”; automatic stop not mentioned2 sources | Automatic stop on a result condition1 source | Not mentionedSee details |
| Gradual rollouts | 25% → 50% → 100%; cautious: 10% → 25% → 50% → 100%Where to check | Percentage rollout; automatic rollback not mentioned2 sources | Winner deployed in one click; automatic rollback not mentioned2 sources | Not mentionedSee details |
| Multi-armed bandit | Thompson on a rate or revenue, per device; read by periodWhere to check | MAB auto-allocation, Pro plan2 sources | Listed as “SOON” (Enterprise)1 source | Not mentionedSee details |
| Revenue and margin per visitor | Revenue and margin per visitorWhere to check | RPV with confidence and interval; margin not mentioned3 sources | Revenue significance; interval and margin not mentioned2 sources | Revenue per user; “True Profit” in beta3 sources |
| Winner’s curse | Corrected by Bayesian shrinkageWhere to check | Not mentionedSee details | Not mentionedSee details | Not mentionedSee details |
| Multivariate testing | Full factorial; effects and interactions corrected by HolmWhere to check | MVT by sections, Pro plan2 sources | All combinations, traffic split evenly2 sources | Advertised; design and interactions not described in the help center1 source |
| Bayesian mode | Optional; prior drawn from your past tests, the plan decidesWhere to check | Chance to win, risk, expected uplift2 sources | Probability to win + time remaining (Pro)2 sources | Bayesian probability (default)1 source |
| Visual editor | Preview, or live site through the script, no extensionWhere to check | Visual editor and code editors2 sources | Visual editor + code, Chrome extension2 sources | Visual editor through an extension, code editor2 sources |
| Widgets | 11 templates (delivery date, survey…); not in the base scriptWhere to check | Custom pop-ups; other widgets not mentioned1 source | Campaign Booster + Shopify checkout components2 sources | Overlays, templates, NPS surveys3 sources |
| Script and flicker | Under 7 KB, no page hiding while loadingWhere to check | Synchronous recommended, anti-flicker by default3 sources | 11.5 KB, sync or async, optional anti-flicker3 sources | Synchronous by default, cutoff at 1 s3 sources |
| Granular permissions | Custom roles by action, no privilege escalationWhere to check | Five access levels per project2 sources | 3 roles (Pro); custom roles on Enterprise2 sources | Access levels combinable per site2 sources |
| Data hosting | Amsterdam (database, computing); safeguarded transfers outside the EUWhere to check | EU servers (AWS), Akamai CDN3 sources | AWS Frankfurt, Cloudflare CDN1 source | EU data center available2 sources |
| Consent by default | Required: nothing before agreementWhere to check | Consent mode to turn on in code3 sources | No cookies; variants can be served before consent3 sources | Optional GDPR setting, 1-year cookies4 sources |
| Pricing model | Free, then €590 to €1,990 per monthWhere to check | From $299/month (100k MTU), paid overage3 sources | Flat fee from €199/month, unlimited traffic (through GA4)3 sources | By tested visitors; pauses above the quota4 sources |
Shopify natives
| Criterion | Probant | Intelligems | Shoplift |
|---|---|---|---|
| Statistical method | Sequential frequentist, plan set at launchWhere to check | Bayesian (Monte Carlo)1 source | Bayesian1 source |
| Peeking | O’Brien-Fleming checkpointsWhere to check | Advice to wait; sequential not mentioned2 sources | Threshold held for several days; sequential not mentioned2 sources |
| Traffic split (SRM) | Continuous check; the verdict is blockedWhere to check | Build it yourself through the API1 source | Manual monitoring through the API1 source |
| Guardrails | Test stops on its own; automatic rollback in rolloutsWhere to check | Automatic pause to build through the API1 source | Automatic pause to build through the API1 source |
| Gradual rollouts | 25% → 50% → 100%; cautious: 10% → 25% → 50% → 100%Where to check | Not mentioned (winner deployment only)1 source | Not mentioned (winner application)1 source |
| Multi-armed bandit | Thompson on a rate or revenue, per device; read by periodWhere to check | Bandit to build through the API1 source | Fixed split during the test1 source |
| Revenue and margin per visitor | Revenue and margin per visitorWhere to check | Gross profit per visitor, with interval6 sources | RPV and AOV with range; margin not mentioned2 sources |
| Winner’s curse | Corrected by Bayesian shrinkageWhere to check | Not mentionedSee details | Not mentionedSee details |
| Multivariate testing | Full factorial; effects and interactions corrected by HolmWhere to check | Up to 5 groups; one group per combination2 sources | A/B; A/B/n not supported for now1 source |
| Bayesian mode | Optional; prior drawn from your past tests, the plan decidesWhere to check | Probability to beat control and to be best2 sources | Probability to win + lift range1 source |
| Visual editor | Preview, or live site through the script, no extensionWhere to check | Built-in visual editor + CSS/JS2 sources | Shopify theme editor + JS2 sources |
| Widgets | 11 templates (delivery date, survey…); not in the base scriptWhere to check | Offer components, checkout blocks2 sources | Lift Assist sections (Advanced plan)1 source |
| Script and flicker | Under 7 KB, no page hiding while loadingWhere to check | ESM module, blocks rendering by default2 sources | Snippet in the head, hides the page ~150 ms3 sources |
| Granular permissions | Custom roles by action, no privilege escalationWhere to check | Access through Shopify; roles not mentioned1 source | Access through Shopify; roles not mentioned1 source |
| Data hosting | Amsterdam (database, computing); safeguarded transfers outside the EUWhere to check | Transfer outside Europe (Canada, United States)2 sources | United States (Google Cloud); EU option not mentioned2 sources |
| Consent by default | Required: nothing before agreementWhere to check | Not mentioned; cookie from the first visit1 source | Measurement after consent, assignment before4 sources |
| Pricing model | Free, then €590 to €1,990 per monthWhere to check | By order volume, from $69/month3 sources | By site visitors, from $99/month4 sources |
Product and data warehouse
| Criterion | Probant | GrowthBook | PostHog | Statsig | Eppo | LaunchDarkly | Amplitude |
|---|---|---|---|---|---|---|---|
| Statistical method | Sequential frequentist, plan set at launchWhere to check | Bayesian by default, frequentist as an option3 sources | Bayesian by default, frequentist as an option2 sources | Frequentist by default, SPRT, and Bayesian2 sources | Sequential by default, fixed, hybrid, Bayesian2 sources | Sequential frequentist by default, Bayesian as an option3 sources | Sequential by default, t-test, and Bayesian2 sources |
| Peeking | O’Brien-Fleming checkpointsWhere to check | Sequential test (frequentist, Pro plan)4 sources | Optional sequential test (frequentist)3 sources | Optional mSPRT, SPRT with no penalty3 sources | Sequential by default (confidence sequences)3 sources | Sequential GAVI (always-valid intervals)2 sources | mSPRT, results valid at any time2 sources |
| Traffic split (SRM) | Continuous check; the verdict is blockedWhere to check | Automatic check, with a warning4 sources | Chi-square check, with a warning4 sources | χ² test and warning; blocking not described2 sources | χ² (α = 0.001), failed diagnostic4 sources | Alert in A/B tests; rollback in rollouts3 sources | Detection and notification; blocking not described3 sources |
| Guardrails | Test stops on its own; automatic rollback in rolloutsWhere to check | Guardrails; automatic stop through Safe Rollouts4 sources | Secondary metrics; automatic stop not mentioned2 sources | Alerts on tests; automatic stop on flags3 sources | Guardrails; automatic pause/stop in canary3 sources | Guardrail metrics; automatic stop outside experiments3 sources | Guardrail metrics; automatic stop not mentioned2 sources |
| Gradual rollouts | 25% → 50% → 100%; cautious: 10% → 25% → 50% → 100%Where to check | Ramps + automatic rollback (Safe Rollouts)4 sources | Scheduled steps; automatic rollback not mentioned2 sources | Scheduled steps, automatic rollback (Pro+)2 sources | Gradual and canary, with automatic rollback2 sources | Gradual rollouts and automatic rollback3 sources | Percentage rollout, manual rollback2 sources |
| Multi-armed bandit | Thompson on a rate or revenue, per device; read by periodWhere to check | Multi-armed bandits (Pro), contextual (Enterprise)3 sources | Not mentionedSee details | Autotune (bandit) and Autotune AI (contextual)2 sources | Adaptive bandit and contextual bandits3 sources | Multi-armed bandits (Thompson Sampling), all plans3 sources | Thompson Sampling, Growth/Enterprise add-on3 sources |
| Revenue and margin per visitor | Revenue and margin per visitorWhere to check | Revenue per visitor and AOV, with intervals3 sources | Revenue per user and AOV, with intervals3 sources | Revenue per user with interval; margin not mentioned2 sources | Revenue with interval; margin through the warehouse3 sources | Numeric metric with interval; margin not mentioned2 sources | Property sum per user, with CI; margin not mentioned3 sources |
| Winner’s curse | Corrected by Bayesian shrinkageWhere to check | Optional prior; James-Stein on overall impact4 sources | Non-informative prior, no documented correction1 source | Correction on the program’s cumulative impact1 source | Empirical Bayes on cumulative impact3 sources | Empirical shrinkage, Bayesian mode only2 sources | Not mentioned; alert on suspicious lifts1 source |
| Multivariate testing | Full factorial; effects and interactions corrected by HolmWhere to check | A/B/n; factorial design not mentioned3 sources | A/B/n up to 20 variants2 sources | A/B/n and interaction detection3 sources | A/B/n and interaction detection2 sources | A/B/n; no documented factorial design3 sources | Mentioned, no documented factorial design1 source |
| Bayesian mode | Optional; prior drawn from your past tests, the plan decidesWhere to check | Default engine, customizable priors3 sources | Default: chance to win, credible interval2 sources | Chance to beat, expected loss, priors2 sources | Normal or Student’s t prior, P(effect > 0)3 sources | Shrinkage prior, expected loss3 sources | Non-informative prior, chance to beat3 sources |
| Visual editor | Preview, or live site through the script, no extensionWhere to check | Chrome extension (AI + manual), Pro plan3 sources | No-code toolbar (text, CSS, HTML)4 sources | Chrome extension in maintenance mode3 sources | Not mentionedSee details | No visual editor documented1 source | Visual editor with overlay; code on Growth+3 sources |
| Widgets | 11 templates (delivery date, survey…); not in the base scriptWhere to check | Not mentionedSee details | Pop-up surveys; other widgets not mentioned1 source | Not mentionedSee details | Not mentionedSee details | Not mentionedSee details | Buttons, banners, modal pop-up2 sources |
| Script and flicker | Under 7 KB, no page hiding while loadingWhere to check | Async script, optional anti-flicker, edge4 sources | Async snippet; anti-flicker not mentioned3 sources | Synchronous script, briefly hides the body3 sources | Non-blocking JS SDK; anti-flicker not mentioned3 sources | JavaScript SDK to code; manual anti-flicker2 sources | Synchronous recommended; async hides the page3 sources |
| Granular permissions | Custom roles by action, no privilege escalationWhere to check | Roles (Pro), custom roles on Enterprise2 sources | RBAC roles limited to Enterprise3 sources | Fixed roles; custom roles on Enterprise2 sources | Custom Datadog roles, per-flag access3 sources | Predefined roles; custom roles on Enterprise3 sources | Standard roles; custom roles on Enterprise1 source |
| Data hosting | Amsterdam (database, computing); safeguarded transfers outside the EUWhere to check | AWS cloud in the United States; self-hosting possible6 sources | EU cloud in Frankfurt (AWS eu-central-1)2 sources | Cloud in the United States; EU option3 sources | EU site (Germany), data in the warehouse4 sources | EU instance (AWS Frankfurt), Enterprise only4 sources | EU data center in Frankfurt (AWS)2 sources |
| Consent by default | Required: nothing before agreementWhere to check | Cookie set by default; can be deferred5 sources | 365-day cookie by default; opt-out possible4 sources | Collects by default; consent to code4 sources | Collects by default, cookie up to 1 year3 sources | No cookies; anonymous key in localStorage3 sources | Consent mode available, off by default3 sources |
| Pricing model | Free, then €590 to €1,990 per monthWhere to check | Per seat: $40/seat/month (Pro)5 sources | Usage-based (flag requests), no seat fee7 sources | Per event: Pro $150/month + overage3 sources | Per launched experiment: from $450/month4 sources | Usage-based: server connections and client MAU3 sources | By event volume; Growth/Enterprise on quote5 sources |
Migrate to Probant
Your AB Tasty, VWO, or Kameleoon history can be imported and reread by Probant’s engine.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway