Probant vs. Eppo
Both tools compared point by point, based on Eppo’s public documentation, accessed November 11, 2026. Every statement about Eppo links to its source.
What Eppo offers.
Eppo is a “warehouse-native” experimentation platform: calculations run in the customer’s data warehouse.1 Acquired by Datadog, it is now sold as Datadog Experiments.23 Sequential testing is its default method; fixed horizon and a Bayesian reading are available.45
Probant is for sites that want A/B test verdicts that hold up, with a plan set before launch and guardrails monitored through rollout. It measures what each conversion is worth: the margin on an order, a qualified or signed lead, a paid subscription.
The differences, point by point.
On the left, what Eppo’s public documentation says, each sentence with its source. On the right, what Probant does. “Not mentioned” means nothing was found in that documentation, which does not prove a feature is missing.
| Criterion | Eppo | Probant |
|---|---|---|
| Statistical method | Four methods: sequential frequentist (default), fixed-sample frequentist, hybrid sequential (sequential, then fixed at the target date), and Bayesian. The default method can be set at the organization level.45 | Frequentist, with a sequential plan set before launch: target effect, duration, and checkpoints. The verdict fits in one sentence, with a 95% range. Where to check |
| Peeking | The default analysis is sequential: “confidence sequence” intervals (Howard et al. normal mixture) valid at every look. Hybrid mode lets you stop early on a degradation and decide on gains at the planned date.678 | Decisions are only made at planned checkpoints, with O’Brien-Fleming boundaries: very strict at first, close to the usual threshold at the end. During a rollout, ranges are always valid (mSPRT), so you can check them at any time. Where to check |
| Traffic split (SRM) | Datadog detects SRM with Pearson’s chi-square test; an imbalance shows as a failed diagnostic on the results page, and the documentation asks you to treat it as blocking before reading results.9101112 | The traffic split is checked with a chi-square test every time the report is read. If the split is off, the test moves to “blocked” status and no winner is declared. Where to check |
| Guardrails | Guardrail metrics (secondary metrics, tolerated degradation threshold). Datadog Feature Flags “canary” rollouts automatically pause or stop the rollout if a guardrail degrades significantly; the experiments docs mention “automated guardrail actions” without details.131415 | Guardrail metrics are tracked with their range; a guardrail is only breached if the whole range crosses the limit. A breached guardrail blocks the test’s verdict, stops a running test on its own, and during a rollout moves visitors back to the original automatically. Where to check |
| Gradual rollouts | Gradual rollouts in scheduled percentage steps; canaries monitor guardrails (errors, latency…) and roll back automatically on regression.1617 | Steps of 25% → 50% → 100% by default, 10% → 25% → 50% → 100% in cautious mode. Automatic rollback when a guardrail is breached; an exposed visitor stays exposed in the following steps. Where to check |
| Multi-armed bandit | Adaptive multi-armed bandit and contextual bandits documented by Eppo; the Datadog Experiments docs only mention bandits in passing (excluded from approvals).181920 | Optimization test: Thompson sampling, reallocated at most once a day, with a minimum share per variant. It maximizes a conversion rate, or an amount per visitor such as revenue: purchase probability and average order value drawn from their distributions, with values capped at a percentile as in an A/B test. Optionally, shares per device (mobile, tablet, desktop): one draw per device, each with its own learning and floor. A visitor always keeps their variant: a new split only applies to visitors who arrive after it. The screen says so: it maximizes conversions (or the amount per visitor) during the test, without proving that a variant is better. The difference is read period by period (and device by device), with no declared winner; the CSV export gives the same reading. Where to check |
| Revenue and margin per visitor | Each metric (including revenue per user) is shown with relative lift and confidence interval; average order value is defined as a ratio metric.2122 Eppo’s About page (an investor quote) mentions margin among business metrics; technically, a margin metric can be defined in SQL if costs are in the warehouse. No dedicated product cost feature is documented.23 | Revenue and margin per visitor, each with its range; the gain is stated in euros. Margin is calculated on the server from reconciled orders, the cost of each product (Shopify or CSV), and store fees; no margin verdict below 90% of revenue covered by known costs. Where to check |
| Winner’s curse | Cumulative impact corrects for the winner’s curse with a hierarchical model (empirical Bayes) fitted on all experiments for a metric; each test’s corrected estimate is shown on its page. At least five eligible experiments are required.242526 | The reported gain is corrected for the winner’s curse by Bayesian shrinkage toward the effects of the store’s past tests. Where to check |
| Bayesian mode | Bayesian mode with a normal or Student’s t prior (which shrinks noisy estimates), credible interval, and probability of a positive effect. Expected loss is not mentioned.272829 | Optional, off by default: a Bayesian reading next to the verdict, with the probability that the variant does better, the credible interval, and the expected loss of each choice. The prior is centered on zero and as wide as the effects measured by the store’s completed tests; while there are too few of them to estimate it, the Bayesian reading is not shown. The decision always follows the sealed plan; when the two readings disagree, the report explains why. Where to check |
| Multiple comparisons | “Preferential” Bonferroni correction across variants × metrics, with a share of alpha reserved for the primary metric. Not available in Bayesian mode.3031 | Segment comparisons are corrected with the Holm method. With several variants, the error risk is split across comparisons (Bonferroni). Where to check |
| Multivariate testing | Several variations per flag (A/B/n, called “multivariate”); no full factorial design documented. Eppo documents detection of interactions between parallel experiments.3233 | Full factorial design generated by Probant: two to four sections, each with two to four levels, up to sixteen combinations, each served as a variant. Main effects of each section across all traffic and interactions between two sections, corrected with the Holm method on the comparisons stated in the plan. Based on the workspace’s measured traffic, the plan says whether the test can reach a conclusion and suggests removing a level or a section; no best combination is declared without a check. Where to check |
| Variance reduction (CUPED) | CUPED on by default, in an extended version (CUPED++) that also uses other metrics and assignment attributes as covariates. No CUPED on segments or ad hoc filters.34353637 | CUPED applied to live test reports, using each visitor’s history before their first exposure; without history, the difference stays unadjusted and the report says so. Where to check |
| Visual editor | A visual editor is not mentioned in their public documentation (as of November 11, 2026). | Two ways to edit, producing the same changes: a page preview loaded by Probant, without its scripts, or the live page on the site, scripts included. On the site, the editor is loaded by the script already installed, through a short-lived signed link for one draft and one variant, with no browser extension. It only sends the chosen changes to Probant. Where to check |
| Widgets | Ready-made widgets are not mentioned in their public documentation (as of November 11, 2026). | Banner, countdown, pop-up, badge, reassurance, action bar, delivery date, social proof, promo code, free shipping, and survey, set up without code in the editor and saved in the workspace library for other tests. A widget is just a set of ordinary changes: escaped text, links over https or to the site, and styles applied without a style attribute, so a strict CSP does not need loosening. The countdown follows the site’s time zone and disappears at the deadline; the pop-in opens on arrival, on exit intent, or after scrolling, closes with the keyboard, and does not reappear during the visit if consent allows. Their small modules only load for visitors in the variant. The delivery date is calculated in the visitor’s browser (cutoff time, business days, public holidays); social proof shows the exact number of purchases received over a rolling week, and only above a threshold. The survey asks up to 8 questions (closed answers, recommendation scale, open question) with conditional logic, on the page or through a link outside the site (email, QR code), based on 70 templates (recommendation, satisfaction, effort, checkout blockers, cancellation, concept test, multi-question surveys…); open answers are redacted before sending (email addresses, phone numbers, cards, IBANs), kept for 180 days, and grouped by theme; each question is read by variant, with its range and response rate, and comparisons across all questions are corrected together; a response through a link is attached to the variant the visitor saw, recalculated by Probant, or stays outside the test. Where to check |
| Script and flicker | No visual snippet is documented: a JavaScript SDK (OpenFeature at Datadog) fetches precomputed assignments; Eppo initialization is non-blocking. No built-in anti-flicker mechanism or published size; a guide suggests hiding elements at first.383940 | Script under 7 KB compressed, loaded asynchronously, that does not hide the page while it loads. Server-side assignment available, for a display with no flicker at all. Where to check |
| Granular permissions | Datadog custom roles and per-flag permissions; Eppo offered four roles plus owning teams.414243 | Four fixed roles, plus custom roles the owner builds from a catalog of named permissions: prepare, launch and stop, roll out, approve a launch, write variant JavaScript, manage flags, metrics, margin, members, and keys. No privilege escalation: billing, roles, transfer, and deletion of the workspace stay with the owner; no one can grant more than their own permissions, and an API key cannot exceed its author’s. Where to check |
| Data hosting | Datadog offers an EU1 site in Germany; Experiments is only excluded from US government sites. In warehouse-native mode, raw data stays in the customer’s warehouse. Eppo ran on Google Cloud.44454647 | Database and compute in Amsterdam, with Scaleway, in the European Union. Scaleway, which hosts the database and compute, is a French company. Published subprocessor list: Scaleway (Amsterdam, Netherlands (nl-ams region), in the European Union), Brevo (European Union: servers in France and Germany, encrypted backups in Belgium (Google Cloud), according to Brevo), Mistral AI (European Union by default (European API endpoint), according to Mistral), ClickHouse (option not used), Stripe (Contract with Stripe Payments Europe, Ltd. (Ireland); data transferred to Stripe, LLC (United States), according to Stripe), Sentry (European Union: Frankfurt (Germany), including backups, according to Sentry). Depending on the feature used, Mistral may temporarily transfer data outside the European Union to the subprocessors it publishes; its data processing agreement limits these transfers to countries the EU recognizes as adequate, or covers them with standard contractual clauses. Stripe transfers payment data to Stripe, LLC, in the United States, under the Data Privacy Framework and, failing that, standard contractual clauses. Whatever region is chosen, Sentry may keep the data of Probant’s Sentry account (members, settings, audit log, project descriptions) in the United States; reported errors stay in Frankfurt. Where to check |
| Consent by default | The RUM SDK (browser-side metrics) collects by default (“granted”) and keeps an anonymous identifier for up to a year in a cookie; opt-out has to be passed by the site. The Eppo SDK can defer sending assignments until consent.484950 | Consent required by default: no cookie, no assignment, and nothing sent before the visitor agrees. Consent given mid-page: the page does not change in front of the visitor; their variant starts on the next page. First-party cookie, lasting 390 days at most, never extended. Where to check |
| Pricing model | Billed per launched experiment (at least 500 subjects and 3 days of data): from $450 per experiment per month billed annually, $575 on demand; unlimited seats. Feature flags billed per million requests beyond the first.51525354 | Creating a Probant workspace is free; paid plans cost €590 to €1,990 per month depending on the number of tested visitors (2 months free with annual billing), with no overage charges. Agency plan: 1,000,000 tested visitors per month, shared across client workspaces, from 3 workspaces, for €1,490 per month or €14,900 per year. Where to check |
What Eppo also does.
Probant shares these points with Eppo.
A sequential test
The default analysis is sequential: “confidence sequence” intervals (Howard et al. normal mixture) valid at every look. Hybrid mode lets you stop early on a degradation and decide on gains at the planned date.678
Automatic stops on guardrails
Guardrail metrics (secondary metrics, tolerated degradation threshold). Datadog Feature Flags “canary” rollouts automatically pause or stop the rollout if a guardrail degrades significantly; the experiments docs mention “automated guardrail actions” without details.131415
A multiple comparisons correction
“Preferential” Bonferroni correction across variants × metrics, with a share of alpha reserved for the primary metric. Not available in Bayesian mode.3031
Variance reduction (CUPED)
CUPED on by default, in an extended version (CUPED++) that also uses other metrics and assignment attributes as covariates. No CUPED on segments or ad hoc filters.34353637
A winner’s curse correction
Cumulative impact corrects for the winner’s curse with a hierarchical model (empirical Bayes) fitted on all experiments for a metric; each test’s corrected estimate is shown on its page. At least five eligible experiments are required.242526
When to choose Eppo over Probant.
Based on Eppo’s public documentation.
Your data already lives in a warehouse
Warehouse-native analysis: calculations run in the customer’s warehouse and metrics are defined in SQL.55
You already use Datadog
Datadog Experiments combines Eppo’s statistical methods with Datadog’s observability, which acts as a real-time guardrail.56
You want contextual bandits
Personalization through contextual bandits (documented by Eppo); their impact is measured in a two-variation experiment, the bandit against a control.1957
What sets Probant apart.
These points are not in Eppo’s public documentation, accessed November 11, 2026. A missing mention does not prove a feature is missing; if these points matter to you, ask Eppo.
Margin per visitor
Probant measures revenue and margin per visitor, each with its range.
EppoEppo’s About page (an investor quote) mentions margin among business metrics; technically, a margin metric can be defined in SQL if costs are in the warehouse. No dedicated product cost feature is documented.23 Margin per visitor in test reports is not mentioned there (as of November 11, 2026).
Learn moreConsent required by default
No cookie, no assignment, and nothing sent before the visitor agrees.
EppoThe RUM SDK (browser-side metrics) collects by default (“granted”) and keeps an anonymous identifier for up to a year in a cookie; opt-out has to be passed by the site. The Eppo SDK can defer sending assignments until consent.484950 Consent required by default is not mentioned there (as of November 11, 2026).
A visual editor
Variants are created without a developer, in a page preview or on the live site, with no browser extension.
EppoA visual editor is not mentioned in their public documentation (as of November 11, 2026).
Learn moreReady-made widgets
Banner, countdown, pop-in, social proof, and survey widgets, set up without code in the editor.
EppoReady-made widgets are not mentioned in their public documentation (as of November 11, 2026).
Frequently asked questions.
How much does Eppo cost?
Billed per launched experiment (at least 500 subjects and 3 days of data): from $450 per experiment per month billed annually, $575 on demand; unlimited seats. Feature flags billed per million requests beyond the first.51525354 Creating a Probant workspace is free; paid plans cost €590 to €1,990 per month depending on the number of tested visitors (2 months free with annual billing), with no overage charges.
Where does Eppo host data?
Datadog offers an EU1 site in Germany; Experiments is only excluded from US government sites. In warehouse-native mode, raw data stays in the customer’s warehouse. Eppo ran on Google Cloud.44454647 Probant hosts its database and application in Amsterdam, with Scaleway.
Does Eppo measure margin?
Eppo’s About page (an investor quote) mentions margin among business metrics; technically, a margin metric can be defined in SQL if costs are in the warehouse. No dedicated product cost feature is documented.23 Probant measures revenue and margin per visitor, each with its range, from orders and the cost of each product.
Can you look at Eppo results while a test is running?
The default analysis is sequential: “confidence sequence” intervals (Howard et al. normal mixture) valid at every look. Hybrid mode lets you stop early on a degradation and decide on gains at the planned date.678 Probant sets its checkpoints before launch (O’Brien-Fleming) and uses always-valid ranges during rollouts.
Sources.
Official pages from Eppo, accessed November 11, 2026. Quotes are reproduced in their original language.
- Source 1: www.geteppo.com
Warehouse-native architecture powers a world class statistical engine without becoming a black box.
- Source 2: www.datadoghq.com/about/latest-news/press-releases/datadog-acquires-eppo-to-expand-its-ai
today announced it has acquired Eppo, a feature flagging and experimentation platform, which will tightly integrate with Datadog’s existing Product Analytics suite.
- Source 4: docs.datadoghq.com/experiments/statistics/analysis_methods
Sequential frequentist analysis is the default because it lets you monitor results and make ship or rollback decisions without inflating the false positive rate.
- Source 5: docs.geteppo.com/statistics/confidence-intervals/analysis-methods
Eppo has four different methods for estimating the expected lift from experiment data and constructing a confidence interval around that estimate.
- Source 6: docs.datadoghq.com/experiments/statistics/analysis_methods
Sequential analysis controls the false positive rate while allowing repeated checks of the results.
- Source 7: docs.datadoghq.com/experiments/statistics/lift_and_interval_calculations
The boundary follows the normal-mixture construction of Howard et al. (2021).
- Source 8: docs.geteppo.com/statistics/confidence-intervals/analysis-methods
We chose to make sequential analysis the default because it allows for maximum flexibility while still making ship/no-ship decisions relatively straightforward: you can peek—and make a decision—at any time, without risking unacceptably high rates of detecting phantom effects.
- Source 9: docs.datadoghq.com/experiments/diagnostics
Datadog detects SRM with Pearson's chi-squared goodness-of-fit test.
- Source 10: docs.datadoghq.com/experiments/diagnostics
Treat SRM as a blocker for interpreting results.
- Source 11: docs.datadoghq.com/experiments/diagnostics
When a check finds an issue, Datadog displays it on the experiment results page as a warning, metric message, or failed diagnostic check.
- Source 12: docs.geteppo.com/experiment-analysis/reading-results/experiment-status
AND there is no traffic imbalance
- Source 13: docs.datadoghq.com/experiments/plan_and_launch_experiments
(Optional) Click the Secondary metrics button to add guardrail metrics, which monitor unintended effects of the experiment on other areas such as performance, engagement, or revenue.
- Source 14: docs.datadoghq.com/feature_flags/use_cases/progressive_rollouts
When the canary detects a statistically significant adverse change in a guardrail metric, it automatically pauses or stops the rollout.
- Source 15: docs.datadoghq.com/experiments/approvals
Secondary metrics that evaluate experiment health are supported. They are separate from automated guardrail actions.
- Source 16: docs.datadoghq.com/feature_flags/use_cases/progressive_rollouts
Progressive rollouts release new functionality gradually by incrementing the percentage of subjects exposed to the feature over time.
- Source 17: www.datadoghq.com/pricing
Automate risk mitigation with telemetry-driven canary rollouts with automatic rollbacks
- Source 18: www.geteppo.com/experimentation
Hone in on winning creatives, AI models, and product changes with an adaptive multi-armed bandit.
- Source 19: docs.geteppo.com/contextual-bandits
Eppo's Contextual Bandits allows us to automatically optimize personalized user experiences.
- Source 20: docs.datadoghq.com/experiments/approvals
Experiment approvals are not supported for canary deployments, bandits, or experiments with externally managed randomization.
- Source 21: docs.datadoghq.com/experiments/reading_results
Confidence interval: A range of lift values consistent with the observed data, shown as a bar centered on the relative lift estimate.
- Source 22: docs.geteppo.com/data-management/metrics/ratio-metric
Average order value is defined as the total amount of revenue divided by the total number of orders:
- Source 23: www.geteppo.com/about-us
Rather than measuring basic click-throughs, Eppo leverages core business metrics such as revenue, margin, and subscriptions
- Source 24: docs.datadoghq.com/experiments/concepts/cumulative_impact
Datadog Experiments performs this correction automatically by fitting a statistical model across all of your team's experiments on a metric and producing a corrected estimate of cumulative impact.
- Source 25: docs.datadoghq.com/experiments/concepts/cumulative_impact
Selecting an experiment from the list opens its result page, where you can see the original measured lift, the shipping decision, and the corrected estimate used in the cumulative calculation.
- Source 26: docs.datadoghq.com/experiments/concepts/cumulative_impact
The model requires at least five eligible experiments.
- Source 27: docs.datadoghq.com/experiments/statistics/analysis_methods
Bayesian analysis uses experiment data to update a prior belief about plausible lift values.
- Source 28: docs.datadoghq.com/experiments/statistics/analysis_methods
Both priors shrink noisy lift estimates toward the prior mean.
- Source 29: docs.datadoghq.com/experiments/statistics/lift_and_interval_calculations
The posterior also gives the probability of a positive effect:
- Source 30: docs.datadoghq.com/experiments/statistics/multiple_testing_correction
Datadog uses preferential Bonferroni correction.
- Source 31: docs.datadoghq.com/experiments/statistics/multiple_testing_correction
Multiple testing correction is available for frequentist analysis methods.
- Source 32: docs.datadoghq.com/feature_flags/guide/headless_cms
Datadog flags also allow for multiple variations on a flag, enabling A/B/n and multivariate tests.
- Source 33: docs.geteppo.com/statistics/interaction-detection
Effect Interaction: The effect of one variant depends on exposure in some other experiment.
- Source 36: docs.geteppo.com/data-management/definitions/overview
Assignment tables contain subject-level properties. These categorical columns are used to filter and segment experiment results, and are also used as additional control variables in Eppo's CUPED++ variance reduction model.
- Source 37: docs.geteppo.com/experiment-analysis/deep-dive/segments
We do not compute CUPED results for neither segments nor ad-hoc filters due to the computational cost of computing CUPED results.
- Source 38: docs.datadoghq.com/feature_flags/client/javascript
Use DatadogProvider for most applications. It fetches precomputed assignments, retrieves updated assignments when the evaluation context changes, and manages built-in telemetry.
- Source 39: docs.geteppo.com/sdks/best-practices/non-blocking-initialization
Eppo's SDK initialization is non-blocking by default.
- Source 40: docs.geteppo.com/guides/marketing/integrating-with-webflow
By making sure our elements are “blank” to start with, we can completely reduce any flicker that would be caused by re-writing and element with Javascript.
- Source 41: docs.datadoghq.com/account_management/rbac
By default, Datadog offers three roles, and you can create custom roles so you can define a better mapping between your users and their permissions.
- Source 42: docs.datadoghq.com/feature_flags/concepts/permissions
Datadog Feature Flags supports organization-level permissions and granular per-flag access controls.
- Source 43: docs.geteppo.com/administration/users-and-permissions
Four roles are provided that can be assigned to users to specify their level of access: "Admin," "Data Owner," "Experiment Editor," and "Viewer."
- Source 45: docs.datadoghq.com/experiments
Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com
- Source 46: docs.geteppo.com/reference/architecture-and-security
Queries to process raw data execute on Customer DW — raw data never leaves Customer DW.
- Source 47: docs.geteppo.com/reference/architecture-and-security
Eppo infrastructure is built entirely using GCP in an environment that is SOC2 Type 2 compliant.
- Source 48: docs.datadoghq.com/real_user_monitoring/application_monitoring/browser/advanced_configuration
"granted" (default): The RUM Browser SDK starts collecting data and sends it to Datadog.
- Source 49: docs.datadoghq.com/real_user_monitoring/application_monitoring/browser/advanced_configuration
Lifetime: Persists for up to one year across sessions in the Datadog session cookie (_dd_s_v2).
- Source 50: docs.geteppo.com/sdks/client-sdks/javascript/gdpr-compliance
In certain situations, you may wish to hold off on logging assignments until a later time, such as the user granting the relevant permission.
- Source 51: www.datadoghq.com/pricing
Datadog Experiments is priced based on the number of successfully launched experiments.
- Source 54: www.datadoghq.com/pricing
An experiment becomes billable once it has launched and reached at least 500 subjects and 3 days of data.
- Source 55: docs.datadoghq.com/experiments
In warehouse native mode, Datadog runs experiment analysis directly in Snowflake, BigQuery, Redshift, or Databricks.
- Source 56: www.datadoghq.com/about/latest-news/press-releases/datadog-experiments-launches
Powered by Datadog’s acquisition of Eppo, Datadog Experiments pairs best-in-class statistical methods with real-time observability guardrails so companies can test what matters, move quickly and ship with confidence.
- Source 57: docs.geteppo.com/bandit-quickstart
To evaluate the impact of the Contextual Bandit, we run an experiment with exactly two variations: the bandit and a control.
Migrate from Eppo
Your Eppo history can be imported as a generic CSV (visitors and conversions per variant) and is reread by Probant’s engine: every verdict is recalculated from visitors and conversions.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway