Probant vs. Statsig
Both tools compared point by point, based on Statsig’s public documentation, accessed November 11, 2026. Every statement about Statsig links to its source.
What Statsig offers.
Statsig presents itself as a unified platform for feature flags, A/B testing, and product analytics.1 After an agreement to join OpenAI, Statsig was acquired by Amplitude on May 5, 2026; the original team stayed at OpenAI.23 Its engine is frequentist by default; sequential tests and a Bayesian reading are available.45
Probant is for sites that want A/B test verdicts that hold up, with a plan set before launch and guardrails monitored through rollout. It measures what each conversion is worth: the margin on an order, a qualified or signed lead, a paid subscription.
The differences, point by point.
On the left, what Statsig’s public documentation says, each sentence with its source. On the right, what Probant does. “Not mentioned” means nothing was found in that documentation, which does not prove a feature is missing.
| Criterion | Statsig | Probant |
|---|---|---|
| Statistical method | Three methods to choose per experiment: frequentist (default, sequential as an option), SPRT, and Bayesian. The engine computes the lift, confidence interval, and p-value for each metric.45 | Frequentist, with a sequential plan set before launch: target effect, duration, and checkpoints. The verdict fits in one sentence, with a 95% range. Where to check |
| Peeking | The frequentist sequential test (mSPRT, wider intervals before the target date) is turned on in advanced settings; an SPRT mode lets you look without penalty and accept the null hypothesis. Statsig advises against looking at results without these corrections.678 | Decisions are only made at planned checkpoints, with O’Brien-Fleming boundaries: very strict at first, close to the usual threshold at the end. During a rollout, ranges are always valid (mSPRT), so you can check them at any time. Where to check |
| Traffic split (SRM) | SRM check by chi-square test on all experiments and feature gates, shown in health checks as a warning or a failure depending on the p-value. The documentation does not describe blocking the results; it recommends investigating or restarting the experiment.910 | The traffic split is checked with a chi-square test every time the report is read. If the split is off, the test moves to “blocked” status and no winner is declared. Where to check |
| Guardrails | In an experiment, guardrail metrics trigger alerts (Rollout / Guardrail Metric Alerts), with no documented automatic stop. Safeguards (Pro and Enterprise plans) automatically roll back, pause, or end a feature gate rollout when an alert fires.111213 | Guardrail metrics are tracked with their range; a guardrail is only breached if the whole range crosses the limit. A breached guardrail blocks the test’s verdict, stops a running test on its own, and during a rollout moves visitors back to the original automatically. Where to check |
| Gradual rollouts | Percentage rollouts and automatic scheduled steps; Safeguards (Pro and Enterprise) can automatically bring a gate back to 0% when a metric alert fires.1415 | Steps of 25% → 50% → 100% by default, 10% → 25% → 50% → 100% in cautious mode. Automatic rollback when a guardrail is breached; an exposed visitor stays exposed in the following steps. Where to check |
| Multi-armed bandit | Autotune is a Bayesian multi-armed bandit that shifts traffic to the best variant for a target metric; Autotune AI is a contextual bandit that personalizes the variant per user.1617 | Optimization test: Thompson sampling, reallocated at most once a day, with a minimum share per variant. It maximizes a conversion rate, or an amount per visitor such as revenue: purchase probability and average order value drawn from their distributions, with values capped at a percentile as in an A/B test. Optionally, shares per device (mobile, tablet, desktop): one draw per device, each with its own learning and floor. A visitor always keeps their variant: a new split only applies to visitors who arrive after it. The screen says so: it maximizes conversions (or the amount per visitor) during the test, without proving that a variant is better. The difference is read period by period (and device by device), with no declared winner; the CSV export gives the same reading. Where to check |
| Revenue and margin per visitor | Each metric, including a sum of revenue per user, is reported with lift, confidence interval, and p-value. Average order value is built as a ratio metric.1819 Margin per visitor is not mentioned in their public documentation (as of November 11, 2026). | Revenue and margin per visitor, each with its range; the gain is stated in euros. Margin is calculated on the server from reconciled orders, the cost of each product (Shopify or CSV), and store fees; no margin verdict below 90% of revenue covered by known costs. Where to check |
| Winner’s curse | “Aggregated Impact” adds up the projected impact of tests, weighted by the risk of a false positive (winner’s curse, Kohavi et al. method). No correction of the lift shown for a single experiment is documented, apart from Bayesian mode priors.20 | The reported gain is corrected for the winner’s curse by Bayesian shrinkage toward the effects of the store’s past tests. Where to check |
| Bayesian mode | Optional Bayesian mode: chance to beat control and expected loss instead of the p-value, an adjustable win threshold, and optional informative priors. The analysis type can no longer change after launch.2122 | Optional, off by default: a Bayesian reading next to the verdict, with the probability that the variant does better, the credible interval, and the expected loss of each choice. The prior is centered on zero and as wide as the effects measured by the store’s completed tests; while there are too few of them to estimate it, the Bayesian reading is not shown. The decision always follows the sealed plan; when the two readings disagree, the report explains why. Where to check |
| Multiple comparisons | Bonferroni correction per metric (with a share of alpha reserved for primary metrics) and the Benjamini-Hochberg procedure per variant and per metric, set in the experiment’s advanced settings.2324 | Segment comparisons are corrected with the Holm method. With several variants, the error risk is split across comparisons (Bonferroni). Where to check |
| Multivariate testing | The pricing page calls A/B/n tests “multi-variate”; no full factorial design is documented. Statsig also measures the interaction between two overlapping experiments.252627 | Full factorial design generated by Probant: two to four sections, each with two to four levels, up to sixteen combinations, each served as a variant. Main effects of each section across all traffic and interactions between two sections, corrected with the Holm method on the comparisons stated in the plan. Based on the workspace’s measured traffic, the plan says whether the test can reach a conclusion and suggests removing a level or a section; no best combination is declared without a check. Where to check |
| Variance reduction (CUPED) | CUPED is applied automatically to all experiments, with a 7-day window before each user’s exposure. CURE, the regression extension, is documented for Warehouse Native; stratification is also available.282930 | CUPED applied to live test reports, using each visitor’s history before their first exposure; without history, the difference stays unadjusted and the report says so. Where to check |
| Visual editor | “Sidecar” visual editor as a Chrome extension on the live site; it is in maintenance mode (no new features or fixes), and Statsig points to Amplitude Web Experiment.313233 | Two ways to edit, producing the same changes: a page preview loaded by Probant, without its scripts, or the live page on the site, scripts included. On the site, the editor is loaded by the script already installed, through a short-lived signed link for one draft and one variant, with no browser extension. It only sends the chosen changes to Probant. Where to check |
| Widgets | Ready-made widgets are not mentioned in their public documentation (as of November 11, 2026). | Banner, countdown, pop-up, badge, reassurance, action bar, delivery date, social proof, promo code, free shipping, and survey, set up without code in the editor and saved in the workspace library for other tests. A widget is just a set of ordinary changes: escaped text, links over https or to the site, and styles applied without a style attribute, so a strict CSP does not need loosening. The countdown follows the site’s time zone and disappears at the deadline; the pop-in opens on arrival, on exit intent, or after scrolling, closes with the keyboard, and does not reappear during the visit if consent allows. Their small modules only load for visitors in the variant. The delivery date is calculated in the visitor’s browser (cutoff time, business days, public holidays); social proof shows the exact number of purchases received over a rolling week, and only above a threshold. The survey asks up to 8 questions (closed answers, recommendation scale, open question) with conditional logic, on the page or through a link outside the site (email, QR code), based on 70 templates (recommendation, satisfaction, effort, checkout blockers, cancellation, concept test, multi-question surveys…); open answers are redacted before sending (email addresses, phone numbers, cards, IBANs), kept for 180 days, and grouped by theme; each question is read by variant, with its range and response rate, and comparisons across all questions are corrected together; a response through a link is attached to the variant the visitor saw, recalculated by Probant, or stays outside the test. Where to check |
| Script and flicker | The Sidecar script is installed as a synchronous tag (loading through a tag manager causes flicker) and briefly hides the <body> during initialization, which can be turned off. The script size is not published.343536 | Script under 7 KB compressed, loaded asynchronously, that does not hide the page while it loads. Server-side assignment available, for a display with no flicker at all. Where to check |
| Granular permissions | Read-Only, Member, Admin, and Owner roles per project; custom roles with granular permissions are limited to Enterprise contracts.3738 | Four fixed roles, plus custom roles the owner builds from a catalog of named permissions: prepare, launch and stop, roll out, approve a launch, write variant JavaScript, manage flags, metrics, margin, members, and keys. No privilege escalation: billing, roles, transfer, and deletion of the workspace stay with the owner; no one can grant more than their own permissions, and an API key cannot exceed its author’s. Where to check |
| Data hosting | Amplitude’s subprocessor list shows Statsig hosted in the United States only (Google Cloud, Azure, MongoDB); the security page lists Azure and AWS. “EU hosting” appears on the pricing page, with no public details.394041 | Database and compute in Amsterdam, with Scaleway, in the European Union. Scaleway, which hosts the database and compute, is a French company. Published subprocessor list: Scaleway (Amsterdam, Netherlands (nl-ams region), in the European Union), Brevo (European Union: servers in France and Germany, encrypted backups in Belgium (Google Cloud), according to Brevo), Mistral AI (European Union by default (European API endpoint), according to Mistral), ClickHouse (option not used), Stripe (Contract with Stripe Payments Europe, Ltd. (Ireland); data transferred to Stripe, LLC (United States), according to Stripe), Sentry (European Union: Frankfurt (Germany), including backups, according to Sentry). Depending on the feature used, Mistral may temporarily transfer data outside the European Union to the subprocessors it publishes; its data processing agreement limits these transfers to countries the EU recognizes as adequate, or covers them with standard contractual clauses. Stripe transfers payment data to Stripe, LLC, in the United States, under the Data Privacy Framework and, failing that, standard contractual clauses. Whatever region is chosen, Sentry may keep the data of Probant’s Sentry account (members, settings, audit log, project descriptions) in the United States; reported errors stay in Frankfurt. Where to check |
| Consent by default | By default, the web SDK creates a stable identifier in localStorage and sends events; it sets no cookie. Consent mode (logging and storage turned off, then back on) has to be coded by the customer.42434445 | Consent required by default: no cookie, no assignment, and nothing sent before the visitor agrees. Consent given mid-page: the page does not change in front of the visitor; their variant starts on the next page. First-party cookie, lasting 390 days at most, never extended. Where to check |
| Pricing model | Usage-based pricing (metered events). Pro: $150 per month with 5M events included, then $0.05 per thousand, billed the following month. Enterprise on quote, based on event or experiment volume.464748 | Creating a Probant workspace is free; paid plans cost €590 to €1,990 per month depending on the number of tested visitors (2 months free with annual billing), with no overage charges. Agency plan: 1,000,000 tested visitors per month, shared across client workspaces, from 3 workspaces, for €1,490 per month or €14,900 per year. Where to check |
What Statsig also does.
Probant shares these points with Statsig.
A sequential test
The frequentist sequential test (mSPRT, wider intervals before the target date) is turned on in advanced settings; an SPRT mode lets you look without penalty and accept the null hypothesis. Statsig advises against looking at results without these corrections.678
A multiple comparisons correction
Bonferroni correction per metric (with a share of alpha reserved for primary metrics) and the Benjamini-Hochberg procedure per variant and per metric, set in the experiment’s advanced settings.2324
Variance reduction (CUPED)
CUPED is applied automatically to all experiments, with a 7-day window before each user’s exposure. CURE, the regression extension, is documented for Warehouse Native; stratification is also available.282930
Revenue per visitor, with its range
Each metric, including a sum of revenue per user, is reported with lift, confidence interval, and p-value. Average order value is built as a ratio metric.1819
Gradual rollouts
Percentage rollouts and automatic scheduled steps; Safeguards (Pro and Enterprise) can automatically bring a gate back to 0% when a metric alert fires.1415
When to choose Statsig over Probant.
Based on Statsig’s public documentation.
You want switchback tests
Switchback tests (alternating over time by zone), without CUPED or sequential testing for now. Synthetic-control geo tests (based on GeoLift) in Warehouse Native, so on the Enterprise plan.495051
You want native session replay
Native session replay (rrweb library), filterable by experiment group, included in the plans (50,000 replays/month on the free plan). No heatmap is mentioned.525354
You want to start for free
Free Developer plan with no credit card (2 million events per month, unlimited seats). Beyond that, metric impact calculations stop unless you move to Pro.5556
What sets Probant apart.
These points are not in Statsig’s public documentation, accessed November 11, 2026. A missing mention does not prove a feature is missing; if these points matter to you, ask Statsig.
Margin per visitor
Probant measures revenue and margin per visitor, each with its range.
StatsigMargin per visitor is not mentioned in their public documentation (as of November 11, 2026).
Learn moreSRM that blocks the verdict
A traffic split imbalance moves the test to “blocked” status, and no winner is declared.
StatsigSRM check by chi-square test on all experiments and feature gates, shown in health checks as a warning or a failure depending on the p-value. The documentation does not describe blocking the results; it recommends investigating or restarting the experiment.910 Automatic blocking of the test is not mentioned there (as of November 11, 2026).
Consent required by default
No cookie, no assignment, and nothing sent before the visitor agrees.
StatsigBy default, the web SDK creates a stable identifier in localStorage and sends events; it sets no cookie. Consent mode (logging and storage turned off, then back on) has to be coded by the customer.42434445 Consent required by default is not mentioned there (as of November 11, 2026).
A test stopped by its guardrails
A breached guardrail stops a running test on its own and blocks its verdict.
StatsigIn an experiment, guardrail metrics trigger alerts (Rollout / Guardrail Metric Alerts), with no documented automatic stop. Safeguards (Pro and Enterprise plans) automatically roll back, pause, or end a feature gate rollout when an alert fires.111213 An automatic stop of the test on a guardrail is not mentioned there (as of November 11, 2026).
Learn more
Frequently asked questions.
How much does Statsig cost?
Usage-based pricing (metered events). Pro: $150 per month with 5M events included, then $0.05 per thousand, billed the following month. Enterprise on quote, based on event or experiment volume.464748 Creating a Probant workspace is free; paid plans cost €590 to €1,990 per month depending on the number of tested visitors (2 months free with annual billing), with no overage charges.
Where does Statsig host data?
Amplitude’s subprocessor list shows Statsig hosted in the United States only (Google Cloud, Azure, MongoDB); the security page lists Azure and AWS. “EU hosting” appears on the pricing page, with no public details.394041 Probant hosts its database and application in Amsterdam, with Scaleway.
Does Statsig measure margin?
Margin per visitor is not mentioned in their public documentation (as of November 11, 2026). Probant measures revenue and margin per visitor, each with its range, from orders and the cost of each product.
Can you look at Statsig results while a test is running?
The frequentist sequential test (mSPRT, wider intervals before the target date) is turned on in advanced settings; an SPRT mode lets you look without penalty and accept the null hypothesis. Statsig advises against looking at results without these corrections.678 Probant sets its checkpoints before launch (O’Brien-Fleming) and uses always-valid ranges during rollouts.
Sources.
Official pages from Statsig, accessed November 11, 2026. Quotes are reproduced in their original language.
- Source 1: docs.statsig.com/welcome
Statsig is a unified platform for feature flags, A/B testing, and product analytics.
- Source 2: www.statsig.com/blog/openai-acquisition
Today, I am excited to share that we’ve signed a definitive agreement for Statsig to join OpenAI.
- Source 3: www.statsig.com/blog/openai-acquisition
Note from Statsig: Since publishing this historic blog, Statsig was acquired by Amplitude on May 5, 2026. While the original team stayed at OpenAI, the product, brand, and customer base are now part of the Amplitude family.
- Source 4: docs.statsig.com/experiments/advanced-setup/bayesian
Experiments use frequentist analysis by default.
- Source 5: docs.statsig.com/experiments/advanced-setup/sequential-testing
Statsig offers three analysis methods, which you choose with the Analytics Type setting under Advanced Settings on the experiment Setup tab.
- Source 6: docs.statsig.com/experiments/advanced-setup/sequential-testing
Statsig uses mSPRT based on the approach that Zhao et al. propose.
- Source 7: docs.statsig.com/experiments/advanced-setup/sequential-testing
On the Setup tab of your experiment, expand Advanced Settings, set Analytics Type to Frequentist, then select Apply Sequential Testing under Frequentist Mode Settings.
- Source 8: docs.statsig.com/experiments/advanced-setup/sprt
You can read results at any time with no peeking penalty, and you can accept the null hypothesis as well as the alternative.
- Source 9: docs.statsig.com/stats-engine/methodologies/srm-checks
Statsig uses a Chi-squared test to identify whether the split of users between groups indicates a sample ratio mismatch.
- Source 10: docs.statsig.com/experiments/monitoring/srm
When Statsig detects SRM, it sets the experiment's health check to a warning or failure state, depending on how extreme the SRM is.
- Source 11: docs.statsig.com/metrics/rollout-alerts
Alerts trigger only if the cumulative metric delta is statistically significantly lower or higher than your threshold, which reduces alert noise.
- Source 12: docs.statsig.com/feature-flags/safeguards-overview
When any alert fires due to metric regressions, Safeguards automatically pause your rollout, roll it back, or finish a rollout based on your settings.
- Source 13: docs.statsig.com/feature-flags/safeguards-overview
Statsig offers Safeguards on the Pro and Enterprise billing tiers.
- Source 14: docs.statsig.com/feature-flags/scheduled-rollouts
Scheduled Rollouts add a time-based scheduling layer to Feature Gates.
- Source 15: docs.statsig.com/feature-flags/safeguards-create
A Safeguard monitors a Feature Gate's targeting rule for regressions and automatically takes an action, such as rolling back to 0%, when an alert fires.
- Source 16: docs.statsig.com/autotune/overview
Autotune and Autotune AI are Statsig's Multi-Armed Bandit solutions.
- Source 18: docs.statsig.com/experiments/statistical-methods/introduction
It turns exposure and metric data into a metric delta, confidence interval, and p-value for each metric, applies variance reduction, and supports frequentist, SPRT, and Bayesian analysis methods.
- Source 20: docs.statsig.com/experiments/exploring-results/aggregated-impact
Statsig sums those values, adjusts for false positive risk (the "winner's curse"), and displays the result as the Aggregated Impact Estimate.
- Source 21: docs.statsig.com/experiments/advanced-setup/bayesian
Bayesian mode reports each metric as a chance to beat control and an expected loss instead of a p-value, and can combine an informative prior with the observed data.
- Source 22: docs.statsig.com/experiments/advanced-setup/bayesian
You can't change the analytics type after the experiment starts.
- Source 23: docs.statsig.com/experiments/setup/analysis-settings
Apply Bonferroni Correction Per Metric: Divides α across the metrics in the scorecard.
- Source 24: docs.statsig.com/experiments/setup/analysis-settings
Apply Benjamini-Hochberg Procedure Per Variant and Per Metric: Control the false discovery rate instead of the family-wise error rate.
- Source 27: docs.statsig.com/experiments/exploring-results/interaction-detection
Interaction Detection measures whether two overlapping experiments change each other's effect on a metric.
- Source 28: docs.statsig.com/experiments/statistical-methods/variance-reduction
Statsig applies CUPED to every experiment automatically, applies winsorization to count and sum metrics, and offers stratified sampling when you set up an experiment.
- Source 29: docs.statsig.com/experiments/statistical-methods/methodologies/cuped
Pre-exposure data is the same metric measured over the 7 days before the user's exposure.
- Source 30: docs.statsig.com/statsig-warehouse-native/cure/introduction
CURE (variance Control Using Regression Estimates) is Statsig's extended implementation of CUPED.
- Source 31: docs.statsig.com/guides/sidecar-experiments/sidecar-v3
Visual Editor v3 is a no-code Chrome extension for building and running A/B tests directly on any website.
- Source 32: docs.statsig.com/guides/sidecar-experiments/sidecar-v3
Sidecar is in maintenance mode. Existing Sidecar setups continue to work, but Statsig doesn't release bug fixes or new features for Sidecar.
- Source 33: docs.statsig.com/guides/sidecar-experiments/sidecar-v3
For visual, low-code experimentation, use Amplitude Web Experiment.
- Source 34: docs.statsig.com/guides/sidecar-experiments/sidecar-v3
Sidecar still runs through a JavaScript script tag, though the package name has changed from earlier versions.
- Source 35: docs.statsig.com/guides/sidecar-experiments/setup
Install Sidecar as a synchronous script tag instead.
- Source 36: docs.statsig.com/guides/sidecar-experiments/sidecar-v3
&reduceflicker=0 disables the brief hiding of the <body> tag while the client initializes.
- Source 37: docs.statsig.com/access-management/projects
Each person invited to a project receives a role that specifies their level of access.
- Source 38: docs.statsig.com/access-management/projects
Custom roles are for Enterprise contracts only.
- Source 39: amplitude.com/subprocessor-list
Cloud hosting and infrastructure provider for the Statsig-branded Amplitude Services (US only)
- Source 40: www.statsig.com/trust/security
Statsig uses industry leading cloud platforms (Azure and Amazon Web Services) to host its production services.
- Source 42: docs.statsig.com/client/javascript-sdk
On first initialization the SDK generates a Stable ID and stores it in localStorage under statsig.stable_id.<SDK_KEY_HASH>.
- Source 43: docs.statsig.com/client/javascript-sdk
browser-only (default): log events from browser environments.
- Source 44: docs.statsig.com/client/javascript-sdk
Start with logging disabled and storage blocked, then enable them after consent.
- Source 47: www.statsig.com/pricing
On Statsig’s Pro tier, you will be charged a baseline fee of $150 per month at the start of each billing period.
- Source 49: docs.statsig.com/experiments/types/switchback-tests
A switchback test switches an entire population back and forth between test and control treatments on a set cadence, instead of splitting the population into two fixed groups.
- Source 50: docs.statsig.com/experiments/types/switchback-tests
Switchback tests don't yet support CUPED and sequential testing.
- Source 51: docs.statsig.com/statsig-warehouse-native/geotests/introduction
Run a geotest when you can't run an A/B test at a per-user level.
- Source 52: docs.statsig.com/session-replay/overview
Session Replay records users interacting with your website or product and lets you play back those sessions.
- Source 54: www.statsig.com/pricing
Watch users who are in specific treatment groups of an experiment or feature gate
- Source 55: www.statsig.com/pricing
When you first sign up for Statsig, you automatically start on Statsig’s Developer Tier, which gives you free access to gates, configs, experimentation and analytics.
- Source 56: www.statsig.com/pricing
If you want to stay on the Developer tier, gate and config targeting will continue to work, but Metric Lift computations will stop.
Migrate from Statsig
Your Statsig history can be imported as a generic CSV (visitors and conversions per variant) and is reread by Probant’s engine: every verdict is recalculated from visitors and conversions.
- Open demo, no sign-up
- Free to start, no overage charges
- Database and compute in Amsterdam, with Scaleway