Automations · every morning

What Probant checks every morning

Automations you turn on reread your data every morning and notify your alert destinations (Slack, signed webhook, or email to chosen members). When one judges a drop, a traffic split, a guardrail, or a checkpoint, the statistical engine makes the call, so ordinary noise doesn’t trigger alerts.

During a test

Post-launch checkon

One day after each launch: exposures received, each variant served, each change displayed (script display reports), traffic split matching the plan, conversions coming in.

No settings: it only reports
Engine decision: traffic split (threshold 0.001)

How Probant decides

The same checks as the test report (chi-square test on the split, at the same strict threshold), run without waiting for someone to open it. One message per test, whether it flags a problem or everything is in place.

“Post-launch check” alert: sent to 2 destinations.

Last run Nov 11 at 06:17 AM · 1 alert sent

Everything is in place for the tests launched the day before.

“Simplified mobile menu”: everything is in place after launch6,792 exposed visitors. Original: 3,412 · Simplified menu: 3,380. Traffic split matches the plan. 475 exposed visitors sent “Commande”.Open

Checkpoint reachedon

Every morning, running tests that reached a checkpoint in their test plan (or their sample size): is the boundary crossed, or should the test continue?

No settings: it only reports
Engine decision: test plan checkpoint boundaries

How Probant decides

Checkpoints come from the test plan locked at launch (O’Brien-Fleming boundaries, triggered by cumulative visitors). Reading the test at those moments, and only then, keeps the stated error risk. One message per checkpoint.

“Checkpoint reached” alert: sent to 2 destinations.

Last run Nov 11 at 06:17 AM · 2 alerts sent

Checkpoints reached; none allows a conclusion yet.

“Simplified mobile menu”: checkpoint 2 of 3 reached, keep goingOn day 9, the lift isn’t clear enough to conclude yet. Let the test run until the next checkpoint, at 69,550 visitors (48,150 today).Open
“Pay-in-3 option highlighted”: checkpoint 1 of 3 reached, keep goingOn day 6, the lift isn’t clear enough to conclude yet. Let the test run until the next checkpoint, at 57,860 visitors (31,560 today).Open

Sample ratio mismatch in a segmenton

Every morning, for each running test: the visitor split between variants, by device, by new versus returning visitors, and by traffic source.

No settings: it only reports
Engine decision: split per segment, corrected for multiple comparisons

How Probant decides

A chi-square test per segment, then the Holm correction across all segments read: a segment is flagged only if the gap holds after correction, at the strict 0.001 threshold. A bot, a redirect, or a browser that breaks a variant often shows up here first.

“Sample ratio mismatch in a segment” alert: sent to 1 destination.

Last run Nov 11 at 06:17 AM

Traffic split as expected in every segment checked.

“Simplified mobile menu”: traffic split matches in every segment5 segments checked (device, new or returning), no mismatch.Open

Overdue testoff

Every morning, running tests that passed the duration in their test plan without reaching their sample size.

No settings: it only reports
Counts, without judging

How Probant decides

The planned end date and progress come from the test report. An overdue test ties up a page area without giving an answer: extend it knowingly, or stop it.

“Overdue test” alert: sent to 1 destination.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

On the site

Conversion drop alerton

Every morning, yesterday’s conversion rate on your rate metrics and the number of tracked visitors, compared with the previous 7 days.

Adjustable: sensitivity, watched metrics, traffic
Engine decision: drop beyond the usual variation

How Probant decides

An alert goes out only if the drop is larger than the usual day-to-day variation explains: the spread of the baseline days widens the margin (quasi-binomial model, and quasi-Poisson for traffic). The false-alarm risk you choose is shared across the watched series.

SensitivityBalancedrecommended setting

Watched rate metrics

Also watch the number of tracked visitors (script removed, broken pages)

“Conversion drop” alert: sent to 2 destinations.

Last run Nov 11 at 06:17 AM · 1 alert sent

1 drop beyond normal day-to-day variation.

Tracked visitors: nothing unusual9,060 on Nov 10, versus 9,474 expected from the previous 7 days (-4%).Open
Drop in “Add to cart” on Nov 10: -29%7.7% on Nov 10, versus 10.9% expected from the previous 7 days. That’s more than the usual variation. Check the journey (checkout, form, stock) and that the event is still sent.Open
Commande: nothing unusual7.1% on Nov 10, versus 7.4% expected from the previous 7 days (-4%).Open

New open-ended answersoff

Every morning, the open-ended answers your test surveys received in the last 24 hours, test by test.

No settings: it only reports
Counts, without judging

How Probant decides

A count, with no answer text in the alert: the answers, redacted on receipt, are read in Probant, where they become observations.

“New open-ended answers” alert: sent to 1 destination.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

After the decision

Rollout guardrailson

Every morning, the guardrails of each running rollout, read at the current step against visitors kept on the original.

No settings: it only reports
Engine decision: always-valid guardrail ranges

How Probant decides

The same reading as the Rollouts screen (always-valid ranges you can check every day without raising the risk). One message when all guardrails are shown to be within their limits (the next step is safe), another when one leans the wrong way without being crossed yet. A crossed guardrail rolls the rollout back to the original without waiting for the morning.

“Rollout guardrails” alert: sent to 1 destination.

Last run Nov 11 at 06:17 AM

1 rollout reviewed.

“Free shipping from €60” (€60 threshold): guardrails still being read2 of 3 shown to be within their limit; the others don’t have enough visitors to decide yet. Based on 41,547 exposed visitors and 41,491 kept on the original, at the 50% step.Open

Flags to removeon

Once a week, on the day you choose: the feature flags your code can drop, and what to keep before archiving.

Adjustable: sending day
Counts, without judging

How Probant decides

The rules of the Feature flags screen: same value served to everyone for 30 days, turned off for 30 days, linked test ended, not evaluated for 30 days, or still in draft.

Sending day

“Flags to remove” alert: sent to 1 destination.

Last run Nov 11 at 06:17 AM · 1 alert sent

3 feature flags to remove from the code.

“Free-shipping threshold” (seuil-livraison-offerte)Has served New threshold (60) to everyone for 43 days. Keep New threshold (60) in the code, then archive the feature flag.Open
“Recommendations on the product page” (recommandations-fiche)Follows the A/B test ““You may also like” recommendations,” which ended. Every visitor now gets Off. Keep the value the A/B test’s verdict chose in the code, then archive the feature flag.Open
“Old search engine” (ancien-moteur-recherche)Off for 69 days: Off is served everywhere. Remove the code path the feature flag protected, then archive it.Open

Account and reporting

Quota almost reachedoff

Every morning, this month’s tested visitors compared with your plan.

Adjustable: share of the plan
Counts, without judging

How Probant decides

A count, not a projection: one alert per month when the chosen share of the plan is passed, with the reset date. Past the quota, running tests continue but no new test can start.

Alert above80% of the plan

“Quota almost reached” alert: sent to 1 destination.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

Weekly reportoff

Once a week, on the day you choose: running tests with their status and progress, and the decisions of the past 7 days.

Adjustable: sending day
Counts, without judging

How Probant decides

Each status is the one on the test page (“Conclusive.”, “Not conclusive yet…”); progress is the share of the planned sample size reached.

Sending day

“Weekly report” alert: sent to 2 destinations.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

Always on

These checks run without setup as events come in, and notify any destination subscribed to them (Settings → Integrations).

Decision readyAs soon as the test plan allows a conclusion: sample size reached or a checkpoint boundary crossed.Sent to 2 destinations.
Sample ratio mismatchEach time a running test is reread, at most every 15 minutes.Sent to 2 destinations.
Guardrails after the decisionEach rollout is checked every 5 minutes and goes back to the original on its own if a guardrail is crossed.Sent to 1 destination.
Change not showingEach change in a running test, checked at most once an hour: missing from day one, or no longer applied.Sent to 1 destination.
Effect fadingA lift that weakens after the first days: a novelty effect may be inflating the result.Sent to 1 destination.
Test interactionTwo overlapping tests whose effects depend on each other.Sent to 1 destination.
Variant at its floorIn an optimization test, a variant that only receives its minimum share.Sent to 1 destination.
Feature flag turned offAn emergency shutoff, as soon as it happens.Sent to 1 destination.
Quota reachedThis month’s tested visitors reach the plan quota.Sent to 2 destinations.

During a test

Post-launch checkon

One day after each launch: exposures received, each variant served, each change displayed (script display reports), traffic split matching the plan, conversions coming in.

No settings: it only reports
Engine decision: traffic split (threshold 0.001)

How Probant decides

“Post-launch check” alert: sent to 2 destinations.

Last run Nov 11 at 06:17 AM · 1 alert sent

Everything is in place for the tests launched the day before.

“Simplified mobile menu”: everything is in place after launch6,792 exposed visitors. Original: 3,412 · Simplified menu: 3,380. Traffic split matches the plan. 475 exposed visitors sent “Commande”.Open

Checkpoint reachedon

Every morning, running tests that reached a checkpoint in their test plan (or their sample size): is the boundary crossed, or should the test continue?

No settings: it only reports
Engine decision: test plan checkpoint boundaries

How Probant decides

“Checkpoint reached” alert: sent to 2 destinations.

Last run Nov 11 at 06:17 AM · 2 alerts sent

Checkpoints reached; none allows a conclusion yet.

“Simplified mobile menu”: checkpoint 2 of 3 reached, keep goingOn day 9, the lift isn’t clear enough to conclude yet. Let the test run until the next checkpoint, at 69,550 visitors (48,150 today).Open
“Pay-in-3 option highlighted”: checkpoint 1 of 3 reached, keep goingOn day 6, the lift isn’t clear enough to conclude yet. Let the test run until the next checkpoint, at 57,860 visitors (31,560 today).Open

Sample ratio mismatch in a segmenton

Every morning, for each running test: the visitor split between variants, by device, by new versus returning visitors, and by traffic source.

No settings: it only reports
Engine decision: split per segment, corrected for multiple comparisons

How Probant decides

“Sample ratio mismatch in a segment” alert: sent to 1 destination.

Last run Nov 11 at 06:17 AM

Traffic split as expected in every segment checked.

“Simplified mobile menu”: traffic split matches in every segment5 segments checked (device, new or returning), no mismatch.Open

Overdue testoff

Every morning, running tests that passed the duration in their test plan without reaching their sample size.

No settings: it only reports
Counts, without judging

How Probant decides

The planned end date and progress come from the test report. An overdue test ties up a page area without giving an answer: extend it knowingly, or stop it.

“Overdue test” alert: sent to 1 destination.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

On the site

Conversion drop alerton

Every morning, yesterday’s conversion rate on your rate metrics and the number of tracked visitors, compared with the previous 7 days.

Adjustable: sensitivity, watched metrics, traffic
Engine decision: drop beyond the usual variation

How Probant decides

SensitivityBalancedrecommended setting

Watched rate metrics

Also watch the number of tracked visitors (script removed, broken pages)

“Conversion drop” alert: sent to 2 destinations.

Last run Nov 11 at 06:17 AM · 1 alert sent

1 drop beyond normal day-to-day variation.

Tracked visitors: nothing unusual9,060 on Nov 10, versus 9,474 expected from the previous 7 days (-4%).Open
Drop in “Add to cart” on Nov 10: -29%7.7% on Nov 10, versus 10.9% expected from the previous 7 days. That’s more than the usual variation. Check the journey (checkout, form, stock) and that the event is still sent.Open
Commande: nothing unusual7.1% on Nov 10, versus 7.4% expected from the previous 7 days (-4%).Open

New open-ended answersoff

Every morning, the open-ended answers your test surveys received in the last 24 hours, test by test.

No settings: it only reports
Counts, without judging

How Probant decides

A count, with no answer text in the alert: the answers, redacted on receipt, are read in Probant, where they become observations.

“New open-ended answers” alert: sent to 1 destination.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

After the decision

Rollout guardrailson

Every morning, the guardrails of each running rollout, read at the current step against visitors kept on the original.

No settings: it only reports
Engine decision: always-valid guardrail ranges

How Probant decides

“Rollout guardrails” alert: sent to 1 destination.

Last run Nov 11 at 06:17 AM

1 rollout reviewed.

“Free shipping from €60” (€60 threshold): guardrails still being read2 of 3 shown to be within their limit; the others don’t have enough visitors to decide yet. Based on 41,547 exposed visitors and 41,491 kept on the original, at the 50% step.Open

Flags to removeon

Once a week, on the day you choose: the feature flags your code can drop, and what to keep before archiving.

Adjustable: sending day
Counts, without judging

How Probant decides

The rules of the Feature flags screen: same value served to everyone for 30 days, turned off for 30 days, linked test ended, not evaluated for 30 days, or still in draft.

Sending day

“Flags to remove” alert: sent to 1 destination.

Last run Nov 11 at 06:17 AM · 1 alert sent

3 feature flags to remove from the code.

“Free-shipping threshold” (seuil-livraison-offerte)Has served New threshold (60) to everyone for 43 days. Keep New threshold (60) in the code, then archive the feature flag.Open
“Recommendations on the product page” (recommandations-fiche)Follows the A/B test ““You may also like” recommendations,” which ended. Every visitor now gets Off. Keep the value the A/B test’s verdict chose in the code, then archive the feature flag.Open
“Old search engine” (ancien-moteur-recherche)Off for 69 days: Off is served everywhere. Remove the code path the feature flag protected, then archive it.Open

Account and reporting

Quota almost reachedoff

Every morning, this month’s tested visitors compared with your plan.

Adjustable: share of the plan
Counts, without judging

How Probant decides

A count, not a projection: one alert per month when the chosen share of the plan is passed, with the reset date. Past the quota, running tests continue but no new test can start.

Alert above80% of the plan

“Quota almost reached” alert: sent to 1 destination.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

Weekly reportoff

Once a week, on the day you choose: running tests with their status and progress, and the decisions of the past 7 days.

Adjustable: sending day
Counts, without judging

How Probant decides

Each status is the one on the test page (“Conclusive.”, “Not conclusive yet…”); progress is the share of the planned sample size reached.

Sending day

“Weekly report” alert: sent to 2 destinations.

Not run yet

Turn it on to run every morning, or run it once to see what it finds.

Manage

Manage

What Probant checks every morning

During a test

Post-launch checkon

Checkpoint reachedon

Sample ratio mismatch in a segmenton

Overdue testoff

On the site

Conversion drop alerton

New open-ended answersoff

After the decision

Rollout guardrailson

Flags to removeon

Account and reporting

Quota almost reachedoff

Weekly reportoff

Always on

What Probant checks every morning

During a test

Post-launch checkon

Checkpoint reachedon

Sample ratio mismatch in a segmenton

Overdue testoff

On the site

Conversion drop alerton

New open-ended answersoff

After the decision

Rollout guardrailson

Flags to removeon

Account and reporting

Quota almost reachedoff

Weekly reportoff

Always on

Manage