Alerts, digest, AI reviews

Three ways to be told instead of checking. All three live on the Alerts tab and the AI tab of a cloud project and need Pro.

Alert rules#

A rule is a scope (one experiment or every running one), a condition with thresholds, and actions. Rules are evaluated every hour and fire at most once per experiment per cooldown window.

Condition	Fires when
srm	the split differs from the configured weights (p < 0.001)
winner	the verdict is winner_found (optionally only once every variant has N visitors)
loser	a variant is at least X% worse than the control with enough sample
exposures	unique visitors on the experiment reach N
conversions	conversions reach N
duration	the experiment has run N days
no_events	no events for N hours while running
quota	the project used X% of its monthly exposures

Actions: send to channels (email addresses or webhooks), pause the experiment, finish it, drop the losing variant (set its weight to 0). A rule can combine them: "SRM → pause and email", "winner → finish and email", "loser after 10k visitors → drop the loser".

Every project starts with sensible rules created disabled; enabling one needs Pro. Test runs the rule as a dry run and lists the experiments it would hit right now.

Channels#

Saved once, reused by rules and the digest: an email address, a webhook URL (JSON POST with the rule, the experiment, the message and the actions taken), more to come. Add them on the Alerts tab; pick them in a rule's "deliver to" multiselect.

The digest#

A scheduled summary, separate from the rules: every day at a chosen hour (UTC), on chosen weekdays, weekly or monthly. It covers running experiments with their verdicts, finished ones since the last digest, quota usage, and what to decide next. Sent to the chosen channels; each member can also switch it off in their profile.

AI reviews#

An analyst's second opinion written by a language model from the experiment setup, aggregate results and available variant screenshots. Raw event rows and visitor ids are not supplied as statistical input:

• On start: a setup review, including available variant screenshots.
• On finish: when an experiment finishes (dashboard, CLI, API, cap, rule), a review is written and attached to the finish email.
• Progress checks: the daily job checks running experiments with at least 100 visitors. It writes a review when the verdict changes from the previous snapshot or a week has passed since the last review, subject to the automatic review cooldown.
• On demand: the AI tab's "Review now".

Reviews are stored with the experiment and readable from the dashboard, npx splithub review [<id>], the MCP tool splithub_review, and GET /v1/reviews[/<id>] with an API key. They are advice; the verdict from the statistics engine stays the source of truth, and the skills tell agents exactly that.

Webhook payload#

{
  "kind": "srm",
  "project": "project-id",
  "subject": "[SplitHub] My app: Broken split (SRM)",
  "text": "Alert summary and dashboard link",
  "rule": "Broken split (SRM)",
  "experiment": "checkout-steps",
  "message": "Sample ratio mismatch in \"checkout-steps\" (p=2.1e-4). The split is not what the config says.",
  "actions": ["paused"],
  "link": "https://app.splithub.io/p/project-id/e/checkout-steps"
}