Statistics and verdicts

@splithub/stats is a zero-dependency package that turns per-variant counts into a verdict. The same code runs in the CLI, in the self-hosted handler, in the cloud, and in the emails, so every surface quotes the same numbers.

What is computed#

For each variant: unique exposed visitors, converted visitors, rate, revenue (when the goal has it). For each non-control variant against the control:

Field	Meaning
absoluteLift	rate − control rate, as a fraction; multiply by 100 for percentage points
relativeLift	(rate − control rate) / control rate
zScore, pValue	two-proportion z-test, two-sided
significant	p < α after Bonferroni correction for the number of comparisons
ci	95% confidence interval of the absolute difference
probabilityToBeat	probability that the variant's true rate is above the control's (normal approximation)

Plus the sample-ratio-mismatch check: a chi-square test of the observed split against the configured weights, flagged when p < 0.001.

The verdict#

status	When	What to do
no_data	no exposures yet	wait, or check the setup with splithub doctor
insufficient_sample	fewer than 100 visitors in a variant or fewer than 30 conversions in total (defaults)	keep running; the message estimates the total visitors needed per variant
srm_detected	the split does not match the weights	the setup is broken (bots, caching, a variant that errors, a redirect loop); fix it, restart as a new test
winner_found	a variant beats the control, or the control beats a significantly worse variant	read winner and message, then finish the test
no_significant_difference	enough sample, no significant difference	ship the simpler variant, finish

message is one sentence written for a person or an agent to act on, for example: Stop: "bold" beats "control" (+18.3%, p = 0.012). Ship "bold". Agents are told in the skills to quote it verbatim and never to call a winner from raw rates.

Sample size#

Rule of thumb for 95% confidence and 80% power: visitors per variant ≈ 16 · p(1−p) / (p·L)² with baseline rate p and relative lift L. Baseline 5% and lift 20% → about 7,600 per variant; baseline 2% and lift 10% → about 78,000. The insufficient_sample message uses the observed rates to estimate a total per variant, not the number still missing.

Peeking#

Checking every day and stopping at the first significant moment inflates false positives several times over. Decide the sample or the duration before starting, run whole weeks, and treat an early "winner" with a confidence interval touching zero as weak. The cloud's AI reviewer and the alert rules apply the same caution: the winner rule waits for the verdict's own minimum sample.

Several variants#

With three or more variants every comparison is against the control and the significance threshold is divided by the number of comparisons (Bonferroni). Pairwise comparisons between non-control variants are not reported; create a new test with the two finalists if you need that.

Segments#

Breakdowns by country, device, browser and host are for hypotheses, not verdicts: slicing after the fact finds noise. If a segment looks interesting, run a targeted experiment on it.

Using the package directly#

import { computeVerdict } from '@splithub/stats';

const r = computeVerdict('hero-title', [
    { variant: 'control', exposures: 4107, conversions: 251 },
    { variant: 'bold', exposures: 4248, conversions: 301 },
  ], { expectedWeights: [50, 50] });
r.status; r.message; r.comparisons[0].pValue;