How to interpret your A/B test result
The calculator uses a two-proportion Z-test with a two-tailed hypothesis. It measures
how many standard errors the difference between the conversion rates is from zero and, from that, computes the
p-value. The decision combines the p-value with the statistical power:
- Low p-value + high power: strong evidence of a difference. The winning version can be rolled out with confidence.
- High p-value + high power: no significant difference. There is no benefit in rolling out the variant.
- High p-value + low power: there may be no difference, but there is not enough data to confirm it.
- Low p-value + low power: there may be a difference, but uncertainty remains. Collect more data.
Good practices: split traffic randomly, run the test for at least 14 days to cover weekly seasonality, and choose
the primary metric before you start.
Formulas
- Conversion rate
CR = conversions ÷ visitors- Uplift
uplift = CRB ÷ CRA − 1- Standard error of each version
SE = √( CR × (1 − CR) ÷ visitors )- Pooled standard error
SEpool = √( p × (1 − p) × (1/nA + 1/nB) ), where p is the conversion rate of both versions combined- Z statistic and p-value
Z = (CRB − CRA) ÷ SEpool and p-value = 2 × P(N(0,1) > |Z|)- Confidence interval
CR ± Zcritical × SE (critical Z = 1.96 for 95%)- Observed statistical power
- Probability that the Z statistic exceeds the critical Z, in either direction, assuming the true difference equals the observed one.
A/B testing FAQ
What is an A/B test?
It is an experiment in which users are randomly split into two groups: one sees the current version (A, control)
and the other sees a change (B, variant). At the end, a statistical analysis shows whether the conversion difference
between the versions is real or due to chance.
What does the p-value mean in an A/B test?
It is the probability of observing a difference as large as the one obtained (or larger) if both versions actually
had the same conversion rate. A p-value below the significance level (0.05 for 95% confidence) indicates a
statistically significant difference.
What is statistical power and why does it matter?
It is the probability that the test detects a difference when one really exists. The usual benchmark is 80% or
higher. With low power, a non-significant result may be a false negative, and the recommendation is to collect more data.
How long should I run an A/B test?
At least 14 days, to cover two full weeks and reduce the effect of weekly seasonality. Low-traffic tests may need
longer to reach the desired statistical power.
Which metric should be the primary one in an A/B test?
The event closest to the change being tested, such as a button click, add to cart or form submission. Purchases are
usually tracked as a secondary metric, because they sit at the end of the funnel and are influenced by many other
factors, which makes significance harder to reach.
Is this calculator one-tailed or two-tailed?
Two-tailed. The alternative hypothesis is that the versions differ, either better or worse, and the same criterion
is used for the p-value, the critical Z, the confidence intervals and the statistical power.
Methodology based on the notebook
AB Test Calculator with Examples
on Kaggle (in Portuguese).