A/B test significance calculator

What does statistical significance actually mean?

When your variant converts at 6.5 percent and your control at 5 percent, two explanations are possible: the variant is genuinely better, or you got lucky. Statistical significance tells you how plausible the luck explanation is. This calculator runs a two-proportion z-test: it pools both samples, computes how many standard errors separate the two conversion rates, and converts that distance into a p-value.

The p-value is the probability of observing a difference at least this large if the two versions were actually identical. The standard convention is to require p below 0.05 — meaning that if there were no real difference, you would see a gap like this less than 5 percent of the time. That is the 95 percent confidence threshold this calculator uses, the same one running inside Growth Pilot's A/B testing engine.

Three traps to avoid. First, small samples: with fewer than 100 visitors per variant, even large gaps are usually noise, which is why the verdict stays cautious below that floor. Second, peeking: if you check the test every day and stop the moment p dips under 0.05, your real false-positive rate is far higher than 5 percent. Decide your sample size up front and hold the line. Third, significance is not importance: with enough traffic, a 0.1 percent uplift can be significant yet not worth shipping. Always read the uplift and the p-value together.

A significant result is not a guarantee — it is a calibrated bet. Over many experiments, the discipline of only shipping significant winners is what compounds into real conversion gains.

Related reading: A/B testing and statistical significance in the glossary.