Method

Sizing and reading an A/B test

The output: how many visitors a two-variant test needs, how long it runs, and whether a finished test shows a difference.

Inputs

The baseline conversion rate in percent, 0.1 to 99.
The smallest relative lift worth detecting in percent, 1 to 500: 10 means 3.0 % becomes 3.3 %.
Power, 80 or 90 %, and confidence, 95 or 99 %.
Visitors a day across all variants.
For a finished test: the visitors and conversions of A and B as whole numbers.

Set the target rate

The target is the baseline raised by the lift. A target of 100 % or more is rejected.
Size one variant

Visitors per variant for a two-sided two-proportion z-test with an equal split and no continuity correction: the confidence z times the pooled spread plus the power z times the per-variant spread, squared, divided by the squared difference of the rates, rounded up.
Turn visitors into time

The total is the per-variant size times the variants; the days are the total over the daily visitors; the run is whole weeks, at least two, so that every weekday counts as often.
Read a finished test

The pooled z-test gives the two-sided p-value; the unpooled standard error gives the 95 % interval for the difference in percentage points.
State the verdict

A p-value of 0.05 or more is unclear. Below it, B is better when its rate is higher and worse when it is lower.

Visitors per variant and in total.
Days of traffic and whole weeks to run.
The rates of A and B, the difference in percentage points and the relative change.
z, the p-value and the 95 % interval.
The verdict: better, worse or unclear.

Reading the test before the planned sample is reached: the p-value crosses 0.05 along the way by chance.
An unequal split between the variants: the sizing assumes equal traffic.
A baseline that converts no one: the relative change cannot be computed.
Traffic that changes during the test, such as a campaign or a holiday: the two-week minimum softens this, it does not remove it.
Several metrics or variants read against one threshold: the chance of a false finding grows with every reading.

The method is the A/B test calculator's code; the example on that page is computed with it at build time.