Sign In

Can You Test This Yet?

How many users an A/B test needs, how long that takes at your traffic, and what to do when the answer is too long

A small lift needs a lot of users before a test can see it. Most new products and small teams do not have that traffic yet. That is normal. It is better to know before you build the experiment than after a month of waiting for a result that never comes.

Enter your numbers below. The calculator uses the usual settings: a 5% chance of a false positive and an 80% chance of seeing the lift if it is real. Users are split evenly between two groups, the current version and the new one.

Your numbers

A conversion rate

Signups, purchases, clicks: each user either does it or not.

From 10% to 11% is a lift of 1 point.

An average per user

Revenue per user, orders per user, minutes per user.

How much users differ from each other. Take it from past data; for revenue it is often larger than the average.

The answer

At 10,500 users a week, you need 3 weeks to see a lift from 10% to 11%.

Users per group
14,748
Users in total
29,496
Weeks to run
3

The same traffic for smaller and larger lifts. Halving the lift needs about four times the users.

Lift Per group Weeks
0.5 points 57,761 12
1 point 14,748 3
2 points 3,838 1
4 points 1,032 1

Weeks are rounded up to whole weeks, so that every day of the week counts equally often. They count the time for users to enter the test; if your metric has an attribution window, such as purchases within 7 days, add that window after the last week. Decide the length before you start and read the result once, at the end.

What to do instead

When the answer is months, the test is not the right tool for this change yet. You still have options:

  • Test a bigger change. A new onboarding flow has a better chance of a large lift than new button text, and twice the lift needs about a quarter of the users.
  • Measure closer to the change. If you change the checkout form, measure checkout completion, not monthly revenue. A metric right next to the change carries less noise from everything else users do, so the same effect is easier to see.
  • Run longer. If the answer is 10 weeks and you can wait, wait. Fix the length now and read the result once at the end.
  • Do not A/B test it. Ship the change behind a feature flag, watch your numbers and the error rate, and turn it off if something breaks. You will not learn the size of the effect, but you will not pretend to either.

The formula

The users needed in each group are

n = 7.849 (σ_C² + σ_T²) / δ²

where δ is the lift and σC2 and σT2 are the variances per user in the current and the new version. For a conversion rate p the variance is p(1 − p). For an average per user, the calculator uses your standard deviation for both versions.

The calculator opens with the worked example from the book: a checkout that converts 10%, and a new one that converts 11%. The variances are 0.1 × 0.9 = 0.09 and 0.11 × 0.89 = 0.0979, so n = 7.849 × 0.1879 / 0.01² = 14,748 users per group. If 3,000 users reach the checkout each day and half of them enter the test, that is 10,500 a week: each group gains 5,250 users a week, and 14,748 / 5,250 = 2.8 rounds up to 3 weeks, the book's 21 days.

The derivation is in the "Power and sample size" section of The Statistics of A/B Tests: The ABMeter Book (PDF). How ABMeter reads the result once the test ends is on How ABMeter Evaluates Experiments.