A space is a pool of users: how ABMeter runs experiments side by side

Say you want to test a new checkout button and a new search box this month. Can both tests see all of your users at the same time? In ABMeter that depends on one choice: whether the two experiments are in one space or in two.

One space or two

A space is all of your users, 100% of them, as one pool. Each experiment in a space takes a share of the pool, and the shares never overlap. A user is in at most one experiment per space.

Put the checkout test and the search test in one space, and they split the pool. Say each takes 50%. Half of your users can see the new checkout, the other half can see the new search, and nobody sees both. Each test gets half of the traffic, so each takes about twice as long to reach a result.

Put them in two spaces, checkout and search, and each test can take 100% of its own pool. Every user is in both tests at once: in one group of the checkout test and in one group of the search test. The group a user gets in one test says nothing about the group they get in the other. Both tests run at full speed.

When two tests belong in one space

A space separates traffic. Causes are yours to separate.

Two experiments in different spaces can still change each other’s results when their effects go through the same thing. A test that adds free shipping at checkout and a test that shows prices in search results both work on how a user feels about price. A user in both tests may react to the pair, not to either one alone. Only you can tell whether two changes work on the same thing. Before you start a test, look at what runs in every space, not only in yours, and ask whether the two effects could meet.

If they could, put both tests in one space. They run slower, and you get two results you can trust.

Why a parameter’s name is global

Each parameter is created in one space. Its name, though, is unique across all of your spaces, and it has one default.

The reason is your code. Your code asks for a parameter by its name, checkout-button-color, and never names a space. The Quick Start shows the call in each language.

If the search space could have its own checkout-button-color, that call would have two possible answers, and your code would have to know which part of the product it runs in. With one name and one default, the line means the same thing everywhere.

The default is also where a test ends. When you pick a winner, ABMeter writes the winner’s values into the parameters’ defaults. From then on, every user outside an experiment or a flag gets the new value.

One test per parameter, checked at start

Because there is one default, ABMeter refuses to start an experiment while another running experiment tests one of its parameters. The check looks at every space.

Picture two tests on checkout-button-color. The first one finishes, green wins, and the default becomes green. The second test’s control group reads the default, so it switches from gray to green in the middle of the test. That test now compares against a moving target, and its result means nothing.

Why at start? ABMeter could let both tests run and refuse the second winner later. Then you would run a test for weeks, reach a result, and learn that you cannot apply it. A refusal at start costs you a wait. A refusal at the end costs you the test.

Ways to cut spaces

• By product area: checkout, search, onboarding. Tests in one area often touch the same decision, so they share a pool. Tests in different areas run in parallel.
• By team: each team gets its own pool and does not wait for another team’s test to finish.
• By platform: web and mobile, when you test the two apps separately.

Start with one space. Add a second one when two tests wait for each other’s traffic and their effects cannot meet.

If your traffic is small, a second space matters more than it seems. A test that needs four weeks on all of your users needs about eight on half of them. Check the numbers with the sample size calculator before you split a pool. If even 100% of your users is not enough for the lift you expect, a space does not fix that: test a bigger change, or measure a metric closer to the change, like a click on the button instead of a purchase.