A variant is a bundle: why a user never gets half an experiment

In Parameters, not flags, one experiment changed one value: the color of a checkout button. Real changes are rarely one value. This article is about the next decision in ABMeter: a variant is a bundle of parameter values, and a user gets the whole bundle or none of it.

Two colors that must agree

Say you want to test a high-contrast theme. It changes two parameters:

• foreground-color: default #555555, high contrast #000000
• background-color: default #eeeeee, high contrast #ffffff

Your page reads both:

user = ABMeter::User.new(user_id: current_user.id)
foreground = ABMeter.resolve_parameter(
  user: user, parameter_slug: 'foreground-color'
)
background = ABMeter.resolve_parameter(
  user: user, parameter_slug: 'background-color'
)

Two calls, two answers. What if they came from different places? A user with the high-contrast foreground and the default background gets black text on light gray. That is readable, but it is not the theme you designed, and it is not the default either. Now flip it: a dark theme with #ffffff text on a #111111 background. Mix one of those with a default and the user gets white text on light gray, a page nobody can read.

Worse, the experiment would measure it. Some users in your “high contrast” group would see something else, and the result would describe a design you never shipped.

The rule: all or nothing

So in ABMeter you do not test two values. You create one variant, high-contrast, that sets both, and test the variant. The rule:

• A user assigned to the variant gets every value it sets.
• A user in the control group gets every default.
• Nobody gets a mix.

This holds per user and per experiment, on every call. It holds across sessions, servers and devices, as long as your code passes the same user ID.

Every variant in one experiment must set the same parameters. If high-contrast sets both colors, a dark variant in the same experiment has to set both too. ABMeter refuses to save an experiment where they differ, because then the groups would not be comparing the same thing.

Bigger changes are bundles too

The theme is a toy. The same idea carries a whole new flow. Say you want to test a shorter checkout:

Each parameter is a value your code already knows how to use, and the variant is the combination you think works best. A user in the test group gets a one-step checkout with autocomplete and “Pay now”. Nobody gets a one-step checkout with a “Continue” button.

Some code still has to branch. A one-step checkout is different markup from a three-step one. The question is what the branch reads:

steps = ABMeter.resolve_parameter(
  user: user, parameter_slug: 'checkout-steps'
)

if steps == 1
  render_single_page_checkout
else
  render_step_checkout(steps: steps)
end

The branch reads checkout-steps, a fact about the checkout. It does not read the variant name. If the short checkout wins, checkout-steps becomes 1 for everyone and the branch still means something. You can delete the old path when you are ready, but nothing breaks if you don’t. If the code read variant == 'short-checkout' instead, it would refer to an experiment that no longer exists.

How ABMeter keeps the bundle together

There is no assignment table. ABMeter does not store which user got which variant and look it up later. Each call to resolve_parameter computes the answer again, from the same inputs, and gets the same result.

For one parameter, the SDK checks in this order:

1. An enabled feature flag whose audience includes the user and whose variant sets the parameter.
2. A running experiment whose variants set the parameter.
3. The parameter’s default.

For the experiment step, the group comes from a hash of a salt and the user ID. The hash does not depend on which parameter you asked for. So the calls for foreground-color and background-color land in the same group, even if they run in different requests on different servers.

Every SDK computes the same answer for the same user, so a Ruby backend and a Go service agree on a user’s group. The browser and React Native SDKs ask ABMeter, which computes the same answer too.

Determinism alone is not enough. The bundle also has to stay the same while the test runs, so ABMeter refuses changes that would break it:

• One experiment per parameter. ABMeter refuses to start an experiment while another running experiment tests one of its parameters.
• A frozen bundle. While an experiment is running or stopped, you cannot edit its variants’ values or its parameters’ defaults, and you cannot swap the variant of a group.
• Flags stay out of the way. ABMeter refuses to start an experiment while an enabled feature flag sets one of its parameters.

One door stays open on purpose. You can enable a flag over a running experiment’s parameter in the web UI, after a warning that you confirm. The API refuses it, and the MCP server has no tool that enables a flag. Flags win per parameter, so if the flag sets only foreground-color, users in the flag’s audience get the flag’s foreground and the variant’s background. That is the mix this article warns about, and ABMeter lets you do it only when you say so. Keep it for a value that hurts users and cannot wait for the test to stop.

Try it with two values

Pick two values in your app that only make sense together: a headline and its subheading, a price and the plan name next to it, a text color and its background. Make each one a parameter, put both in one variant, and run the variant against control. Every user in the test sees the pair you designed, and the result tells you about that pair.