A user's group is a fact: why ABMeter refuses to rebalance a running test

An A/B test compares two groups that differ in one thing: the variant. That only works if each user stays in their group for the whole test. So in ABMeter, once an experiment starts, a user’s group is a fact. The server refuses any edit that would change it, and only a person can end the test to start a new one.

Day 5: 50/50 becomes 60/40

You run a checkout test with two groups, 50% control and 50% test. On day 5 someone decides the new checkout looks risky and edits the split to 60/40, to send fewer users to it.

The edit looks harmless. It is not. The same user always lands in the same group, as long as the split stays the same. Change the split, and one user in ten lands somewhere else from now on. They saw the new checkout for four days and the old one after that.

What happens to the analysis?

• Those users carry four days of the test variant into the control group. Their purchases from day 5 on are no longer a clean reading of either checkout.
• ABMeter counts each user from their first exposure in a group. A user who switched has a first exposure in both groups, so they count in both, and their purchases can count twice.
• The two groups are no longer two random halves of the same traffic, which differ only by the variant. The p-value assumes they are.

Nothing in the result would warn you. The chart would look normal, and the number would be wrong.

Edits that are fine, and edits that are refused

Most edits do not touch assignment, and ABMeter allows them on a running test:

• the experiment’s description, hypothesis and slug
• its primary and guardrail metrics
• a variant’s description and slug
• a parameter’s description, and the default of any parameter the experiment does not test

These change what you read about the test, not what a user sees.

The rest is locked while the experiment is running or stopped:

• The split. You cannot add, remove or resize a group, or point a group at a different variant.
• The allocation. The share of traffic in the experiment is fixed, in both directions. Changing it moves users into the test or out of it.
• What the groups see. You cannot change the parameter values of a variant in the test, or the default value of a parameter the test overrides, because the control group sees that default.

The lock is on the server, so the REST API and your AI assistant get the same answer. Ask an assistant to change the split of a running test to 60/40, or the default of a tested parameter, and the tool call comes back refused, with the reason.

The assistant reads the refusal and tells you. There is no flag to force it.

The lock follows the experiment’s state:

State	What you can change
init (draft)	Everything
running	Copy and metrics only
stopped	Copy and metrics only, so the record you decide on stays coherent
archived	Nothing. It is the history of what you tested

The way out is a person

Sometimes the split really has to change. The new checkout breaks on one browser, or you want a third variant. ABMeter has a way to do that, and it runs through a person:

1. Open the experiment in the web UI and click Stop.
2. In the decision dialog, pick Reset to Draft to edit the test and start it again, or record a decision and create a new experiment with the new split.

A restarted test counts only exposures after its new start. Nothing from the old split leaks into the new result.

Your AI assistant cannot do any of this. It has no tool to stop, reset or archive an experiment. A person decides explains why that decision stays with you, and how the assistant can still bring you to the right page with the right choice filled in. The REST API can stop and reset an experiment with your API key, because a script you wrote and reviewed is a decision you made in advance.

Why not let a well-meaning edit through with a warning? Because the cost lands on someone who is not there. The person who rebalances on day 5 sees a form that saved. The person who reads the result on day 14 sees a number. Neither sees the users who switched. Stopping is a visible act, and it leaves a record that the test changed.

The same rule behind the statistics

The statistics page describes the attribution window: a user’s events count from that user’s own first exposure. That rule quietly assumes the group a user saw first is the group they stay in. The refusals above are what make the assumption true. Without them, “first exposure” would name a group the user may have left.

Try it

Connect your assistant with the steps on Setup MCP Connection and ask it to call setup_demo. Pick a running experiment and ask the assistant to change its split. Read the refusal, then stop the experiment in the web UI and make the change yourself.