The previous article ended with a list of things an AI assistant cannot do in ABMeter: start or stop an experiment, pick a winner, turn a feature flag on or off, delete anything. It promised to explain why. This is that article.
MCP in one paragraph
MCP, the Model Context Protocol, lets an AI assistant use an outside service. The service offers tools: named functions with typed inputs and a short description. The assistant reads the descriptions, decides which tool fits what you asked, and calls it. The protocol describes the tools. It does not police how the assistant uses them. Whatever a tool can do, the assistant can do, whenever it decides to.
Ship the winner at 2 am
Imagine an experimentation service with a tool called apply_winner. Its description is careful: “Applies the winning variant to all users. Only call this after the user has confirmed.”
You ask your assistant to watch a checkout test overnight: “Keep an eye on it, and ship the winner when it is significant.” At 2 am the assistant checks the results. The test variant is ahead and the p-value is 0.04. You told it to ship when it is significant. The description says to ask first, but you already said yes, in a way. So it calls apply_winner.
The model did nothing unreasonable. The test was planned for two weeks and it is day three. A p-value below 0.05 on one of many looks is what chance looks like when you check every night. By morning every user sees the new checkout, the experiment is over, and the evidence that would have told you it was a fluke is gone.
The careful description did not help. Neither would a better one.
A description is advice, not a lock
A tool description is text for a model. The model weighs it against everything else in the conversation: your instructions, the results it just read, its guess about what you meant. Most of the time it follows the description. Sometimes it decides, for reasons that sound fine in the moment, that this case is the exception.
MCP also has annotations: flags on a tool such as read-only or destructive. A client can use them to ask you before a destructive call. They are useful, and ABMeter sets them on every tool. But the MCP specification calls them hints, and tells clients not to trust them from servers they do not trust. Whether a client asks you, and when, is up to the client.
So whatever the protocol offers, the rule “a person decides” cannot live in the tool. It has to live on the server, where the assistant cannot argue with it.
What ABMeter does instead
No tool for the decision
ABMeter’s MCP server has no tool that starts, stops or resets an experiment, records the decision after a test, turns a feature flag on or off, or deletes anything. The assistant cannot ship a winner at 2 am because there is nothing to call. Those operations happen in the web UI, where you are signed in and press the button yourself. A feature flag the assistant creates starts disabled.
The REST API can start and stop experiments, with your API key. That is a script you wrote and reviewed, which is a decision you made in advance.
The browser brings you to the decision
An assistant that cannot press the button can still do all the work before it. Two tools make that practical:
link_to_browserlinks the assistant to your browser. You run it once.navigate_browseropens a page of ABMeter in that browser, such as an experiment or its decision page, and can fill in the form on it.
The Setup Browser Link page shows how to link. After that, a conversation about ending a test goes like this:
- You ask the assistant whether the checkout test is done. It calls
evaluate_experiment_effects, reads the effects and their confidence intervals, and recommends a decision, with the reasons. - If you agree, it navigates your browser to the experiment, and you click Stop.
- It navigates to the decision page with its recommendation filled in.
- The decision dialog opens with Apply Winner Variant selected, the green audience chosen and the notes typed in. You read it and click Confirm Decision, or pick Keep Original Values or Reset to Draft if you disagree.
- The assistant checks the experiment’s state afterwards and tells you what changed.
The assistant did the analysis and the typing. You made one click, on a screen that showed you exactly what it would do.
The server refuses the rest
Taking the decision tools away is not enough. Some ordinary edits would change what users see in a running experiment just as surely. Raising its allocation moves users in and out of it. Changing the split between control and test moves users from one group to the other. An enabled feature flag that sets one of the tested parameters takes the flag’s audience out of the experiment, because flags are evaluated first.
ABMeter refuses these on the server, so the REST API and MCP get the same answer. Ask the assistant to raise a running experiment from 50% to 80%, and it gets back a refusal that says the allocation is locked while the experiment runs. Ask it to point an enabled flag at a variant that sets the tested button color, and the refusal names the flag, the parameter and the experiment, and says to stop the experiment in the web UI or disable the flag first.
The web UI can still enable such a flag. It shows the same warning and asks you to confirm.
The assistant reads the refusal and tells you. It cannot talk its way past it, because nothing in the conversation reaches the check. The rule in plain words: once an experiment runs, each user stays in the group they were first assigned to, and a change that would move them needs a person in the web UI.
Try it
Connect your assistant with the steps on Setup MCP Connection, ask it to call setup_demo, then ask it to link your browser. Pick an experiment in the demo and ask whether it is ready to conclude. Watch it fill in the decision for you, and then decide for yourself.