A/B tests are the unit tests of product decisions

A unit test tells you the code does what you meant. An A/B test tells you the feature does what the customer needed.

For years the first question took most of a developer’s day. Now an AI assistant writes much of the code, and it writes the tests too. The first question got cheap. The second one did not, and it is the one left for you.

Small teams already have a habit and a tool for the first question: a test next to every change, run on every commit, red or green. For the second question most of them have nothing. These principles are about making the A/B test as ordinary as the unit test.

Two catalog images in one afternoon

For example, you are building an app for small online shops. A seller uploads a photo of a product and asks for a clean catalog image. Your assistant wires up an image model in an afternoon, and now you have a choice: a photoreal variant (model A with a studio-lighting prompt) or an illustrated one (model B with a flat-style prompt). Both look good in the demo. Which one do sellers actually use? Every generated image also costs money, so the answer matters.

Without a test, the loudest opinion in the room decides. Or the one you liked best at 6 p.m.

With a test, each seller gets one of the two variants, and you count the sellers who save a generated image into a listing. Sellers log in, so their account id is the user id, and a seller sees the same variant every time. The photoreal settings are the parameters’ defaults, so photoreal is the control group. Say each variant gets 600 sellers:

The illustrated variant has a save rate 8 points higher, and the 95% confidence interval of that difference runs from 2.7 to 13.3 points. It does not include zero, so the difference is significant: the p-value is about 0.003. You can recompute the interval with the formula on the statistics page.

So you ship the illustrated variant, and when someone asks why, you have a number, not an opinion.

Look at the numbers, though. 600 sellers per variant was enough only because the difference was large. I come back to that below.

A practice, not a project

At most companies an A/B test is a project: a ticket, a data person, a dashboard, a meeting about the dashboard. That is why small teams skip it. A practice has to be cheaper than that. It is one test per feature, set up from the same loop where you write the code, read in minutes, and forgotten once it ends.

Two things in ABMeter make that possible.

First, your code reads parameter values, never variant names. The image tool reads two parameters, the model and the prompt, and passes them to the image model:

seller = ABMeter::User.new(user_id: current_account.id)
model = ABMeter.resolve_parameter(
  user: seller, parameter_slug: 'image-model'
)
prompt = ABMeter.resolve_parameter(
  user: seller, parameter_slug: 'image-prompt'
)
render_catalog_images(photo, model, prompt)

When the seller saves an image into a listing, one more line records it:

ABMeter.track_event('catalog-image-saved', current_account.id, {})

The code does not know whether an experiment is running. The illustrated variant sets the model and the prompt together, and a seller gets both or neither. Model B with the studio-lighting prompt is a combination nobody designed, and no seller ever sees it (why a variant is a bundle). You integrate once, and the next experiment on the same tool is a change in ABMeter, not in your code.

Second, a finished test leaves nothing behind in the code. When you pick the winner, ABMeter makes its value the parameter’s default, and every user gets it from then on. There is no if experiment branch to clean up later, the way there is no unit test to delete after it passes.

What it costs

A practice is only worth it if you know the price. There are three.

Traffic. A small lift needs many users before a test can see it. For a lift from 30% to 38%, the sample-size check asks for about 550 sellers per variant, which is why the example had 600. If 100 sellers a week try the image tool, that takes 11 weeks. A lift from 30% to 32% needs about 8,400 sellers per variant, more than three years at the same pace. Before you build an experiment, put your numbers into the sample-size check. When the answer is too long, it says what to do instead: test a bigger change, measure closer to the change, run longer, or ship behind a feature flag without pretending to measure.

Discipline. You decide what you measure before you start: one primary metric, such as the share of sellers who save an image. You also name things as parameters with values, not as experiment labels in the code. That costs a few minutes of thinking per test, and it is the part an assistant cannot do for you.

Patience. The test is fixed horizon. You choose the length before the start and read the result once, at the end. If you check every day and stop at the first good-looking p-value, you will ship noise several times more often than the 5% you signed up for. The statistics page explains why.

What ABMeter does to make it a habit

An assistant sets up and reads the test. Connect the ABMeter MCP server to Claude, Cursor or another MCP client. In the same conversation where you write the feature, the assistant creates the parameter and the experiment, adds the SDK call to your code, and reads the results with you when the test ends. The assistant has no tool to start or stop an experiment: you click Start and Stop in the browser, so it never changes live traffic on its own.

The SDKs stay out of your app’s way. The server SDKs pick a variant inside your process from a cached copy of the configuration, so a page does not wait on an ABMeter request. Exposures and events go out in the background. An SDK error is logged or returned, not raised into your request handler. There are SDKs for Ruby, Python, Node.js, Go, browser JavaScript and React Native.

The statistics are checkable. Every result is a two-sample z-test on per-user metrics, with confidence intervals and p-values. The method is on the statistics page, and the math behind it is in The Statistics of A/B Tests, a short book you can read in an evening. Nothing in the result is a black box.

Run one real test this week

Pick a change you are going to ship this week anyway: a new onboarding step, a different headline, a shorter form. Check on the sample-size page that your traffic can answer the question. If it can, run the change as an experiment instead of shipping it to everyone. ABMeter is free, and the first test takes an afternoon.

If your traffic cannot answer it yet, that is a useful result too. You learned it in five minutes, not after a month of waiting for a number that never comes.