Every ABMeter SDK has unit tests. They tell you each function does what it should. They do not tell you that a user who saw the green button and then bought something shows up in the right group of the result. That trip crosses the SDK, the network, the server and the statistics.
So ABMeter tests the whole trip the way you would test a scale: put a known weight on it. Each SDK runs users through an experiment whose answer is decided in advance, and ABMeter must report that answer back.
Ten users and a coin
Here is the known weight. Each simulated user sees a button, blue for control or green for treatment. Then the user flips a coin: in control it says “buy” 10% of the time, in treatment 15%. The true lift is +5 points, because I wrote it into the code.
Now run ten users, five per group. On average, control makes half a purchase and treatment three quarters of one. In about three runs out of four, at least one group has no purchase at all. One purchase moves a group’s rate by 20 points. Whatever lift you read, it says nothing about the +5 points in the code.
Run 5,000 users instead, 2,500 per group. Control buys about 250 times and treatment about 375. The measured lift now changes by about one point from run to run, and in nearly every run it lands between 3 and 7 points. At this size, a broken pipeline and a working one look different.
One truth, one app per SDK
The truth lives in one place: the two purchase rates and the split. A small app per SDK uses the SDK the way a customer would. It asks for the button color, flips the coin for the group it got, and tracks a purchase when the coin says so. A driver sends users through the app and then asks ABMeter for the result.
Adding an SDK means adding one small app. The truth and the checks stay the same.
Three checks, cheapest first
The driver checks three things in order, so a failure says where to look.
- Every user arrived. The number of exposed users ABMeter reports must equal the number of users the driver sent. Exactly, not approximately.
- The data did not change on the way. The app knows which users bought, so it knows the lift of its own run. ABMeter’s lift for that run must match it. This check does not depend on luck: if the two differ, something between the app and the result is wrong.
- The truth comes back. Over many runs, ABMeter’s lift must land near +5 points, and closer on larger runs than on smaller ones. This is the only statistical check.
The first two need no statistics at all, which is why they come first. A user lost on the way, or a purchase filed under the wrong user, fails them even on a small run.
Do the same for your integration
You can plant a lift in your own app before your first real test:
- Start a throwaway experiment on a test parameter and a test event, so made-up users never mix with real ones.
- Write a script that sends a few hundred made-up users through your real code: resolve the parameter, then track the event with a different probability per group, for example 10% and 30%. Record what the script did.
- Drain the SDK’s queue so everything is sent, then compare. The exposed user count must match exactly. The per-group event counts and the lift must match what your script recorded.
A few hundred users will not reach significance, and they do not need to. You are checking the plumbing, not the product. For the real test, put your numbers into the sample-size check first. If your traffic is too small, it says so and tells you what to do instead.