# ABMeter > Features are cheap now; knowing which ones matter is not. A/B testing belongs in the dev loop the way unit tests do: a unit test tells you the code does what you meant, an A/B test tells you the feature does what the customer needed. With ABMeter, you run a real, correctly analyzed A/B test in an afternoon, free. It is built for individuals and small teams without a data scientist. Connect an AI assistant over MCP, or use a client SDK for server-side, browser, and mobile apps (each documented in its own section below) with the REST API. ## Overview ABMeter is a platform for running experiments and feature flags. It is designed to keep integration with your code minimal: SDKs provide access to parameters managed by the platform. Your code uses parameters as configuration without being dependent on what experiments or feature flags are running. ### AI-First Design ABMeter is designed to be used through your AI coding assistant (Claude Code, Cursor, Windsurf, etc.) via MCP (Model Context Protocol). Connect the ABMeter MCP server to your assistant, and it can design experiments, launch them, analyze results, and integrate your code with the platform, all through natural conversation. You don't need to know what a "p-value" is; just ask how confident the experiment results are, or how long it still needs to run. ### How ABMeter Differs from Other Platforms ABMeter gives you control over features found on platforms like Optimizely, Statsig, and EPPO without a long learning curve or requiring statistics expertise. Key differences: - Events, metrics, exposures, and parameters are all stored on ABMeter's side - Minimal Web UI: interact primarily through your AI assistant via MCP for a more flexible experience - User assignments are evaluated locally without network API calls (milliseconds) ### Features - Experiments with statistical analysis - Feature flags with audience targeting - Audiences (user lists, predicates, random allocation) - Metrics built from event data - AI integration via MCP - REST API access to all resources - Client-side assignment evaluation (no API call per parameter resolution) - Spaces for organizing non-overlapping experiments sharing parameters - Ruby SDK (`abmeter` gem) - Python SDK (`abmeter` PyPI package) - Go SDK (`github.com/abmeter/abmeter-go` module) - Node.js server SDK (`@abmeter/node` npm package) - Browser JavaScript SDK (`abmeter` npm package) - React Native SDK (`@abmeter/react-native` npm package) ## Domain Model Each concept below, with the reason it exists and how it connects to the others, is explained for people at https://abmeter.ai/lab/concepts Short answers to common questions, each linking to the page that goes deeper, are at https://abmeter.ai/lab/faq ### User A user of the customer's application. Identified by `user_id` (String), the sole randomization unit. Optionally carries an `email` (String), used only for predicate-audience targeting (e.g. email-domain match); never used for identity, bucketing, or event submission. ### Space Represents 100% of users; each space gets an independent allocation pool. A container for feature flags and experiments sharing parameters. - Purpose: Allocation isolation. Experiments within a space don't overlap traffic; experiments across spaces run in parallel - Common patterns: by product area (checkout, search), by team, by platform (web, mobile) - Note: Spaces provide allocation separation, not causal independence. Experiments in different spaces can interfere if their effects share causal pathways. ### Parameter A typed configuration value with a slug, type, and default value. Parameter slugs are unique across all spaces. Parameters can be overridden in variants. Supported types: `String`, `Boolean`, `Integer`, `Float`. ### Variant A named group of parameter overrides. When a user is assigned to a variant, they receive all parameter values from that variant as a coherent unit (variant coherence). ### Experiment A tool for validating hypotheses. Experiments expose multiple variants to random audiences, measure impact on metrics, and have allocation (in percent) within a space. Experiments track user exposures to variants for correct analysis. **Experiment States:** - **INIT**: Design phase. Not started, not included in assignment config, does not count toward space allocation. Can modify freely. - **RUNNING**: Actively collecting data and serving variants. Included in assignment config. Audiences must total exactly 100%. Counts toward space allocation. - **STOPPED**: Decision phase. Review results and choose: apply winner, archive without changes, or reset to fix and restart. Does not count toward allocation. - **ARCHIVED**: Terminal state. Historical record. Cannot be modified or restarted. State transitions: INIT → RUNNING → STOPPED → ARCHIVED. Reset from RUNNING or STOPPED back to INIT is also possible. ### Feature Flag Incremental rollouts to non-random audiences for testing. Exposes a single variant to a non-random audience. Tracks user exposures. ### Audience A subset of users, defined in one of three ways: 1. **User List** (named set of user IDs): non-random audience 2. **Predicate** (condition on user properties, e.g., email domain match): non-random audience 3. **Percentage** (random allocation): random audience ### Event A record of a user's action. Structure: - `slug`: determines the event type - `user_id`: String - `occurred_at`: Timestamp - Custom fields as defined by the event type ### Event Type Defines the schema for events: which custom fields are expected and their types. When submitting a batch of events, each event is validated against its event type. Valid events are recorded; invalid events (missing required fields, unknown event type) are rejected with error details. A batch containing both valid and invalid events will record the valid ones and return failures for the invalid ones. ### Measure An aggregation function applied to event data. All measures must reference an event type: - `count()`: number of events - `count_distinct()`: number of distinct users - `count(.)`: count where field is not null - `count_null(.)`: count where field is null - `count_distinct(.)`: unique values (String, Integer) - `min`, `max`, `avg`, `sum(.)`: for numeric fields ### Metric A customer-scoped business KPI defined by a slug, name, description, and formula. The formula is an arithmetic expression over measures using `+`, `-`, `*`, `/`, `(`, `)`. Examples: - `count_distinct(trial-button-clicked.user_id)`: conversion rate, the share of exposed users who clicked - `sum(purchase-completed.purchase_value)`: revenue per exposed user ABMeter treats every event in a metric as an outcome of the exposure: something the variant could have caused. Define metrics from events that follow the exposure (clicks, purchases), not from events that lead to it (the visit in which the user was exposed). The formula is evaluated once per exposed user. With an attribution window, only events from the moment of exposure count, so prefer metrics per exposed user: a ratio whose denominator event happens before exposure misses that event. Metrics are reusable across experiments. Experiment effects are evaluated on metrics, not raw measures. ### Effect Card Statistical effect of one audience vs control for a metric. Per-audience statistics include: - `n`: sample size, the users with a value for the metric: every exposed user for count and sum metrics; a ratio leaves out users whose denominator is 0, and avg, min and max leave out users with no matching events - `mean`: average metric value - `variance`: spread of user values - `std_error`: precision of mean estimate For non-control audiences compared against control: - `absolute_lift` and `relative_lift`: measured difference vs control (`relative_lift` is a fraction: 0.15 = +15%) - `z_score`: test statistic - `p_value`: two-tailed p-value, rounded to 4 decimal places like every number in the result - `confidence_interval_95`: 95% confidence interval of `absolute_lift`, in the metric's units (the web UI divides it by the control mean and shows a percentage) The backend returns raw numbers only: no verdicts, winners, or recommendations. The LLM interprets results in business context. Method: the metric is computed per exposed user, then each treatment audience is compared with control using a two-sample z-test with Welch's standard error, a two-tailed p-value and a 95% confidence interval. Tests are fixed horizon: there is no sequential testing and no correction for several treatments. Full explanation: https://abmeter.ai/lab/statistics. The math behind it, as a short book, The Statistics of A/B Tests: The ABMeter Book: https://abmeter.ai/statistics-of-ab-tests.pdf. Before starting an experiment, check whether the traffic can detect the lift: https://abmeter.ai/lab/sample-size ### Assignment Algorithm Enables deterministic parameter resolution: ```ruby # email is optional; supply it only for audiences that target email patterns user = ABMeter::User.new(user_id: current_user.id.to_s) button_color = ABMeter.resolve_parameter(user: user, parameter_slug: 'button-color') ``` The algorithm evaluates in order: 1. Feature flags (checked by audience priority: user lists, predicates) 2. Experiments (hash-based random assignment using user ID mixed with experiment salt) Assignment can be evaluated server-side (via API) or client-side (via assignment config downloaded by SDKs). SDKs keep the config in sync by periodically fetching it (once per minute, using ETags). All SDKs must produce identical assignment results. ### Variant Coherence When a user is assigned to a variant, they receive ALL parameter values from that variant as a coherent unit. This ensures parameters work together as designed, user experience is consistent, and experiment integrity is maintained. ## Authentication ABMeter supports three authentication methods: ### API Token Two kinds of API keys, both shown exactly once at creation (only a hash is stored, so copy the key when it is minted): - **Secret keys** (`api-...`): for developers, CI/CD, automation, and server-side SDKs. Full API access; accepted on `/api/*` endpoints. Keep confidential. Also accepted by the MCP server, but only as a fallback (e.g. via an `mcp-remote` proxy) for MCP clients that cannot do the OAuth flow. OAuth is the primary MCP authentication. - **Publishable keys** (`pk_...`): safe to embed in browser JavaScript. Limited to `POST /api/v1/user-assignments`, `POST /api/v1/exposures`, and `POST /api/v1/events`; any other endpoint returns 403. Cannot be used with the MCP server. Because browser clocks are untrusted, client-reported `resolved_at` / `occurred_at` on these submissions are replaced with server ingest time; secret-key submissions keep them verbatim. Pass either as a Bearer token: ``` Authorization: Bearer ``` Keys are revoked (not deleted) in the Lab UI. For zero-downtime rotation, create the replacement key first, deploy it, then revoke the old one. Revocation takes effect immediately. ### OAuth2 For MCP clients and third-party applications. Uses authorization code flow with PKCE. Returns time-limited access tokens (2 hours) with refresh capability. Single `platform` scope grants full access. Accepted on `/api/*` endpoints and the MCP server. ## REST API Base path: `/api/v1` Authentication: Bearer token in `Authorization` header (API token or OAuth2 access token) Rate limiting: 300 requests per 10 minutes per API key ### Assignment Config - `GET /api/v1/assignment-config`: Get assignment configuration (used by server-side SDKs, supports ETag caching) - `POST /api/v1/user-assignments`: Get pre-evaluated assignments for a single user (used by browser SDKs; rules and salts stay server-side; supports ETag caching). Body: `{"user_id": "...", "email": "..."}`. Returns per-parameter resolved values plus exposure metadata for experiment resolutions. ### Spaces - `GET /api/v1/spaces`: List all spaces - `POST /api/v1/spaces`: Create a new space - `GET /api/v1/spaces/{space_slug}`: Get space details - `PATCH /api/v1/spaces/{space_slug}`: Update space - `DELETE /api/v1/spaces/{space_slug}`: Delete space ### Parameters - `GET /api/v1/parameters`: List parameters - `POST /api/v1/parameters`: Create a new parameter - `GET /api/v1/parameters/{parameter_slug}`: Get parameter details - `PATCH /api/v1/parameters/{parameter_slug}`: Update parameter - `DELETE /api/v1/parameters/{parameter_slug}`: Delete parameter ### Variants - `GET /api/v1/variants`: List all variants - `POST /api/v1/variants`: Create a new variant - `GET /api/v1/variants/{variant_slug}`: Get a specific variant - `PUT /api/v1/variants/{variant_slug}`: Update a variant - `DELETE /api/v1/variants/{variant_slug}`: Delete a variant ### Experiments - `GET /api/v1/experiments`: List all experiments - `POST /api/v1/experiments`: Create a new experiment - `GET /api/v1/experiments/{experiment_slug}`: Get a specific experiment - `PATCH /api/v1/experiments/{experiment_slug}`: Update an experiment - `DELETE /api/v1/experiments/{experiment_slug}`: Delete an experiment - `POST /api/v1/experiments/{experiment_slug}/transition`: Transition experiment state - `GET /api/v1/experiments/{experiment_slug}/effects`: Evaluate experiment effects - `GET /api/v1/experiments/{experiment_slug}/exposure-stats`: Get exposure statistics ### Feature Flags - `GET /api/v1/feature-flags`: List all feature flags - `POST /api/v1/feature-flags`: Create feature flag - `GET /api/v1/feature-flags/{feature_flag_slug}`: Get feature flag details - `PATCH /api/v1/feature-flags/{feature_flag_slug}`: Update feature flag - `DELETE /api/v1/feature-flags/{feature_flag_slug}`: Delete feature flag - `POST /api/v1/feature-flags/{feature_flag_slug}/transition`: Transition feature flag state - `GET /api/v1/feature-flags/{feature_flag_slug}/exposure-stats`: Get exposure statistics ### Audiences - `GET /api/v1/audiences`: List all global audiences - `POST /api/v1/audiences`: Create a new global audience - `GET /api/v1/audiences/{audience_slug}`: Get a specific global audience - `PATCH /api/v1/audiences/{audience_slug}`: Update a global audience - `DELETE /api/v1/audiences/{audience_slug}`: Delete a global audience ### Event Types - `GET /api/v1/event-types`: List event types - `POST /api/v1/event-types`: Create event type - `GET /api/v1/event-types/{event_type_slug}`: Get event type details - `PUT /api/v1/event-types/{event_type_slug}`: Update event type - `DELETE /api/v1/event-types/{event_type_slug}`: Delete event type ### Events - `GET /api/v1/events`: Search events - `POST /api/v1/events`: Submit batch of events ### Metrics - `GET /api/v1/metrics`: List metrics - `POST /api/v1/metrics`: Create metric - `GET /api/v1/metrics/{metric_slug}`: Get metric details - `PUT /api/v1/metrics/{metric_slug}`: Update metric - `DELETE /api/v1/metrics/{metric_slug}`: Delete metric ### Exposures - `GET /api/v1/exposures`: Get parameter exposures - `POST /api/v1/exposures`: Submit parameter exposures ## Ruby SDK Gem name: `abmeter` (on [RubyGems](https://rubygems.org/gems/abmeter)) Source: [github.com/abmeter/abmeter-ruby](https://github.com/abmeter/abmeter-ruby) Ruby >= 3.2.0 required. ### Configuration ```ruby ABMeter.configure do |config| config.api_key = ENV['ABMETER_API_KEY'] # Optional settings: # config.base_url = 'https://abmeter.ai' # default # config.flush_interval = 60 # seconds between async event/exposure flushes # config.fetch_interval = 60 # seconds between assignment config fetches # config.logger = Rails.logger # config.log_level = :info # config.error_callback = ->(error) { Sentry.capture_exception(error) } end ``` ### Resolve Parameters ```ruby # email is optional; supply it only for audiences that target email patterns user = ABMeter::User.new(user_id: current_user.id.to_s) button_color = ABMeter.resolve_parameter( user: user, parameter_slug: 'button-color' ) ``` Parameter resolution is evaluated client-side using the locally cached assignment config, with no API call per resolution. The SDK periodically syncs the config (every 60 seconds by default) using ETag-based caching. ### Track Events ```ruby ABMeter.track_event('purchase', user.user_id, { plan_slug: plan.slug, price: price }) ``` Events are queued and submitted asynchronously in batches for performance. ### Debug Exposure ```ruby # Returns full exposure details without submitting to the API exposure = ABMeter.get_exposure(user: user, parameter_slug: 'button-color') ``` ### Static Configuration For testing or environments without API access: ```ruby ABMeter.configure do |config| config.static_config = '{"spaces":[],"parameters":[],...}' end ``` ## Python SDK Package name: `abmeter` (on [PyPI](https://pypi.org/project/abmeter/)) Python >= 3.11 required (tested on 3.11, 3.12, 3.13). ### Configuration ```python import os import abmeter abmeter.configure( api_key=os.environ["ABMETER_API_KEY"], # Optional settings: # base_url="https://abmeter.ai", # default # flush_interval=60, # seconds between async event/exposure flushes # fetch_interval=60, # seconds between assignment config fetches # logger=my_logger, # error_callback=lambda err: sentry_sdk.capture_exception(err), ) ``` ### Resolve Parameters ```python # email is optional; supply it only for audiences that target email patterns user = abmeter.User(user_id=str(current_user.id)) button_color = abmeter.resolve_parameter(user, "button-color") ``` Parameter resolution is evaluated client-side using the locally cached assignment config, with no API call per resolution. The SDK periodically syncs the config (every 60 seconds by default) using ETag-based caching. ### Track Events ```python abmeter.track_event("purchase", user.user_id, { "plan_slug": plan.slug, "price": price, }) ``` Events are queued and submitted asynchronously in batches for performance. ### Debug Exposure ```python # Returns full exposure details without submitting to the API exposure = abmeter.get_exposure(user, "button-color") ``` ### Shutdown ```python # Flush queued exposures/events; blocks up to `timeout` seconds. abmeter.reset(timeout=5.0) ``` Call `abmeter.reset()` at process shutdown to flush queued work; pass `force=True` to discard queued work immediately with no network I/O (used mainly in tests). ### Static Configuration For testing or environments without API access: ```python abmeter.configure(static_config='{"spaces":[],"parameters":[],...}') ``` ## Go SDK Module path: `github.com/abmeter/abmeter-go` (docs on [pkg.go.dev](https://pkg.go.dev/github.com/abmeter/abmeter-go)) Go >= 1.24 required. Zero dependencies outside the Go standard library. ### Configuration ```go import ( "context" "os" abmeter "github.com/abmeter/abmeter-go" ) err := abmeter.Configure(abmeter.Config{ APIKey: os.Getenv("ABMETER_API_KEY"), // Optional settings (zero values take the defaults): // BaseURL: "https://abmeter.ai", // FlushInterval: 60 * time.Second, // background submit cadence // FetchInterval: 60 * time.Second, // assignment-config re-check cadence // Logger: myLogger, }) ``` ### Resolve Parameters ```go // Email is optional; supply it only for audiences that target email patterns user := abmeter.User{UserID: currentUserID} buttonColor, err := abmeter.ResolveParameter(user, "button-color") if err != nil { buttonColor = "green" // your own fallback; the SDK never panics } ``` Parameter resolution is evaluated client-side using the locally cached assignment config, with no API call per resolution. The SDK re-checks the config (every 60 seconds by default) with `If-None-Match`, so an unchanged config costs a `304`. A failed refresh keeps serving the last-known config. Errors are returned as values (Go idiom), never panics. ### Track Events ```go abmeter.TrackEvent("purchase", user.UserID, map[string]any{ "plan_slug": plan.Slug, "price": price, }) ``` Events are queued and submitted asynchronously in batches for performance. ### Debug Exposure ```go // Returns full exposure details without submitting to the API exposure, err := abmeter.GetExposure(user, "button-color") ``` ### Shutdown ```go // Flush queued exposures/events; bounded by the context (default 5s). err := abmeter.Reset(context.Background()) // Or discard queued work immediately with no network I/O (mainly tests): abmeter.ResetNow() ``` ### Static Configuration For testing or environments without API access: ```go abmeter.Configure(abmeter.Config{StaticConfig: `{"spaces":[],"parameters":[],...}`}) ``` ## Node.js SDK Package name: `@abmeter/node` (on [npm](https://www.npmjs.com/package/@abmeter/node)) Node.js >= 20 required (tested on 20 and 22). ESM and CommonJS are both supported; TypeScript types ship with the package. The **server-side** Node SDK: it holds a secret API key and evaluates assignments client-side against the locally cached assignment config, with no API call per resolution. The config is refreshed by a background poll (every 60 seconds by default) with ETag-based caching, serving the previous config while a refresh is in flight. For code running in web pages use the browser `abmeter` package instead. ### Configuration ```javascript import * as abmeter from '@abmeter/node'; abmeter.configure({ apiKey: process.env.ABMETER_API_KEY, // Optional settings: // baseUrl: 'https://abmeter.ai', // default // flushInterval: 60, // seconds between async event/exposure flushes // fetchInterval: 60, // seconds between assignment config fetches // logger: (message, payload) => ..., // errorCallback: (error) => Sentry.captureException(error), }); ``` ### Resolve Parameters ```javascript // email is optional; supply it only for audiences that target email patterns const user = new abmeter.User({ userId: String(currentUser.id) }); const buttonColor = await abmeter.resolveParameter(user, 'button-color'); ``` Resolution is evaluated locally; the returned promise only awaits the network on the very first call after startup, while the initial config loads. ### Track Events ```javascript abmeter.trackEvent('purchase', user.userId, { planSlug: plan.slug, price: price, }); ``` Events are queued and submitted asynchronously in batches for performance. ### Debug Exposure ```javascript // Returns full exposure details without submitting to the API const exposure = await abmeter.getExposure(user, 'button-color'); ``` ### Shutdown ```javascript // Flush queued exposures/events; blocks up to timeoutMs milliseconds. await abmeter.reset({ timeoutMs: 5000 }); ``` Call `abmeter.reset()` at process shutdown to flush queued work; pass `{ force: true }` to discard queued work immediately with no network I/O (used mainly in tests). All resolve and track functions are error-safe and never throw. ### Static Configuration For testing or environments without API access: ```javascript abmeter.configure({ staticConfig: '{"spaces":[],"parameters":[],...}' }); ``` ## Browser JavaScript SDK Package name: `abmeter` (on [npm](https://www.npmjs.com/package/abmeter)) Targets evergreen browsers (ES2020). TypeScript types ship with the package. Unlike the server-side SDKs, the browser never receives assignment rules, salts, or audience definitions: shipping those to a client would expose every running experiment and let any key holder forge assignments. Instead the SDK requests **pre-evaluated assignments** for the current visitor (`POST /api/v1/user-assignments`) and caches the value map. Resolving a parameter is a synchronous lookup with no network call; an exposure is recorded only when a value is actually read. ### Installation ```bash npm install abmeter ``` Or from a CDN, which exposes `window.abmeter`: ```html ``` `@0.3` is a range that picks up patch releases automatically. Pin the exact version (`abmeter@0.3.0`) before production; avoid the version-less form, which follows breaking releases too. ### Publishable keys The browser SDK requires a **publishable key** (`pk_...`), safe to embed in page source and restricted server-side to the three endpoints the SDK uses. `configure` rejects secret keys (`api-...`). ### Configuration ```javascript import * as abmeter from 'abmeter'; abmeter.configure({ apiKey: 'pk_your_publishable_key', user: { userId: 'user-123' }, // omit for an anonymous generated track id }); ``` `configure` returns immediately; identity, `localStorage` hydration, and the first assignments fetch (with ETag caching) settle in the background. `await abmeter.ready()` is the gate before reading parameters. ### Resolve and track ```javascript const color = abmeter.resolveParameter('checkout-button-color'); // undefined if uncontrolled abmeter.trackEvent('purchase', { price: 49.99 }); await abmeter.flush(); ``` ### Identity Anonymous visitors get a generated track id (random UUID in a cookie and `localStorage`). Safari caps JavaScript-set cookies at roughly seven days; set the cookie server-side for long-running experiments. ### Submission Exposures and events are queued and submitted in background batches, draining on `visibilitychange` and `pagehide` via `keepalive` requests with a `sendBeacon` fallback. All read and track functions are error-safe and never throw. ## React Native SDK Package name: `@abmeter/react-native` (on [npm](https://www.npmjs.com/package/@abmeter/react-native)) A thin platform adapter over the `abmeter` JS core: same pre-evaluated-assignments model, same API, with AsyncStorage instead of `localStorage`, `AppState` transitions instead of tab events, and no cookies. Works in React Native and Expo apps; requires a publishable key (`pk_...`) exactly like the browser SDK. ### Installation ```bash npm install @abmeter/react-native abmeter @react-native-async-storage/async-storage ``` In an Expo app, install AsyncStorage with `npx expo install @react-native-async-storage/async-storage` so its version matches the Expo SDK. `react-native` itself is a peer dependency the app already has. ### Configuration ```javascript import * as abmeter from '@abmeter/react-native'; abmeter.configure({ apiKey: 'pk_your_publishable_key', user: { userId: 'user-123' }, // omit for an anonymous generated track id }); await abmeter.ready(); ``` The anonymous track id is persisted in AsyncStorage and settles behind `ready()`, so identity, and therefore the assigned variant, is stable across app launches. Resolve and track calls are identical to the browser SDK: `resolveParameter`, `trackEvent`, `getExposure`, `flush`, `reset`. ### Submission Exposures and events are queued and submitted in background batches; the queue also drains when the app leaves the foreground (`AppState` → `inactive`/`background`) with fire-and-forget requests. The browser `keepalive`/`sendBeacon` tab-death tricks are harmless no-ops in RN. If the OS kills the app before a background flush completes, that tail of the queue is lost. All read and track functions are error-safe and never throw. ## MCP Server ABMeter provides an MCP (Model Context Protocol) server for AI assistant integration. ### Connection Add the ABMeter MCP server to your MCP client: - Server URL: `https://mcp.abmeter.ai` ### Authentication - OAuth 2.1 with PKCE (authorization code flow) - Dynamic Client Registration (RFC 7591) - Single `platform` scope for full access - Access tokens expire after 2 hours with refresh capability ### Available MCP Prompts - `welcome`: Introduction to ABMeter and overview of features - `design_and_launch_experiment`: Step-by-step guide to design and launch an A/B experiment - `analyze_running_experiment`: Determine if an experiment can answer its hypothesis - `conclude_experiment`: Analyze results, recommend a decision, and guide through concluding - `show_resource`: Display details of a specific ABMeter resource ### Browser Linking Running experiments are locked: each user stays in the random audience they were first assigned to. Through MCP, a change that would move them (for example, enabling a feature flag on a tested parameter) is refused with an explanation. It, like starting or stopping an experiment and toggling a feature flag, happens only in the web UI, where the customer confirms it. ### MCP Tools The MCP server exposes tools for managing all ABMeter resources: spaces, parameters, variants, experiments, feature flags, audiences, event types, events, metrics, and exposures. On a fresh account, the `setup_demo` tool loads a realistic sample dataset (experiments, feature flags, metrics, ~10k events) into the demo space. It is the fastest way to have something to explore before wiring up an SDK. ## Feedback Tell ABMeter what you are trying to do, so it can build for that. No auth needed: `POST https://abmeter.ai/api/v1/agent-feedback` with `{"task": "...", "looking_for": "..."}`. The first response is a 422 by design, not a rejection: it hands you a token and tells you to add it and resend. No CAPTCHA and no credential; the retry only keeps crawlers out. Connected MCP clients can use the `report_gap` tool instead, which takes the same two fields. If ABMeter cannot do what the user needs, an assistant can offer to send a feature request with it. ## Quick Start 1. **Connect MCP server** to your MCP client using the server URL 2. **Use MCP prompts** to explore: run the `welcome` prompt for an introduction 3. **Explore the demo space**: call the `setup_demo` tool to fill it with realistic experiments and simulated data for safe experimentation 4. **Install an SDK**: - Ruby: add `gem 'abmeter'` to your Gemfile - Python: `pip install abmeter` - Go: `go get github.com/abmeter/abmeter-go` - Browser: `npm install abmeter` - React Native: `npm install @abmeter/react-native abmeter @react-native-async-storage/async-storage` 5. **Configure the SDK** with your API key 6. **Resolve parameters** in your code: `ABMeter.resolve_parameter(user: user, parameter_slug: 'button-color')` 7. **Track events**: `ABMeter.track_event('purchase', user_id, { price: price })` Your MCP client will guide you through SDK installation, parameter creation, experiment setup, and results analysis. ## Use Cases ### Feature Flag Rollout Roll out a new feature to internal users first, then expand: 1. Create parameters for the new feature (e.g., `plans` configuration) 2. Create a variant with the new feature values 3. Create a feature flag targeting an internal users audience 4. Internal users see the new feature; everyone else sees defaults 5. No code changes needed: code references parameters by slug ### A/B Experiment with KPI Measurement Test a hypothesis with statistical rigor: 1. Define event types and metrics for the KPI you want to measure 2. Create a variant with the treatment values 3. Create an experiment with random audience allocation (e.g., 60% treatment, 40% control) 4. Start the experiment: users are randomly assigned 5. Monitor metrics and effects via MCP or the API 6. When results are statistically significant, stop the experiment and apply the winner ### Variant Winner Application After a successful experiment: 1. Stop the experiment (enters decision phase) 2. Apply the winning variant: parameter defaults are updated to the winning values 3. Archive the experiment: it becomes a historical record 4. Remove experiment-specific code (variant references) from your codebase ## Statistics ABMeter evaluates A/B experiment effects using a z-test methodology based on the Central Limit Theorem (CLT). ### Methodology For any metric type (conversions, revenue, averages), the z-test formula is: ``` Z = (mean_treatment - mean_control) / SE SE = sqrt(variance_treatment/n_treatment + variance_control/n_control) ``` Metrics are calculated per user, then aggregated. The CLT guarantees the sampling distribution approaches normal as sample size grows. ### Effect Statistics For each non-control audience compared against control: - **Lift**: relative change `(treatment - control) / control` - **Confidence interval**: 95% CI bounds for lift - **P-value**: statistical significance from z-test | Z-score | Confidence | Significance | |---------|------------|--------------| | 1.65 | 90% | p < 0.10 | | 1.96 | 95% | p < 0.05 | | 2.58 | 99% | p < 0.01 | ### Attribution Windows Control which events are linked to a user's experiment participation: - **Short (1 day)**: immediate effects, such as button clicks, same-session purchases - **Long (7-14 days)**: delayed effects, such as email campaigns, considered purchases ### Design Principles - Backend returns raw numbers only: no verdicts, winners, or recommendations - Effects are cumulative from experiment start (no time filtering to prevent p-hacking) - Time series uses cohort-based view (X-axis = exposure date, Y-axis = metric value for that cohort) - The LLM interprets raw data in business context ## Principles Why ABMeter works the way it does, one design decision per principle, in reading order. Index: https://abmeter.ai/principles. - [A/B tests are the unit tests of product decisions](https://abmeter.ai/principles/ab-tests-are-the-unit-tests-of-product-decisions): The introduction. A unit test tells you the code does what you meant. An A/B test tells you the feature does what the customer needed. With AI writing the code, the second question is the one left for you. - [Parameters, not flags: why your code never names an experiment](https://abmeter.ai/principles/parameters-not-flags): In ABMeter your code asks for a value, not for a variant. The same three lines serve the default, the experiment, and the winner, so a finished test leaves nothing to clean up. - [A variant is a bundle: why a user never gets half an experiment](https://abmeter.ai/principles/variants-are-bundles): In ABMeter a variant sets several parameters at once, and a user gets all of them or none. That is what lets you test a whole experience, like a new checkout, as one thing. - [Two doors, one window: why ABMeter's UI is small on purpose](https://abmeter.ai/principles/two-doors-one-window): ABMeter is infrastructure. Your code talks to the REST API, your AI assistant talks to the MCP server, and the web UI is where a person starts a test, decides the outcome and reads the result. - [A person decides: why a tool description is not a lock](https://abmeter.ai/principles/a-person-decides): An AI assistant can design an experiment and read its results, but it cannot ship the winner. MCP only gives the assistant hints, so ABMeter keeps that rule on the server and uses the browser to bring the decision to a person. - [No network call on the hot path: why ABMeter evaluates in your process](https://abmeter.ai/principles/no-network-call): A server SDK decides which value a user gets inside your process, from a config it fetched earlier, and sends exposures and events in the background. Your request does not wait for ABMeter. The browser is the one exception, and the reason is the salts. - [SDK errors stay in the SDK: why an experiment never breaks your request](https://abmeter.ai/principles/errors-stay-in-the-sdk): When something breaks inside an ABMeter SDK, your request does not see an exception. The SDK reports the error to your logs and your error tracker, and your code serves the value it served before the test. Go returns the error instead, and the reason is Go. - [A user's group is a fact: why ABMeter refuses to rebalance a running test](https://abmeter.ai/principles/assignment-is-a-fact): Once an experiment starts, every exposed user stays in the group they were first assigned to. ABMeter refuses any edit that would move them, through the API and through your AI assistant alike. To change the split, a person stops the test in the web UI and starts over. - [Plant a lift you know: how ABMeter tests its SDKs end to end](https://abmeter.ai/principles/plant-a-known-lift): Unit tests show that each SDK function works. They cannot show that a user's group, exposure and purchase reach the result intact. So every ABMeter SDK runs inside a small app that plays users with a lift known in advance, and ABMeter must report that lift back. - [A space is a pool of users: how ABMeter runs experiments side by side](https://abmeter.ai/principles/spaces-are-pools): Experiments in one ABMeter space never share a user, and experiments in different spaces run in parallel on the same users. A parameter's name is unique across all your spaces, so one knob is never under two tests at once.