If your A/B test says “+6% conversion,” the first question I ask to ensure trustworthy results isn’t “Is it significant?” It’s “Did you actually perform random allocation the way you think you did?”
Sample ratio mismatch (SRM) is the quiet failure mode that turns clean-looking results into expensive mistakes. It shows up when variants don’t receive the expected share of eligible users, like a 50/50 test that lands 53/47. Selection bias is often the culprit. That sounds small. In practice, it often means assignment broke, filtering changed after assignment, or tracking dropped unevenly.
When I’m on the hook for revenue, SRM is a stop sign. I’d rather throw away a week of data than ship a pricing or onboarding change based on corrupted randomization.
Why sample ratio mismatch is a business problem, not a stats detail
Sample Ratio Mismatch matters because it attacks the core promise of A/B testing: comparable groups. Once that’s gone, your measured lift can come from who got in, not what you changed.
Here’s the money version. Suppose you run a checkout test on 200,000 sessions/month. Your baseline conversion is 2.5%, and it suggests a +4% relative lift (to 2.6%). If you ship it, that might mean about 200 extra orders per month. If AOV is $120, that’s $24,000/month you’ll attribute to the change.
Now imagine Sample Ratio Mismatch happened because paid traffic hit Variant B more often, and paid traffic converts differently. Allocation imbalance leads to a false positive. You didn’t discover a behavior change, you mixed two audiences. Your “lift” reflects this experimental flaw, which compromises data integrity, and your decision making gets anchored to a fake win. The downstream cost isn’t only shipping the wrong UI. It’s also the opportunity cost of not testing a better idea, and the trust loss when teams realize results don’t replicate.
This shows up a lot in startup growth because teams move fast. Traffic sources shift daily, with paid versus organic audiences entering unevenly and creating variation bias or survivorship bias. SDKs drop events after app updates. Product-led growth loops create weird edge cases (deep links, referrals, invites) that don’t behave like homepage traffic.
If the split is wrong, treat every metric as suspect, including “neutral” results. SRM can hide both winners and losers.
If you want deeper background on what SRM is and common triggers, I like Microsoft’s write-up on diagnosing sample ratio mismatch in A/B testing. It’s practical, not hand-wavy.
The SRM check I run before reading any lift
I keep this simple because speed matters. Before I look at conversion, I verify traffic allocation.
Step 1: Confirm the expected split for the eligible population
If you ramped from 10% to 50%, don’t check the whole date range as one block. Check by stable segments (each ramp period), otherwise you’ll flag “SRM” that is just ramp math.
Also confirm the expected unit: users, devices, sessions. I’ve seen teams assign on user_id, then analyze on sessions, and wonder why splits drift.
Step 2: Compare expected vs observed, then run a quick significance check
Most tools surface SRM automatically. If yours doesn’t, use a chi-squared goodness-of-fit test. As a rule of thumb, I get nervous when deviation is over 1% to 2% with large sample size, or when the SRM p-value falls under 0.01 for statistical significance.
Here’s a simple example for a 50/50 test:
| Variant | Expected | Observed | Absolute gap |
|---|---|---|---|
| A | 50,000 | 51,500 | +1,500 |
| B | 50,000 | 48,500 | -1,500 |
That’s a 3% relative imbalance. With 100,000 total users, it’s rarely “random noise.” It usually means something structural.
Step 3: Decide what you will do if SRM is present
This is the part most teams skip. SRM affects the balance between control and treatment groups, so my default is harsh: I don’t trust lift when SRM is real. I either (a) fix the cause and rerun, or (b) exclude the corrupted time window and re-check allocation.
If you need a second opinion on practical thresholds and prevention, this guide on Sample Ratio Mismatch and what to do is a decent reference.
Root causes that create SRM: root cause diagnosis (and how I triage them fast)
Sample Ratio Mismatch isn’t one bug. It’s a category. When I’m under time pressure, I triage in the same order every time because it finds the high-frequency failures.
Randomization unit doesn’t match analysis unit
If user randomization happens on user_id but your funnel is session-based, cookie churn and multi-device behavior can skew observed splits. This gets worse in mobile web and in markets with high privacy tooling. Fix is boring: align the unit, or analyze on the assignment key.
Filtering after assignment (the silent killer)
This is where teams hurt themselves without noticing. Someone builds an audience like “US, iOS, returning users,” but applies parts of it after assignment in analytics. Now Variant A and B pass through different filters, creating uneven filtering that leads to interaction effects masking the true treatment effect. The result looks like SRM, and it also breaks causal claims.
This is where behavioral science creeps in. If Variant B changes page speed or error rates, users drop before the event that defines “eligible,” so your filtering step becomes treatment-affected. At that point, your SRM is a symptom, not the disease.
Traffic sources or entry points aren’t evenly distributed
Paid vs organic, email vs push, deep links vs homepage, all of these can route through different stacks. One route might fire the assignment call earlier, or fail it more often. If your growth strategy depends on channel mix, you need to segment SRM checks by source.
Instrumentation and runtime issues (especially with applied AI)
Bots, ad blockers under varying triggering conditions, data loss, caching, CDN quirks, and mid-test config changes can all bias exposure counts. I now add lightweight anomaly detection in my analytics pipeline to flag sudden changes in assignment rate by browser, geo, and referrer. It’s not fancy AI. It’s simply automated “this looks different than yesterday.” Continuous monitoring of ratios is better than one-off checks.
For more prevention ideas, this SRM prevention guide covers common fixes like consistent bucketing and avoiding late assignment.
My decision rule (so you can move fast without guessing)
When SRM shows up, I perform an SRM check and use one rule to protect conversion and credibility:
If the chi-square statistic indicates SRM is statistically unlikely (p < 0.01) or operationally large (split off by more than ~1% to 2% at scale), I stop interpreting lift, isolate the cause, then rerun or exclude the bad period.
While some prefer a Bayesian method for analysis, the chi-square check remains the standard for SRM.
If you need an actionable next step today, do this in the next hour:
- Recompute observed splits by day and by top 3 traffic sources.
- Perform a cohort analysis to find the first day the split drifted.
- Inspect releases, ramp changes, targeting edits, and tracking deploys on that day.
That small habit keeps experimentation honest, protects product-led growth bets, and improves your financial outcomes over time because you ship fewer “wins” that vanish in production.
In other words, I’d rather be slower on one test than wrong on ten. SRM discipline is how I stay fast without lying to myself, guarding against Sample Ratio Mismatch to deliver trustworthy results in A/B testing.
Leave a Reply