A/B Testing

A/B testing compares two versions of something - a webpage, email, pricing page, or product feature - by randomly showing each version to different user groups and measuring which performs better. It replaces opinions with evidence. The method is straightforward; the hard part is ensuring the data behind the test is trustworthy.

The approach works because randomization balances confounding variables across the groups. Take an illustrative test: 10,000 users are randomly split between a blue button (control) and a green button (variant), and the green button produces a 12% higher click-through rate that is statistically significant. Because assignment was random, the difference can be attributed to the color change rather than to time of day or user demographics, with the remaining risk that it is a false positive bounded by the significance level you chose.

How A/B Testing Works

The process follows six steps, each with a specific purpose.

1. Form a hypothesis. "Changing the CTA button from blue to green will increase click-through rate on the pricing page." A hypothesis without a specific, measurable prediction is not testable.

2. Define the metric. Click-through rate on the pricing page CTA. This is the primary metric - the one that determines whether the test wins or loses. Secondary metrics (bounce rate, time on page) provide context but do not drive the decision. Ideally, these metrics map to governed KPIs.

3. Calculate sample size. Based on the baseline conversion rate, the minimum detectable effect (how small a difference matters), and the desired statistical power (typically 80%). Running with too few users produces unreliable results. Running with too many wastes time and traffic.

4. Split traffic randomly. 50/50 between control and variant. Randomization must be user-level - the same user sees the same variant every time they visit. Cookie-based or user-ID-based assignment handles this.

5. Wait for significance. Run the test until the pre-calculated sample size is reached. Do not peek at results early and stop when they look good - with standard fixed-sample tests this inflates the false positive rate well above the nominal level. If you need to monitor continuously, use a sequential method designed for it, such as the always-valid p-values described by Johari, Pekelis and Walsh.

6. Analyze and decide. Did the variant beat the control by a statistically significant margin? If yes, implement. If no, iterate or discard. If the result is ambiguous, the test may have been underpowered for the effect you care about, and a rerun with a larger sample may be needed.

Statistical Foundations

Four concepts separate rigorous A/B testing from guesswork.

Statistical significance. The p-value is the probability of seeing a difference at least as large as the one observed if there were truly no difference between the variants. A p-value below 0.05 is the usual threshold for calling a result significant. It is not the probability that the null hypothesis is true, nor the probability that the result was "due to chance", as the American Statistical Association's 2016 statement on p-values spells out. A 95% confidence interval has a similar reading: the procedure that produces it captures the true effect in 95% of repeated experiments, which is not the same as a 95% chance that this particular interval contains it.

Sample size. Calculated before the test starts, based on the baseline conversion rate, the minimum detectable effect (the smallest improvement worth detecting), the significance level, and the desired statistical power. To detect the same relative improvement (say, 5%), a page with a 2% conversion rate needs far more users than a page with a 20% conversion rate. Online calculators, such as Evan Miller's sample size calculator, do the arithmetic.

Type I errors (false positives). Declaring a winner when there is no real difference. A common cause is peeking at results before the test reaches its planned sample size. Each peek is an implicit statistical test, and running many tests inflates the false positive rate, so a test checked daily for 14 days and stopped at the first significant reading has a much higher false positive rate than the nominal 5%.

Type II errors (false negatives). Missing a real effect because the sample is too small. If a test has 60% power for a given effect size, there is a 40% chance of missing a real improvement of that size. Underpowered tests are risky because a non-significant result is easily misread as proof that a change "doesn't work."

Common Mistakes That Invalidate Tests

Stopping early. The most common and most damaging mistake. In a typical scenario, a product manager checks the dashboard on day 3, sees a 15% lift with p=0.03, and calls the test. But p-values fluctuate widely while samples are small, and an early "significant" reading often fades as more data arrives. The fix: commit to a sample size before the test starts and do not stop early.

Testing too many variables at once. Changing the headline, hero image, and CTA button in a single test makes it impossible to attribute the result. If the variant wins, which change drove it? If it loses, which change hurt? Run one change per test, or use multivariate testing with a sample size large enough to detect interaction effects.

Ignoring segment effects. In an illustrative case, a test shows a 5% overall improvement, but mobile users saw a 10% decline while desktop users saw a 15% lift. The overall number looks positive, but the mobile experience got worse. Check results across key segments such as device, geography, and new vs. returning users. Slicing into many segments also multiplies the chance of a false positive, so treat a surprising segment result as a hypothesis to confirm in a follow-up test.

Sample ratio mismatch. If a test designed as a 50/50 split ends up with a ratio that a chi-squared test says is very unlikely under that design (on a large sample, even 48/52 can qualify), something is wrong with assignment or logging - a bot, a caching layer, or a redirect that selectively drops users from one group. A sample ratio mismatch is a symptom of a data quality problem that usually biases the comparison, so check for it before analyzing results.

Novelty and primacy effects. Users may react to change itself, not to the variant. A new design gets more clicks for a week because it looks different, then engagement returns to baseline. Run tests long enough to capture steady-state behavior, not just the novelty spike.

Beyond Basic A/B Testing

Multivariate testing tests multiple elements simultaneously - headline, image, and CTA in all combinations. This reveals interaction effects (for example, a headline that works better with the large image but worse with the small one). The cost: sample size requirements grow multiplicatively. A test with 3 headlines and 3 images needs 9 variants, and each needs enough traffic for significance.

Bandit algorithms dynamically allocate more traffic to the variant that appears to be winning. Instead of a fixed 50/50 split, the algorithm might shift to 70/30 as evidence accumulates. This reduces the "cost of learning" - fewer users see the losing variant - but makes statistical analysis more complex. Best for short-lived decisions like ad creative rotation.

Bayesian A/B testing combines the data with a prior to give a probability that one variant is better than another ("there is an 89% probability that B outperforms A by at least 2%") instead of a binary significant/not-significant verdict. The answer depends on the prior chosen, so the prior should be agreed before the test starts. This is often more useful for business decisions, where stakeholders want to know "how likely is this better?" rather than "can we reject the null hypothesis?"

Data Quality: The Hidden Dependency

A/B test results are only as reliable as the data pipeline that feeds them, and a data problem can look exactly like a real effect. Booking.com, for example, computes common metrics in two separate pipelines built by different engineers and alerts on discrepancies, because a bug in either one would otherwise go unnoticed. The problems below are illustrative of what such checks catch.

Duplicate event tracking. A JavaScript event fires twice on page load due to a race condition. Conversion rates appear doubled. The test declares a winner based on inflated numbers. The real conversion rate is unchanged. This is a data observability problem, not a testing problem.

Inconsistent user identity. A user visits on their phone, gets assigned to variant A. The same user visits on their laptop, gets a new cookie, and is assigned to variant B. They convert on the laptop. The test attributes the conversion to variant B, but the user's initial exposure was variant A. Assigning variants on an identity that holds across devices, such as a logged-in account, avoids this where a product has one.

Delayed data. The analytics pipeline has a 24-hour lag. A test is stopped "at significance" based on data that is actually one day stale. The true numbers - including the last day of traffic - might tell a different story. Real-time or near-real-time data pipelines matter for test integrity.

Metric definition disagreements. Marketing defines "conversion" as a form submission. Product defines it as a completed purchase. Both teams analyze the same test and draw opposite conclusions. This is a data governance problem: the organization needs a single, governed definition for each metric used in experimentation.

Tools and Platforms

Experimentation platforms: Optimizely, LaunchDarkly (feature flags with built-in experimentation) and Statsig (product analytics and experimentation combined). These handle randomization, metric tracking, and statistical analysis end-to-end.

Analytics-integrated: Amplitude Experiment ties experiments to product analytics events. Google Analytics no longer runs experiments itself: Google Optimize, Google's free testing tool, was discontinued on 30 September 2023, and Google pointed users to third-party testing tools that integrate with Google Analytics. This route suits organizations already using these analytics tools and wanting to add experimentation without a separate platform.

Open source: GrowthBook offers Bayesian and frequentist statistics engines. Intuit's Wasabi was open-sourced as an A/B testing service, but its repository now states it is no longer under active development. Open source suits organizations that want control over their experimentation infrastructure and data.

How Dawiso Supports Experimentation

Dawiso does not run experiments or calculate their results; it documents the definitions and data that experiments depend on. Reliable A/B testing requires consistent metric definitions across tools, and Dawiso's business glossary holds business indicators - conversion rate, revenue per user, engagement score - with a description, a calculation, an owner and a steward. Definitions can move through an approval workflow from draft to review to approved, so when marketing and product disagree about what "conversion" means, both teams can point to one approved definition and build it into their experimentation platforms and analytics dashboards.

The data catalog documents which datasets feed experimentation pipelines, who owns them, and which quality checks are defined on them and in which external tool those checks run. The check results themselves stay in that tool, so teams confirm there that the event data was complete, timely and deduplicated before trusting a result.

When an A/B test produces surprising results, data lineage helps teams trace the experiment metric back through the analytics pipeline toward the raw event source. Dawiso builds lineage from supported connectors, dbt, SQL parsing (not on by default) and manually added relations, so how far a trace reaches depends on which of those cover the pipeline.

Conclusion

A/B testing is the simplest form of causal inference available to product and marketing teams. The statistical method is well-established. The tooling is mature. What most organizations underestimate is the data quality dependency: an experiment is only as trustworthy as the data pipeline that measures it. Governed metric definitions, reliable event tracking, and consistent user identity are the prerequisites that make experimentation work.

Sources

• Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.
• Ronald L. Wasserstein and Nicole A. Lazar, The ASA Statement on p-Values: Context, Process, and Purpose, The American Statistician, 2016.
• Ramesh Johari, Leo Pekelis and David J. Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing, arXiv, 2015.
• Aleksander Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners, KDD 2019.
• Raphael Lopez Kaufman, Jegar Pitchforth and Lukas Vermeer, Democratizing online controlled experiments at Booking.com, arXiv, 2017.
• Ashok Chandrashekar, Fernando Amat, Justin Basilico and Tony Jebara, Artwork Personalization at Netflix, Netflix Technology Blog, 2017.
• Evan Miller, Sample Size Calculator.
• Google Analytics Help, [Sunset September 2023] Google Optimize.
• GrowthBook Docs, Statistics overview.
• Intuit, Wasabi repository, GitHub.
• Dawiso Help, Business glossary.
• Dawiso Help, Workflows overview.
• Dawiso Help, Sources of data lineage.