Optimizely Sample Size Calculator

Calculate Your A/B Test Sample Size

Use this Optimizely Sample Size Calculator to determine the number of visitors you need for each variation in your A/B tests to achieve statistically significant results.

Table 1: Sample Size Requirements for Different MDEs

MDE (%)	Sample Size (Current Settings)	Sample Size (90% Power, 95% Significance)

What is an Optimizely Sample Size Calculator?

An Optimizely Sample Size Calculator is a specialized tool designed to help A/B testers, marketers, and product managers determine the minimum number of participants (or visitors) required for each variation in an experiment to detect a statistically significant difference, if one truly exists. In the world of conversion rate optimization (CRO) and experimentation, tools like Optimizely are used to run A/B tests. Before launching any test, it’s crucial to know how many users you need to expose to each version of your experience to ensure your results are reliable and not due to random chance.

This calculator helps prevent two common pitfalls: running tests for too short a period with too few users (leading to inconclusive or misleading results) or running tests for too long (wasting resources and delaying implementation of winning variations). By inputting key metrics like your baseline conversion rate, desired minimum detectable effect, statistical significance, and power, the Optimizely Sample Size Calculator provides the necessary sample size per variation.

Who Should Use an Optimizely Sample Size Calculator?

• A/B Testers & CRO Specialists: To plan experiments effectively and ensure valid results.
• Product Managers: To make data-driven decisions about new features or UI changes.
• Marketers: To optimize landing pages, email campaigns, and ad creatives.
• Data Analysts: To validate experiment designs and interpret results accurately.
• Anyone running online experiments: From small businesses to large enterprises, understanding sample size is fundamental to reliable testing.

Common Misconceptions about Sample Size

• “More data is always better”: While more data can reduce variance, there’s a point of diminishing returns. Over-collecting data beyond the required sample size is inefficient.
• “Just run it for a week”: Experiment duration should be determined by sample size requirements, not arbitrary timeframes. A week might be too short or too long depending on traffic and MDE.
• “I can stop the test as soon as I see a winner”: This is known as “peeking” and can inflate Type I error rates, leading to false positives. Tests should run until the predetermined sample size is reached.
• “Sample size only matters for conversion rates”: While often applied to conversion rates, sample size calculations are critical for any metric you’re trying to optimize, such as click-through rates, engagement, or revenue per user.

Optimizely Sample Size Calculator Formula and Mathematical Explanation

The calculation of sample size for A/B testing, particularly for proportions (like conversion rates), is rooted in statistical power analysis. The goal is to determine how many observations are needed to detect a specific effect size (your MDE) with a given level of confidence (significance) and probability (power).

The formula used by this Optimizely Sample Size Calculator is a common approximation for comparing two proportions (e.g., control vs. variation) and determines the sample size required for *each* group:

n = [(Z_α/2 + Z_β)² * (p₁(1-p₁) + p₂(1-p₂))] / (p₂ – p₁)²

Where:

• n: Sample size required per variation (group).
• Z_α/2: The Z-score corresponding to the desired statistical significance level (alpha). For a two-tailed test, we use α/2. For example, for 95% significance (α=0.05), Z_α/2 is 1.96.
• Z_β: The Z-score corresponding to the desired statistical power (1 – beta). For example, for 80% power (β=0.20), Z_β is 0.84.
• p₁: The baseline conversion rate (as a decimal).
• p₂: The expected conversion rate of the variation, assuming the Minimum Detectable Effect (MDE) is achieved. Calculated as p₁ * (1 + MDE).
• MDE: The Minimum Detectable Effect (as a decimal), representing the smallest relative improvement you want to be able to detect.

Step-by-Step Derivation:

1. Define your parameters: Set your baseline conversion rate (p1), desired MDE, significance (alpha), and power (1-beta).
2. Calculate p2: Determine the target conversion rate for the variation by applying the MDE to p1.
3. Find Z-scores: Look up the Z-scores for your chosen alpha/2 and power. These values come from the standard normal distribution table.
4. Calculate the numerator: This part of the formula accounts for the desired confidence and the variability within your data.
5. Calculate the denominator: This part represents the square of the absolute difference you want to detect (the effect size). A larger difference requires a smaller sample size.
6. Divide and round: Divide the numerator by the denominator and round up to the nearest whole number, as you can’t have a fraction of a visitor.

Table 2: Key Variables for Sample Size Calculation

Variable	Meaning	Unit	Typical Range
Baseline Conversion Rate (p1)	Your current conversion rate for the metric being tested.	% (decimal in formula)	0.1% – 50%
Minimum Detectable Effect (MDE)	Smallest relative improvement you want to detect.	% (decimal in formula)	5% – 25% (relative)
Statistical Significance (α)	Probability of a false positive (Type I error).	% (decimal in formula)	5% (0.05) or 1% (0.01)
Statistical Power (1-β)	Probability of detecting a real effect if one exists.	% (decimal in formula)	80% (0.80) or 90% (0.90)
Number of Variations	Total versions in your experiment (e.g., Control + 1 Variation).	Integer	2 – 5
Average Daily Visitors	Your site’s daily unique visitors.	Integer	1,000 – 1,000,000+

Practical Examples (Real-World Use Cases)

Example 1: Optimizing a Landing Page CTA

Imagine you’re optimizing a landing page’s Call-to-Action (CTA) button. Your current (baseline) conversion rate for clicking the CTA is 8%. You believe a new CTA design could improve this, and you want to be able to detect at least a 15% relative improvement (MDE) with 95% statistical significance and 80% statistical power. You plan to test one new variation against your control (2 variations total). Your site gets about 5,000 average daily visitors.

• Baseline Conversion Rate (p1): 8% (0.08)
• Minimum Detectable Effect (MDE): 15% (0.15)
• Statistical Significance (Alpha): 95% (0.05)
• Statistical Power (Beta): 80% (0.80)
• Number of Variations: 2
• Average Daily Visitors: 5,000

Using the Optimizely Sample Size Calculator:

• p2 = 0.08 * (1 + 0.15) = 0.092 (9.2%)
• Z_α/2 (for 95% significance) = 1.96
• Z_β (for 80% power) = 0.842
• Sample Size per Variation: Approximately 10,500 visitors
• Total Sample Size: 10,500 * 2 = 21,000 visitors
• Absolute MDE: 9.2% – 8% = 1.2%
• Estimated Experiment Duration: 21,000 / 5,000 = 4.2 days (round up to 5 days to ensure full cycles)

Interpretation: You would need to expose about 10,500 unique visitors to your control CTA and 10,500 to your new CTA. Given your traffic, this test would run for approximately 5 days. This ensures that if the new CTA truly provides a 15% relative lift (or 1.2% absolute lift), you have an 80% chance of detecting it as statistically significant.

Example 2: Testing a New Checkout Flow

You’re redesigning a critical step in your e-commerce checkout process. Your current checkout completion rate (baseline) is 60%. Due to the high traffic volume and importance of this step, you want to detect even a small 5% relative improvement (MDE) with high confidence: 99% statistical significance and 90% statistical power. You’re testing one new flow against the existing one (2 variations). Your site receives 50,000 average daily visitors.

• Baseline Conversion Rate (p1): 60% (0.60)
• Minimum Detectable Effect (MDE): 5% (0.05)
• Statistical Significance (Alpha): 99% (0.01)
• Statistical Power (Beta): 90% (0.90)
• Number of Variations: 2
• Average Daily Visitors: 50,000

Using the Optimizely Sample Size Calculator:

• p2 = 0.60 * (1 + 0.05) = 0.63 (63%)
• Z_α/2 (for 99% significance) = 2.576
• Z_β (for 90% power) = 1.282
• Sample Size per Variation: Approximately 14,500 visitors
• Total Sample Size: 14,500 * 2 = 29,000 visitors
• Absolute MDE: 63% – 60% = 3%
• Estimated Experiment Duration: 29,000 / 50,000 = 0.58 days (round up to 1 day)

Interpretation: For this high-traffic, high-stakes test, you’d need about 14,500 visitors per variation. With 50,000 daily visitors, this test could conclude in about 1 day. The higher significance and power levels, combined with a smaller MDE, still require a substantial sample size, but your high traffic allows for a quick test. This demonstrates the importance of balancing MDE, significance, and power with your available traffic.

How to Use This Optimizely Sample Size Calculator

Our Optimizely Sample Size Calculator is designed for ease of use, providing quick and accurate estimates for your A/B testing needs. Follow these steps to get your results:

Step-by-Step Instructions:

1. Enter Baseline Conversion Rate (%): Input your current conversion rate for the metric you are testing. For example, if 10% of users complete a form, enter “10”. This is your p₁.
2. Enter Minimum Detectable Effect (MDE) (%): This is the smallest *relative* improvement you want to be able to detect. If your baseline is 10% and you want to detect a 10% relative lift, the new rate would be 11% (10% + 10% of 10%). Enter “10” for 10% MDE.
3. Select Statistical Significance (Alpha) (%): Choose your desired confidence level. 95% (Alpha = 0.05) is standard, meaning there’s a 5% chance of a false positive.
4. Select Statistical Power (Beta) (%): Choose the probability of detecting a real effect if one exists. 80% is common, meaning you have an 80% chance of seeing a true winner.
5. Enter Number of Variations: Specify how many different versions (including the control) are in your experiment. For a standard A/B test, this is “2”.
6. Enter Average Daily Visitors: Provide your website’s average daily unique visitor count. This helps estimate the duration of your experiment.
7. View Results: The calculator will automatically update the results in real-time as you adjust the inputs.

How to Read Results:

• Sample Size per Variation: This is the primary result, indicating the minimum number of unique visitors required for *each* version of your experiment (e.g., 10,000 for Control, 10,000 for Variation A).
• Total Sample Size: The sum of visitors across all variations (Sample Size per Variation * Number of Variations).
• Absolute MDE: This shows the Minimum Detectable Effect in absolute percentage points. For example, if your baseline is 10% and relative MDE is 10%, the absolute MDE is 1% (meaning you want to detect a change from 10% to 11%).
• Estimated Experiment Duration: This provides an approximate number of days your experiment needs to run to gather the total required sample size, based on your average daily visitors.

Decision-Making Guidance:

The results from the Optimizely Sample Size Calculator are crucial for planning. If the estimated duration is too long, you might need to:

• Increase your MDE (aim to detect a larger effect).
• Decrease your statistical significance (e.g., from 99% to 95%).
• Decrease your statistical power (e.g., from 90% to 80%).
• Reconsider running the test if traffic is too low for a meaningful duration.

Conversely, if the duration is very short, you might consider decreasing your MDE to detect smaller, potentially valuable changes, or increasing your significance/power for higher confidence.

Key Factors That Affect Optimizely Sample Size Calculator Results

Several critical factors influence the sample size required for your A/B tests. Understanding these helps you make informed decisions when using an Optimizely Sample Size Calculator and designing your experiments.

1. Baseline Conversion Rate:

   This is your current conversion rate for the metric you’re testing. Lower baseline conversion rates generally require larger sample sizes to detect the same *relative* effect. For example, detecting a 10% relative lift on a 1% baseline (0.1% absolute lift) is harder than on a 10% baseline (1% absolute lift).

2. Minimum Detectable Effect (MDE):

   The MDE is the smallest *relative* improvement you want to be able to detect. A smaller MDE means you want to detect a more subtle change, which requires a significantly larger sample size. Conversely, if you’re only interested in large, impactful changes, you can set a higher MDE, reducing the required sample size. This is often a trade-off between the cost of running the experiment and the value of detecting a small improvement.

3. Statistical Significance (Alpha):

   Also known as the alpha level, this is the probability of making a Type I error (false positive) – concluding there’s a difference when there isn’t one. A common significance level is 95% (alpha = 0.05). To increase your confidence (e.g., to 99% significance), you need a larger sample size, as you’re demanding stronger evidence to declare a winner.

4. Statistical Power (1 – Beta):

   Power is the probability of making a correct decision when there is a real effect – detecting a difference when one truly exists (avoiding a Type II error, or false negative). A common power level is 80%. Increasing power (e.g., to 90% or 95%) means you want to be more certain you won’t miss a real winner, which requires a larger sample size. This is crucial for high-stakes tests where missing a true improvement would be costly.

5. Number of Variations:

   The more variations you include in your experiment, the larger the total sample size required. While the sample size *per variation* might not change drastically for a fixed MDE, the total traffic needed is multiplied by the number of variations. More variations also increase the risk of Type I errors if not properly accounted for (e.g., using Bonferroni correction or similar methods).

6. Traffic Volume (Average Daily Visitors):

   While not directly affecting the *calculated* sample size, your average daily visitors significantly impact the *duration* of your experiment. High traffic allows you to reach the required sample size faster. Low traffic might mean your experiment needs to run for weeks or even months, making it impractical to detect small MDEs or achieve high confidence levels. This is a practical constraint that often forces adjustments to MDE, significance, or power.

Frequently Asked Questions (FAQ)

Q1: Why is sample size important for A/B testing?

A: Sample size is crucial because it ensures your test results are statistically reliable and not due to random chance. Without an adequate sample size, you risk making incorrect decisions, either by declaring a false winner (Type I error) or missing a true winner (Type II error).

Q2: What is a good MDE to aim for?

A: A “good” MDE depends on your business context, traffic volume, and the potential impact of the change. For high-traffic sites, a smaller MDE (e.g., 5-10% relative) might be feasible. For lower-traffic sites, you might need to accept a larger MDE (e.g., 15-25% relative) to complete tests in a reasonable timeframe. Consider the business value of detecting a specific lift.

Q3: What are typical values for statistical significance and power?

A: The most common statistical significance level is 95% (alpha = 0.05). For statistical power, 80% is widely accepted. However, for critical experiments, you might increase significance to 99% or power to 90-95% to reduce the risk of errors, understanding that this will increase your required sample size.

Q4: Can I stop my test early if I see a clear winner?

A: No, stopping a test early (known as “peeking”) is a common mistake that can invalidate your results. It inflates the Type I error rate, making it more likely to declare a false positive. Always run your test until the predetermined sample size is reached, or use sequential testing methods if you absolutely need to monitor results continuously.

Q5: How does the number of variations affect sample size?

A: The sample size *per variation* is calculated based on the comparison between two groups (control vs. one variation). If you add more variations, the total sample size for the experiment increases proportionally (e.g., 3 variations means 3x the sample size per variation). This also increases the complexity of analysis and the risk of false positives if not handled correctly.

Q6: What if my estimated experiment duration is too long?

A: If the duration is too long, you have a few options: 1) Increase your MDE (aim for a larger detectable effect), 2) Decrease your statistical significance (e.g., from 99% to 95%), 3) Decrease your statistical power (e.g., from 90% to 80%), or 4) Reconsider running the test if your traffic simply isn’t sufficient for a meaningful experiment.

Q7: Does this calculator work for all types of metrics?

A: This specific Optimizely Sample Size Calculator is designed for binary metrics (proportions), like conversion rates (e.g., yes/no, converted/not converted). For continuous metrics (e.g., average revenue per user, time on page), a different sample size formula based on means and standard deviation would be required.

Q8: How often should I use an Optimizely Sample Size Calculator?

A: You should use an Optimizely Sample Size Calculator before launching *every* A/B test. It’s a fundamental step in experiment design to ensure your tests are properly powered and your results are trustworthy. Recalculate if your baseline metrics or MDE expectations change significantly.

Related Tools and Internal Resources

Explore other valuable tools and guides to enhance your experimentation and conversion rate optimization efforts:

• A/B Testing Calculator: A general calculator for analyzing A/B test results and determining statistical significance.
• Conversion Rate Optimization Guide: A comprehensive guide to improving your website’s conversion rates.
• Statistical Significance Checker: Verify the significance of your experiment results post-test.
• Experiment Duration Estimator: Estimate how long your tests will need to run based on traffic and desired sample size.
• Power Analysis Tool: Dive deeper into statistical power and its impact on experiment design.
• MDE Calculator: Understand and calculate your Minimum Detectable Effect more precisely.