A/B Testing Calculator

Created:January 8, 2026

Last Updated:May 9, 2026

This comprehensive A/B testing calculator provides both Frequentist and Bayesian analysis for your experiments. Analyze conversion rates or continuous metrics, calculate required sample sizes, and make data-driven decisions with confidence.

Whether you're testing website conversions, email click-through rates, or average order values, this calculator helps you determine statistical significance and practical impact of your experiments.

Calculator

Parameters

Mode

Metric Type

Control Group (A)

Sample Size

Conversions

Treatment Group (B)

Sample Size

Conversions

Significance Level (α)

Results

Configure parameters and click calculate to see results

Need to cite or share this calculator?

Get properly formatted citations for academic work •20k+ calculations in the past 30 days

APA 7th Edition • Most Common

StatsCalculators Team. (2026). A/B Testing Calculator. StatsCalculators. Retrieved May 20, 2026 from https://statscalculators.com/calculators/experimental-design/ab-testing-calculator

Learn More: A/B Testing Guide

What is A/B Testing?

A/B testing (also known as split testing) is a method of comparing two versions of a webpage, app feature, or other product element to determine which performs better. By randomly showing version A to some users and version B to others, you can statistically determine which version leads to better outcomes.

Key Components of A/B Testing:

Control (A): The current version or baseline
Treatment (B): The new version being tested
Metric: The outcome you're measuring (e.g., conversion rate, click-through rate, revenue)
Sample Size: Number of users in each group
Statistical Significance: Confidence that the results aren't due to chance

Frequentist vs Bayesian Approaches

Frequentist Approach

Key Concepts:

P-value: Probability of observing results if no real effect exists
Confidence intervals: Range where true value likely lies
Fixed sample size required
Binary decision (significant or not)

Advantages:

Well-established and widely accepted
Clear decision threshold (α level)
Standard in academic research

Limitations:

Doesn't provide probability of hypothesis
Sensitive to peeking at results
Binary decision can be inflexible

Bayesian Approach

Key Concepts:

Probability that B beats A
Credible intervals: Range where value lies with X% probability
Can peek at results safely
Continuous probability estimate

Advantages:

Direct probability interpretation
More intuitive for decision-making
Accounts for prior knowledge
Flexible stopping rules

Limitations:

Requires understanding of priors
Less standardized in practice
Can be computationally intensive

Formulas and Examples

Conversion Rate (Proportion) Test

Frequentist Formula:

Test statistic:

z = \frac{p_2 - p_1}{\sqrt{\hat{p}(1-\hat{p})(\frac{1}{n_1} + \frac{1}{n_2})}}

where:

$p_1, p_2$ = observed proportions
$\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}$ = pooled proportion
$n_1, n_2$ = sample sizes

Bayesian Formula:

Posterior distribution:

p_1 | \text{data} \sim \text{Beta}(\alpha_0 + x_1, \beta_0 + n_1 - x_1)

p_2 | \text{data} \sim \text{Beta}(\alpha_0 + x_2, \beta_0 + n_2 - x_2)

With uniform prior: $\alpha_0 = \beta_0 = 1$

Example:

Website redesign test with 1000 visitors per variant:

Control (A): 100 conversions (10% rate)
Treatment (B): 120 conversions (12% rate)
Relative lift: 20%

Frequentist: p = 0.153 (not significant at α = 0.05)

Bayesian:92.3% probability that B > A

Sample Size Calculation

Formula:

n = \frac{(z_{\alpha/2} + z_\beta)^2 [p_1(1-p_1) + p_2(1-p_2)]}{(p_2 - p_1)^2}

where:

$z_{\alpha/2}$ = critical value for significance level (1.96 for α = 0.05)
$z_\beta$ = critical value for power (0.84 for 80% power)
$p_1$ = baseline conversion rate
$p_2$ = target conversion rate

Example Calculation:

Parameters:

Baseline rate: 10%
Minimum detectable effect: 20% relative lift
Target rate: 12%
Significance level: α = 0.05
Power: 80%

Required sample size: 3,841 per group (7,682 total)

A/B test sample size calculator setup showing the inputs used to get n equals 3841 per group for the example calculation. — Calculator setup for the example above: 10% baseline, 0.02 effect size, 80% power, and α = 0.05 gives 3,841 users per group.

Verification

A/B Testing Calculator

Created:January 8, 2026

Last Updated:May 9, 2026

Whether you're testing website conversions, email click-through rates, or average order values, this calculator helps you determine statistical significance and practical impact of your experiments.

Calculator

Parameters

Mode

Metric Type

Control Group (A)

Sample Size

Conversions

Treatment Group (B)

Sample Size

Conversions

Significance Level (α)

Results

Configure parameters and click calculate to see results

Need to cite or share this calculator?

Get properly formatted citations for academic work •20k+ calculations in the past 30 days

APA 7th Edition • Most Common

StatsCalculators Team. (2026). A/B Testing Calculator. StatsCalculators. Retrieved May 20, 2026 from https://statscalculators.com/calculators/experimental-design/ab-testing-calculator

Learn More: A/B Testing Guide

What is A/B Testing?

Key Components of A/B Testing:

Control (A): The current version or baseline
Treatment (B): The new version being tested
Metric: The outcome you're measuring (e.g., conversion rate, click-through rate, revenue)
Sample Size: Number of users in each group
Statistical Significance: Confidence that the results aren't due to chance

Frequentist vs Bayesian Approaches

Frequentist Approach

Key Concepts:

P-value: Probability of observing results if no real effect exists
Confidence intervals: Range where true value likely lies
Fixed sample size required
Binary decision (significant or not)

Advantages:

Well-established and widely accepted
Clear decision threshold (α level)
Standard in academic research

Limitations:

Doesn't provide probability of hypothesis
Sensitive to peeking at results
Binary decision can be inflexible

Bayesian Approach

Key Concepts:

Probability that B beats A
Credible intervals: Range where value lies with X% probability
Can peek at results safely
Continuous probability estimate

Advantages:

Direct probability interpretation
More intuitive for decision-making
Accounts for prior knowledge
Flexible stopping rules

Limitations:

Requires understanding of priors
Less standardized in practice
Can be computationally intensive

Formulas and Examples

Conversion Rate (Proportion) Test

Frequentist Formula:

Test statistic:

z = \frac{p_2 - p_1}{\sqrt{\hat{p}(1-\hat{p})(\frac{1}{n_1} + \frac{1}{n_2})}}

where:

$p_1, p_2$ = observed proportions
$\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}$ = pooled proportion
$n_1, n_2$ = sample sizes

Bayesian Formula:

Posterior distribution:

p_1 | \text{data} \sim \text{Beta}(\alpha_0 + x_1, \beta_0 + n_1 - x_1)

p_2 | \text{data} \sim \text{Beta}(\alpha_0 + x_2, \beta_0 + n_2 - x_2)

With uniform prior: $\alpha_0 = \beta_0 = 1$

Example:

Website redesign test with 1000 visitors per variant:

Control (A): 100 conversions (10% rate)
Treatment (B): 120 conversions (12% rate)
Relative lift: 20%

Frequentist: p = 0.153 (not significant at α = 0.05)

Bayesian:92.3% probability that B > A

Sample Size Calculation

Formula:

n = \frac{(z_{\alpha/2} + z_\beta)^2 [p_1(1-p_1) + p_2(1-p_2)]}{(p_2 - p_1)^2}

where:

$z_{\alpha/2}$ = critical value for significance level (1.96 for α = 0.05)
$z_\beta$ = critical value for power (0.84 for 80% power)
$p_1$ = baseline conversion rate
$p_2$ = target conversion rate

Example Calculation:

Parameters:

Baseline rate: 10%
Minimum detectable effect: 20% relative lift
Target rate: 12%
Significance level: α = 0.05
Power: 80%

Required sample size: 3,841 per group (7,682 total)

A/B Testing Calculator

Calculator

Parameters

Control Group (A)

Treatment Group (B)

Results

Need to cite or share this calculator?

Related Calculators

Sample Size & Power Analysis Calculator

Two Proportion Z-Test Calculator

Difference in Proportions CI Calculator

Randomization Calculator

Learn More: A/B Testing Guide

What is A/B Testing?

Key Components of A/B Testing:

Frequentist vs Bayesian Approaches

Frequentist Approach

Key Concepts:

Advantages:

Limitations:

Bayesian Approach

Key Concepts:

Advantages:

Limitations:

Formulas and Examples

Conversion Rate (Proportion) Test

Frequentist Formula:

Bayesian Formula:

Example:

Sample Size Calculation

Formula:

Example Calculation:

Verification

View Verification Details

A/B Testing Calculator

Calculator

Parameters

Control Group (A)

Treatment Group (B)

Results

Need to cite or share this calculator?

Related Calculators

Sample Size & Power Analysis Calculator

Two Proportion Z-Test Calculator

Difference in Proportions CI Calculator

Randomization Calculator

Learn More: A/B Testing Guide

What is A/B Testing?

Key Components of A/B Testing:

Frequentist vs Bayesian Approaches

Frequentist Approach

Key Concepts:

Advantages:

Limitations:

Bayesian Approach

Key Concepts:

Advantages:

Limitations:

Formulas and Examples

Conversion Rate (Proportion) Test

Frequentist Formula:

Bayesian Formula:

Example:

Sample Size Calculation

Formula:

Example Calculation:

Verification

View Verification Details