Hypothesis Testing Simulation

Simulations

Hypothesis Testing Simulation

Parameters

Set the null hypothesis, the true population mean, and see how often we reject H₀

Null Hypothesis Mean (μ₀)

True Population Mean (μ)

Population Std Dev (σ)

Sample Size (n)

Significance Level (α)

Tail Type

Number of Tests

Scenario: H₀: μ = 0 vs H₁: μ ≠ 0. The null hypothesis is FALSE (true μ = 0.5 ≠ 0). Failures to reject are Type II errors.

Learn More

Type I and Type II Errors

	H₀ is True	H₀ is False
Reject H₀	Type I Error (α)	Correct (Power = 1 − β)
Fail to Reject H₀	Correct (1 − α)	Type II Error (β)

Set the true mean equal to the null mean to observe Type I errors. Set them differently to observe power and Type II errors.

Statistical Power

Power is the probability of correctly rejecting a false null hypothesis. It depends on three factors:

Effect size (|μ − μ₀|/σ): larger effects are easier to detect
Sample size (n): more data gives more power
Significance level (α): a larger α increases power but also increases Type I error risk

\text{Power} = P(\text{Reject } H_0 \mid H_0 \text{ is false})= 1 - \beta

For a z-test with known σ, the power for a two-tailed test is:

\text{Power} = 1 - \Phi\left(z_{\alpha/2} - \frac{\mu - \mu_0}{\sigma/\sqrt{n}}\right) + \Phi\left(-z_{\alpha/2} - \frac{\mu - \mu_0}{\sigma/\sqrt{n}}\right)

P-Value Distribution

A key insight from this simulation:

When H₀ is true: p-values follow a uniform distribution between 0 and 1. This means exactly α proportion of p-values fall below α by chance.
When H₀ is false: p-values pile up near 0. The stronger the effect, the more concentrated they become near 0.

This is why the p-value histogram is one of the most important diagnostic plots in statistics — it reveals whether there is a signal or just noise.

Things to Try

Set μ = μ₀ (null is true) and run 1,000 tests — the rejection rate should be close to α
Gradually increase the difference |μ − μ₀| and watch the power increase
Keep the effect fixed and increase sample size — see how power grows
Compare α = 0.01 vs α = 0.10 — see the trade-off between Type I and Type II errors
Compare one-tailed vs two-tailed tests — one-tailed has more power in the specified direction

Related Calculators

P-Value Calculator

Sample Size & Power Analysis Calculator

One Sample Z-Test Calculator

P-Value Simulation

Hypothesis Testing Simulation

Simulations