This calculator determines if there's a significant difference between groups without assuming your data follows any specific distribution. Unlike t-tests or ANOVA that require normality assumptions, this test works by randomly shuffling your data thousands of times to create a null distribution, making it ideal for small samples, non-normal data, or data with outliers.
When to use this test: Choose permutation tests when you have small samples (n < 30 per group), non-normal data, outliers, or when parametric assumptions are questionable. For large samples with normally distributed data, parametric tests (t-testorANOVA) may be more powerful.
Ready to test your groups? to see how it works, or upload your own data to get distribution-free, exact results without worrying about statistical assumptions.
Setting a seed ensures reproducible results across multiple runs
Permutation Test is a non-parametric statistical method used to determine if there's a significant difference between groups by randomly shuffling (permuting) the data between groups many times to create a null distribution, then comparing the observed difference to this distribution.
Permutation tests are particularly useful in these situations:
This could be a difference in means, medians, or any other statistic of interest.
For a two-sided test, we count how many permuted statistics are as or more extreme than the observed statistic.
| Group A | Group B |
|---|---|
| 75 | 85 |
| 72 | 86 |
| 80 | 83 |
| 78 | 87 |
| 76 | 84 |
Combine all data: 75, 72, 80, 78, 76, 85, 86, 83, 87, 84
Randomly shuffle and split into groups many times (10,000 permutations)
Calculate the difference for each permutation
In our example, if we assumed that we found 19 out of 10,000 permutations where the absolute difference was greater than or equal to 8.8, then the p-value would be:
p-value = 19/10,000 = 0.0019
Since the p-value (0.0019) is less than our significance level (0.05), we reject the null hypothesis. There is statistically significant evidence to conclude that the groups differ.
Cohen's d can be used to measure effect size:
where
Guidelines:
For our example:which indicates a very large effect.
library(tidyverse)
set.seed(42)
group1 <- c(75, 72, 80, 78, 76)
group2 <- c(85, 86, 83, 87, 84)
observed_diff <- mean(group2) - mean(group1)
print(str_glue("Observed difference: {observed_diff}"))
combined <- c(group1, group2)
n1 <- length(group1)
n <- length(combined)
n_perm <- 10000
perm_diffs <- numeric(n_perm)
for (i in 1:n_perm) {
perm <- sample(combined, n, replace = FALSE)
perm_group1 <- perm[1:n1]
perm_group2 <- perm[(n1+1):n]
perm_diffs[i] <- mean(perm_group2) - mean(perm_group1)
}
p_value <- mean(abs(perm_diffs) >= abs(observed_diff))
print(str_glue("Permutation test p-value: {p_value}"))
# plot permuted differences with observed difference
ggplot(data.frame(perm_diffs), aes(x = perm_diffs)) +
geom_histogram(aes(y = after_stat(density)), bins = 30, fill = "lightblue", color = "black") +
geom_vline(aes(xintercept = observed_diff), color = "red", linetype = "dashed", linewidth = 1) +
geom_vline(aes(xintercept = -observed_diff), color = "red", linetype = "dashed", linewidth = 1) +
labs(title = "Permutation Test: Distribution of Permuted Differences",
x = "Difference in Means",
y = "Density") +
theme_minimal()import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
np.random.seed(42)
group1 = np.array([75, 72, 80, 78, 76])
group2 = np.array([85, 86, 83, 87, 84])
# observed difference
observed_diff = np.mean(group2) - np.mean(group1)
print(f"Observed difference: {observed_diff}")
combined = np.concatenate([group1, group2])
n1 = len(group1)
n = len(combined)
n_perm = 10000
perm_diffs = np.zeros(n_perm)
for i in range(n_perm):
# Randomly permute the combined data
perm = np.random.permutation(combined)
# Split into two groups of original sizes
perm_group1 = perm[:n1]
perm_group2 = perm[n1:]
# Calculate and store the difference in means
perm_diffs[i] = np.mean(perm_group2) - np.mean(perm_group1)
# p-value
p_value = np.mean(np.abs(perm_diffs) >= np.abs(observed_diff))
print(f"Permutation test p-value: {p_value}")
# plot the permuted differences with observed difference
plt.figure(figsize=(10, 6))
sns.histplot(perm_diffs, kde=True, color='lightblue', edgecolor='black')
plt.axvline(x=observed_diff, color='red', linestyle='dashed', linewidth=2, label='Observed difference')
plt.axvline(x=-observed_diff, color='red', linestyle='dashed', linewidth=2)
plt.title('Permutation Test: Distribution of Permuted Differences')
plt.xlabel('Difference in Means')
plt.ylabel('Density')
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()How permutation tests compare to other common statistical methods:
This calculator determines if there's a significant difference between groups without assuming your data follows any specific distribution. Unlike t-tests or ANOVA that require normality assumptions, this test works by randomly shuffling your data thousands of times to create a null distribution, making it ideal for small samples, non-normal data, or data with outliers.
When to use this test: Choose permutation tests when you have small samples (n < 30 per group), non-normal data, outliers, or when parametric assumptions are questionable. For large samples with normally distributed data, parametric tests (t-testorANOVA) may be more powerful.
Ready to test your groups? to see how it works, or upload your own data to get distribution-free, exact results without worrying about statistical assumptions.
Setting a seed ensures reproducible results across multiple runs
Permutation Test is a non-parametric statistical method used to determine if there's a significant difference between groups by randomly shuffling (permuting) the data between groups many times to create a null distribution, then comparing the observed difference to this distribution.
Permutation tests are particularly useful in these situations:
This could be a difference in means, medians, or any other statistic of interest.
For a two-sided test, we count how many permuted statistics are as or more extreme than the observed statistic.
| Group A | Group B |
|---|---|
| 75 | 85 |
| 72 | 86 |
| 80 | 83 |
| 78 | 87 |
| 76 | 84 |
Combine all data: 75, 72, 80, 78, 76, 85, 86, 83, 87, 84
Randomly shuffle and split into groups many times (10,000 permutations)
Calculate the difference for each permutation
In our example, if we assumed that we found 19 out of 10,000 permutations where the absolute difference was greater than or equal to 8.8, then the p-value would be:
p-value = 19/10,000 = 0.0019
Since the p-value (0.0019) is less than our significance level (0.05), we reject the null hypothesis. There is statistically significant evidence to conclude that the groups differ.
Cohen's d can be used to measure effect size:
where
Guidelines:
For our example:which indicates a very large effect.
library(tidyverse)
set.seed(42)
group1 <- c(75, 72, 80, 78, 76)
group2 <- c(85, 86, 83, 87, 84)
observed_diff <- mean(group2) - mean(group1)
print(str_glue("Observed difference: {observed_diff}"))
combined <- c(group1, group2)
n1 <- length(group1)
n <- length(combined)
n_perm <- 10000
perm_diffs <- numeric(n_perm)
for (i in 1:n_perm) {
perm <- sample(combined, n, replace = FALSE)
perm_group1 <- perm[1:n1]
perm_group2 <- perm[(n1+1):n]
perm_diffs[i] <- mean(perm_group2) - mean(perm_group1)
}
p_value <- mean(abs(perm_diffs) >= abs(observed_diff))
print(str_glue("Permutation test p-value: {p_value}"))
# plot permuted differences with observed difference
ggplot(data.frame(perm_diffs), aes(x = perm_diffs)) +
geom_histogram(aes(y = after_stat(density)), bins = 30, fill = "lightblue", color = "black") +
geom_vline(aes(xintercept = observed_diff), color = "red", linetype = "dashed", linewidth = 1) +
geom_vline(aes(xintercept = -observed_diff), color = "red", linetype = "dashed", linewidth = 1) +
labs(title = "Permutation Test: Distribution of Permuted Differences",
x = "Difference in Means",
y = "Density") +
theme_minimal()import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
np.random.seed(42)
group1 = np.array([75, 72, 80, 78, 76])
group2 = np.array([85, 86, 83, 87, 84])
# observed difference
observed_diff = np.mean(group2) - np.mean(group1)
print(f"Observed difference: {observed_diff}")
combined = np.concatenate([group1, group2])
n1 = len(group1)
n = len(combined)
n_perm = 10000
perm_diffs = np.zeros(n_perm)
for i in range(n_perm):
# Randomly permute the combined data
perm = np.random.permutation(combined)
# Split into two groups of original sizes
perm_group1 = perm[:n1]
perm_group2 = perm[n1:]
# Calculate and store the difference in means
perm_diffs[i] = np.mean(perm_group2) - np.mean(perm_group1)
# p-value
p_value = np.mean(np.abs(perm_diffs) >= np.abs(observed_diff))
print(f"Permutation test p-value: {p_value}")
# plot the permuted differences with observed difference
plt.figure(figsize=(10, 6))
sns.histplot(perm_diffs, kde=True, color='lightblue', edgecolor='black')
plt.axvline(x=observed_diff, color='red', linestyle='dashed', linewidth=2, label='Observed difference')
plt.axvline(x=-observed_diff, color='red', linestyle='dashed', linewidth=2)
plt.title('Permutation Test: Distribution of Permuted Differences')
plt.xlabel('Difference in Means')
plt.ylabel('Density')
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()How permutation tests compare to other common statistical methods: