Why You Shouldn't Peek at Your A/B Test

Each experiment sent 10,000 visitors to each of two identical variants converting at 5%, and analyzed the running results with a two-proportion z-test at evenly spaced checkpoints, stopping the moment the test reached significance. Because the variants are identical, every "winner" is a false positive.

10,000 simulated A/A experiments per condition, seeded and reproducible. Peeking 50 times on a 10,000-visitor test means checking roughly every 200 visitors — about what "glancing at the dashboard every day" looks like for a typical two-week test.

A significance level of α = 0.05 means: if you run the test once, there is a 5% chance of a false positive. Each additional look is another chance for noise to cross the threshold, and a stop-at-significance rule keeps every lucky crossing while ignoring the recovery that would have followed.

• The p-value of a running test is not a settled fact — it wanders as data arrives (watch the trajectory chart above), and early looks are the noisiest because samples are small
• Consecutive looks are correlated, so k peeks inflate the false-positive rate by less than k × α — but the rate still climbs without bound as looks accumulate
• With continuous monitoring and no sample-size limit, a "significant" result is eventually guaranteed, even when no difference exists
• This is the same mechanism as testing many metrics or segments at once — multiple comparisons, spread over time instead of across variables

• Fix the sample size in advance. Use a power analysis to compute how many visitors you need, run until you reach that number, and analyze once. This is the simplest fully valid procedure.
• Use a group-sequential design.Methods like Pocock and O'Brien–Fleming let you schedule a few interim looks and spend your α across them with stricter per-look thresholds, keeping the overall false-positive rate at 5%.
• Use always-valid inference. Sequential methods such as the mSPRT (used by several commercial A/B platforms) produce p-values that remain valid under continuous monitoring, at the cost of needing more data for the same power.
• Peeking to check for bugs is fine.Look at your data quality any time — just don't make the stop-or-continue decision based on significance unless your method was built for it.

Note: Bayesian dashboards are not automatically immune — stopping the moment a posterior probability crosses a fixed threshold has an optional-stopping problem of its own.