When to stop an A/B test: don't call a winner early
Calling an A/B test early is a coin toss with a dashboard.
Flip a fair coin three times. Three heads in a row happens once in every eight tries, on average. 100% heads. Beautiful dashboard. Still a fair coin.
A test that was worth waiting for
I worked as fractional CMO on a coin game where we A/B tested a scripted user journey through the tutorial, the first-time user experience every new player sees.
The result: overall ROAS improved by 14.5%.
A change inside the first few minutes of the product changed the return on the money spent acquiring players. That is why I care so much about what happens after the install. It is also why I want more than an early lead before calling a test.
The trap inside "wait for significance"
"Wait for statistical significance" sounds like the safe rule. It can become another way to fool yourself.
If you check the result every day and stop the moment it crosses the line, you will call false winners far more often than the significance level suggests. Every peek is another flip of the coin. Keep flipping and sooner or later you get three heads.
The rules I use
For a fixed-sample test:
- Set the sample size and the decision rule before launch. Write down the metric, the minimum effect worth acting on, and how many users each group needs.
- Do not stop early because you like the answer. Stop when the planned sample is reached.
- Give both groups the same time to generate revenue. A ROAS test needs the same payback window for each group. Comparing a 14-day cohort with a 7-day cohort is not a test.
If you need to decide continuously, use a method built for that, such as a sequential test, instead of peeking at a fixed-sample test.
Why this matters more for FTUE tests
Onboarding and tutorial changes affect every new user, so a wrong call spreads across all future cohorts and all future ad spend. The cost of a false winner is not one bad feature. It is a worse first session for everyone you pay to acquire.
Bottom line
I want faster learning. I also want the result to survive beyond the screenshot.
Three heads can win a coin toss. Make your experiment earn the rollout.
FAQ
How long should I run an A/B test?
Until it reaches the sample size you calculated before launch, and long enough to cover full weekly cycles. For revenue or ROAS tests, both groups also need the same amount of time to generate revenue.
Why is it wrong to stop an A/B test when it reaches significance?
Checking repeatedly and stopping at the first significant result inflates false positives. Each check is another chance for random noise to look like a win.
What should I A/B test first in a mobile game?
The first-time user experience is often the highest-impact area, because it affects every new player. In one coin game, a scripted tutorial journey improved overall ROAS by 14.5%.
What if I need to make decisions before the test ends?
Use a sequential testing method designed for continuous monitoring, rather than peeking at a fixed-sample test.