Why your customer surveys mislead you: the people who left don't answer
In 1936, a magazine polled 2.4 million Americans and called the election for the wrong man. George Gallup asked about 50,000 and got it right.
The Literary Digest mailed ten million ballots to its own readers, car owners and telephone users. In the middle of the Depression, that list leaned rich, and rich leaned Landon. The Digest called it for Landon. Roosevelt won every state except Maine and Vermont.
More answers from the wrong people just makes you wrong with more confidence.
The magazine folded within 18 months. Polling your own subscribers turned out to be bad for subscribers.
You are polling the phone book
Your NPS score, your app-store reviews, your customer survey: every one of them comes from people who stayed. The ones who churned do not fill in forms.
So the feedback you collect describes the users your product already works for. The users it failed, the ones whose answers would actually change the roadmap, are missing from the sample.
It gets worse inside your own data
The same problem shows up in analytics. Before you read any cohort, check who is in it: test accounts, team members, fraud and bots can make up a surprising share of a small sample. A cohort read on the wrong people is a confident answer to a question about someone else.
Who to ask instead
- Talk to the users who cancelled last month. Not a survey link: a short call or a two-question email.
- Keep going until their answers start repeating. That is usually when you have heard the main reasons.
- Ask about the moment, not the opinion. "What were you trying to do when you stopped using it?" beats "How satisfied were you?"
- Compare with your best retained users. What did they do in their first session that the churned users did not?
- Clean the sample before you read any cohort. Remove internal, test and flagged accounts first.
Bottom line
Stop counting answers. Start choosing who you ask.
Sources: The Literary Digest.
FAQ
Why is NPS misleading?
NPS only reflects the users who are still around to answer. Users who churned rarely respond, so the score describes the people the product already works for and misses why others left.
What is survivorship bias in customer research?
It is drawing conclusions only from the customers who stayed, while ignoring those who left. Because the ones who left often had different needs or problems, the result can point the roadmap in the wrong direction.
How do I get feedback from churned users?
Reach out soon after they cancel with a short, personal message or a brief call. Ask what they were trying to do and what made them stop. Keep going until the reasons start repeating.
How many churn interviews do I need?
There is no fixed number. Keep talking to churned users until their answers start repeating, which often happens after a modest number of conversations.