Wikipedia Deep Dive

Participation bias

11 min read

The Poll That Predicted a Landslide—for the Wrong Candidate

In 1936, The Literary Digest conducted one of the most ambitious polling efforts in American history. They mailed out ten million questionnaires asking Americans who they planned to vote for in the upcoming presidential election. When 2.38 million responses flooded back—an enormous sample by any standard—the magazine confidently predicted that Republican Alf Landon would crush the incumbent Franklin D. Roosevelt, winning 57 percent of the popular vote and 370 electoral votes.

They were catastrophically wrong.

Roosevelt won in one of the most lopsided victories in American electoral history. Landon received just 37 percent of the vote and carried only two states, collecting a mere eight electoral votes. The Literary Digest's reputation never recovered, and the magazine folded within two years.

What went wrong? The answer reveals one of the most insidious problems in gathering information about any group of people: participation bias, also called non-response bias. The people who actually fill out surveys, answer polls, or show up to studies are often systematically different from the people who don't. And those differences can completely distort your results.

Why the People Who Answer Aren't Like the People Who Don't

The Literary Digest's mailing list came from automobile registrations and telephone directories. In 1936, during the depths of the Great Depression, these were markers of relative affluence. Poorer Americans—who overwhelmingly supported Roosevelt's New Deal—were underrepresented from the start.

But the bigger problem was who actually mailed back the questionnaires. Research published decades later, in 1976 and 1988, concluded that non-response bias was the primary culprit. Even among the people who received the survey, those who took the time to respond weren't a random cross-section. They were disproportionately the kind of people who fill out and return political surveys—which correlated with the kind of people who supported Landon.

This pattern appears everywhere researchers look. A study on AIDS-related surveys found that people who refused to participate tended to be older, more religious, more skeptical about survey confidentiality, and less willing to discuss sexual topics. They weren't just randomly busy or forgetful. They had specific characteristics that shaped their decision not to participate—characteristics that also related to the very things the survey was trying to measure.

The Workload Paradox

Consider a deceptively simple scenario. You want to know how much work managers at a company are actually doing, so you send out a survey asking about their workload. Seems straightforward enough.

But think about who will actually respond.

Managers drowning in work may not have time to fill out yet another survey. Their heavy workload literally prevents them from reporting their heavy workload. Meanwhile, managers with light workloads might avoid responding too—what if their bosses find out they're not that busy? Even if the survey claims to be anonymous, anonymity can be compromised, and people know it.

So your results could skew low, because the overworked people didn't respond. Or they could skew high, because the underworked people stayed quiet. Or, in a cosmic coincidence, these two effects might cancel out and give you the right answer for entirely wrong reasons.

There's a certain dark humor in imagining the perfect survey question that captures this problem: "Agree or disagree: I have enough time in my day to complete a survey." Anyone who disagrees probably won't be around to tell you.

The Dropout Problem in Long-Term Studies

Participation bias becomes especially troublesome in longitudinal research—studies that follow the same people over months, years, or even decades. These studies are invaluable for understanding how things change over time, from the progression of diseases to the long-term effects of educational interventions.

But people drop out. They move, they lose interest, they get sick, they die. And crucially, they don't drop out randomly.

In medical research, younger patients drop out more often. So do people from poorer communities. And so do people who are less satisfied with their treatment—which is a significant problem if you're trying to measure whether a treatment actually works. The people who stay in your study long enough to give you data may be precisely the ones for whom things are going well.

This creates a survivor bias layered on top of participation bias. You're not just missing the people who never showed up. You're systematically losing the people whose experiences might contradict your emerging conclusions.

Testing Whether You Have a Problem

Researchers have developed several techniques to detect non-response bias, though none are perfect.

One common approach is to compare early responders to late responders. The theory is that people who answer right away might be different from people who need multiple reminders, and those who barely respond at all might be more similar to those who never respond. If you find significant differences between your first and fourth quartiles of responses, that's a warning sign.

Another technique works when you already know something about everyone in your sample before they respond. In email surveys sent within a company, for instance, you might know each person's age, department, and tenure. You can then compare these characteristics between responders and non-responders. If the groups look similar on these known variables, it's at least suggestive that they might be similar on unknown variables too. Suggestive, but not certain.

The most direct approach is to chase down the people who didn't respond and ask them a few key questions by phone. If their answers don't differ significantly from the people who completed the full survey, you might be in the clear. Researchers call this "non-response follow-up," and it's one of the few ways to actually peek behind the curtain at what you're missing.

The Response Rate Myth

For decades, researchers operated under a seemingly reasonable assumption: higher response rates mean less bias. If you can get 80 percent of your sample to respond instead of 40 percent, surely you're capturing a more representative picture. Right?

The evidence doesn't support this intuition nearly as strongly as you might expect.

Robert M. Groves conducted a meta-analysis—a study of studies—examining 30 methodological investigations into non-response bias. He found that response rate explained only about 11 percent of the variation in non-response bias. That's a remarkably weak relationship. You could have a 70 percent response rate and still have significant bias, or a 30 percent response rate with relatively little.

Another meta-analysis of 44 studies delivered an even more counterintuitive finding. Methods designed to boost response rates—like sending advance notifications or offering incentives—didn't necessarily reduce non-response bias. Sometimes they even increased it. Paying people to respond might bring in more respondents, but it might disproportionately attract people who are motivated by small payments, which introduces its own systematic skew.

The Sixty Percent Rule and Its Discontents

Despite this shaky relationship between response rates and bias, many academic journals have established minimum response rate thresholds for publishing survey research. The Journal of the American Medical Association, known as JAMA, requires a 60 percent response rate. The reasoning is straightforward: we need some standard, and 60 percent seems rigorous enough to matter.

Critics compare this to the widespread use of 0.05 as the threshold for statistical significance. In both cases, a somewhat arbitrary number has become a gatekeeping mechanism that researchers must satisfy, regardless of whether it actually indicates what we care about. A study with a 59 percent response rate gets rejected; one with 61 percent gets through. Yet the actual risk of bias in each case might be identical—or even reversed.

The pursuit of higher response rates can become counterproductive. Organizations spend substantial resources on reminder mailings, incentive payments, and follow-up calls, all to push their response rate above some magic threshold. Those resources might be better spent on other aspects of research quality. Worse, valid and valuable surveys sometimes get rejected not because they're actually biased, but because they fail to satisfy a heuristic that doesn't reliably measure bias in the first place.

Cousins and Impostors: Related Types of Bias

Participation bias has several relatives in the taxonomy of research problems, and they're easy to confuse.

Self-selection bias is a close cousin. It occurs whenever people voluntarily sort themselves into groups. If you're studying the effects of a new exercise program, and you let people choose whether to join, the joiners might already be more health-conscious, more motivated, or more optimistic than non-joiners. Any benefits you observe might reflect these pre-existing differences rather than the program itself. The act of choosing to participate is itself informative—and potentially distorting.

Response bias sounds like it should be the opposite of non-response bias, but it's actually something different entirely. Response bias refers to systematic inaccuracies in the answers people give, not in whether they answer at all. People might tell you what they think you want to hear, give answers that make them look good, or misremember past events in ways that fit their current beliefs. A person with response bias showed up and participated—they just didn't participate accurately.

Selection bias is the broader category encompassing all of these problems. Whenever your sample isn't truly representative of your target population, for any reason, you have selection bias. Non-response bias is just one path to this destination. You might also have a flawed sampling frame (like the Literary Digest's reliance on car owners and phone subscribers), or coverage errors where some people in your target population have no chance of being sampled at all.

Why This Matters Beyond Academia

Participation bias isn't just a methodological headache for researchers. It shapes public discourse and policy in ways we rarely notice.

Online reviews suffer from it badly. The people who leave reviews on Amazon or Yelp are disproportionately those with very positive or very negative experiences. The satisfied-but-unremarkable middle rarely bothers to write anything. This creates a bimodal distribution of opinions—five stars or one star—that doesn't reflect the actual distribution of customer experiences.

Political polling wrestles with it constantly. The voters who answer unknown numbers and patiently respond to a pollster's questions are not the same as voters who hang up or don't answer at all. Pollsters use sophisticated weighting techniques to adjust for known demographic differences, but they can't adjust for unknown ones. Some analysts believe that differential non-response helps explain why polls have underestimated certain candidates in recent elections.

Employee satisfaction surveys at companies face the workload paradox and more. The most disengaged employees might not bother responding—or might fear retaliation despite promises of anonymity. The results then look rosier than reality, leading management to believe things are fine when they're not.

Even medical research on treatment effectiveness can be skewed. If sicker patients drop out of clinical trials because they're too ill to continue, the remaining sample looks healthier than it should. The treatment appears more effective than it actually is.

Living with Imperfect Information

There's no perfect solution to participation bias. You can't force people to respond to surveys, nor would it be ethical to try. You can minimize the problem through good study design, careful follow-up, and honest analysis of who might be missing from your data. But you can't eliminate it.

What you can do is stay aware of it. When you see a poll result or a study finding, ask yourself: who actually participated in this? What kind of person would have taken the time to respond? What kind of person would have refused, and how might their views differ?

The Literary Digest's spectacular failure in 1936 taught researchers a lesson they're still learning. The size of your sample matters far less than its representativeness. Two million responses mean nothing if they're systematically skewed. A smaller, more carefully selected sample—like the one George Gallup used that same year to correctly predict Roosevelt's victory—can outperform a massive but biased one.

In an age of ubiquitous online surveys and instant polls, participation bias is arguably more relevant than ever. It's easier than ever to gather responses. It's harder than ever to know if those responses represent anything beyond the kind of people who answer online surveys.

The next time someone cites a statistic about what "people" think or want or do, remember: they're really telling you what participating people think, want, or do. And participating people are never quite like everyone else.