Opinion poll

Based on Wikipedia: Opinion poll

The Magazine That Destroyed Itself by Asking the Wrong People

In 1936, The Literary Digest magazine was riding high. For two decades, it had correctly predicted every presidential election winner by mailing out millions of postcards and counting the returns. Woodrow Wilson, Warren Harding, Calvin Coolidge, Herbert Hoover, Franklin Roosevelt—the magazine had called them all. That year, they sent out a staggering 2.3 million postcards asking Americans who they planned to vote for.

The results came back clear: Alf Landon would defeat Roosevelt.

Roosevelt won in one of the largest landslides in American history, carrying 46 of 48 states. Within two years, The Literary Digest went out of business, its reputation destroyed by the most spectacular polling failure the country had ever seen.

What went wrong? The answer reveals everything you need to know about how opinion polls actually work—and why they can still go spectacularly wrong today.

The Birth of Scientific Polling

The first recorded opinion poll wasn't conducted by a research firm or a statistics department. It was a simple tally published in two newspapers—the Raleigh Star and North Carolina State Gazette, along with the Wilmington American Watchman and Delaware Advertiser—before the 1824 presidential election. The count showed Andrew Jackson leading John Quincy Adams by 335 votes to 169.

Jackson did win the popular vote that year, both in that state and nationally (though he lost the presidency when the House of Representatives chose Adams instead). These informal "straw votes" gradually caught on, but they remained local affairs, usually confined to a single city.

The Literary Digest changed that in 1916 when it launched a national survey. Partly, this was a circulation-building gimmick—respond to our poll and we'll send you a sample issue! But it worked. For twenty years, the magazine's simple approach of mailing postcards and counting returns produced accurate predictions.

The fatal flaw was hiding in plain sight the whole time.

Who Gets the Postcards Matters More Than How Many

The Literary Digest sent its postcards to names drawn from automobile registration lists and telephone directories. In 1936, owning a car or telephone still marked you as relatively affluent. These were people who had weathered the Great Depression better than most—and who were more likely to vote Republican.

But there was another problem, one that haunts polling to this day. George Gallup, an advertising executive who had started his own polling operation, analyzed what went wrong. He found that Landon supporters were simply more enthusiastic about returning their postcards. This is called participation bias, or sometimes response bias—the people who bother to respond to a survey aren't necessarily representative of the broader population.

Imagine you're running a poll about whether a city should build a new stadium. The people who feel strongly about it—either for or against—are far more likely to fill out your survey than people who are ambivalent. Your results will overrepresent passionate views on both sides and underrepresent the shrugging majority in the middle.

The Gallup Revolution

While The Literary Digest was mailing millions of postcards, three researchers—George Gallup, Archibald Crossley, and Elmo Roper—were taking a radically different approach. Instead of trying to reach as many people as possible, they focused on reaching the right people.

All three correctly predicted Roosevelt's victory in 1936. Their samples were far smaller than the Literary Digest's millions, but they were carefully constructed to represent the American population as a whole. This is the core insight of scientific polling: a small sample chosen correctly beats a huge sample chosen poorly.

It's counterintuitive. How can asking 1,000 people tell you anything about 100 million? The answer lies in one of the most elegant results in mathematics: the central limit theorem.

The Math Behind the Magic

Here's the basic idea. Imagine you have a jar containing millions of marbles, some red and some blue. You want to know what percentage are red without counting every single marble. So you reach in, grab a handful at random, and count the colors in your hand.

If you grab enough marbles—and this is the crucial part—grab them truly at random, the proportion in your hand will be very close to the true proportion in the jar. The math proves this works even when the jar contains millions of marbles and you've only grabbed a few hundred.

The technical way to say this: the distribution of sample proportions follows what statisticians call a binomial distribution, which approximates a normal distribution (the famous bell curve) as your sample gets larger. This means you can calculate exactly how confident you should be in your result.

Let's say you poll 1,000 randomly selected voters and 650 say they support Candidate A. You can calculate that you're 95% confident the true level of support falls somewhere between 62% and 68%. That range—plus or minus 3%—is your margin of error.

Why Sample Size Has Diminishing Returns

Here's something that surprises most people: to achieve a margin of error of plus or minus 3% with 95% confidence, you need roughly 1,000 respondents. It doesn't matter whether you're polling a city of 500,000 or a country of 300 million. The math is the same.

This is why national polls don't need millions of respondents. If you want that 3% margin of error, about 1,000 randomly selected people will do it. If you want to tighten the margin to 2%, you'll need roughly 2,400 respondents. For 1%, you'd need nearly 10,000. Each incremental improvement in precision requires exponentially more effort.

The catch, of course, is that word "randomly." True random sampling is extraordinarily difficult to achieve, and this is where modern polling struggles.

The Landline Problem

For decades, pollsters relied on a technique called random digit dialing. A computer would generate random phone numbers, and interviewers would call them, ensuring that every household with a telephone had an equal chance of being selected. This worked beautifully when most American households had landlines.

Then cell phones happened.

Today, more than half of American households have no landline at all. These households skew younger, more urban, and more Democratic. A poll that only reaches landlines will systematically undercount these groups—a problem eerily similar to The Literary Digest's reliance on automobile registrations and telephone directories.

Modern pollsters have adapted by calling cell phones too, but this creates new challenges. People are less likely to answer calls from unknown numbers on their cell phones. Response rates that once hovered around 30% have plummeted to single digits. This means pollsters are increasingly talking to a self-selected group: people willing to answer an unknown call and spend fifteen minutes answering questions.

Are these people representative of the broader population? The Literary Digest's ghost whispers a warning.

Weighting: The Art of Correction

To compensate for these sampling challenges, pollsters use a technique called weighting. If your sample contains too few young people compared to the general population, you give each young respondent's answers more weight in your calculations. If you have too many college graduates, you reduce their weight.

This works well for demographic factors you can measure, like age, education, and race. But what about factors you can't easily measure—like enthusiasm to participate in polls, or trust in institutions? These "hidden" biases may explain why polls in recent elections have systematically underestimated support for certain candidates.

Weighting also assumes you know what the true population looks like. In a presidential election, who exactly counts as a "likely voter"? This seemingly simple question involves complex modeling that different pollsters answer differently—and that can dramatically affect results.

How Polling Shaped America

The rise of scientific polling in the late 1930s coincided with a broader transformation in American capitalism. The Great Depression had devastated the advertising industry and shaken public faith in business. The New Deal promoted consumer protection and questioned whether advertising served any useful purpose.

Advertising executives like George Gallup—who served as vice president of the major agency Young & Rubicam—recognized that polls could rehabilitate their industry's reputation. If you could scientifically measure what consumers wanted, advertising wasn't manipulation; it was responding to genuine public preferences. The concept of "consumer sovereignty"—the idea that consumers, not producers, should drive the market—became advertising's defense against its critics.

This ideological work intensified during World War II, when the advertising industry threw itself into the war effort. Pollsters helped the government understand public opinion about everything from rationing to bond drives to attitudes toward allies. In the process, they helped define what the historian Jackson Lears calls "the American Way of Life"—a vision centered on free enterprise, consumer choice, and democratic participation measured through surveys.

By the end of the war, polling had become inseparable from both commerce and politics. Gallup's organization expanded internationally, launching a British subsidiary that became famous for being virtually alone in correctly predicting Labour's surprising victory in the 1945 general election. Everyone else expected Winston Churchill's Conservatives to win; the polls said otherwise.

Polling as a Tool of Occupation

One of the most revealing uses of early polling came in postwar Germany. The Allied occupation authorities established survey institutes in all the Western occupation zones between 1947 and 1948. The purpose wasn't academic curiosity—it was to guide denazification.

How deeply had Nazi ideology penetrated German society? Were former Nazis genuinely reformed or merely hiding their views? Polling offered a way to peer beneath the surface, to measure attitudes that people might be reluctant to express openly. It was an optimistic vision of social science as a tool for democratization.

By the 1950s, polling had spread to most democratic societies. France's first survey institute, the Institut Français d'Opinion Publique, had actually been founded in 1938 by Jean Stoetzel after he met Gallup. Stoetzel's first political poll, conducted in summer 1939, asked the French public about the provocative question then being debated in appeasement circles: "Why die for Danzig?"

Within months, France was at war, and the question had become tragically moot.

When Polls Go Wrong

The Literary Digest's failure wasn't the last great polling disaster. In 1948, every major poll predicted that Thomas Dewey would defeat Harry Truman for the presidency. The famous photograph of Truman holding up a newspaper with the headline "Dewey Defeats Truman" has become an enduring symbol of polling hubris.

What went wrong? Several things. Pollsters stopped surveying too early, missing a late swing toward Truman. They used quota sampling—instructing interviewers to find specific numbers of men and women, young and old—rather than true random sampling. And they underestimated the challenge of predicting who would actually show up to vote.

More recent failures have included polls that underestimated support for Brexit in the 2016 United Kingdom referendum and support for Donald Trump in both 2016 and 2020 in the United States. In each case, analysts have proposed explanations involving response bias, turnout modeling, and the difficulty of reaching certain demographic groups.

But a pattern emerges: polling failures tend to undercount people who are skeptical of institutions, including the polling industry itself. If you don't trust polls, you're less likely to participate in them—which means polls will undercount people like you.

The Approval Rating: A Special Kind of Poll

Among the most watched polls are presidential approval ratings, which track how the public evaluates a sitting president's job performance. Unlike election polls, which ask about a future event, approval ratings measure current sentiment. They provide a running score, a constant feedback loop between leaders and the public.

Different pollsters phrase the question differently—some ask if you "approve or disapprove" of the president's job performance, while others ask if you think the president is doing an "excellent, good, fair, or poor" job. These subtle differences can produce meaningfully different numbers, which is why analysts typically look at averages across multiple polls rather than any single result.

Approval ratings have become central to political analysis, used to predict everything from midterm election outcomes to a president's ability to advance legislation. A president with high approval ratings has political capital to spend; one with low ratings faces resistance from even their own party.

But approval ratings are also snapshots, not prophecies. They can change rapidly in response to events—rallying after national crises, collapsing after scandals, drifting with economic conditions. A president's approval rating six months before an election may tell you little about the ultimate result.

The Irreducible Uncertainty

Modern polling is far more sophisticated than George Gallup's early efforts. Pollsters use complex weighting schemes, adjust for likely voter turnout, aggregate results across multiple surveys, and employ statistical models to account for various sources of error.

And yet uncertainty remains irreducible. A poll with a 3% margin of error is not "wrong" if the actual result falls 2.9 percentage points away from the prediction—that's exactly what the margin of error means. In a close race, the polling average might show a candidate leading by two points with a margin of error of three points. Mathematically, that's almost a coin flip.

The public often misreads this uncertainty as failure. When a candidate who was "behind in the polls" wins, observers declare that "the polls were wrong," even when the result was well within the stated margin of error. This reflects a fundamental misunderstanding of what polls actually claim to measure.

Polls don't predict the future. They measure the present, imperfectly, within stated bounds of uncertainty. They're more like weather forecasts than prophecies—useful guides that sometimes get things wrong, not because the method is flawed, but because measuring human intentions is genuinely hard.

The Future of Asking Questions

As response rates continue falling and new communication technologies emerge, pollsters are experimenting with alternatives to phone surveys. Online panels, where respondents opt in to answer surveys regularly, have become increasingly common. Text message surveys, social media analysis, and even analysis of anonymized cell phone location data offer new ways to gauge public behavior and opinion.

Each approach has tradeoffs. Online panels struggle with the same participation bias that doomed The Literary Digest—people who join survey panels may differ systematically from those who don't. Social media analysis captures only what people choose to share publicly, which may not reflect their private views or actual behavior.

But the fundamental challenge remains what it has always been: finding a representative sample. The Literary Digest reached millions of people but asked the wrong ones. George Gallup reached thousands but asked the right ones. The method matters less than the representativeness.

Opinion polling began as a simple question: who do you plan to vote for? Two hundred years later, it has become a multibillion-dollar industry that shapes political strategy, guides corporate decisions, and influences how we understand ourselves as a society. Its failures make headlines, but its routine successes—the countless surveys that accurately measure everything from consumer preferences to pandemic attitudes—rarely do.

The next time you see a poll result, remember The Literary Digest. Remember that the confidence interval isn't a flaw to be dismissed but the whole point—an honest acknowledgment of what we can and cannot know about each other. And remember that even the most sophisticated poll is ultimately just a structured way of asking questions, with all the limitations that implies.

We are still learning how to listen to each other at scale. Two centuries in, we're getting better at it. But the jar still holds millions of marbles, and we're still reaching in with our imperfect hands.