Survey methodology

Based on Wikipedia: Survey methodology

The Art of Asking Questions

Here's a puzzle that sounds deceptively simple: How do you find out what millions of people think by talking to just a few hundred of them?

This is the central challenge of survey methodology, and getting it wrong has led to some spectacular failures. In 1936, a magazine called Literary Digest mailed out ten million questionnaires to predict the presidential election. They received over two million responses—an astonishing number. Their prediction? Alf Landon would crush Franklin Roosevelt in a landslide.

Roosevelt won by the largest margin in over a century.

What went wrong? The magazine had pulled its mailing list from telephone directories and automobile registration records. In the depths of the Great Depression, these were luxuries. They had surveyed the wealthy and assumed they were surveying America.

Why Sample Size Isn't Everything

The Literary Digest disaster illustrates a counterintuitive truth: who you ask matters far more than how many you ask. A carefully chosen sample of one thousand people can predict an election more accurately than a sloppily chosen sample of one million.

This brings us to the concept of representativeness. A representative sample is like a miniature portrait of the larger population. If the population you're studying is 75 percent women and 25 percent men, your sample should roughly mirror those proportions. If it doesn't, you've got what researchers call selection bias—a systematic error that skews all your results.

Think of it like tasting soup. You don't need to drink the whole pot to know if it needs salt. But you do need to stir it first. If all the salt has settled at the bottom, and you only taste from the top, you'll make a bad judgment.

Stratified random sampling is the stirring mechanism of survey research. Instead of picking people at random from the entire population, researchers first divide the population into meaningful subgroups—called strata—and then sample randomly from each stratum. This ensures that no important group gets accidentally left out or overrepresented.

The Sampling Frame: Where It All Begins

Before you can select a sample, you need something to select from. This is called a sampling frame—typically a list of everyone in the population you want to study. It might be a voter registration database, a company's customer records, a school's enrollment list, or even a map of geographic areas.

The quality of your sampling frame determines the quality of your entire survey. If your frame is incomplete—if it systematically excludes certain types of people—no amount of statistical wizardry can fix the problem. This is why modern pollsters spend enormous effort thinking about who might be missing from their lists.

Consider the challenge of surveying all adults in a country. Once, telephone directories served this purpose reasonably well. Most households had landlines, and most landlines were listed. Today? Many people have only mobile phones with unlisted numbers. Some rely entirely on internet communication. The very concept of a comprehensive list of adults has become elusive.

How You Ask Changes What You Learn

Once you've figured out who to survey, you face the question of how to reach them. This choice isn't neutral—it fundamentally shapes what kinds of answers you'll get. Researchers call this mode effects.

Telephone surveys were the gold standard for decades. Interviewers could clarify confusing questions, probe for deeper answers, and cajole reluctant participants into completing the survey. But telephone surveys are expensive—each interview requires a trained human being—and response rates have plummeted as people screen unfamiliar callers.

Mail surveys are cheaper but slower. They give respondents time to think carefully about their answers, which can be either an advantage or a disadvantage depending on what you're measuring. Spontaneous reactions look different from considered judgments.

Online surveys have revolutionized the field through their speed and low cost. A researcher can collect thousands of responses in days rather than months. But online surveys face their own representativeness problems. Not everyone has internet access. Those who respond to online surveys may differ systematically from those who ignore them.

In-person surveys, conducted by interviewers visiting homes or intercepting people in shopping malls, remain the most flexible option. Interviewers can show visual materials, read body language, and adapt to unexpected situations. But they're expensive and increasingly difficult to conduct as people become more reluctant to engage with strangers.

Many modern surveys use mixed modes—perhaps starting with an online invitation, following up by mail with non-responders, and making phone calls to hold-outs. This mixing helps maximize response rates while controlling costs, though it introduces complexity in ensuring that answers are comparable across modes.

The Time Dimension

Surveys don't just vary in how they're administered—they vary in when they're administered, and how often.

The simplest design is cross-sectional: you survey a group of people once, at a single point in time. This gives you a snapshot. It can tell you that sixty percent of people support a particular policy, but it can't tell you whether support is growing or shrinking.

To track change over time, researchers use successive independent samples. At regular intervals—perhaps monthly or yearly—they draw fresh random samples from the same population. By comparing results across these snapshots, they can observe trends. But there's a catch: because different people are surveyed each time, you can't tell whether individuals are changing their minds or whether the population's composition is shifting.

Longitudinal studies solve this by surveying the same people repeatedly over months or years. This reveals individual change. If someone supported a candidate last year and opposes them now, a longitudinal study captures that directly. Researchers can then investigate what experiences preceded the change.

But longitudinal studies are difficult. They require participants to commit to months or years of involvement. People move, lose interest, or simply become unreachable. This gradual departure—called attrition—isn't random. The people who drop out often differ systematically from those who stay, gradually eroding the sample's representativeness.

One clever workaround involves self-generated identification codes. Instead of collecting names, researchers ask participants to create codes from personal facts—first initial of mother's maiden name, last digit of birth year, that sort of thing. Participants can remain anonymous while researchers can still link their responses across survey waves. Recent approaches have moved toward even less personal information, asking for things like the name of your first pet.

The Questionnaire: Harder Than It Looks

You might think writing survey questions is easy. Ask what you want to know, and people will tell you.

In reality, questionnaire design is fiendishly difficult. Small changes in wording can produce large changes in answers. Consider the difference between asking whether the government should "forbid" public speeches against democracy versus whether it should "not allow" them. Logically identical questions, yet "forbid" sounds harsher, and more people oppose forbidding than not allowing.

Words carry connotations that vary across cultures, generations, and social groups. A question that reads clearly to a college-educated professional might confuse a teenager or someone for whom English is a second language. The recommendation is to keep questions under twenty words and use the simplest vocabulary that accurately conveys the intended meaning.

Questions come in two basic types. Open-ended questions let respondents answer in their own words: "What do you think about the president's economic policy?" Closed-ended questions provide fixed options: "Do you approve, disapprove, or have no opinion about the president's economic policy?"

Open-ended questions capture nuance and surprise. Respondents might mention concerns you never anticipated. But they require extensive coding to convert free-text responses into analyzable data, and different coders might interpret the same response differently.

Closed-ended questions are cleaner for analysis but risk forcing respondents into categories that don't quite fit their views. They also require you to anticipate all the relevant answer options in advance—and if you leave out an important option, you'll never know.

Reliability and Validity: Two Ways to Be Wrong

A survey question can fail in two distinct ways. It can be unreliable, giving different answers each time you ask it. Or it can be invalid, consistently measuring the wrong thing.

Imagine a bathroom scale that shows a different weight each time you step on it, varying by ten pounds in either direction. That's unreliable. Now imagine a scale that always shows exactly the same weight—but it's fifteen pounds too high. That's invalid but reliable.

Reliability is easier to assess. Give the same questionnaire to the same people twice, separated by a couple of weeks. If their responses are similar—not identical, but similarly ranked—your measures are reliable. More items measuring the same concept tend to increase reliability, as do clearer instructions and distraction-free testing environments.

Validity is trickier. How do you know if a question measures what you think it measures? If you're asking about height, you can check against a ruler. But if you're measuring something like "job satisfaction" or "political conservatism," there's no ruler. You have to build a web of evidence: Does the measure correlate with other measures it should theoretically relate to? Does it fail to correlate with things it shouldn't relate to? Do experts agree it captures the intended concept?

The Order of Questions Matters

People don't answer questions in isolation. Each question is colored by the questions that came before it.

Researchers discovered this through experiments. If you first ask people whether they think Japan should be allowed to set limits on American goods sold in Japan, and then ask whether America should be allowed to set limits on Japanese goods sold in America, you get different answers than if you ask the questions in reverse order. The first question about Japan primes thoughts about fairness and reciprocity that carry over to the second question.

This is called a survey response effect. It means that the same question, asked in different contexts, can yield different results.

The practical implications depend on how the survey is administered. For self-administered questionnaires—paper forms or online surveys that respondents complete independently—the most engaging questions should come first to capture attention, with demographic questions (age, income, education) saved for the end, when respondents are committed enough to finish.

For interviewer-administered surveys, the opposite often works better. Starting with easy demographic questions builds rapport and confidence before tackling more complex or sensitive topics.

Translation: More Than Finding the Right Words

As survey research has gone global, translation has become crucial. A survey that works beautifully in English might fail completely in Spanish or Mandarin—not because of poor translation, but because the concepts themselves don't transfer.

Consider asking people to rate their satisfaction on a scale from one to five. In some cultures, people gravitate toward middle values, avoiding extremes as impolite. In others, strong opinions are freely expressed. The resulting numbers aren't directly comparable.

Best practice in survey translation follows a model called TRAPD: Translation, Review, Adjudication, Pretest, and Documentation. Developed originally for the European Social Survey—a massive coordinated effort to gather comparable data across dozens of countries—TRAPD emphasizes teamwork and iteration.

A translator produces an initial version. Reviewers critique it, thinking about whether the translation captures the same meaning in the target culture. An adjudicator resolves disagreements. The translated questionnaire is pretested with real respondents who can flag confusing or awkward phrasing. Everything is documented so that future researchers understand the choices made.

The goal isn't literal translation but equivalent communicative effect. If a source question uses a colloquial expression, the translation should use an equivalent colloquialism in the target language—even if a word-for-word translation would be grammatically correct but awkwardly formal.

Survey Error and Survey Cost

Survey methodology, as a scientific discipline, focuses on understanding and reducing survey errors. These errors come in many flavors.

Coverage error occurs when your sampling frame excludes part of the population. Sampling error arises from the inherent randomness of selecting a sample—even a perfect sampling procedure will, by chance, sometimes oversample certain views. Non-response error emerges when the people who decline to participate differ systematically from those who cooperate. Measurement error reflects the gap between what a question is supposed to measure and what it actually measures.

All of these can be reduced, but reduction costs money. More exhaustive sampling frames, larger samples, more persistent follow-up with non-responders, more extensive pretesting of questions—every improvement has a price tag.

Survey methodologists think in terms of tradeoffs. Given a fixed budget, where should you invest to maximize data quality? Given a target level of quality, what's the cheapest way to achieve it? These optimization problems have no universal answers—they depend on the specific survey's goals, population, and constraints.

Why This Matters Beyond Academia

Survey data shapes the world in ways you might not notice. Government surveys determine how trillions of dollars in funding are allocated. Census data dictates political representation—which is why census methodology is perpetually controversial. Public health surveys guide vaccination campaigns and hospital construction. Market research surveys influence which products get developed and how they're marketed to you.

When surveys are done well, they give voice to populations that would otherwise be invisible in decision-making. When done poorly, they create illusions of knowledge that may be worse than admitting ignorance.

The Literary Digest failure wasn't just embarrassing for the magazine—it contributed to false confidence in survey methods that took years to rebuild. Today's pollsters work hard to avoid similar disasters, but the fundamental challenges haven't changed: finding representative samples in an increasingly fragmented society, getting people to respond honestly to increasingly skeptical ears, and translating complex human opinions into quantifiable data without losing what makes them meaningful.

Every time you see a headline about public opinion or consumer preferences, remember: behind those numbers is a complex apparatus of sampling, questioning, and statistical adjustment. The numbers are not pure facts emerging from the population. They are constructions, shaped by thousands of methodological choices. Understanding survey methodology is understanding how much of our collective knowledge about ourselves gets made.