Survey Methodology

Phone, Online, and Mixed-Mode

For decades, telephone surveys using random digit dialing were the gold standard. Pollsters called randomly generated phone numbers, reaching a near-random sample of the population. Response rates in the 1970s exceeded 70%. By 2020, Pew Research reported telephone response rates of roughly 6%, raising serious questions about who still answers the phone.

Online surveys have filled the gap. Firms like YouGov and Ipsos maintain large panels of respondents who agree to take regular surveys. Online methods are faster and cheaper but face their own challenges: people who join online panels are not naturally representative of the population, requiring heavy statistical adjustments. Mixed-mode surveys combine phone, online, and sometimes text-to-web recruitment, attempting to reach different demographic groups through different channels.

How to Evaluate Poll Quality

Not all polls are created equal. Quality indicators include the organization conducting the poll (established firms with track records versus partisan outfits), sample size (at least 800-1,000 for national polls), methodology transparency (do they disclose their methods?), and weighting approach (how do they adjust for non-response bias?).

FiveThirtyEight historically maintained pollster ratings based on demonstrated accuracy and methodological standards. Highly rated pollsters such as Pew Research, Marist, Monmouth, and Siena College (which partners with The New York Times) consistently produce accurate results. Partisan polls, conducted by firms affiliated with political parties or campaigns, tend to show a systematic bias toward their side and should be interpreted with skepticism, especially when released selectively to shape a narrative.

Worked Example: Correcting a Skewed Sample

Suppose a pollster wants a national sample where 18-29 year-olds should be 20% of respondents. They complete 1,000 interviews but only reach 100 young adults (10%) because young people rarely answer the phone. To make the sample match the population, each young respondent must be counted more heavily.

The weight is the target share divided by the achieved share: 0.20 / 0.10 = 2.0. So each of the 100 young respondents is weighted as if they were 2 people, contributing an effective 200 'weighted' respondents. Older respondents, meanwhile, are weighted down (their weight is below 1.0) so the totals still add to 1,000.

Now watch the risk. Imagine those 100 young respondents split 70-30 for Candidate A. After doubling their weight, that 70-30 split drives twice as much of the topline as it should if the 100 are unusual. If the young people who actually answered the phone were more politically engaged (and more pro-A) than typical young voters, the weighting has faithfully amplified a biased subsample. This is why heavy weights are a red flag: they multiply both signal and error.