How Different Are People Who Don't Respond to Pollsters?

Report Read time:
How Different Are People Who Don't Respond to Pollsters?

In mid-2007, Princeton Survey Research Associates conducted a major survey for the Pew Internet Project and the University of Illinois-Urbana-Champaign about how people solve certain kinds of problems (Information Searches that Solve Problems, 12/30/2007). Funding for this survey was provided by a federal agency called the Institute for Museum and Library Services. All surveys supported by the federal government are reviewed by the White House's Office of Management and Budget. In the process of evaluating our survey, OMB officials asked if we could do additional work to see if non-respondents to the initial poll were different from those who did respond. The following article excerpts the major findings of that additional analysis.

Summary of findings

In the past two decades, the research profession has faced increasing numbers of Americans who fail to complete interviews. The levels of such non-response raise questions about the representativeness and validity of surveys and the data they provide. The additional analysis produced at the request of the White House's Office of Management and Budget (OMB) had three elements.

First, the interviewing effort on a sub-sample of the telephone numbers generated through computerized Random Digit Dialing (RDD) methods was doubled. RDD is the practice of pollsters to generate results from a random, representative subset of a population. The original survey design called for a maximum of 10 calls to each number. We selected 1,500 of the phone numbers that did not yield complete survey interviews after 10 calls and phoned them at least 20 times. The results of that extra effort were compared with the results of the standard 10-call effort.

Second, an analysis was conducted of survey results from the base 10-call design by segmenting interviews by the amount of effort actually required to get results.

Third, the total sample of telephone numbers and the subset that provided completed interviews using the 10-call design were analyzed to determine what kinds of communities are under- and over-represented in completed interviews.

The results of the analysis found:

  • Doubling the interviewing effort to 20 calls produced 84 additional interviews. The results from these extra-effort interviews varied only occasionally and marginally from results in the base study.
  • The extra interviewing effort drew responses from those who are usually harder to reach in surveys: Younger adults, working adults and those with college degrees were a larger share of the extra-effort completed interviews.
  • Analyzing the original survey by the level of effort required to achieve an interview revealed few statistically significant differences. In other words, there were few meaningful differences between those who were reached in the first wave of calls, compared with those who were reached on the ninth or tenth try to a phone number. Analyzing the complete RDD sample and the "extra-effort" completed interviews by community characteristics showed that interviews are hardest to complete in urban areas and easiest to complete in rural areas. While there appear to be no significant variations across communities by average household income, areas with higher minority populations (both Hispanic and African-American) were less productive in terms of interviewing, paralleling the finding on urban areas.

Read the complete survey results How Different Are People Who Don't Respond to Pollsters? on the Pew Research Center Web site.

America’s Overdose Crisis
America’s Overdose Crisis

America’s Overdose Crisis

Sign up for our five-email course explaining the overdose crisis in America, the state of treatment access, and ways to improve care

Sign up
Quick View
America’s Overdose Crisis

America’s Overdose Crisis

Sign up for our five-email course explaining the overdose crisis in America, the state of treatment access, and ways to improve care

Sign up

MORE FROM PEW

A worker in a seafood market dumps bright orange shrimp from a blue bin onto a bed of ice between a lot of other seafood laid out for sale. Other workers are visible in the background, as are the arms of shoppers placing orders from the other side of the ice bed. Seafood Industry Benefits From Greater Vessel Transparency
An aerial view of a vast wetland, with scattered vegetation visible on sandy islands. In Western Australia, a Conservation Model for the World
A presentation being given by a businesswoman to a group of participants at a business conference. Four Plan Features Can Boost 'Retirement Readiness'
A large group of people in formal or traditional attire stands together on a stage, with a digital background depicting marine life and the large text “WTO Agreement on Fisheries Subsidies.” WTO Needs to Redouble Its Effort on Fishing Subsidies
Composite image of modern city network communication concept

Learn the Basics of Broadband from Our Limited Series

Sign up for our four-week email course on Broadband Basics
Quick View

How does broadband internet reach our homes, phones, and tablets? What kind of infrastructure connects us all together? What are the major barriers to broadband access for American communities?

Pills illustration
Pills illustration

What Is Antibiotic Resistance—and How Can We Fight It?

Sign up for our four-week email series The Race Against Resistance.
Quick View
Pills illustration

Antibiotic-resistant bacteria, also known as “superbugs,” are a major threat to modern medicine. But how does resistance work, and what can we do to slow the spread? Read personal stories, expert accounts, and more for the answers to those questions in our four-week email series: Slowing Superbugs.