How Different Are People Who Don't Respond to Pollsters?

In mid-2007, Princeton Survey Research Associates conducted a major survey for the Pew Internet Project and the University of Illinois-Urbana-Champaign about how people solve certain kinds of problems (Information Searches that Solve Problems, 12/30/2007). Funding for this survey was provided by a federal agency called the Institute for Museum and Library Services. All surveys supported by the federal government are reviewed by the White House's Office of Management and Budget. In the process of evaluating our survey, OMB officials asked if we could do additional work to see if non-respondents to the initial poll were different from those who did respond. The following article excerpts the major findings of that additional analysis.

Summary of findings

In the past two decades, the research profession has faced increasing numbers of Americans who fail to complete interviews. The levels of such non-response raise questions about the representativeness and validity of surveys and the data they provide. The additional analysis produced at the request of the White House's Office of Management and Budget (OMB) had three elements.

First, the interviewing effort on a sub-sample of the telephone numbers generated through computerized Random Digit Dialing (RDD) methods was doubled. RDD is the practice of pollsters to generate results from a random, representative subset of a population. The original survey design called for a maximum of 10 calls to each number. We selected 1,500 of the phone numbers that did not yield complete survey interviews after 10 calls and phoned them at least 20 times. The results of that extra effort were compared with the results of the standard 10-call effort.

Second, an analysis was conducted of survey results from the base 10-call design by segmenting interviews by the amount of effort actually required to get results.

Third, the total sample of telephone numbers and the subset that provided completed interviews using the 10-call design were analyzed to determine what kinds of communities are under- and over-represented in completed interviews.

The results of the analysis found:

• Doubling the interviewing effort to 20 calls produced 84 additional interviews. The results from these extra-effort interviews varied only occasionally and marginally from results in the base study.
• The extra interviewing effort drew responses from those who are usually harder to reach in surveys: Younger adults, working adults and those with college degrees were a larger share of the extra-effort completed interviews.
• Analyzing the original survey by the level of effort required to achieve an interview revealed few statistically significant differences. In other words, there were few meaningful differences between those who were reached in the first wave of calls, compared with those who were reached on the ninth or tenth try to a phone number. Analyzing the complete RDD sample and the "extra-effort" completed interviews by community characteristics showed that interviews are hardest to complete in urban areas and easiest to complete in rural areas. While there appear to be no significant variations across communities by average household income, areas with higher minority populations (both Hispanic and African-American) were less productive in terms of interviewing, paralleling the finding on urban areas.

Read the complete survey results How Different Are People Who Don't Respond to Pollsters? on the Pew Research Center Web site.