Perils of Polling in Election '08
The analysis of total survey error has evolved over many decades to consider a wide variety of potential threats, including concerns about the contribution of both bias and variance, and an attention to errors of both observation and non-observation (Groves 1989). The validity of public opinion polling in the presidential election of 2008 was thought to be seriously imperiled by a wide range of these potential errors. Among these were coverage error due to the growth of the wireless-only population, nonresponse error potentially caused by differential nonresponse among Republicans and racially conservative voters, and measurement error potentially resulting from racially-related understatement of support for the Republican candidate and greater-than-usual difficulties in forecasting turnout and identifying likely voters.
Despite these obstacles, polls performed very well, with 8 of 17 national polls predicting the final margin in the presidential election within one percentage point and most of the others coming within three points. Both at the national and state levels, the accuracy of the polls matched or exceeded that of 2004, which was itself a good year for the polls. The performance of election polls is no mere trophy for the polling community, for the credibility of the entire survey research profession depends to a great degree on how election polls match the objective standard of election outcomes. The consequences of a poor performance were dramatically demonstrated in the reaction to the primary polls' inaccurate prediction that Barack Obama would win in New Hampshire, portrayed as one of polling's great failures in the modern political era ((AAPOR 2009 (PDF)).
The Pew Research Center examines the challenges of potential coverage bias from excluding cell phones and potential measurement and non-response bias due to race in detail using data from a wide range of sources, including a summary analysis of state and national pre-election polls, six telephone surveys conducted among both landline and cell phone samples, and a comparison of a survey conducted by landline with reluctant and elusive respondents with a survey conducted at the same time with a fresh sample using standard methodology. The conclusion is that some of the threats were very real but overcome by the techniques normally employed in surveys to address potential bias from various sources of error, while other threats turned out to be less serious than some anticipated.
Read the full report Perils of Polling in Election '08 on the Pew Research Center's Web site.
