Are Survey Innovations a Threat to Horserace Polls?
Ever since the infamous Literary Digest straw poll which wrongly predicted that Landon would win the 1936 Presidential Election and led to the reliance on random, representative surveys, pollsters have been very careful about putting polls in the public domain that are reliable. Incidents of actual or perceived poll failure (the British election of 1992; the 2004 Canadian election) to predict election outcomes linger in our collective memory. Nevertheless, a number of companies have introduced innovations into the horserace poll. Can we trust these innovations? Do they threaten to undermine public confidence in the industry?
Since almost every household had a phone and the survey could be conducted easily and quickly, the telephone survey was a natural choice over in-person and mail survey options for most polling in Canada. Of course, landline phones have become less universal, due to wireless substitution (replacing one`s landline phone with a cell phone) and response rates have declined. So innovations in horserace polling are often framed in terms of the perceived deficiencies of telephone surveys. Or, at least the veracity of the telephone survey is said to itself be undermined by these issues.
Two particular types of innovations are deserving of more attention.
The online survey innovation
Online surveys are ubiquitous in market research so it is not surprising that they have been introduced to horserace coverage. While there are advantages and disadvantages of this mode of data collection, the fundamental difficulty of moving surveys online relates to sampling. First, there is the fact that not all Canadians are online. Second, there is the problem of selecting a random sample.
Most online surveys are restricted to people who are online with the accompanying age and education biases that are associated with having Internet access. It is theoretically possible to recruit people randomly and provide them with Internet access and equipment but this is cost prohibitive and not used by any of the horserace polls in Canada.
Even with the restriction to those online, there is no way to draw a representative sample that meets the statistical properties necessary to infer that the results are a sample of the true population. To be part of an email sample, a person must have given the survey firm consent to email them in advance (usually by joining a panel of respondents). Panels of this sort are by definition convenience samples. Survey firms use quotas and weighting (often proprietary) to adjust their survey findings to resemble the Canadian population.
Building a truly representative survey of online Canadians must be done through a random method (e.g. telephone) first, which is costly. Even here, surveys based on these methods are still potentially influenced by panel learning effects if they are used more than once.
The IVR innovation
Some in the industry are also using the telephone in new ways to conduct horserace polls. Instead of interviewers trying to reach people and convince them to participate, Interactive Voice Response technology uses a machine to dial the numbers and administer the survey. Surveys of this type are short, can reach cell phones and a live interviewer can be accessed by the respondent if needed.
The IVR approach seems to offer real potential for conducting horserace polls if we can rely on respondents to be engaged with this method. Since it is relatively new, caution is in order and there are still potential issues of coverage (response rates are higher but some people may still be less likely to respond) and weighting, especially when the sample includes both landline and cellular numbers.
The implication for the poll consumer
In Canada, there are at least four firms which regularly release a horserace poll and a number of other firms which release them on an intermittent basis (particularly around elections). Of the four main surveys which are released, two use traditional telephone surveys, one uses an online panel and one uses IVR. Notably there are consistent differences over time in the results of different companies.
The online panel approach is the clearest departure from the principles of representative surveys, while the IVR approach represents an attempt to improve on the telephone survey. In fairness, the innovators can point to evidence that their results have delivered as well or better than telephone surveys when it comes to the closeness of their predictions to actual vote returns in elections. Nevertheless, past performance is not always a good predictor of future success and without a statistical basis for making inferences there is always a danger.
For the poll consumer, the following are key things to look at when it comes to horserace polls:
1.Only compare polls by the same company.
2.Look closely at the time of the field of the surveys since an announcement or some bad news can create temporary bumps with no long-run impact.
3.Check the question wording and the allocation of leaners (if asked).
4.Consider the proportion that did not provide a vote intention. Often, the change in the proportion who do not have an intention is itself important.
Richard Jenkins, Ph.D. is President of Jenkins Research Inc. He can be reached at rjenkins@jenkinsresearch.ca
Comments
Post new comment