The Perils of Opinion Polls
D. Thomas
November 2001
First, sampling of opinion can be misleading for those who depend upon it for strategic decision making. Second, politicians, in making policy on the basis of best pleasing opinion (and maximising their chance of re-election) do not necessarily make the best policy for the future of the country.
A few days before the election, three polling companies released their latest survey results, all taken at the same time. The predicted two-party preferred proportions were:
- ACNielsen: 48% Labor, 52% Coalition;
- Newspoll: 49% Labor, 51% Coalition;
- Roy Morgan Research: 54.5% Labor, 45.5% Coalition.
The election result (49% Labor, 51% Coalition) was very close to the predictions of ACNielsen and Newspoll, but Roy Morgan Research was considerably in error. Any one basing decisions on the latter company's predictions would be very unhappy.
Is this error just due to chance or are there systematic biases at work?
The probability of a random sample of 1710 voters (the sample size employed by Roy Morgan Research) indicating a Labor vote of 54.5% when the population figure is 49.0% is less than one in a thousand. So the Roy Morgan Research figure is very unlikely to be due to chance sampling error. Especially since their previous poll was almost as much in error.
So their sample was not random or there is measurement error.
The chief of Roy Morgan Research, Gary Morgan, defended his figures before the election by saying that his methodology was the only one that employed face-to-face interviews (the other companies employ random digit dialling phone surveys). He claimed that face-to-face was superior because the respondent is more likely to reveal their true voting intention (he also cast slurs on the other companies by stating the Newspoll was part owned by a media company and that ACNielsen was better at managing grocery and media panels of consumers).
Neither measurement approach is ideal in terms of creating an actual voting environment of a secret ballot, but telephone polls are more anonymous. Voting is compulsory in Australia so turnout variations are not an issue. The treatment of people who are undecided about how they will vote could be a problem. Also, interview length and placement of the voting intention question within a long interview could be a problem.
Most companies running face-to-face interviews select respondents in clusters in order to keep interviewer travelling costs affordable. They select a number of starting points at random and then interview at 5, 10, or 20 residences nearby. But, if the opinions of respondents within a cluster are highly correlated, this reduces the effective sample size. For example, if the voting intentions within clusters of 10 were perfectly correlated, a sample size of 1710 is effectively a sample size of 171. Voting intentions are highly correlated within small areas. Random digit dialling does not suffer from this clustering effect.
But more important than the problems of sample surveys is the making of policy to maximise a favourable opinion with voters. Of course, there is nothing worse than arrogant, out-of-touch politicians, so it is important to tap into public opinion. But it has become so pervasive recently in Australia that some people are using opinion polls to blackmail politicians (rather like a protest vote in a by-election). Even worse, talk back radio is also monitored and analysed despite being a very non-random sample of opinion.
The easy option for a politician is to make policy on the run, based on these measures of opinion. This seems to have happened in Australia on several issues and has left the country internally divided and internationally an outcast.
Much better for a leader to have a vision of the future (based on the aspirations of the community) and to carry public opinion through effective and well communicated policies.
That is my wish for the future but I don't like my chances of it coming true anytime soon.
Postscript
Shortly after the election, Roy Morgan Research recontacted a sample of the respondents who had earlier said that they intended to vote Labor. They found that many had changed their mind between the survey the week before the election and the election itself. Thus, it was claimed that there was no problem with the research methodology - rather that enough prospective labor voters changed their mind at the last moment to swing the result. But this is at best a flawed research design or a worst, a design contrived to achieve the desired result. In particular, there was no follow-up survey of intending Coalition voters to see if they had changed their mind. There may have been enough of these to offset the shifting prospective Labor voters.
The Bulletin magazine has since showed good judgement and terminated their association with Roy Morgan Research.