THe Blog

September 18, 2015

Predictions that never come true

Man’s fascination for predicting the future is an ancient weakness. Thomas Malthus’ disastrous forecast of the demise of the human race couched in scientific logic was made 60 years before Charles Darwin could explain the very evolution of that human race. This temptation to predict, albeit with an ignominious track record of doing so, continues unabated. It is no surprise that our own electoral prediction season has kicked off yet again in its characteristic pomp and splendour with the sounding of the Bihar poll bugle. In true Malthusian tradition, these electoral surveys are often wrong, but unlike Malthus, unaccountable and unashamed yet.

 

Of the 82 electoral surveys that surveyed more than 1.25 million voters for 13 Lok Sabha and State Assembly elections since 1996 that we analysed, zero (yes, zero) have been fully accurate and 15 have been partially accurate. We define “fully accurate” as being within a +/-5 per cent range in terms of seats predicted vis-à-vis actual results for both the winner and the runner-up. The 5 per cent test is a fairly standard statistical norm to determine if a certain outcome is purely by chance or by design. We classify partial accuracy as those that predicted either the winner or the runner-up but not both, within the range of 5 per cent. If we relax the +/-5 per cent accuracy levels to 10 per cent, then 9 out of 82 surveys have been fully accurate and 27, partially accurate. Accuracy levels of these surveys show no signs of an improving trend since 1996 though sample sizes and number of surveys have increased substantially. Similarly, there is no evidence of these surveys being able to predict a State Assembly election better than a Lok Sabha election or vice-versa. Neither is there proof of any specific survey organisation(s) getting their predictions consistently right vis-à-vis others. Accuracy levels of these poll surveys are understandably as random as electoral outcomes. Thus, at best, one in three poll surveys over the last two decades have been partially accurate and at worst, zero has been fully accurate. Either way, they are of no Nostradamical proportions. Yet, over the next few weeks, we will be inundated with an opinion poll blitzkrieg in the run-up to the five phase Bihar elections. Many will tout a successful track record by claiming credit for predicting a winner correctly in previous elections, which in most bi-polar alliance situations, is a simple 50/50 chance, akin to a coin toss. Many others will claim use of more sophisticated techniques such as stratified random sampling, better vote share to seat share predictive models etc. The naked truth is 7 out of 10 polls are likely to get it wrong, as per our analysis and definitions of accuracy.

 

We compiled from various sources, information on electoral surveys since 1996 of both Lok Sabha and State Assembly elections (raw data below). Of the 82 surveys, 48 were pan-India surveys for six Lok Sabha elections since 1996, and 34 for Assembly elections of Rajasthan, Madhya Pradesh, Bihar and Delhi. Only 41 out of these 82 disclosed their sample sizes and methodology. Many pollsters acknowledge the hopelessness of predicting seats accurately even if one predicts vote shares correct. While psephology may be an established science in other electoral democracies, India’s complex plurality mixed with an inappropriate first-past-the-post electoral system and a generic human fallibility for voting along identity lines renders this science useless in the Indian context. A multi-cornered contest with no requirement for winners to secure at least 50 per cent of the votes makes electoral seat predictions a futile exercise in India. Not only do opinion polls amplified by their noisy media partners routinely get it wrong, but they have the potential to inflict damage to the very notion of a free and fair election.

 

Opinion polls play an inordinately profound role in shaping perceptions of likely voters. The average Indian voter is generally deemed to be vulnerable. This prompted the Election Commission recently to include photographs of candidates on the ballot to prevent confusion from multiple similar names. In this context, it is not difficult to fathom the potential for inaccurate and misleading opinion polls to sway voters. It is not possible for the average voter to distinguish between deliberate or unintended imprecision of these opinion polls, i.e. sins of omission versus commission.

 

Though bans are in vogue these days, this is not to imply a ban on election surveys. Perhaps, we can start by issuing some basic guidelines for publishing opinion polls by requiring pollsters to disclose their past accuracy track record, as per a standard definition. It is similar to airlines being asked to publish their on-time performance data periodically. It is imperative we acknowledge that opinion polls have a definitive impact (intended or otherwise) on voter behaviour, especially in a largely illiterate democracy such as ours. Publishing accuracy track record and making voters aware of this can be a baby step in the direction to restrict the abilities of opinion polls to impinge on fair electioneering. To be sure, the increasing complexity of electoral outcome predictions is not unique to India as we witnessed in the recent U.K. elections. What is unique is the size and complexity of the diversity of India’s voter base and the relatively lower education levels of the average voter. To curb infringement on fair electioneering, it is necessary to build accountability and transparency to the opinion poll process. Creating a measurement framework and making publicly available, accuracy data of opinion polls can be an important first step in this process.

 

 

(A version of this article appeared in the Hindu. Swapnil Bhandari and Rithika Kumar contributed to this story.)

The full list of surveys used for this analysis can be found below:

Poll AgencyElectionWinner Seat PredictionRunner-up Seat predictionSample SizeSource
India Today ORG MARGLok Sabha - 1996178 - 197168 - 17713150http://bit.ly/1K9tUso
India Today: ORG-MARGLok Sabha - 1998234 - 249 150 - 16517434http://bit.ly/1K9u17n
CMS-1 Dec 3 - 5Lok Sabha - 1998195 - 23580 - 1202800http://bit.ly/1K8Cqpk
CMS-2 Jan 7 - 10Lok Sabha - 1998235 - 245140 - 1508400http://bit.ly/1i9ZLzz
Frontline CMS -3 Jan 27 - 30Lok Sabha - 1998225 - 235145- 1558,900http://bit.ly/1i9ZLzz
Asianet - CMS 4 Feb 4 - 7Lok Sabha - 1998230 - 240145 - 1553325http://bit.ly/1i9ZLzz
CSDSLok Sabha - 1998220 - 241140 - 16015500http://bit.ly/1K9u17n
DRSLok Sabha - 1998249155 http://bit.ly/1K8D5Hr
Outlook/AC NeilsenLok Sabha - 1998238149 http://bit.ly/1K8D5Hr
India Today/CSDSLok Sabha - 1998214164 http://bit.ly/1K8D5Hr
Timespoll/ DRSLok Sabha - 199928417923098A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
India Today/ InsightLok Sabha - 1999322 - 336132 - 14620640http://bit.ly/1UOzhp4
India Today Lok Sabha - 199930513518960http://bit.ly/1UQlmcJ
Outlook/ CMSLok Sabha - 1999319 - 329135 - 14514220http://bit.ly/1QzwtGn
Hindustan Times-LokmatLok Sabha - 1999265 - 270210 - 21515199A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
HT-AC NeilsenLok Sabha - 199930014613959A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Pioneer DRILok Sabha - 1999313 - 318140 - 14514850A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
NDTV- Indian Express- AC NielsenLok Sabha - 2004289 - 307142 - 16345478http://bit.ly/1FHcsaB
India Today, Aaj Tak, Dainik Bhaskar by ORG MARGLok Sabha - 200428216550632http://bit.ly/1QFhyKX
Outlook Lok Sabha - 2004280 - 290159 - 16912249http://bit.ly/1Oa1cLM
Sahara TV-DRSLok Sabha - 200427116865089A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-TNSLok Sabha - 2004230 - 26518517513A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Star News-C VoterLok Sabha - 2004263 - 275174 - 18612000http://bit.ly/1K8D5Hr
Zee - TaleemLok Sabha - 200426519613000http://bit.ly/1NyvaJs
India Today-ORG MARG opinion poll, Aug 2003Lok Sabha - 2004242 - 252175 - 18517500http://bit.ly/1KN0yUa
Star News- NielsenLok Sabha - 200920319150400http://bit.ly/1Kl3EMS
India TVLok Sabha - 200917818750000http://bit.ly/1NywEn4
Star-NielsenLok Sabha - 200925718450402http://bit.ly/1Y1UWJn
CNN IBN- CSDSLok Sabha - 2009215-235165-18520000http://bit.ly/1EXdwfH
India Today-ORG-ACLok Sabha - 2009195-200150-17012374A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
NDTV GFKLok Sabha - 2009205-215160-17051390http://bit.ly/1ip5ylB
The Week-CvoterLok Sabha - 200923418630318A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Times Now- India Tv- C VoterLok Sabha - 201418611724284http://bit.ly/1EXdAfr
ABP- NeilsenLok Sabha - 201420613633408http://bit.ly/1UOBHUy
The Week- Hansa ResearchLok Sabha - 2014197184130000http://bit.ly/1Oa2VRf
CNN IBN- The Hindu by CSDSLok Sabha - 2014172-180149-15719062http://bit.ly/1iIrHel
ABP News- NielsenLok Sabha - 201422610164006http://bit.ly/1Y1VA9R
CNN IBN- Lokniti- CSDSLok Sabha - 2014211-231107-12718591http://bit.ly/1FHgccd
NDTV HansaLok Sabha - 201423012846571http://bit.ly/1K8HJ8m
NDTV HansaLok Sabha - 201427511124000http://bit.ly/1FHghMO
Times Now- India Tv- C VoterLok Sabha - 201422710114000http://bit.ly/MOiZeh
ABP News- NielsenLok Sabha - 20142369229000http://bit.ly/1EXekB7
CNN IBN- The Week by CSDSLok Sabha - 2014187-195134-142 http://bit.ly/1EXel8g
India TV-Times Now-C Voter Opinion PollLok Sabha - 201415613636914http://bit.ly/1M5SLhk
CNN-IBN-CSDS-Lokniti-The WeekLok Sabha - 2014234-246111-12320957http://bit.ly/1Oa3MBF
News 24-ChanakyaLok Sabha - 201434070 http://reut.rs/1UOD1Xz
India Today-CICEROLok Sabha - 2014261-283150-162 http://reut.rs/1UOD1Xz
Times Now-ORG MargLok Sabha - 2014249146 http://reut.rs/1UOD1Xz
Star-Nielson pollBihar 20105022 http://bit.ly/1FHhxQb
The Week-C VoterBihar 20105226 http://bit.ly/1FHhxQb
Star-Nielson ExitpollBihar 20105614 http://bit.ly/1Kht0a7
Zee News-C VoterDelhi 200343 - 5114 - 22 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Times of India-DRSDelhi 20033728 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Hindustan Times-CSDSDelhi 20035810 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Sahara Samay-DRSDelhi 20034026 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-TNSDelhi 200346 - 5411 to 20 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Outlook-Zee News-AC NeilsonDelhi 20035611 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Outlook-Zee nEws-ORG MargDelhi 200350 - 6510 to 20 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
India Today-Aaj Tak-C VoterDelhi 200340 - 4821 - 29 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Times of India-ORG MARGDelhi 200345 - 5510 to 20 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Hindustan Times-CINSDelhi 200342 - 4618 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Star News-AC NeilsonDelhi 20083829 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-C VoterDelhi 200836-4228 - 34 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Star News-AC NeilsonDelhi 20083927 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Zee News-C VoterMadhya Pradesh 2003133 - 14162 - 70 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Sahara Samay-DRSMadhya Pradesh 200316055 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-TNS ModeMadhya Pradesh 2003165 - 19519 - 33 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Outlook-Zee News-AC NeilsonMadhya Pradesh 200314758 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
India Today-Aaj Tak-ORG MARGMadhya Pradesh 2003140 - 15055 - 65 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Zee News-C VoterMadhya Pradesh 2003122 - 13077 - 85 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
India Today-Aaj Tak-ORG MARGMadhya Pradesh 2003115 - 13575 - 95 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Indian Express-Internal Congress SurveyMadhya Pradesh 2003114 - 145108 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-C VoterMadhya Pradesh 2008118 - 128100 - 110 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Zee News-C VoterRajasthan 200396 - 11280 - 96 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Sahara Samay-DRSRajasthan 20039585 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-TNS ModeRajasthan 2003125 - 13555 - 60 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Outlook-Zee news-AC NeilsonRajasthan 200311367 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
India Today-Aaj Tak-ORG MARGRajasthan 2003120 - 13055 - 65 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Zee News-C VoterRajasthan 2003107 - 11577 - 85 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
India Today-Aaj Tak-ORG MARGRajasthan 2003110 -12073 - 83 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
Hindustan Times-MMRRajasthan 200310589 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
The Week-C VoterRajasthan 200894 - 10694 - 106 A Handbook on Poll Surveys: Dr. N. Bhaskar Rao
x Close Window

Please verify your email address to access this content