It has been over a month since the US elections are over and going by Trump’s own booking of the after-party hall, he was as much surprised by his own victory, as were the pollsters and the media pundits. His hall was small, more suited for a concession speech rather than a victory speech. If 2012 US elections’ outcome was the rising of pollster Nate Silver’s FiveThirtyEight.com (the name draws from the number of votes in the US presidential electoral college) then 2016 was the undoing of most pollsters, including the Nate Silver. Though he did change the modelling and poll prediction on the date of voting giving Trump a higher odds’ ratio than most others did. But if there was one comprehensive undoing of the poll prediction as a science, then it was 2016 US election outcome. Poetic justice, considering this is the land where polls’ prediction originated, reached its frenzy and has now reached its ignominy too. Not that pollsters in other parts of the world are faring any better. Brexit outcome was also wrongly predicted, so was our own Bihar and Delhi elections, And lest we forget, the Scottish referendum for separation from United Kingdom and the 2015 British polls too!
The US pollstering profession is going through a tectonic shift. The American Association of Public Opinion Research (most pollsters are affiliated to this association) are doing a deep dive post mortem analyses. The oldest pollster Gallup Inc and one of the older and credible surveyors Pew Research Centre have decided to withdraw from horse race predictions (i.e. during the run-up to the elections, calling the candidates’ performance on course to elections) and have decided to only do final outcome predictions.
From sample size, respondents’ bias and representativeness of respondents to the impact of not being able to call on mobile phones (since US poll surveys mandate fixed line calls only and mostly fixed lines are with rich and elderly demography) to increased number of non-respondents, all is under scrutiny.
Heavy-weight names like CNN, Huffington Post, Reuters have apologised for getting the predictions horribly wrong. But no such luck with pollsters in India. While most get election outcomes routinely wrong, an apology is distinctly out of question. Two elections which were wrongly called, by light-years, but never got us Indians any apologies from the pollsters or their media partners, were the Delhi Assembly elections in early 2015 and the Bihar Assembly elections in late 2015. A year full of mistakes, but dare we desire an apology! In fact the irony was when, the lone exit poll, Axis Exit Poll actually got it right in Bihar but its media partner refused to publish the results.
But before we make much ado of the poll predictions’ mistakes and write the obituary of polling as a profession, we need to remember that it is after all a behaviourial science. It is based on voters’ responses, which are likely to change and no amount of modelling will give the prediction the exactness of natural sciences like physics. The sooner pollsters quit the pretention of being physicists; the better it will be for all of us. We need the pollsters to be humble that they are dealing with human opinions which are prone to change, inexact. The very real possibility of voters’ not wanting to be seen with the trending loser, while they might be voting
otherwise, is not acknowledged often enough.
There is the additional complexity of pollsters not declaring their own subjective bias. Mona Chalabi, a data journalist with Guardian, writes in an insightful piece called “Yes the election polls were wrong, here’s why” that, pollsters, other than behaving like physicists, are also more keen to predict the voters’ behaviour than understand and explain their voting preferences. They seem to have dug themselves into the echo chamber manufacturing a self-(un)fulfilling-prophecy than actually doing election analysis. The hierarchy and halo built around data-scientists, pollsters in this case, where the mainstream media treats their predictions uncritically and unquestioningly also contributes to the quantum of mistakes made. The media and the pollsters are equally to blame because the moment pollsters declare uncertainty of their models and probability of their results, they are less likely to be taken seriously and the hagiography built around them would be gone. But she also blames the readers who are interested in echo-chamber articles and pressing refresh-button on pollsters’ sites than in having an inconvenient conversation with a person who has a contrarian opinion or voting preference. A must read for all interested in politics and the science of poll predictions.
Ole J Forseberg, in his seminal piece in The Conversation, “Reports of the death of polling greatly
exaggerated” further buttresses the point about the importance of probabilities in poll predictions and never ever confusing the results as certainties. He also makes the case of burden of responsibility by recommending that the journalists covering elections should take a course in statistics or polling. Seems like a lot of work, and in the era of post-truths, rather difficult too! Ole J Fosberg’s piece is essential reading.
For people like us, deeply interested in political economy and citizen-state engagement, we can’t make what the hullabaloo about the poll predictions is really about, unless it is agenda driven, about influencing voters and outcomes and messing with the body-politick of the very democracy it is trying to predict really. Else, it is just a discipline at the intersection of statistics and politics which is more about probabilities than certainties.
It is pertinent to mention that, pollstering is not the only stream suffering from physics’ envy. Economics and health researchers are no less. From late nineties to 2013, the Nobel Prize of Economics is replete with speculative modelling, game theories, financialisation and behaviorial voodoo theories getting awarded than solid meta-data analysis, consumption expenditure and public policy designs. That Angus Deaton’s Nobel in 2015 and Oliver Hart and Bengt Holmstorm’s Nobel win in 2016 for Contract Theory is undoing the romance with financial speculation, behaviourial voodoos that refused to recognise the power asymmetry, political economy in Economics applications. If 2008 financial melt-down was not a wake up call, then Thomas Piketty’s land-mark work Capital in 21st century was the much-needed rude wake up call against Economists pretending to be physicists and refusing to recognise the inequality and the elite capture.
Health researchers are more interested in behaviour change and behaviourial scientists are attracting more funding than anyone else in the health equity stream. Again, this is predicated on patronising power relationship where communities do not behave in the best interest of their health and hence need to be taught otherwise. But most progressive public health experts will say predictable, high quality health services are the best guarantee for sustainable behaviour change. This piece (https://www.theguardian.com/global-development-professionals-network/2016/ mar/04/world-bank-nudging-attitudes-health-hygiene) in Guardian on World Bank nudging the attitude on health and sanitation makes the case and also highlights the danger of victim-blaming that patronising health researchers can unleash. It also makes a strong case for humility amongst behavioural scientists.
Looks like humility amongst pollsters, economists and health researchers can go a long way. And media asking more critical questions and demanding transparency in methodology would take us a long way too. But most important will be for pollsters, economists and health researchers to stop pretending to physicists and media to stop treating them as such.
(Biraj Swain is an international development expert, a media watcher and behavioral sciences’ sceptic. She works in South Asia, East Africa and globally and can be contacted at biraj_swain@hotmail.com)
(Azhar Hasan Abbas, an art curator and auctioneer and a political animal hailing from Uttar Pradesh, believes every UP-ite is a default pollster and election analyst)
Leave a Comment