When polls mislead

I am surprised that the Hindustan Times and the Mumbai Mirror recently chose to publish statistics from an unknown poll, targeted at an ambiguously defined respondent population, and conducted by an anonymous group with a spokesperson who identifies herself only by her first name.  (Hindustan Times: “Ordinary voices: Findings from a Bohra online poll,” September 12, 2014 by Manoj Nair and Mumbai Mirror: “Disquiet among Bohras,” September 12, 2014 by Jyoti Punwani.)

Due diligence would have quickly verified that the poll results are absurd at best and insidious at worst.  Combine that with the fact that the articles do not give voice to the Dawoodi Bohra head office in Mumbai and one would be forgiven for wondering whether the poll is part of a deliberate effort to defame the head of the Dawoodi Bohras, Syedna Mufaddal Saifuddin.

Be that as it may, it is worthwhile to dissect the reported results and let facts emerge.

Incongruence among poll statistics

Given that both newspapers are covering the same poll, a first glance shows some glaring dissimilarities in the results across the two newspapers.  Consider these wildly different statistics:

Respondents who:	Hindustan Times	Mumbai Mirror
Have given oath freely to Syedna Mufaddal Saifuddin	~ 75%	38%
Believe Syedna Mufaddal Saifuddin is the successor	~ 20%	32%
Believe Khuzaima Qutbuddin is the successor	< 20%	46%
Stay in community out of fear	~ 12%	77%

Even if we are to ignore the disparity of results between the two articles, there is cause to question the validity of the responses as stated within each newspaper. 

According to the Hindustan Times, the vast majority of the participants, approximately 75%, have freely given their oath to Syedna Mufaddal Saifuddin but only 20% appear to actually believe in him as the rightful successor.  This implies that 55% of the respondents freely gave their oath to someone who they do not believe to be their leader, a result that is obviously erroneous.

Similarly, the Mumbai Mirror reports that 77% of respondents stay in the community out of fear implying that 23% are doing so out of free will.  Yet at the same time the newspaper reports that 38% have given their oath freely to Syedna Mufaddal Saifuddin.  This leaves us with 15% of the respondents who were inconsistent by choosing both fear and free will as a reason for their oath implying that the survey language did not make the tradeoff clear enough.

The discrepancies between the two newspapers and among the statistics reported within each newspaper article are indicative of polling issues that can plague the best-intentioned of polls, never mind those that appear to have an ostensible agenda.  A robust approach has unfortunately been shown the door for the sake of expediency.

Lack of a scientific approach in the poll

The first thing a statistician will tell you about polls is that their results are completely dependent on the randomness of the sample, the sample size, and the phrasing of the questions.  This particular poll fails to deliver on all of those accounts.

The pollsters confess that because the poll was posted online it reflects the views of only the “educated and the computer literate.”  But the confession does not go far enough because no details are provided on how the respondents were actually selected.  Was there a random selection from a current community mailing list?  Was it done via social media?  Or was it based on the pollsters’ own networks? 

Given that several of the polled were “former Bohras who left the community” it is clear that the respondents were not selected from a current community mailing list which would by definition not contain former Bohras.  Can one then truly check the pulse of a community by polling those who do not belong to it?   

Additionally, the Mumbai Mirror asserts there were 659 responses of which 69 were disqualified and 195 did not complete the survey yielding as many as 399 responses.  No explanation is given as to why the 69 respondents were disqualified and why the 195 individuals may not have answered all questions.  Given the 395 respondents who were qualified and completed the poll, using 399 responses opens the door to the possibility of pollsters cherry picking among the responses to support their cause.  For all we know these 264 respondents did not agree with the pollsters and therefore chose to not participate or drop out after beginning the poll.  Put another way, the pollsters ignore a likely self-selection bias in their poll. 

Another characteristic that jumps out is the lack of information that makes a statistic meaningful (or in technical terms provides insights on the statistical significance of the quoted numbers).  Say I were to tell you that a poll shows that 50% of all people believe there is life outside earth.  What scenario would be more compelling: a poll of 2 persons or one of 2,000?  Surely you would agree that the poll with 2,000 respondents is more likely to be accurate (i.e., have a low margin of error). 

Thus, it goes without saying that a well-executed, scientific poll must state for each statistic: (1) the actual number of respondents, and (2) the resulting margin of error.  Neither of these metrics has been provided by either newspaper leaving the reader no basis by which to interpret the numbers.

As for survey phrasing, how the questions were worded is anyone’s guess.  My requests to both the Hindustan Times and the Mumbai Mirror for the questionnaire and the raw data—since the responses were anonymous there should be no issues with public disclosure—remain unanswered. 

That said, Farida, the single-named spokesperson for the polling group, reflects their obvious bias: according to her the poll was in response to the “turmoil and unease after the split in the leadership. People were talking, but only to friends and family. We wanted to capture that.” 

In other words, instead of beginning with an unbiased hypothesis such as “to what extent is there turmoil and unease?” the pollsters chose to begin with the belief that there already was turmoil and unease that needed to be captured by a poll.  No surprise then that the pollsters found themselves in an echo chamber.

Poll results are not upheld by reality

Let us for a moment assume the poll results to be true.  According to the Mumbai Mirror, close to two thirds (62%) of respondents are “reluctantly” tied to the leadership of Syedna Mufaddal Saifuddin out of “fear” whereas close to half (46%) support the claimant Khuzaima Qutbuddin.  Yet these numbers are not in sync with past reports published in both the Mumbai Mirror and the Hindustan Times. 

Mumbai Mirror’s article on January 18, 2014 “We’ve lost our father today” reported on the hundreds of thousands of Dawoodi Bohras who filled up Mumbai’s streets from Malabar Hill to Bhendi Bazar to attend Syedna Mohammed Burhanuddin’s funeral and to express their condolences to Syedna Mufaddal Saifuddin. 

The Hindustan Times in its coverage of the event published the following photograph on January 19, 2014 with the caption “More than one lakh [100,000] people were part of the funeral procession that started from Saifee Mahal at 9 am and ended at Saifee Masjid in Bhendi Bazar around noon.”

Getty Images / HindustanTimes, January 19, 2014. 

The funeral, let us be reminded, was led by Syedna Mufaddal Saifuddin and the only noteworthy no-show in the sea of grievers was Khuzaima Qutbuddin.  If there was a gathering of an equal or larger magnitude at Qutbuddin’s residence in Thane to mourn the demise of Syedna Mohammed Burhanuddin, neither the Hindustan Times nor the Mumbai Mirror deemed it important enough to cover it.

It is also illogical to think that an overwhelming majority (77%) of the Dawoodi Bohra population continues to remain in the fold because of “family, cultural identity, and fear” and “most are suffering in their private corners, fearful of what’s going to happen to them.”  Any Dawoodi Bohra will tell you that the cultural identity of the community is defined by the incumbent head.  Does it not seem ironic that so many respondents would have an affinity for the cultural identity created by a leader who they do not accept?

Also, is it really credible to think that such a large majority should have to fear social boycott from a minority?  United by its common fears and suffering, one would expect this allegedly oppressed majority to quickly come out of its private corners and regroup and create an independent social and cultural order its constituents so covet.

Furthermore, newspaper coverage of Dawoodi Bohra events shows that Dawoodi Bohras have only continued to increase their attendance counts in community events after June 2011 when Syedna Mohammed Burhanuddin publicly declared Syedna Mufaddal Saifuddin as his successor in Raudat Tahera in Mumbai.  It should be remembered that after this declaration, all events presided over by Syedna Mohammed Burhanuddin had sermons delivered by Syedna Mufaddal Saifuddin.

In fact, in its last year’s report on the annual event of Moharram the Hindustan Times stated on November 14, 2013:  70,000 Dawoodi Bohras came to Mumbai from other countries such as North America and Australia, while about 30,000 Dawoodi Bohras have come in from across the country. The rest [100,000] are Mumbaiites [who are attending] the discourses by the Syedna’s heir apparent Syedi Mufaddal Bhaisaheb Saifuddin at Saifi Masjid in Bhendi Bazaar.

A year later covering the annual event, on October 27, 2014, the Times of India (owned by The Times group that also owns the Mumbai Mirror) reported:  2.5 lakh Dawoodi Bohras from across the country and abroad have descended on [Surat, Gujarat] to attend the ‘vaiz’ (religious sermon) to be delivered for nine continuous days by [the] 53^rd [Dai al-Mutlaq] Syedna Mufaddal Saifuddin.

It would be difficult to argue that the ever-growing crowds congregating to Syedna Mufaddal Saifuddin year after year from all over India and the world are doing so out of coercion or fear.  If we are to resort to statistics to back an argument or a person, the strength of attendance counts quoted by the very newspapers in question only confirm Dawoodi Bohras’ unwavering commitment to Syedna Mohammed Burhanuddin and their unshakable belief in and support for his chosen successor Syedna Mufaddal Saifuddin.

Capturing reality

If the Hindustan Times and Mumbai Mirror are serious about understanding the truth about Dawoodi Bohras I urge them to resist the temptation to rely on mysterious and sensational polls and instead visit Dawoodi Bohra neighborhoods in Mumbai, be they in Bhendi Bazar, Mazgaon, Fort, or Malabar Hill. 

There they will be hard pressed to find any disquiet, turmoil, or unease.  If anything they will find exactly the opposite.  Our peace of heart and mind comes from knowing that Syedna Mohammed Burhanuddin chose Syedna Mufaddal Saifuddin as his successor in a ceremony that is engraved in our collective memory.  Every single Dawoodi Bohra man, woman, and child witnessed it either in person or via video relay on that fateful day on June 20, 2011.