Shucking Oysters: And the Survey Says…
By Alex Allen
Did you know that half of Canadians and over half of British Columbians support the new bitumen pipeline project between Alberta and BC? When I heard these stats, my first reaction was one of surprise. My next, one of suspicion. Even with rising political tensions, environmental concerns, and Indigenous push back, polling firms, Leger and Angus Reid, still reported that 50% of Canadians and 54% British Columbians were favourable of the pipeline project? Immediately you should ask, 50% and 54% of what? De omnibus dubitandum est.
Despite all of our technological advances, we haven’t even come close to inventing some device that could read our messy, little minds. Instead, we resort to what seems to be the next best thing: ask people about what they believe and value – surveys and polls. Except we humans often lie about ourselves, which raises the question of whether surveys are even accurate. Talk is cheap and in surveys, there are no consequences in misrepresenting ourselves.
Today, more news media are incorporating polling data in their news feeds than ever. Along with an increase in the number of polling companies and polls, the sampling methods have also changed over the years. Many polling companies, including Angus Reid, Nanos and Leger, have abandoned random probability sampling and have chosen to employ opt-in sampling methods.
With opt-in sampling, people are invited to volunteer their time. The downside of this approach is that a margin of error, which is calculated for the whole population being surveyed, does not apply. In essence, without a margin of error, one has to be cautious about how representative the figure is in relation to a larger “true” population value. The result might be “an estimate based on intuition, convenience, or a flawed methodology, rather than a statistically robust one.” Many news outlets, including the CBC, are overlooking this fact. And it’s not enough to tell readers about a comparative margin of error when most of us have no clue what that even means, let alone how it differs from a normal margin of error.
Canadian pollster and political pundit, Allan R. Gregg, said in a recent interview that when pollsters started back in the 80s, everyone had a landline and the response rates were in the 75% range. While not perfect, everyone had an equal chance to be selected for a survey and, as a result, the responses reflected the population from which they were drawn. Weighting and “rejigging the sample” was rarely necessary. Today, it’s a lot harder to develop a methodology that will guarantee a reliable result.
Recently, there have been renewed concerns about the quality of opt-in sampling. Response rates today already tend to be low, highlighting that those who actually choose to participate in a survey have a “propensity-bias” for completing surveys. What’s more, many opt-in panels have “career respondents” who collect rewards for every survey they complete and may be motivated to lie in their screening or qualifying questions, so they can complete as many surveys as possible.
So let’s dive into the Leger pipeline poll, which was conducted among 1,548 Canadian residents, aged 18 or older, who were randomly recruited through an online panel. That’s 1,548 responses out of a population of 41,494,132 people – a response rate of 0.003875%. This is an exceptionally low metric, representing roughly one response for every 25,806 attempts. In terms of data collection, the figure is far below industry standards. Half of 1,548 responses = 774 Canadians in favour of the pipeline out of over 41 million.
What you don’t read…Support for the pipeline was strongest among Albertan men, over the age of 55, who voted Conservative. Those who viewed the pipeline as essential to national growth, once again strongest among old Alberta men who voted Conservative. Fast-tracking environmental reviews under Bill C-5 for projects in the “national interest,” including the controversial pipeline project? Again, the most support, old Conservative Albertan men.
Across the country however, 68% said it was important to obtain the consent of Indigenous communities before construction. Highest support came from Quebec, older Canadians, Green Party voters, closely followed by NDP and Liberal voters. Not old Conservative farts from Alberta.
The Angus Reid poll showed a slim majority of British Columbians (54%) in favour, conditional on Indigenous ownership, environmental protections and the tanker ban. Once again a red flag on the methodology. First, the survey was conducted among a randomized sample of 1,851 Canadian adults who were recruited from an opt-in Angus Reid Forum. And second, only 517 BC respondents were surveyed out of a population of 5.7 million, a response rate of 0.00907%, below even the lowest benchmarks.
Typically, pollsters ask questions that will elicit yes or no answers. In his book The Surrender of Culture to Technology, Neil Postman asks: “Is it necessary to point out that such answers do not give a robust meaning to the phrase public opinion?” Postman writes that the technique of polling promotes the assumption that an opinion is a thing inside people that can be extracted by questions. But an opinion is not a momentary thing; it is a process of thinking, “shaped by the continuous acquisition of knowledge and the activity of questioning, discussion, and debate.” We don’t really “have” opinions, rather, we are involved in what Postman termed “opinioning” which tells us nothing really. Polling essentially ignores what people know about subjects.
In an open letter in The Walrus in December, “Mr. Carney, about That Pipeline Deal – We Need to Talk,” Julian Brave NoiseCat wrote: “Reconciliation is, in a way, kind of a perfect encapsulation of Canada’s proclivity to say ‘sorry.’ One of the most important human qualities that Canada has to offer its citizens and the world: integrity. Otherwise, what is this country really?”
As NoiseCat reminds us: “Reconciliation rooted in truth, backed by action – that is the path Canada vowed to walk. And not as long as public opinion supported it, but in perpetuity.”
