Thursday, April 12, 2012

Age Bias and Polling Methods

A few years ago, in one of my research methods classes in grad school, a professor I had asked us to raise our hand if we had a cell phone.  

Everyone raised their hands.  

Then he asked people to keep their hands up if they had a land line as well.  

Many hands went down.  

For those left, he asked how many answered it regularly or had caller ID and screened calls.  

Pretty much everyone.

This of course then led in to a discussion of political polling and how many of us had ever considered who was actually answering these questions.  It was an interesting discussion, as pretty much the entire class admitted they would have self excluded.  The Pew Research center suggests this was not an anomaly, and that this is actually a problem that's becoming more acute in political polling.  

While many large national polling organizations have started calling cell phones as well, on the state level this is not often corrected for.  This can, and has, resulted in some inaccurate polls, as the sample of people home, with a landline, willing to answer a pollsters call, does not always reflect the general population.  Actually, I think there's good reason to question the representativeness of a sample willing to answer their phone for an unknown number, but that could be disputed (those interested enough to pick up the phone also might be more likely to actually go vote).  

Anyway, none of this is new.  What is new this (presidential) election cycle is that news organizations are now starting to put up stats on Twitter and Facebook status updates.  I decided to take a look and see exactly how skewed these stats are, and found that Twitter is most popular in the 18-29 demographic.  Of course, this is the least likely demographic to actually vote.  Interestingly, the poll on Twitter usage did not include people under 18, but these are not excluded when they are compiling trends.  

So two different ways of tracking elections, two different sets of flaws.  Pick your poison.


  1. The famous example of polling problems happened in 1948 when all the polls said Dewey would be our next president. When Truman won, there was a lot of soul searching. They determined that more Republicans had telephones than Democrats so the sampling was biased. Back then, telephones were still somewhat of a luxury (phones had a lusury tax until the 70's) so it was far from a random sample.

  2. Bad sampling...

    I actually think there might be something to the idea that conservatives score lower than liberals and moderates. Lord knows I’ve met my share of unthinking conservatives. I can at least follow a train of thought that says Conservatism can be defined as preference for the status quo, and that has fewer intellectual requirements. I don’t know what that means when liberalism is the status quo, or if there are limits to all of this. And “acceptance of hierarchy” seems odd. What’s that mean in their minds? But I don’t reject the idea out-of-hand.

    But this population sample? Seriously? You have decimal points in your Mean, and do a regression analysis, but don’t control for what friggin’ bar you went to? Like, they must be all the same, right? (The study design and interpretation is bad enough, and I already have deep suspicion of all academic psychologists, but I’m not talking about those here.)

    85 New England Bar Patrons?

    1. But this population sample? Seriously? You have decimal points in your Mean, and do a regression analysis, but don’t control for what friggin’ bar you went to?

      Ha! This reminds me of the part of "How to lie with statistics" where he talks about someone (Stalin maybe?) who gave some talk where the logic was "if we presume people work for 8 hours, then presume they work 300 days a year, etc etc we get each person providing 3.51 something something". Anyway, he probably spent a page afterwards railing about how the whole stat was based on presumptions and rounding, and then was somehow magically able to predict to two significant figures what was going on.

      As for the politics of drinking....I'm sad they didn't repeat the study on pot smokers. I'm pretty sure the results would have shown some impaired thinking favors liberals.