Wednesday, April 4, 2012

Opinions, everybody's got one

I was listening to a management podcast recently where a man named John Blackwell was being interviewed.  He was talking about how he was constantly reading things about how the whole workplace was changing, but he was getting curious as to why he felt like the companies he worked with weren't reflecting this.  When he tried to investigate, he found out that the ongoing surveys commonly used in British management journals (can't find a link) were being done on the "up and coming business leaders".  When he looked in to what that meant, he realized it was people who were second year MBA students.

The problem with this, of course, was that this was asking people not in the workforce what the workforce was going to look like 10 years from now.  They found, not surprisingly, that young people in grad school tend to be very optimistic about things like "working from home" or "flex time" when they're in school, but when they got in to business, they towed toed the line.  Thus, every survey done was essentially useless.  

This all reminded me of a conversation I got in to several years ago when I was working the overnight shift.  Someone had brought in a magazine (People or Vogue or something like that) and they had a ranking of the 100 most beautiful women in Hollywood.  Drew Barrymore was number one that year, and one of my (young, male) coworkers was actively scoffing at that.  "She's unattractive," he stated definitively.  "All the guys I know think so too."

Now, I was feeling a little feisty feminist that night, so I thought about how to challenge him on that.  Leaving aside that "Hollywood unattractive" would still turn heads in any average crowd (and be more attractive than any girl he'd dated), something about his comment irked my data side.  "So maybe the voting was done by women," I replied.  

He was floored.

I noted that it was not a men's magazine that ran the story, so really women's opinions of other women's attractiveness would actually be more relevant to this list.  Furthermore, as most of the leading women in Hollywood make their money on romantic comedies, professionally women's opinions of their attractiveness (which presumably included a certain likeability factor) would actually matter more than men's.

I was fascinated that this clearly disturbed him.  It had clearly never occurred to him that straight men may not be the target audience for female attractiveness, or even that the relevance of his opinion might get questions.  He wasn't trying to be a jerk, he was legitimately confused at the whole idea.

A long intro, but the bigger point is important.  In any opinion survey or research, it's important to figure out whose opinion is most relevant to what you're trying to get at and why.  When it comes to law and public policy questions, I think every voter is relevant.  When it comes to workplace trends?  You may need to narrow your sample.

Sampling bias is a huge problem in many contexts, but my primary one for today's post is when the survey was not conducted with the end in mind.  For any sample, you have to figure out how much your subject's opinions actually matter given what you're trying to find out.  In social conversation it may be interesting to find out what a particular person thinks of a topic, but for good data, show me why I care.


  1. Ack! It's toed the line, as in your little piggies touching the line on the gym floor. Other than that, what an interesting post. Much to think about here.
    Oh, and you might care that the first in the Shatterworld Trilogy should be coming out next month.
    I think about you and your comment about Pacific Northwest drivers every time I wave in a car trying to get in my lane.

    1. Fixed! I actually had no idea that was the wrong way of spelling that for the idiom, I just looked up the whole history. Fascinating. I think I had thought it was about tugboats.

      I'm glad to hear my observations on drivers made an impression, and very happy to hear about the book! It's good to be blogging again!

  2. "...but for good data, show me why I care." You go, girl.

    Second-year MBA students. Give me a friggin' break. Researchers are at colleges. When they need subjects, where do they look? Not only are college students who volunteer for stuff a limited sample, each college has a culture and the students self-select. You get different answers from BU students and North Park students, and different answers between majors.

    Not that it matters, really. Popular magazines are trying to get the words "most attractive women" onto the cover, and run pictures. And everyone loves numbered lists. Inter-rater reliability is hardly a concern for them. The data could be generated by chimps pressing buttons for all they care - unless word got out about the chimps, in which case you would have to sell the article to Cracked.