Opinions, everybody's got one

I was listening to a management podcast recently where a man named John Blackwell was being interviewed.  He was talking about how he was constantly reading things about how the whole workplace was changing, but he was getting curious as to why he felt like the companies he worked with weren't reflecting this.  When he tried to investigate, he found out that the ongoing surveys commonly used in British management journals (can't find a link) were being done on the "up and coming business leaders".  When he looked in to what that meant, he realized it was people who were second year MBA students.

The problem with this, of course, was that this was asking people not in the workforce what the workforce was going to look like 10 years from now.  They found, not surprisingly, that young people in grad school tend to be very optimistic about things like "working from home" or "flex time" when they're in school, but when they got in to business, they towed toed the line.  Thus, every survey done was essentially useless.  

This all reminded me of a conversation I got in to several years ago when I was working the overnight shift.  Someone had brought in a magazine (People or Vogue or something like that) and they had a ranking of the 100 most beautiful women in Hollywood.  Drew Barrymore was number one that year, and one of my (young, male) coworkers was actively scoffing at that.  "She's unattractive," he stated definitively.  "All the guys I know think so too."

Now, I was feeling a little feisty feminist that night, so I thought about how to challenge him on that.  Leaving aside that "Hollywood unattractive" would still turn heads in any average crowd (and be more attractive than any girl he'd dated), something about his comment irked my data side.  "So maybe the voting was done by women," I replied.  

He was floored.

I noted that it was not a men's magazine that ran the story, so really women's opinions of other women's attractiveness would actually be more relevant to this list.  Furthermore, as most of the leading women in Hollywood make their money on romantic comedies, professionally women's opinions of their attractiveness (which presumably included a certain likeability factor) would actually matter more than men's.

I was fascinated that this clearly disturbed him.  It had clearly never occurred to him that straight men may not be the target audience for female attractiveness, or even that the relevance of his opinion might get questions.  He wasn't trying to be a jerk, he was legitimately confused at the whole idea.

A long intro, but the bigger point is important.  In any opinion survey or research, it's important to figure out whose opinion is most relevant to what you're trying to get at and why.  When it comes to law and public policy questions, I think every voter is relevant.  When it comes to workplace trends?  You may need to narrow your sample.

Sampling bias is a huge problem in many contexts, but my primary one for today's post is when the survey was not conducted with the end in mind.  For any sample, you have to figure out how much your subject's opinions actually matter given what you're trying to find out.  In social conversation it may be interesting to find out what a particular person thinks of a topic, but for good data, show me why I care.