Thursday, March 22, 2012

When in Doubt, Blame the Journalist

Within minutes of hitting "publish post" on my mission statement, I found an article that reminded me of one of my worst pet peeves when it comes to data/science/studies of all types.  The headline read  "Keeping Your Name? Midwesterners Are Judging You".  My ears (eyes?) perked up at this headline, as I am among those women who declined to change her name post-nuptial.  Despite knowing that Jezebel is not often the best place for unbiased reporting, I gave it a read.  

The article linked to a much more well nuanced article here, but the basics are as follows: students at a small midwestern college feel that women who don't change their last names when they get married are less committed to their relationships than those who do.  This was interesting in part because the number of people who felt negatively about this quadrupled between 1990 and 2006.  

For the personal reasons listed above, I find this interesting.  However, when you look at the numbers (2.7% of 256 and 10.1% of 246 which Jezebel did include) and do a little math, you realize that this "jump" is a difference of 18 people.  

A few things to consider about this:
  1. I couldn't find that this was published anywhere.  It seemed to be a sort of "FYI for the headlines".
  2. Apparently there's no data on whether or not this perception is true.  My bias would be that it's not, but I couldn't find data actually saying if the perception was correct.  This happens in many "perception" studies....they quote percentages who believe something with the implication that a certain belief is wrong without ever proving it.
  3. There wasn't a gender breakdown of who those 18 people were.  If most were female, then isn't their perception likely to be based on experience?  As in "well if I didn't do it, it would be because I wasn't committed"?  That not judgement of others, that's judgement of self.
  4. Have any of their professors (or TV shows, or other media sources) recently made disparaging remarks about this?  18 people who all very well might know each other (the university surveyed was under 1000 students) could easily be influenced in their answer  by even one strong source.
  5. As college students, presumably very few of those polled were actually married.  From my experience in college, I would conjecture that this is a phase of life during which people are very idealistic regarding their future mates without having many real experiences to back it up.  I put much more stock in what people who are actually married use to feel out level of commitment than what someone who's never walked down that aisle thinks.
All that being said, it looked like the study authors were careful to address several of these points (especially the "this is not a representative sample" point.  It was only in the translation that conclusions were drawn that were more dubious.  

Scientists have very little incentive to exaggerate the meaning of their findings.  They are in a profession where that could be very damaging.  Reporters for both old and new media have EVERY incentive to spin things in to good headlines.  Remember that.

3 comments:

  1. Yup. Other possible confounding factors. Colleges have become less regional since 1970, when nearly everyone stayed in region. I don't know how 1990-2006 fits into that, but it could mean that the two populations were not as similar as might superficially appear. Second: the African-American community is starting to use hyphenated names for children when the parents are not married (and not married is a pretty good proxy for less committed, I would think.) Though that is the children, not the mothers, the association for aspirational whites might have some negative spillover.

    You are most correct about the perception-and prediction polls. "Do you think earthquakes are more likely now because of climate change?" What we think has nothing to do with anything. The earthquakes will happen according to their own rules. Similar questions about social issues all the time. Drives me nuts.

    There may be actual data on the more/less committed question, but it would be tough. One, some of your friends will hate you, no matter what happens - and if it's your boss or professor, so much the worse. Second, marrying earlier or marrying later, years of education, and number of children would all have to be factored out.

    ReplyDelete
  2. I've seen a little tendency to exaggerate a bit when talking to the funding agencies, but the agencies manage that by using other scientists to do the reviews. After a while you get to know who just throws up the best resolution plot and who puts up typical plots.

    But when a reporter gets ahold of the story it's like a whole game of telephone compressed into one writer.

    ReplyDelete
  3. BK, I don't know the stats on this, but I would also wonder how many of those surveyed are married or in some form of long term committed relationship themselves. That certainly colors their commentary for me.

    ReplyDelete