5 rules for reading "scientific papers" online

This weekend I stumbled in to a very dark place on the internet.

It wasn't political, oh no, it was something much much scarier, with far more demeaning adherents.

It was the world of parenting theory.

I'm not going to name the theory, because quite frankly, I'm afraid they'd find me....but lets just say it's considered pretty fringe by most people.  Don't tell them that though, they went after critics (and even people just asking questions about tiny modifications) with a viciousness that made CNN commenters look rational.

Anyway, as a response to many questions, they kept linking to a paper by a doctor who was part of their movement to prove any and all points.  Finally, I decided to go see what all the fuss was about.

I won't link to it here, because I am seriously scared of these folks, but the paper got me thinking about how often this sort of thing appears on the internet, and things people should look for when consuming "scientific research" that shows up on random websites.  Many readers of this blog are probably already good at this, but for those who may not be so savvy, here we go:

1. Remember that pretty much anything can call itself a "scientific paper". When I clicked on the link above, I was surprised to find it was really just an opinion piece by a doctor.  While I'll take it at face value that she actually exists and has the credentials she claimed, this was not a peer reviewed paper published in a journal.  That's fine, but calling it a "scientific paper" definitely gave it a little more credit than it deserved....it's like calling something in the supermarket "natural".  It sounds good, and you think you know what it means, but there's really no standard for it.
2. Beware people who jump around a lot.  I don't mean to be picky, but for the life of me I couldn't figure out what the point of this paper was.  There was no thesis statement, just lots of disjointed "some critics say this" followed by some anecdotes and then the next topic.  It doesn't sound that bad, but believe me, it was.  When I finished I went back through and realized that she was giving just enough rebuttal to raise doubt, but moving on before she had to make a full argument.  Any one of her comebacks would have fallen apart if it had been responded to, but by jumping around with her topics it made you forget what you were objecting to.  Haven't we all had an in person argument like this?  Where the person never responds to a counter argument, they just throw another objection at you?
3. Follow up on citations.  I had a few minutes to kill while I was reading this paper, so I decided to actually follow the citation trail she gave for her points.  It was amazing how few of them said what she said they said.  Many of them were close, but a few were downright deceptive.  This was uncovered merely by reading the abstract.  I can only conclude they were either mistaken links or the author thought no one would follow up.
4. Don't presume the citation supports the entire sentence.  This was the most deceptive part of the paper.  At least twice there was a sentence that said something like "You should do A because A leads to B which helps with C and that prevents D from happening, as researcher X has proved"  Now none of these were intuitive connections, all were based on her particular strategy.  When I followed up on researcher X's work, all the paper said was that C prevented D....which was the least controversial statement.  She offered no actual proof that her methods A and B would actually increase C...but her links implied it was there.
5. Watch out for nitpicking. I know, I'm the last one who should call someone out for nitpicking other people's studies.  However, this is a favorite technique of those trying to prove a point.  They take opposing research, point out a weakness, then proclaim the study invalid because of this.  The problem is that some of it doesn't actually make studies invalid.  Sometimes it's minor things like "those studies on gender pay gap didn't include transgendered persons so they don't really prove anything!".  In this case she faulted the opposition for using too narrow of a definition for one of her recommended practices, and then two paragraphs later mentioned that the narrow definition was really the one she advocated for.

While I stumbled on these in parenting theory, I've seen similar on nutrition/lifestyle websites in particular.  I'm sure there are other examples and other rules.  Would love to hear your thoughts.

Moral obligations and Lazy Truth

I was going to include this in a Friday link post, but I really felt it deserved it's own spotlight.  

There's a new gmail gadget called "Lazy Truth" that promises to send you a fact check email every time you receive a (forwarded) email it deems to be of dubious content.

I haven't tried it, so I'm not sure what it's set up to flag, or how accurate the "fact check" email is, but I was immediately intrigued.  I've actually been working on a much longer post that covers just this topic, so it's something I've been giving a lot of thought.

I've been mulling over the rise of Facebook/email/Twitter lately, and wondering.....for those of us who value our integrity and our truthfulness, and do not believe ends justify means, what exactly are the moral implications of hitting forward or share on information that we could have easily proven to be false if we'd checked?

I was wondering if I was the only one worried about this, when I came across a blog post from Dr Michael Eades.  He's a pro-low carb physician, who spends much of his time critiquing nutritional research.  In a post about the book "The China Study", he describes finding what he consider a great critique of it on another person's blog.  Then this:

.... I had fallen victim to the confirmation bias.  My bias was that Dr. Campbell was wrong, so I was more than happy to uncritically accept evidence confirming his error without lifting a finger to double check said evidence myself.  I knew that if a blogger somewhere had come out with a long post describing an analysis of the China study demonstrating the validity of all of Dr. Campbell’s notions of the superiority of the plant-based diet, I would’ve been all over it looking for analytical errors.  But since Ms. Minger’s work accorded with my own beliefs, my confirmation bias ensured that I accepted it at face value. 

Once the fact that I had succumbed to my confirmation bias settled in around me, I became suffused with angst.  I had tweeted and retweeted Ms. Minger’s analysis a number of times, giving the impression that I had at least minimally checked it out and had approved it.  I had emailed it to a number of people, many of whom, I’m sure, had forwarded it on.  I’m sure I played a fairly large role in the rapid dissemination of the anti Campbell/China study info.

In the end, he went back and realized that the post was good, but his panic attack was intriguing to me.  How many of us have had this same panic?  How many of us should have?  How many lousy graphs rip through Facebook like wildfire because no one bothers to double check if they're even valid?  Is the liar the person who created the graph, or do those who share it share some blame?

I don't pretend I have an answer for this.  I feel most of the people interested enough to read this blog probably do not fall in the category of those who would easily share skewed information without thinking about it, but I am hoping for some thoughts/feedback from you all.

Are we so used to hearing politicians of all stripes seamlessly repeat bad data that we've come to view it as acceptable?  Is this just a fact of life?  Is it possible that we will be saved by widgets like the one above?   Does religion matter, or is this an overall moral issue? Does confrontation work with this sort of thing?  Or is this something I just have to learn to live with?

Arizona Immigration and fake statistics

In case you haven't heard, the Supreme Court ruled on Arizona's immigration law today.

I was not surprised to see this show up on some of the feminist blogs I read, as they generally have a pro-immigration slant, but I was more than a little surprised to see that Amanda Marcotte considers this a women's issue.

In a blog post for the XX blog on Slate.com, she argues that the laws surrounding checking IDs will likely result in racial profiling (certainly) and probably target the young (highly likely) but that this will also target women more than men (wait, huh?).

Her reasoning:

.....women, especially in poor or rural communities, are also much more likely to be out and about without legal identification than men, especially if they don't drive or drive often. Women that are poor or undereducated are much more likely to be stay-at-home mothers with few resources, which makes it very easy to let concerns about up-to-date licenses or ID slip, especially if you don't drive a car much because someone else in the household is using it for work. If your daily life is dedicated to running errands for your family, you may not have much cause to worry about keeping all your papers in order generally, until it's too late and you're finding yourself in jail for not being able to prove citizenship on the spot.

A few comments:

1. I have searched for 20 minutes for any study or proof that women leave the house without their ID more often than men.  I can't find it.  Maybe the idea is that women walk around more than men?
2. Women that are poor and undereducated are not more likely to be stay at home mothers.  56% of SAHM have at least some college education or more.
3. I can't find any hard data on which gender lets their license expire more often, but I also can't find proof that it's women.

I hate statistics based on bad data, but I really hate statistics just pulled from thin air.  Some assertions are self evident for sure....I don't know that many people would argue that a group of teenage boys out on a corner is more likely to be stopped by the police than a group of 70 year old men....but the paragraph above states quite definitively several things that don't seem at all definitive.  I could be wrong, but there weren't any sources attached to check with.  When you factor in the idea that men are probably more likely to be stopped than women, it's hard to figure out where this particular point is coming from.

If you disagree with Arizona's law, that's fine....but don't make up statistics about it's impact on women to justify that.  If it's wrong, it's wrong because it impacts people in general, not women in particular.

Back It Up!

One of the more thought provoking moments of my high school career came from a youth pastor who decided to find an amusing way of calling out a bunch of church kids.  It seemed he had at some point grown weary of hearing too many good church going adolescents start sentences with "Well it says in the Bible...." when what they actually mean was "I heard in a sermon/my Dad says/my mom believes/I read this book once/I'm pretty sure this is true".   Anyway,  he was a clever sort of youth pastor, and he realized that calling out and/or publicly shaming offenders would probably lead to lots of discord, hurt feelings, and possibly calls from parents, so he decided to take a different tack.

Starting with a few key young gentleman, he began to tell everyone that whenever they heard the phrase "It says in the Bible" they were all allowed (in fact encouraged) to all yell "BACK IT UP!!!!"  At this point whoever had made the claim had to stop and find a verse to back themselves up and read it to the group.  If they couldn't find a verse quickly, the conversation continued, and they were condemned to sitting rifling through the concordance until they either admitted they couldn't find it, or they stayed quiet for the rest of the conversation.

I bring this up because I WANT THIS TO BE A THING.

Wouldn't it be great if we could do this with research?  I bet the story I mentioned in my post this morning wouldn't have happened if every time we heard/read someone saying "Research shows" we could all scream "BACK IT UP!!!" and then silence them until they found the proper citation (and no, Wikipedia and Malcolm Gladwell would not count as actual citations).

For the printed word on the internet, we need some sort of meme for this that people could leave in comments sections of articles with vague "research" claims...perhaps a gif of some sort (where are the 4channers when I need them???).  I took a poke around the internets and this is the best I could find was this lady:

GIFSoup

I think this could work.  There has to be an unemployed journalist or two out there who could help me spread this around.