5 rules for reading "scientific papers" online

This weekend I stumbled in to a very dark place on the internet.

It wasn't political, oh no, it was something much much scarier, with far more demeaning adherents.

It was the world of parenting theory.

I'm not going to name the theory, because quite frankly, I'm afraid they'd find me....but lets just say it's considered pretty fringe by most people.  Don't tell them that though, they went after critics (and even people just asking questions about tiny modifications) with a viciousness that made CNN commenters look rational.

Anyway, as a response to many questions, they kept linking to a paper by a doctor who was part of their movement to prove any and all points.  Finally, I decided to go see what all the fuss was about.

I won't link to it here, because I am seriously scared of these folks, but the paper got me thinking about how often this sort of thing appears on the internet, and things people should look for when consuming "scientific research" that shows up on random websites.  Many readers of this blog are probably already good at this, but for those who may not be so savvy, here we go:

1. Remember that pretty much anything can call itself a "scientific paper". When I clicked on the link above, I was surprised to find it was really just an opinion piece by a doctor.  While I'll take it at face value that she actually exists and has the credentials she claimed, this was not a peer reviewed paper published in a journal.  That's fine, but calling it a "scientific paper" definitely gave it a little more credit than it deserved....it's like calling something in the supermarket "natural".  It sounds good, and you think you know what it means, but there's really no standard for it.
2. Beware people who jump around a lot.  I don't mean to be picky, but for the life of me I couldn't figure out what the point of this paper was.  There was no thesis statement, just lots of disjointed "some critics say this" followed by some anecdotes and then the next topic.  It doesn't sound that bad, but believe me, it was.  When I finished I went back through and realized that she was giving just enough rebuttal to raise doubt, but moving on before she had to make a full argument.  Any one of her comebacks would have fallen apart if it had been responded to, but by jumping around with her topics it made you forget what you were objecting to.  Haven't we all had an in person argument like this?  Where the person never responds to a counter argument, they just throw another objection at you?
3. Follow up on citations.  I had a few minutes to kill while I was reading this paper, so I decided to actually follow the citation trail she gave for her points.  It was amazing how few of them said what she said they said.  Many of them were close, but a few were downright deceptive.  This was uncovered merely by reading the abstract.  I can only conclude they were either mistaken links or the author thought no one would follow up.
4. Don't presume the citation supports the entire sentence.  This was the most deceptive part of the paper.  At least twice there was a sentence that said something like "You should do A because A leads to B which helps with C and that prevents D from happening, as researcher X has proved"  Now none of these were intuitive connections, all were based on her particular strategy.  When I followed up on researcher X's work, all the paper said was that C prevented D....which was the least controversial statement.  She offered no actual proof that her methods A and B would actually increase C...but her links implied it was there.
5. Watch out for nitpicking. I know, I'm the last one who should call someone out for nitpicking other people's studies.  However, this is a favorite technique of those trying to prove a point.  They take opposing research, point out a weakness, then proclaim the study invalid because of this.  The problem is that some of it doesn't actually make studies invalid.  Sometimes it's minor things like "those studies on gender pay gap didn't include transgendered persons so they don't really prove anything!".  In this case she faulted the opposition for using too narrow of a definition for one of her recommended practices, and then two paragraphs later mentioned that the narrow definition was really the one she advocated for.

While I stumbled on these in parenting theory, I've seen similar on nutrition/lifestyle websites in particular.  I'm sure there are other examples and other rules.  Would love to hear your thoughts.