Monday, September 17, 2012

5 rules for reading "scientific papers" online

This weekend I stumbled in to a very dark place on the internet.

It wasn't political, oh no, it was something much much scarier, with far more demeaning adherents.

It was the world of parenting theory.

I'm not going to name the theory, because quite frankly, I'm afraid they'd find me....but lets just say it's considered pretty fringe by most people.  Don't tell them that though, they went after critics (and even people just asking questions about tiny modifications) with a viciousness that made CNN commenters look rational.

Anyway, as a response to many questions, they kept linking to a paper by a doctor who was part of their movement to prove any and all points.  Finally, I decided to go see what all the fuss was about.

I won't link to it here, because I am seriously scared of these folks, but the paper got me thinking about how often this sort of thing appears on the internet, and things people should look for when consuming "scientific research" that shows up on random websites.  Many readers of this blog are probably already good at this, but for those who may not be so savvy, here we go:


  1. Remember that pretty much anything can call itself a "scientific paper". When I clicked on the link above, I was surprised to find it was really just an opinion piece by a doctor.  While I'll take it at face value that she actually exists and has the credentials she claimed, this was not a peer reviewed paper published in a journal.  That's fine, but calling it a "scientific paper" definitely gave it a little more credit than it deserved....it's like calling something in the supermarket "natural".  It sounds good, and you think you know what it means, but there's really no standard for it.
  2. Beware people who jump around a lot.  I don't mean to be picky, but for the life of me I couldn't figure out what the point of this paper was.  There was no thesis statement, just lots of disjointed "some critics say this" followed by some anecdotes and then the next topic.  It doesn't sound that bad, but believe me, it was.  When I finished I went back through and realized that she was giving just enough rebuttal to raise doubt, but moving on before she had to make a full argument.  Any one of her comebacks would have fallen apart if it had been responded to, but by jumping around with her topics it made you forget what you were objecting to.  Haven't we all had an in person argument like this?  Where the person never responds to a counter argument, they just throw another objection at you?
  3. Follow up on citations.  I had a few minutes to kill while I was reading this paper, so I decided to actually follow the citation trail she gave for her points.  It was amazing how few of them said what she said they said.  Many of them were close, but a few were downright deceptive.  This was uncovered merely by reading the abstract.  I can only conclude they were either mistaken links or the author thought no one would follow up.
  4. Don't presume the citation supports the entire sentence.  This was the most deceptive part of the paper.  At least twice there was a sentence that said something like "You should do A because A leads to B which helps with C and that prevents D from happening, as researcher X has proved"  Now none of these were intuitive connections, all were based on her particular strategy.  When I followed up on researcher X's work, all the paper said was that C prevented D....which was the least controversial statement.  She offered no actual proof that her methods A and B would actually increase C...but her links implied it was there.
  5. Watch out for nitpicking. I know, I'm the last one who should call someone out for nitpicking other people's studies.  However, this is a favorite technique of those trying to prove a point.  They take opposing research, point out a weakness, then proclaim the study invalid because of this.  The problem is that some of it doesn't actually make studies invalid.  Sometimes it's minor things like "those studies on gender pay gap didn't include transgendered persons so they don't really prove anything!".  In this case she faulted the opposition for using too narrow of a definition for one of her recommended practices, and then two paragraphs later mentioned that the narrow definition was really the one she advocated for.
While I stumbled on these in parenting theory, I've seen similar on nutrition/lifestyle websites in particular.  I'm sure there are other examples and other rules.  Would love to hear your thoughts.

10 comments:

  1. You are very mysterious, making me very curious. It's not an intentional manipulation on your part, but remember it's how some folks work. I wonder who this is, but if I had to guess, it would be Scientology, because really, who else goes looking for people that disagree with them in order to hurt them? You can blame me for saying it out loud, and you don't have to agree or acknowledge.

    I think you describe well each of the problems with this type of brainwashed scientism, but that may be because I've thought and read some along those lines before. You seem clear to me, but I don't know if others will get your general point, as it is abstract without examples.

    #2, Jumping around a lot. I use the metaphor of the game whack-a-mole. I accuse liberals of it all the time, but really, lots of folks use it. I probably do it too.

    ReplyDelete
    Replies
    1. No, it was a group centered around just parenting theory. No dissent or you're a terrible selfish person with lousy kids. I'm too hormonal to risk attracting them to this blog at the moment.

      Delete
  2. OK first of all, just stop searching for parenting advice online. I don't know why you are doing it, but virtually nothing good can come of it. Stop. Now. ;-)

    You are MUUUCCCHHH better off just talking to people you know and trust who have been there and done that. But...you are sensible. You probably know this already.

    ReplyDelete
    Replies
    1. I didn't mean to I swear. I started looking at a book on a very specific topic and then saw a nasty comment and then followed the rabbit trail and it all went down hill from there.

      Sigh.

      Delete
  3. I'm with AVI: insatiable curiosity and you're taunting me!

    I don't know which is worse: the pretense of science or the raw terror of those parenting support websites where all the "experts" tear a new one into unsuspecting moms seeking advice because you
    A) Let your child cry
    B) Didn't let your child cry
    C) Didn't feed your child organic yak's milk from a glass bottle with an organic hemp nipple AND/OR
    D) Dared let a bottle enter your child's breastfed life before the age of 2 1/2

    I avoid all internet advice (and most friends'/coworkers'/colleagues' advice as well...nothing seems to divide people like parenting talk) and stick to a few close friends whom I respect. And just for kicks, I check in with @HonestToddler for some reality checks on my expectations.

    ReplyDelete
    Replies
    1. Option C almost made me choke on my water.

      The thing that killed me about this site is that people were like "hey I did the 86 things you're in favor of, but then I hit number 87 and it seems like something else really works for my child, is that okay?" and then the teeth came out about how the person was an idiot, they were reading their child wrong, etc, etc.

      I'm pretty sure if that's how they treat their supporters, I'd get eviscerated.

      Delete
  4. I tried googling for "just parenting" and was warned that the second site listed "may harm your computer." Number one was about "socially just parenting", which would harm my temper.

    ReplyDelete
    Replies
    1. I googled "socially just parenting" and the second result was something called "Cinderella ate my daughter". I had no idea Disney characters turned vicious. Glad I had a boy.

      Delete
  5. The secret is to have excellent children that make you look good. Remind them that they are expected to do this.

    ReplyDelete