Thursday, March 29, 2012

Why most nutritional research is useless

Nutrition research is big money these days.  Our national obsession with weight loss is at a fever pitch, and any new or interesting research is sure to make headlines.

Here's some basic guidelines on what to look for in nutritional research (any study, not just this one):
  1. Was the data self reported? Even CNN brought this up in their article.  People, especially those embarrassed about their weight, don't accurately assess what they eat.  My mother, skinny little thing that she is, could eat one peppermint patty and tell you she'd had a serving of chocolate.  I don't think I'd even count it until I had 3 or so.
  2. How much was "more"? I actually can't find this for this study.  Is it the difference between 1 and 2 servings per week?  Or the difference between 1 and 5?  Both would produce statistically significant correlations, but the practical outcome would be different.  In 2005, researchers made news by saying that eating more fruits and veggies did not, in fact, prevent cancer.  The cancer treating establishment (which I work in, btw) promptly responded by pointing out that they compared people who ate half a fruit per day to those who at 1-2 fruits per day.  It was all reported in grams too, so the data look extra impressive "Those eating less than 114 grams showed no difference from those eating 367 grams".  The link gives more examples, but 250 grams is one medium apple.  Watch out for this.
  3. Who classified people as "normal weight" or "overweight"?  If this was also self reported (and in this study, it looks like there were clinic visits), then look out.  There's a great study I can't find right now that shows that women tend to lie about weight, and men tend to lie about height.  Both lies will screw up the BMI calculation (the most common metric for assessing "normal").
  4. Were the overweight people actively (or even somewhat) trying to modify their diets to lose weight?  A few years ago, I heard about the study that suggested diet soda was linked to obesity.  I remember my first reaction was "are we sure they're all not just on diets?".  This seemed like a classic correlation/causation issue.  All the analysis seemed to presume they were overweight because they drank diet soda.  I wondered why they never seemed to look at the idea that they could be drinking diet soda because they were overweight.  That's one of the first swaps most people I know make when they try to lose weight.  
  5. Don't even get me started if it's a population study.  That's a big topic for another time, but lets just say they're really really tricky.
If you ever want a fabulous crash course in how nutrition research can be skewed, pick up two diet books that contradict each other, and read through their parts on research.  Take something like Atkins (high protein, low carb) and Joel Fuhrman (nearly vegan), and watch them rip to shreds the research the other one builds their whole case on.  

He may have his own controversy, but this is why I like Michael Pollan.  The book I linked to has a great crash course in why most nutritional research just sees what it wants to.  He refused to take a strict nutritional stance and instead condensed it down to a few "rules" that he gleaned from quizzing nutritionists on "what they could say for sure".  The answer? Eat real food, not too much, mostly plants.  


  1. I like the food guideline you end with! Though I must point out that from a scientific perspective, words like "real", "too much" and "mostly" leave a lot of room for interpretation...

    When it comes to nutritional studies, my rule of thumb is that there are only two reasons that they'd be widely enough publicized for me to hear about them:

    1) Somebody wants me to buy something
    2) Someone wants to attract attention and gain readership, so that they can sell advertising, so that I'll buy something

    Hey, what do you know? It's really only one reason!

    Anyway, my point is that I am extremely suspicious of the motives behind any nutrional study that comes to my attention, no matter what it purports to find. Because of this, my inclination is to ignore them and simply try to follow the rule of "moderation in all things" when menu-planning.

    Except that I've found that carbs give me indigestion. Drat! And I love bread...

  2. By evolution, we weren't designed to handle much fat - not many animals in the wild can afford the loss of speed - and grains are pretty recent. Next, not many of our ancestors ever had superabundance. Our bodies are programmed to keep eating stuff that they don't really know what to do with. So our brains cobble together various plans to correct for that, all of them at least partly unnatural.

    An impressive percentage of us are living to be 80, as opposed to the 48 that has been the norm for the last 200,000 years. It's hard to see how all this is a crisis.

    1. Those of us who are going to be stuck with the baby boomers Medicare bills have no trouble viewing it as a crisis.

  3. As someone who studies nutrition a lot and not for weight loss purposes (see point one, I'm the skinny mom), my life depends on avoiding gluten. Fortunately for me, I am part of a new gluten free craze! Business is booming in the gluten free world and I benefit from all the interest. It seems it doesn't take much to make people an expert on diets. They have studies and anecdotal stories to back them up. Everyone wants to "feel better" so I look at my nutrition/diet/cook books and review diet recommendations that guarantee you'll be looking and feeling younger, be full of energy and enjoy a life bursting with new vitality. For some of them it is in the advertising, if I could look like her why wouldn't I drink green juice every morning. (see Natalia Rose book ). I actually have a couple of Christian books on Biblical theories leading you to a healthier life style with supplements to buy and retreats to attend. So now I'm saying to myself "how can I get into this business?" I'm pretty sure the pay back would be better then my current job. Maybe I'll do a study or maybe I'll stick with anecdotal stories. All sorts of possibilities!

  4. Very amused. My first thought in reading even the article was that if the data were good, the likely reason was that the chocolate eaters were just a bit slimmer. Cutting down on candy is another thing most people do when they are trying to lose weight.

    Also the 3.5 hours of exercise a week sounded over the national average. Someone who's exercising an hour every other day is probably not the norm. Also the study group was selected to be people without metabolic disease, which may account for the 3.5 hours of exercise a week.

    I don't mean to be mean, but if you deliberately select a disease free group, don't expect your findings to be viable for the general population!

    The comments on your link are wonderful:
    Proof is in the (chocolate) pudding… Interesting. I was a 2-3 x per day dark-chocolate addict. I’m a 52 year old active male at 5’8” and 150 lbs. My new year’s resolution was to stop all chocolate. Now chocolate free for 88 days with no other change in diet, I have put on 4 pounds and have my waist size has increased 1.5”. First 3 weeks was extremely challenging with mood swings, headaches and tremors (especially after meals). My goal was to reach 100 days without Chocolate, but now... Is Doctor Godiva in the house?
    This information is exactly right. I am 145 years old (or maybe 144, I forget) and I have been eating dark chocolate as long as I can remember. It is not harmful and might be beneficial.

    The first rule for dealing with bad data has got to be to develop a good sense of humor!

    1. A good sense of humor certainly helps!

      As for the deliberately disease free group....there's been some interesting discussion of that in the cancer prevention research. Those who are willing to be part of a study assessing eating habits are probably more likely to be good eaters, because who wants to admit their bad eating habits?

  5. Yeah, chocolate is a little psychoactive, so that throws things off a bit, too.

    imle should indeed write a gluten nutrition book. The trick would be to give people actual good information while disguising it with anecdotal crap.