It's not the question, it's how you ask it

Data gathering is a lot harder than most people imagine.  It's an interesting exercise to take a study and prior to reading it start asking yourself "how would I, if pressed, get the data they claim to have gotten?".  It's amazing how many fall apart quickly when you realize how bad the source data is.

I face this all the time at work.  The simplest questions...what is our demand for transplants? can be a never ending labyrinth of opinion, observation, anecdote, and data....all completely enmeshed.  I spend much of my day trying to untangle these strings, and I never underestimate how difficult getting a simple answer can be.

Factcheck.org ran a great piece today illustrating this challenge.  In a post titled "How Many Would Repeal Obamacare?"  they review 4 different surveys that all try to get to the same number: how many people think healthcare reform should be repealed?

It's a great article that covers sampling practices, question phrasing, date of the poll, and history of the polling organization.

If you looking at the numbers, it shows up pretty quickly that when given dichotomous choices (repeal/keep), people often look like they gave a strong opinion.  In the polls where more moderate answers are given ("it may need small modifications, but we should see how it works), people trend towards that answer.

The phrasing was extremely intriguing though:
“Turning to the health care law passed last year, what is your opinion of the law?”
“If a Republican is elected president in this November’s election, would you strongly favor, favor, oppose, or strongly oppose him repealing the healthcare law when he takes office?”
“Do you think Congress should try to repeal the health care law, or should they let it stand?”

In one, the question focuses on personal opinion, in the next the focus is the presidency, in the third it's Congress.  All of this for a law that most Americans have yet to feel the effects of in any practical way.

Of course this is not to say that a public opinion poll (or 4) makes one side right or wrong. If constitutionality or effectiveness are your concern, nothing here addresses either.  I am enjoying it immensely for the educational value though, and kind of wishing I was teaching a class so I could use this as an example.  Those of us in Massachusetts do have the luxury of sitting back and just sort of pondering all of this....as this has been our world for 7 years now.

That reminds me....were these samples controlled for that????