Defending the CNN Poll, a Little

CNN's poll has Obama dropping 10 points in 100 days.

Ed criticizes the poll, which still puts Obama at an unexpectedly-high 56% support, but he's off-base in this specific criticism, I'm pretty sure:

Interviews with 1,136 adult Americans, including an oversample of African-Americans, conducted by telephone by Opinion Research Corporation on July 31-August 3, 2009. The margin of sampling error for results based on the total sample is plus or minus 3 percentage points.

Did they correct for that oversample? Did they weight the responses? CNN’s polling data doesn’t explain at all, which leads one to the conclusion that they kept the oversample in the final data.

Oversampling does not mean overweighting.

If you want to, say, conduct a broad national poll but also make specific conclusions about what a minority might think, you have to oversample them. I think the minimum -- minimum -- number of respondents you need to put out numbers for any question is around 400. The academic/business organization that sets standards for this sort of thing is the AAPOR, sets 400 as the minimum number for meaningful results for any poll question -- and that includes a sub-segment of the poll's total universe of respondents, too.

So, to get meaningful results from black respondents specifically, you need 400 of them. At a minimum.

Now, blacks are like 12% of the population so if you are polling 1000 Americans you're likely to only get 120 blacks. That's fine, if you don't want to report specific responses from the black population.

But if you want to report those responses, and want them to actually be statistically meaningful, you need to get that sub-cohort up around 400.

Does that mean they didn't weight black responses back down? No, of course they did. Any polling company which didn't would be setting itself up for not only ridicule but loss of business, as its competitors pointed out the gross error of postulating a 37% black US population.

They weighted them back down to 12%, or 13%, or whatever the current Census figures indicate.

As for polling adults rather than voters: Well, it's true that adults are the least predictive sample... if you're predicting election results. If you don't care about who wins an election, but rather only what adults think, whether they vote or not, it's the appropriate sample.

In this case, their decision was likely made not according to what they really wanted to measure -- adult mood or likely future election results -- but according to cost. Anytime you screen out respondents, it costs you money. You spend more on your callers' salaries (paid by the hour, mostly) because they're making many more phone calls than they're actually generating responses (i.e., stuff you can use in your poll).

In this case, CNN had already made one costly decision -- calling lots of people to try to get those 400 black respondents -- and cheaped out on making it more costly by screening for voters (or, even more costly, likely voters).

At any rate, there are a lot of ways that a pollster can manipulate results -- the biggest assumption being about current party identification figures, those being kinda anyone's guess and therefore pretty variable between pollsters -- but grossly overweighting a well-known and easily-checked stat like "percentage of blacks in the US" isn't one of them.

I mention this not to call Ed out, but because it's, well, true, and I've been seeing stray references to an "oversampled CNN poll" in the comments.

By the Way: The minimum 400 rule gets violated all the time.

The media violates it all the time when they report statistically-insignificant stuff like "unmarried Asian females," a group their poll sampled like, what, three of? You asked three unmarried Asian females a question and now you're gonna get all conclude-y about them?

But polling companies themselves aren't supposed to. They too will write lots of analysis about tiny sample-size questions and pass those to clients, to demonstrate they're "getting their money's worth," but they're not supposed to. A lot of those micro-sized break-out questions are absolutely meaningless and the AAPOR says not to report them at all.

One Other Way the Media Biases Polls: They bury polls they don't like and trash their own polls.

I forget which poll it was, but one of the big networks found a lot of support for McCain/Palin, and simply didn't report the poll. When they eventually did have to report it (embarrassed into it), they trashed it.

Now that poll was probably a bad poll.

But when a network finds support for Obama and the Democrats at a suspiciously-high level, they don't bury or trash their poll. They vigorously promote it, claiming it heralds a major come-back or blah blah blah.

If they were being honest and consistent, they'd either report all polls equally enthusiastically, or equally bury/trash polls that were out of kilter with other polls.

They don't. They don't mind when their polls come in overly-optimistically for the Democrats. To them it's win-lose -- hey, it's a garbage poll, but we win, because we inject a little optimism into our Democratic friends.

A poll coming in too high for the GOP is lose-lose, and they trash it.

Heh: Ed's right about this:

One of the questions asked, “How do you personally feel about Barack Obama being president?”, has the following options for response: Thrilled, Happy, Don’t Care, Unhappy, Depressed, No Opinion. Depressed? CNN offers two kinds of responses for happy, one for unhappy, and one that asks people to diagnose themselves with a clinical disease.

Not just a disease, but a sin, isn't it? Don't Catholics teach that allowing yourself to stew in depression is self-indulgent and sinful?

Biased. Why not "very displeased"?

Or maybe CNN can really cook the books and ask,

"How do you personally feel about the President?"

Happy, plus I'm really good-looking

Unhappy, and I've gone all pear-shaped and I smell like funky cheese

Angry, crazed, and racist, and also my dick is the size of watch-battery and/or I've got chronic swamp-ass

posted by Ace at 01:30 PM

| Access Comments