Wow: Daily Kos Admits Highly Publicized Research 2000 Polling Was "Likely Bunk"

To say he "admits" it isn't quite right; this isn't a grudging admission, but a statement that he feels he was defrauded. So, he publicized bunk polls, but at least he's copping to it.

I have just published a report by three statistics wizards showing, quite convincingly, that the weekly Research 2000 State of the Nation poll we ran the past year and a half was likely bunk.

Since the moment Mark Grebner, Michael Weissman, and Jonathan Weissman approached me, I took their concerns seriously and cooperated fully with their investigation. I also offered to run the results on Daily Kos provided that they 1) fully documented each claim in detail, 2) got that documentation peer reviewed by disinterested third parties, and 3) gave Research 2000 an opportunity to respond. By the end of last week, they had accomplished the first two items on that list. I held publication of the report until today, because I didn't want to partake in a cliche Friday Bad News Dump. This is serious business, and I wasn't going to bury it over a weekend.

We contracted with Research 2000 to conduct polling and to provide us with the results of their surveys. Based on the report of the statisticians, it's clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don't know if some or all of the data was fabricated or manipulated beyond recognition, we know we can't trust it. Meanwhile, Research 2000 has refused to offer any explanation. Early in this process, I asked for and they offered to provide us with their raw data for independent analysis -- which could potentially exculpate them. That was two weeks ago, and despite repeated promises to provide us that data, Research 2000 ultimately refused to do so. At one point, they claimed they couldn't deliver them because their computers were down and they had to work out of a Kinkos office. Research 2000 was delivered a copy of the report early Monday morning, and though they quickly responded and promised a full response, once again the authors of the report heard nothing more.

While the investigation didn't look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else. I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases. I hereby renounce any post we've written based exclusively on Research 2000 polling.

Okay, that's a clear admission.

But note he's renouncing any post he's written.

Not any book.

I point this out because he's got a book about "The American Taliban" (that would be you all) coming out in two months, and it seems to me very likely that his various Research 2000 polls which painted conservatives in an extremist light were part of his case there.

I guess he'll probably delete references to the Research 2000 polls while keeping all the conclusions, because, let's face it, it's not as if his beliefs were based on polling and other objective indicators anyway.

Thanks to Slublog and Drew.

Falsified? The actual statistical case against the polls is not one about mere methodological failure, but of simply making stuff up.

1. A large set of number pairs which should be independent of each other in detail, yet almost always are either both even or both odd.

2. A set of polls on separate groups which track each other far too closely, given the statistical uncertainties.

3. The collection of week-to-week changes, in which one particular small change (zero) occurs far too rarely. This test is particularly valuable because the reports exhibit a property known to show up when people try to make up random sequences.

The statistical analysis suggests that these numbers were simply made up on the fly, and the reason for these odd occurrences is that people screw up when trying to make things look random.

For example, in point number 2, a smaller sub-sample should show much wilder variation than your main (big) sample, because you're polling many fewer people in that subgroup and hence the Margin of Error is of course larger. But Research 2000 consistently seems to find what you'd call "likely numbers" for these groups -- numbers that seem to make sense, that accord with your gut instinct.

But that's a mistake, because, statistically, with these small subgroups, you should be getting what looks like clearly outside-the-margin-of-error stuff very frequently. In other words, if your numbers are genuine, statistics demands that a lot of these small sub-group numbers look like complete and utter garbage.

If they all keep looking like what you'd expect them to look like, that's because they're not real, but rather someone making up numbers that accord with your gut.

Not So Random Walk?: I think for the past year people have cited these polls, after an initial skepticism, on the theory that they sort of looked like the other polls, only a little more skewed liberal.

Well, it turns out that maybe they looked like the other polls, only skewed a bit more liberal, because they just took the other polls and pushed the numbers a little more liberal.

I have joked around with a pollster I know that I can do just that for far cheaper than her company can -- hey, pay me $10,000, I'll just take a blend of polls from Real Clear Politics and push the numbers around slightly. I give you a discount because I'm just making this up based on other polls and it takes me only a few hours to fake it up; but, on the other hand, let's face it, it's almost as good as the real thing, right?

I mean, honestly, how far off will my fake numbers be from the real ones?

I don't know what's happened here and am reluctant to malign a company with an allegation this large (which, if true, would surely destroy the company entirely).

But they haven't responded to Kos about these problems for two weeks.

Will the MFM Retract? The media didn't use a lot of these polls, but they did use them on occasion. Like, Ezra Klein, lefty blogger hired by the Washington Post.

TPM isn't real media but the lefty media treats it like it is; they loved themselves some R2K polls, too.

Thanks to Slublog for that, again.

Kos: I Can't Wait For You To Read My Book Because It's Loaded With Great Research 2000 Polling Info: I noted that his book American Taliban had been partly based on R2K polling.

From NRO, here's Kos pimping that aspect of it.

As I've mentioned before, I'm putting the finishing touches on my new book, American Taliban, which catalogues the ways in which modern-day conservatives share the same agenda as radical Jihadists in the Islamic world. But I found myself making certain claims about Republicans that I didn't know if they could be backed up. So I thought, "why don't we ask them directly?" And so, this massive poll, by non-partisan independent pollster Research 2000 of over 2,000 self-identified Republicans, was born.

The results are nothing short of startling.

It's a long poll, so the results are summarized below the fold. For a direct link to the poll's crosstabs, click here.

Ultimately, these results explain why it is impossible for elected Republicans to work with Democrats to improve our country. Their base are conspiracy mongers who don't believe Obama was born in the United States, that he is the second coming of Lenin, and that he is racist against white people. They already want to impeach him despite the glaringly obvious lack of high crimes or misdemeanors. If any Republican strays and decides to do the right thing and try to work in a bipartisan fashion, they suffer primaries and attacks. Even the Maine twins have quit cooperating out of fear of their homegrown teabaggers.

Given what their base demands, and this poll illustrates them perfectly, it's no wonder the GOP is the party of no.

"Given what their base demands, and this poll illustrates them perfectly, it's no wonder the GOP is the party of no."

Wow! It's almost like the poll knew exactly what you wanted it to say, dude!

Man did you get lucky!!!11!eleventy!!

posted by Ace at 02:10 PM

| Access Comments