Weighting on Party: Why It's Dangerous

Conservatives have long complained that polls often oversample Democrats. They get angry when a poll fails to "properly" weight the numbers according to assumed party identification -- reducing the proportions of Dems and increasing the proportions of Reps to the guestimated partisan breakdown.

I used to say this myself. I believe I was wrong, and that those who urge this are wrong now.

Here's the thing: Party identification is not inelastic. It does not remain fixed at 38 Dem/32 Rep/30 Independent (or whatever ratio you might guess at). It changes all the time. Many people are right on the edge of being Democrats or Independents -- when the Democrats please them, they say they're Democrats. When the Democrats piss them off, bang, they're Independents.

Same deal on the Republican side. People kept complaining that 2006 polls didn't include enough Republicans, because the fraction of Republicans was 5% or so lower than longterm historical averages, or what recent (2004) numbers would suggest. But most polls, including Gallup, nailed the 2006 results.

Republicans were not undersampled, mostly. What was going on was that a lot of people who were formerly self-identifying Republicans were disgusted by the Mark Foley scandal, Katrina, and the worsening situation in Iraq and told the pollsters they were Independents. Because, actually, they were. The affinity they once had with the GOP had too much abused of late.

I'm told by someone I trust a great deal that one of the most dangerous things to weight on is Party ID. This is a pollster saying this. Yes, some demographics can be weighted on (age gets weighted a lot, almost always, because there are always too many older people being called and too few young people) and sometimes other demographic designators whose proportional representation in the population we know for a certain fact out to the thirtieth decimal place -- sex, race, economic bracket, education level.

These things do not change. We always know how many Asian people there are in the country. (And thank god for that -- gotta keep tabs. They know karate and ninja skillz.)

But party ID is never known with a great deal of accuracy, and furthermore can change from week to week, sometimes fairly dramatically.

You want to know why Zogby is sometimes very accurate and sometimes seems like he's making up bizarre numbers randomly? Because he weights on party ID. Sometimes his guess as to the current ratio of partisan ID is accurate and he nails a spot-on poll and embarrasses the competition.

But very often his guess is wrong and the numbers he winds up producing seem to have come in from Mars.

It's not so much a bad practice as a dangerous practice. Well, I guess that makes it a bad practice, actually.

Now, here's the weird thing this cycle: You know how you guys have often urged pollsters to weight on party? And correct for oversized samples of Democrats?

The thing is -- now pollsters are getting, it seems, oversized samples of Republicans. And pollsters who do weight on party will knock those self-identifying Republicans down in numbers to get them closer to the long-term average or best guess as to current representation.

But what if Sarah Palin has made many actual new self-identifying Republicans, just as Mark Foley made many self-identifying Independents and Democrats?

Pollsters that weight on party will have less dramatic swings and show closer contests, as they will be massaging down and diminishing numbers that actually reflect the true state of affairs.

Case in point: Rasmussnen. Rasmussen weights on party.

For a variety of reasons, the Rasmussen Reports daily Presidential Tracking Poll is less volatile than some other polls and always shows a somewhat smaller convention bounce than reported by others. This is primarily because we weight our results by party identification (see methodology). Looking at the data before adjusting for partisan identification, the Republican convention appears to have created a larger surge in party identification than the Democratic convention the week before. If this lasts, it could have a significant impact on Election 2008.

Note: "If it lasts." At the moment they're assuming this is either bad numbers, or some other anomaly. "If it lasts" presumably they'll change how they weight the various party IDs and give Republicans a larger fraction of the population -- but only if the trend continues long enough to convince them it's real.

Anyway, I think it's interesting, that. Rasmussen used to show better-than-usual numbers for Republicans, because they would weight on party ID and boost Republican affiliation while diminishing Democratic affiliation, guessing at the right "real" mix.

And now, with Hurricane Sarah blowing a lot of Independents to the Republican side, they're doing the opposite, now correcting for "too many" Republicans. That's why they're showing the race dead even, while Gallup shows a 5 point lead (or even a 10 point one).

I don't think it's established, clearly, that Zogby and Rasmussen are wrong to do this. But how accurate their weighting depends, ultimately, on their guess as to the "real" partisan breakdown in America -- and sometimes they can just be off, and sometimes they can fail to take into account a real shift in partisan ID.

Or at least not take it into account until weeks after the trend has been detected in other polls.

Something to keep in mind when looking at the much-touted WSJ/NBC poll tonight, too.

PS: Ed Morrissey found a poll a few months ago where not only did the jagoffs doing the poll sample far more Democrats than Republicans, but on top of that they weighted the Democrats into an even higher proportion -- up to something like 43% or thereabouts.

So sometimes the partisan weighting is not merely dangerous, but positively dishonest.

posted by Ace at 06:03 PM

| Access Comments