Polling Bias

Pardon this long post. I've been meaning to write this for a while, and would up, as I usually do, stating the obvious with an excess of words.

But it's one day from the election, so if I don't put this up now, I have to wait another two years.

...

Do the polls really mean very much?

Consider that the response rate for telephone polls is about 20%. Twenty. Percent. I knew it was low, but I didn't know it was that low. (And even that's just 20% of the people who have published phone numbers, which is itself a declining proportion of adults.)

We therefore have a huge problem with polling. We're not polling Americans, per se. We're polling a minority of Americans willing to actually sit through a twenty minute to half hour conversation with a complete stranger (or, in some cases, with a precorded human voice).

All polls assume this minority is actually reflective of the actual majority of Americans who will not spend such a large amount of wasted time on the phone. Is that assumption correct? And if it's not, can we determine if this minority is more likely to be a Democrat than a Republican?

Self-selection is always considered a killer in polling. Polls have to be random, not just those who choose to answer a poll. That's why no one at all puts any stock at all in Internet polls-- they're self-selecting. You haven't been randomly called out of a blue from a list of telephone numbers. You chose to answer the Internet poll.

Furthermore, you found the poll via a non-random method. You may have been directed to the poll by FreeRepublic, or perhaps Democratic Underground. Obviously people stumbling upon the polls by such non-random methods will be skewed heavily this way or that.

So such internet polls, with no random selection, and complete self-selection by respondants, are worthless.

But telephone polls, again, have only a 20% response rate. Twenty percent! This is itself a very self-selected group. How much more representative is a self-selceted twenty-percent minority than Internet polls, which we all immediately dismiss as meaningless?

In college I picked up some money working for a "market research group," i.e., a polling company. We did a little politics, but mostly we polled about public perceptions of companies and products, to determine, kinda-sorta, if their advertising was working, if the public had a good opinion of them, etc.

The response rates were horrendous. Who the hell wanted to talk to us?

Generally, from my memory, those who answered were either quite old and maybe not minding a little diversion, or pretty uneducated (e.g., service members).

No, really. Pretty uneducated.

I was polling one time -- a list of about thirty boring repetitive questions -- about the loan company Beneficial. One respondant was willing to spend forty five minutes with me talking about Benefical (it was a loooong poll), comparing Beneficial to other loan companies, telling me how high an opinion she had of Beneficial.

Or, rather, "Beneficiary," which is what she kept calling it, no matter how many times I clearly said "Beneficial" (it was named in the prompt of most questions). What was the loan company she most trusted? Beneficiary. If she was considering a college loan to finance her child's education, which company would she choose? Beneficiary. Was she aware Beneficial was now offering highly competitive rates for second mortgagas? Hell's yeah, she was!

It was completely obvious the woman had never heard of Beneficiary -- er, Beneficial -- but was just parroting "Benificiary" as the answer to every question I asked her. She became bored by the end of this process, and began answering "Beneficiary" before I finished a question.

"Now rate the following loan providers in terms of which you've heard the most about. Beneficial, Fanny Mae--"

"Beneficiary."

"Wait, there's more."

"Beneficiary. I don't know about the others."

I should have terminated the call, but "live ones" were pretty uncommon and hell, we just wanted anyone, ANYONE to answer these stupid things. Hey, if my supervisor had a problem with the interview (listening on the line, as they do), he could have said something. He didn't. He had quotas to meet, too.

So, according to one of the very few people willing to answer questions for the better part of an hour about Beneficary -- I mean Beneficial -- Beneficial was the world's best, most trusted loan company and all of its advertising for the past two years had been a screaming success. Particularly among people who had plainly never heard of Beneficial at all and almost certainly were not in the market for a college or home-refinance loan.

So let's think about most of the people who answer political polls. They're not all like Ms. Beneficiary, of course; I know smart people who answer them, because they want their opinions out there, in order, mostly, to shape the public opinion.

And that's the problem. The sort of people who answer these polls tend to either be 1) drawn from the ranks of those with far too much time on their hands or 2) those actively wishing to shape public opinion.

And they're going to tell you what you want to hear. Especially when it comes to being a likely voter. If you haven't voted in years, if you have such little civic spirit as to not even know where you local polling station is, are you likely to admit it? No, you're not. You might feel a little embarrassed about it. Or you might just want to make sure your voice gets counted.

After all, if you're one of the small minority of people self-selecting themselves to answer questions which may tend to shape public opinion, are you going to let preliminary likely-voter questions screen you out?

Nope -- of course you're going to vote! You're very certain to vote. Of course you voted in the last election, and the last midterm! Of course you know precisely where your polling station is! Why, it's right next to Beneficiary's corporate headquarters!

Even the elderly who answer pollsters aren't necessarily likely voters. True, older Americans are famously dilligent voters. But some of them don't vote. I can't help but imagine older folks who answer these questions tend not to be the spry or vigorous sort of senior citizens, folks who really don't have the time or inclination to dick around on the phone for half an hour. Again, the older respondants probably skew towards the the inactive and the semi-ambulatory -- those seniors unlikely to vote due to failing health and general disengagement.

Dean Barnett points out that the maximum perecntage of Americans voting even in a big Presidential election year is 60%, and yet pollster's typically find that 80% of their respondants are "likely voters."

Okay, "likely," not "certain." Still, poll respondants seem to seriously overstate their likelihood of actually voting, don't they?

Okay, so repsondants to these polls aren't really very representative of most of the American population. But if they just so happen to nonetheless perfectly mirror the entirety of the populace in terms of party ID and partisan sentiment, they would still serve as an accurate barometer of the greater, non-shut-in, non-mutant, non-determined-to-influence-elections population.

Do they perfectly -- or even closely -- mirror the greater population?

Of course they don't. We already know that these polls skew Democratic. They overstate Democratic support by five, six, seven, sometimes even ten raw points above Democrats' actual representation in the polling booths.

Why? Well, Dean Barnett says that liberals want to share their opinions, while conservatives have lives. That's, of course, a deliberately tendentious way to put it, but the fact of the matter is that poll respondants do tend to be those with an inordinate number of free time on their hands. The jobless, the inactive fraction of the elderly.

As well as those who are just all fired up to talk some politics to a stranger.

Who are these people all fired up to talk politics to a stranger? Perhaps this must-read New York Times article from a week or two ago yields a clue. Called "The Elephant In The Room," it notes the fact that friendships and coworker relationships are more and more strained over political differences.

While the New York Times strains to find Democrats who don't feel comfortable discussing politics with their conservative friends, most of the anecdotes are about "closet Republicans," Republicans who attempt to avoid political discussions and even lie about their beliefs due to the fear of ostracism by their eagerly-to-mock-Bush liberal friends.

I don't know if this is a nationwide phenomenon; I don't want to apply a Blue State bias to this. For all I know, the situation is reversed in more conservative parts of the country, with conservatives acting as the bullying smack-talkers and liberals hiding their views. But in the Northeast, I know for a fact it's common for Republicans to obfuscate their politics or even outright lie about them. Or, at least, attempt to deflect and avoid the topic entirely. And that it's the liberals who are always gung-ho to get into it.

After all, they've finally found one of those strange creatures called "conservatives" they've heard so much about. How can they pass on chance to interrogate this bizarre thing's alien mindset, and find the answer to the question that burns within them -- "What the Hell is wrong with you??!!"

Even if this situation were reversed in the South and heartland or mountain states, this effect would still cause a big skew in polling -- because it's not the South or heartland or mountain states where Republicans are in trouble. Almost exclusively, incumbent Republican congressmen are threatened in the Northeast and Midwest, from Yankee Connecticut through the farm and factory belt from Pennsylvania to Michigan. All blue, or at least blue-purple, areas.

I also wonder if conservatives are less willing to answer because of the economics of the situation After all, in answering these polls, you're providing a valuable service, gratis, to companies that are not, in fact, charities. They couldn't exist without the public being willing to give them the polling information they need, and yet do not pay for. "Jerry" from ARG doesn't come down to your place of employment and help you out for a half an hour with your job; why return the favor, then, for Jerry? If he wants to come down to your office, maybe you'd be more willing to talk politics while you both sort out your inventory figures.

A final bias may be due to the sorts of people asking the question. The 2004 election polls were skewed, most likely, because most exit pollsters were young college-age women. Now, you'd have to be pretty dumb not to know this is a demographic that skews left. And most people are nonconfrontational -- they'd rather not share their conservative opinions with someone they're pretty sure is very liberal. So again-- they self-select, or self-de-select. If polling questioners are largely women, or largely college-aged, or disproportionately minority, etc., people are just going to be less likely to say they're Republican (also known as "The Party Workin' Hard To Keep You Down.")

They may not say they're voting Democratic, but they might just say "I'm undecided." When they're not. (Dave Chappelle has a funny bit about white people refusing to tell him whom they voted for -- which Chappelle, of course, takes to mean they voted for Bush. And he's probably right.)

So what do the polls tell us at all? They tell us that a small fraction (20%) of the population, which tends to skew either demographically Democratic or partisan-Democratic due to an ebullient eagerness to share their liberal opinions with strangers, considers itself 80% likely to vote (when the actual figure is closer to 50%) and, furthermore, identifies itself as Democratic far out of proportion to actual figures for actual Democratic voters and, get this, intends to vote Democratic in the upcoming election.

Candidates and news organizations pay thrity or forty thosuand dollars a pop for that kind of vital information. The sort of information I could have provided for them, based on nothing but gut and guesswork, for about twenty bucks.

I could do the same work, putting it into spreadsheets and pie graphs, without calling a single person, just sort of using my instinct to fake up the numbers so that they seem reasonable. Just throwing together some bullshit that I think can pass as a fair guess at what people would say if I actually bothered to do the calls.

Kind of like John Zogby, for example.

Forget About Getting The Wealthy To Answer Questions... One poll we did was for a new Jaguar offering. We specifically called upper-income towns to guage what they thought about the car.

Response rates were horrible. They just did not want to be bothered. One guy was willing to talk at length about it, and seemed pretty damn knowlegeable. Because he seemed like a bit of a car-freak, and just liked the topic. But, if memory serves, I ended up getting a buttload of hang-ups and many of those who answered at all did not seem likely to be in the upper-income brackets we were specifically looking for. If you've never heard of Jaguar, it's kind of hard to beleive you're making the mint you just told me you were.

On the phone, as on the internet, almost everyone is fantastically attractive and makes $100,000 in a bad year.

And, of course, absolutely take-it-to-the-bank lead-pipe-cinch certain to vote.

The 20% Figure: Quiggs wants to know where I got that from.

I got the 20% figure from Geraghty, who seems to know what he's talking about.

Now polling companies won't say what their actual response rates are. So here's how Geraghty got that figure -- I think. This is a bit of a fudge, but I think this is his evidencet, and it makes sense to me.

Robotic polls are denigrated because they only get 20% (or less) response rates. That's the charge from conventional person-on-the-line pollsters. However, in their defense, robotic polling companies counter-charge that conventional pollsters get no better a response rate, or barely better.

Now it may be that robotic polling companies are lying. However, conventional pollsters refuse to release their response rates -- and there must be a reason for that.

The 20% figure may be too low for conventional person-on-the-phone polls, but I figure it's ballpark. If these companies had significantly higher response rates, they'd say so.

Based on my limited experience in this area, 20% sounds right to me. Actually, it sounds a bit high, but then, I was asking people about Benefiary, which no one wants to talk about. Even people who work at Beneficiary don't want to talk about Beneficiary.

posted by Ace at 01:39 PM

| Access Comments