May 3, 2003
From the desk of Jane Galt:And here's why surveys and polling are such a black art.
One of the biggest ones isn't even covered: it's getting harder and harder to get people on the phone. I participated in a health survey the city of New York was doing; it took me fifteen minutes and I figured, what the hell, here's some civic duty. But I talked to any number of other people who'd hung up on the survey takers. This is getting more and more common with polls and surveys, and it's a huge problem, because the people who hang up aren't simply randomly distributed, so you end up getting samples hugely weighted with the old and unemployed, and underweighted with those who have better things to do than answer questions about public policy.
The article does, however, have some troubling ideas about our poll-driven democracy:
Economists have gone even further in explaining/excusing public sloth in regard to political beliefs and actions. George Mason University economist Bryan Caplan recently has posited the appropriateness of rational irrationality, whereby we choose an optimal amount of absurd and counterfactual things to believe based on what it costs us to hold these unrealistic beliefs.Caplan’s concept would have helped clarify Weissberg’s findings, which show that people seem to credulously accept the endless possibilities of government goodies, believing they will all deliver exactly the benefits they promise. Weissberg argues that most polls are systematically biased toward manufacturing a vox populi that clamors for an ever-growing welfare state.
To test this thesis, he designed and executed a pair of surveys that he thinks provide a more sophisticated and accurate way of gauging an intelligent, informed decision -- not just an ignorant wish. He used these polls to retest public support for a couple of Clinton-era government expansions: shrinking public school class size by hiring tens of thousands of new teachers, and increasing government-supported day care.
Weissberg found exactly what he was looking for (and one wonders how often that happens in social science research -- there’s a poll whose results I’d like to see). If you give longer, more detailed polls that demand citizens balance costs within a necessarily limited total budget, and inform them of both the possibilities of failure and the real dimensions of the problem allegedly being solved, previous apparent support for government action and spending quickly fades. For example, if respondents were told that the new teacher program could lead to cutbacks in other school programs, 71 percent of the support evaporated; when informed that an expenditure of $1.2 billion would lower average class size only from 17.8 to 17, 43 percent of supporters changed their minds.
Underlying Weissberg’s argument is the dire hint that, to the extent that politicians speak and act in reaction to polls, we are in effect living in a plebiscitory democracy. His larger point is that the people are far too stupid to be heeded by politicians.
Which goes back to an argument I had about funding in the comments of another post: the commenter argued that it was perfectly okay to have a large majority voting themselves things to be paid for by only a small minority. But when that happens, you get the majority of voters looking only at one side of the equation: the benefits. Several liberal emailers and commenters have suggested that it was some sort of wacky, fascist notion that one should evaluate the benefits of a program only in connection with the costs. If I blow my head off to cure my headache, the headache's gone, but so's the head. And if I decide to spend 10% of GDP on social programs I don't really care about because what the hell, I'm not paying for it -- it's that rich guy I don't like! -- the 10% of GDP is nonetheless gone, and not being spent on cell phones or televisions or trade paperback versions of Das Capital.
Polling certainly has it's problems, although it's unfair to lump focus groups in with more scientific methods. The possible margin of error of a poll is infinite, according to this piece I always assigned my undergrads.
Still, professionally done polls have excellent predictive value. They weren't very accurate in the 2000 election, to be sure, but the results were so close as to be within sampling error. Plus, measuring opinion and measuring behavior are not quite the same thing.
Polls can only be predictive when the question is concrete and the true answer will come out, ie, "who will you vote for" can be at least partially tested by who wins. Although election stock markets (run at U of Iowa and U of British Columbia I think) do a better job of predicting.
Megan is dead-right about polls on questions about government programs. When I lived in Canada a few years back I gave some poor pollster hell over the question "which of the three should the Federal government make its highest priority: health care, education, or unemployment". I kept telling her "less of all the above", or "shrink the government in all these areas". Then I heard a child crying in the background and decided this was a single mom doing poll work for $5 an hour and just answered the questions quickly and, of course, uselessly.
Cell phones change polling. First, the person called has to pay for the call and the resentment ends the poll; Second when I worked at a large corporation last year in a department with 500 people ALL of the young people had cells and most of them had no land line at all. We are in a society of cells and pagers; polling is toast. Or at least accurate polling is toast.
The there is the problem of the "loaded" question, and this is really a bummer. Almost all the "liberal" polls I've seen have loaded questions in them. I assume the Right Wing polls are the same. No difference: cell phones make polling useless.
I think that virtually everybody believes that if people are taught more about a subject, their opinions will then converge on the "correct" belief. I know I do!
On the subject of people voting for programs they don't have to pay for: there's some merit to this idea, but it conveniently ignores the fact that rich people already have enormous resources at their control to influence public opinion and the political process. The idea that the masses are going to simply vote themselves bread and circuses misunderstands both the history of bread and circuses and the ability of the rich and powerful to fight these tendencies among the hoi polloi.
Jane Galt:
As I explained in the other thread, most taxes are already paid by the rich, and the primary direct beneficiaries of social programs are mainly the poor or middle class. The majority is already in a position, under existing tax law, to vote itself big new government benefits that it won’t have to pay for, or that it won’t have to pay much for. There is no systematic relationship between the value of goods and services a citizen receives from the government and the amount of money that citizen pays to the government in taxes and fees.
Also, you have argued that raising taxes on the rich reduces investment and innovation and thereby imposes costs not just on the rich but on the poor and middle class, too. So, by that argument, the burden of new spending would not be borne only by the rich even if a majority of citizens were exempt from the taxes needed to pay for that spending and were the primary beneficiaries of it.
"I think that virtually everybody believes that if people are taught more about a subject, their opinions will then converge on the "correct" belief. I know I do!"
hmm no, perhaps you are joking? When people are taught more their opinions diverge. It is only when they are intentionally taught less, by excluding contrary information, that opinions converge. Where you stand depends in part on where you sit, in part on temperament and in part on information. Persuasion is an art, a black art.
'I gave some poor pollster hell over the question "which of the three should the Federal government make its highest priority: health care, education, or unemployment". I kept telling her "less of all the above"'
During the 2000 campaign, I started to answer two polls about the campaign - but I got fed up and quit partway into each one. The turned out to be a disguised campaign presentation for a congressional candidate. It's been too long to remember the details, but it went something like this:
Q. Did you know that Mr. X eats babies?
A. errr, no
Q. Would knowing that make you less likely to vote for Mr. X?
A. No. (My mind was already made up; if the Libertarian party candidate didn't get on the ballot I'd go eeny-meeny-minie-mo through the other third parties rather than give my vote to either of the major party fasco-socialists.)
Q. Did you know that Mr. Y promises to introduce legislation to outlaw baby eating?
A. Isn't that already covered by the laws about murder?
Q. Please answer yes or no
A. (click)
Maybe I should have played along, giving bogus answers and letting them think this ploy was actually working so they'd waste more time and money on it, but it was time to get supper on the stove.
The other poll was a serious one conducted by professionals in the pay of one of the major news organizations, but it was actually more irritating and irrelevant than the pollo-mercial. The polltaker seemed to be following a script where certain answers would trigger other questions, but the script didn't allow for libertarian opinions at all (or for the opinions held by any of the men who signed the Declaration of Independence). It seemed to assume that I would either be voting for Bush because I was terrified of being free and hoped he would move a couple steps closer to a police state, or be voting for Gore because I was terrified of the responsibility of spending my own money and hoped he would tax it all away. And after I hung up they called back and pleaded for me to complete all the questions, something like this:
Q. Please don't hang up. If we don't complete all the questions we have to throw it out, and that increases our margin of error.
A. GOOD! (click)
In retrospect, I did it all wrong. The proper response: "Oooh, supper is burning, please hold." And leave the phone off the hook. ;-)
The way to get around cell phone is to have online polling which is a) easier and b) a lot cheeper (because you don't have to pay some poor high school kid $5 per hour to ask the questions...I know because when I was 17 this is what I did for the summer. I can still remember the script, "HI, I'm Kate XXXXX calling from the SR&B business research center, we're calling to ask CFOs such as yourself about banking needs in your area. So if you have a few moments I'd like to begin by asking.....").
But I was surprised by then contention that the only people who fill out these forms or answer these telephone surveys are old and underemployed and therefore you don't have an accurate sample. Usually you weed out pleople based on some questions you ask on a preliminary basis just so you don't get a skewed sampling. While I have no idea how it is determined, since I am not a statistician, I know that polls, even those requiring a "random sampling" need to have that sampling actually be somewhat random, that the poll is incorrectly done if it's made up of 50% elderly and 50% deadbeats.
On another note, the piece cited to in Megan's post reminds me of a study I heard from the 1960s or 1970s (and no, I have no idea where I read this) where if you asked people whether they agreed with the first amendment of the US constitution (which was then included in the poll) they would say "yes" almost without fail. However if you asked people whether anyone should have the right to (and I'm making this up, because I don't remember the study very well) go on protest marches or make unpleasent anti-war statements, some huge number did not think citizens should have the right to do that.
I find that a facinating concept.
The problem isn't that they don't adjust for the sample skew -- it's that the samples of people who are not elderly or unemployed are too small, making the information about those people more unreliable.
The problem with online polling is that you don't pick the respondants. That leads to two problems: the people who respond are the people who are more motivated, and they're more likely to give you bad information to control the samples.
Then again, if you told them how much their taxes would actually increase (.1%?, 1%?) instead of throwing around dollar figures, support would probably increase.
Last time I answered a telephone poll (~15 years ago) I was asked 5 minutes of set-up questions, and finally one blatently biased question about abortion rights. I felt like a chump for sitting through it.
Since then, I haven't participated in any polls, but I did work for one start-up company that failed miserabley despite Gallup's assurances that there was a huge market for our product. There wasn't.
There is no incentive to create a poll that might give the wrong answer.
I roll dice at the beginning of a phone poll. The result decides how much truth they're getting.
I wonder if they'd be willing to pay me to be honest, or at least to use a more truth-favorable die.
"the people who hang up aren't simply randomly distributed, so you end up getting samples hugely weighted with the old and unemployed, and underweighted with those who have better things to do than answer questions about public policy."
Much like jury duty, although at least the hammer of legal compulsion and a sense of social obligation helps partially even out the balance for juries.
Telephone polls have become too abused to be useful. Too many are fronts for sales pitches, politicians pushing an agenda, etc. for me to waste any time with them.
Perhaps a solution is something along the lines of the TV Nielsen system: a picked group of pollees whose demographic is carefully established to be as "median" as possible, and given some incentive to respond to the poll. I suspect that this is already in place for most serious polling organizations, which makes my (random) participation even less likely.
Megan,
I've been thinking about your post all last night and I wanted to point out a few things.
1) Obviously politically biased polls do not give correct information (questions like "do you know that XXX eats dead babies?"). Those polls have an agenda and therefore are invalid on their face.
2) Polling is expensive and time consuming. The information it conveys, whether politically or from a marketing standpoint is being collected for it's accuracy. To put it in a non-polical context, if I make a product and I want to know the correct way to market it, who my demographic is, etc. I might take a poll. If that poll does not create correct data, then there is no point to my collecting it in the first place, since I could have been equally incorrect had I just made wild guesses. This is a huge expense which would not be necessary were the poll results generally inaccurate.
3) I would imagine political polls, ones that are not partisan at least, collect similar types of data. This data is of no use to the candidate or political officer if it is not correct. From an economic standpoint, why would someone continue to pay for a service which constantly gave them incorrect information?
Perhaps I am simplifying this too much, but your argument, in a general way, doesn't make much sense to me.
Polling isn't dead yet. . . but the consensus is that it will be, as more and more people refuse to help out. I should have made that clear. Eventually it will get to the point where it's no longer very accurate, but I'd bet that politicians will continue to use polling somewhat after this point, as markets, as good as they are, rarely clear with perfect. efficiency. ;-)
A cynic might add to Jane's last comment that politicians aren't famously good at understanding markets. A more likely response: "people aren't answering poll questions or believing their results? Clearly, government intervention is needed!"
I once helped a CDC team that was investigating an outbreak of a rare strain of Salmonella, in conjunction with my state health department. (It turned out that the transmission was via frogs.)
We wanted to interview a bunch of people just like the ones who had gotten infected, to try to identify differences between them. We dredged up a list of everyone shown as residing in one of the target counties at the time of their birth over a certain year range. (E.g., we wanted every currently three-to-five-year old born in X county as our controls.)
Well, with the resources of the state health department behind you, that data is easily available. Combine it with looking in the phone book, and you have a pool of numbers to call.
Pick up the phone. Now, at this point, if you say "Mrs. Jones, I'd like to ask you a few questions about Alice, your two-year-old daughter whose birthday is April 12 and was born at X County General hospital, particularly about whether she plays outside and what the environment around your house looks like," people will freak out.
So we always had to introduce ourselves as being from the state health department, explain what we were studying, and then ask if there was anyone in the specific age range we needed in the household.
You also always lied about how long the survey would take, because nobody will do a survey over 15 minutes. (Ours took about 20, so it wasn't too bad).
Anyway, Jane, depending on what exactly they asked you, they very well might have been seeking you out somewhat specifically. You might even have ended up in MMWR.
Kate argues that you can expect real polls (not disguised marketing) to be unbiased because they're expensive and inaccurate results are useless. I see two problems with that:
1) Deliberately slanted polls can indeed be useful - to persuade politicians that your position has more public support than it really does, or simply in the hopes that if the bogus results are trumpeted loudly enough and often enough, sheeple will join the bandwagon. Kate's a logical person, which makes it hard for her to understand the bandwagon effect - but I've been around a long time, and it does work.
2) The designers of the polls may be simply too biased themselves to realize how they are biasing the questions, or too unimaginative to recognize all the alternatives.
"Oh, but they're professionals..." Lots of professionals are incompetent, especially outside the fields of medicine and engineering where reality checks are unavoidable and obvious. For many polls cover things that will never be measured any other way. The ones that actually do attempt to predict hard real-world data - marketing and election predictions - work with so many uncontrolled variables that there are always plenty of excuses for their failures. "This despicable last minute desperation tactic swung the election, in spite of the heroic efforts by our campaigners in the last week." "People often change their minds on the way into the voting booth." "When we said customers preferred green, we didn't mean baby-poop green." "Styles are unpredictable."
Comments are Closed.