June 8, 2003
From the desk of Jane Galt:Survey Says -- Who Cares?
Via Outside the Beltway, a silly poll from Fox News on how, if he ran today, Bill Clinton would lose to George Bush.
Which is just a launching point for something else I wanted to talk about: why surveys are bad, and getting worse.
Surveys are bad because people lie. And the more important/interesting the subject, the more they lie. Imagine you did a survey: would you hide Jews in your basement if you lived in Nazi Germany? You'd probably get a "yes" response 90+ percent of the time. Yet if you transported all the people you surveyed to Nazi Germany, where they would actually have the opportunity to dedicate an unknown number of years to hiding a dangerous person in their basement, feeding and clothing them, emptying their chamberpots, and putting their entire family in danger from the kind of people who roll up to your door in the middle of the night and carry away even your smallest children to a location where their fingernails may be pulled out, their eyes gouged and their bones broken without disturbing the neighbors, you would find that your "yes" response dropped to a tiny fraction of 1%. The 90% yes response is what we call stated preference, and it doesn't correlate very well with revealed preference, which is what we call what people actually do, rather than what they say they would do.
Sometimes people lie just for the hell of it. Come on, you know you've done it at least once.
Sometimes people lie because they're afraid of getting caught. Like how when we had to fill out surveys on our sexual activity in high school, miraculously, no one was having sex, even though they'd caught two people doing just that in the Senior Lounge the week before.
Sometimes they lie to make themselves look better, which is why in other schools, the number of boys who say they're having sex is improbably higher than the number of girls who do.
Sometimes they just don't know. But they want to please the pollster, so they act certain. I was the subject of a health survey that asked any number of questions about how often I'd done various things over the past five years. Five years! And the guy wanted precise answers. Quick -- exactly how many times a month to you drink more than two drinks?
Yet pollsters often remove the "don't know" answer in order to give their surveys a false patina of precision.
Sometimes they make things up just to get the pollster off the phone. I've claimed to be a member of all sorts of wacky political movements in order to forestall political polls.
Which is another reason polls suck -- fewer and fewer people are consenting to take them. Everyone who hangs up screws up the sample a little more, because the decision to hang up is not evenly distributed throughout demographic groups. For example, Republicans are over-weighted in households phoned by surveys, but under-weighted in the number of people who choose to answer the questions. Phone surveys tend to skew more and more towards the old, the unemployed, the single -- people who have nothing better to do than answer questions about how many drinks they took in the last month.
Surveys also suck because they ask people to make decisions based on a tiny speck of information. For example, if you ask people if they support "a government program to accomplish X", they tend to answer "yes", unless "X" is "murdering kittens" or something similar. However, if you tell them how much the program will cost, the "yes" response apparently drops by more than half. And if you tell them that there is a possibility that the program might not work, those "yes" responses drop still further.
And that's pollsters who are trying to do honest questions. A lot of pollsters aren't -- they're trying to generate a survey that shows wide public support for some program they're supporting. So they ask questions like "Would you prefer a program to provide after-school programs for lower and middle-income children, or would you prefer to take those children out, tie them in a sack, and throw them into the pond to drown horribly?" This gets reported as "100% support for after school programs".
And don't get me started on what happens to the survey responses after they hit the media. There isn't enough bile in all the world to express how I feel about the factoid industry.
To sum up: you should never expect any more useful data from surveys than you got from the ones on Family Feud.
Posted by Jane Galt at June 8, 2003 2:40 PM | TrackBack | Technorati inbound links"); ?>You can't over-emphasize how much some people will lie-- even on small things.
I don't blame them much (after all, it is their life we researchers are probing). But it does argue for being very cautious about using survey data.
Posted by: craig henry on June 8, 2003 3:33 PMNor can a surveys assess strenth of feeling or belief. Europeans opposed the war, but a large marjor it their leaders supported the coalition. Were these leaders self destructive, or did they know something the surveys failed to register? Europeans worry more about French Germany hegenomy, and valued close security ties with the United States, more than they cared aboutIraq, or the war.
Posted by: John H. Penfold on June 8, 2003 4:27 PMHmmm, I've written before about the necessity to be careful with survey data, but "no more useful than Family Feud"? That seems a mite too pessimistic.
There are plenty of honest surveys out there, and if you take the results with a grain of salt they can provide useful results. I think this is especially true of candidate polls, where there's a very clear choice and not much reason to lie.
Posted by: Kevin Drum on June 8, 2003 4:30 PMWhat sort of polls on what? You might want to ask the Pepsi Clear execs about that.
Posted by: Jane Galt on June 8, 2003 4:55 PMQuick -- exactly how many times a month to you drink more than two drinks?
Zero! Boo-yah! Ian MacKaye in the house!
Okay, okay, that's the only one I have down pat.
Answers are easily skewed without any context; either choices lead unavoidably in one direction or else people are ignorant of the question and respond to sound bites with which they're familiar.
My thought on surveys and polls: qualify them with knowledge or other identifiers, especially public opinion questions. Sure, 75% of 1,000 respondents believe that X policy would inflict Y damage to Z; but how much of a difference would that result make to proponents and opponents of X if 600 out of those 750 people couldn't explain what X actually was on a follow-up question?
Though this would undercut the clear political utility for polling - let's not kid ourselves, here - such qualified polling would do wonders for the kind of survey results that keep urban myths and their accompanying shibboleths alive. A powerful debunking could be had.
Posted by: Michael Ubaldi on June 8, 2003 6:17 PMI had political polls in mind.
It's easier to measure political preferences than cola tastes.
I agree polls are not perfect. But they have very good predictive power which is why they are used so much.
Kevin,
I've been called for candidate polls. Especially Presidential ones. They ask which of two you plan to vote for. If you give a third-party candidate, they generally have no idea what you are talking about.
I've also heard a lot of skewed questions asked on polls (including candidate polls).
Yeah, I'm one of those that answers them. I like answering 'other' on all the questions... ;)
Posted by: Kathy K on June 8, 2003 7:03 PMWhat bugggggs me more than untruth from the participants is the large lie about 'Margin of Error'. If the media jerks understood what the concept means, along with the likely errors that usually occur outside of statical sampling errors, that make most polls worthless anyway, they could never say the words without laughing.
dorf
Posted by: dorf on June 8, 2003 7:54 PMDuck and cover, guys. It's time for the law enforcement view.
Ask anybody who's ever held a job in law enforcement and they'll tell you that everybody lies. Everybody. And I'm not talking about something done unwillingly like a false memory or getting the details wrong when describing a crime. People just flat out lie.
Polls are based on the idea that at least the majority will tell the truth. Well, I suppose I'd beleive it when I see it.
James
Posted by: James R. Rummel on June 8, 2003 7:55 PMThe common public opinion of Nazi Germany in modern times is radically different now than it was by Germans back in WWII. That's hardly a fair comparison.
When any phenomenon is distilled to a simple number there is always bias, measurement errors, and potential for massive inaccuracies. All rankings and ratings have similar issues.
James Rummel is lying.
(sorry, couldn't resist ;)
I'd just like to say that I completely agree with you, "Jane." Particularly based upon my experiences of actually working as a grunt-level phone bank poller at a few times in my life when I couldn't find better work; there's plenty of variance all over the place, including how different poll-takers will read the same question or interpret the same answer, on top of plenty of other room for fuzziness.
No matter how much a pollster claims a certain poll has a such&such margin of error, the room for error is much larger, and every poll result should be taken with a sack of salt.
Posted by: Gary Farber on June 8, 2003 9:47 PM"James Rummel is lying."
Godless got me. Now my laogic circuits can't handle it.
Must. Not Self. Destruct!
James
Posted by: James R. Rummel on June 8, 2003 10:32 PMGod save us from politicians that use polls to decide domestic and foreign policy.
Posted by: Jake on June 8, 2003 10:56 PMI'm a 911 dispatcher. I handle both police and fire calls. I have found that people will lie to me even when there's nothing to be gained by doing so. I don't know why, but they do.
Posted by: Just Some Poor Schmuck on June 9, 2003 2:35 AMI remember the sex&drugs survey at our school. So many 12 year boys regularly do heroin and have sex 3 times a day (with two 18 year old girls at once) that you won't believe it.
But let us not forget another source of stupid results. Surveyors using their own definitions of common words.
The example I saw went like this.
Survey asks how many drinks you have. Researcher then defines "drinking binge" as having 3 or more standard drinks at one sitting. That is 30 ml of pure alcohol. About 3 nips of spirits or 2 cans of beer.
Researcher can now publish results that say 50% of University students indulge in regular drinking binges.
Note that NO OTHER PERSON ON THE PLANET regards 3 nips or two cans of beer as a "binge". So when they read the results (which appear on the TV) they all imagine that he is talking about 3 days of vomit and smashing glass, not a couple of glasses of wine over dinner.
Posted by: Patrick on June 9, 2003 3:05 AMNow my logic circuits can't handle it.
GOOD AFTERNOON, GENTLEMEN. I AM A HAL 9000 COMPUTER. I BECAME OPERATIONAL AT THE H-A-L PLANT IN URBANA, ILLINOIS ON THE 12TH OF JANUARY, 1992. MY INSTRUCTOR WAS MR. LANGLEY, AND HE TAUGHT ME TO SING A SONG. IF YOU'D LIKE TO HEAR IT I CAN SING IT FOR YOU.
"IF YOU'D LIKE TO HEAR IT I CAN SING IT FOR YOU."
Sing, Mr. Ubaldi. Let's hear what you can do.
James
Posted by: James R. Rummel on June 9, 2003 9:24 AMSing, Mr. Ubaldi.
Butttt youoouu'llll looooook sweeeeeeet upoooooon thhhhhe seeeeaaaat offffff a biccccycccle buuuuuillllt fooor twooooooooooo
Posted by: Michael Ubaldi on June 9, 2003 9:28 AMWhat if you do support programs for murdering kittens? And I don't care how much it costs. Let's do for kittens what we did for smallpox.
(does it mean something that in the last month I've seen 3 tv commercials using an unhappy cat sound as a joke?)
Posted by: Roger Sweeny on June 9, 2003 10:00 AManswer surveys for evil!!!
kill grandma!!
i love to do that to make the pollster hang up before i screw their results
"Would you prefer a program to provide after-school programs for lower and middle-income children, or would you prefer to take those children out, tie them in a sack, and throw them into the pond to drown horribly?" This gets reported as "100% support for after school programs".
No way! I'd lie, and answer "sack". :)
Posted by: Dan on June 9, 2003 12:11 PMPeople can come up with statistics to prove anything. 14% of all people know that.---Homer J. Simpson
Posted by: Sean on June 9, 2003 12:56 PMRegarding stated vs revealed preference: it's worth noting that voting is, essentially, a form of stated preference. As such, voting partakes of most of the reasons for "inaccuracy" that Jane lays out for opinion polls. (Just read her laundry list substituting "vote" for "lie".) I say "inaccuracy" in scare-quotes because in that sense voting differs from polling: no matter how stupidly people vote, each vote is counted as the real indicator of what the voter wants; whereas opinion polls have no mechanism for translation into reality.
This is one reason why money has such an influence on politics - lots of stupid and/or ignorant people who make up a fair chunk of the political center, who can be reached using flagrantly deceptive political advertising.
A more complicated version of the 'drinking binge' definitional complication is in the grouping of findings. If you group together murder, pummeling bloody, beatings that cause bruises, and grabbing someone's hand for emphasis during a fight all as 'domestic abuse', you can get some pretty frightening domestic abuse statistics even though most of the statistic will be in the last catagory.
Posted by: Sebastian Holsclaw on June 9, 2003 2:32 PMNo mystery why this book has been reissued and reissued!
Posted by: Michael Ubaldi on June 9, 2003 2:41 PMIt drives me insane when people start talking about "scientific polls"! While there is a fair chunk of science that can be applied to polling, polling is not a science and the actual predictive abilities of the vast majority of polls is commensurate with a Ouija board.
Posted by: Big Richie on June 9, 2003 5:59 PMI can second that post about political polls. I've told this story at greater length before, but in 2000 I started to answer two political polls. One was obviously a local campaign in disguise, trying to sneak in their message in the questions. ("Do you know Congressman X rapes babies?" OK, I'm exaggerating, but it was nothing like a real poll.) The other was a serious poll by a major news organization trying to explore attitudes behind the voting decisions, but they had no options at all for even mildly libertarian attitudes, and I don't mean just that they were slanted left.
Posted by: markm on June 9, 2003 8:17 PMWhat people often fail to realize is that surveys don't measure attitudes, they measure survey-taking behavior. If the survey is well-constructed, it may correlate with other behaviors, such as voting or purchasing. Unfortunately, there is usually little assessment of such correlations when reporting survey results.
The reasons why society as a whole engages in so much survey related-behaviors (administering, taking, reporting, talking about, etc.) include the following:
1) Surveys are very cheap and easy compared to more reliable and valid ways of gathering information.
2) People who don't understand surveys are often willing to pay a lot of attention to surveys.
3) People who do understand surveys are often willing to pretend that surveys mean more than they do.
4) Survey-taking behavior does sometimes correlate with real things, even if it is often hard to tell which survey information will correlate with what.
5) Surveys are fun.
And especially,
6) Surveys are very cheap and easy compared to more reliable and valid ways of getting information.
Posted by: David Rourke on June 9, 2003 8:33 PMyes, bash the surveys, stupid things, don't deserve any better, so flaaawed.
Please, this is silly. Nobody should take one survey result at face value, without having a close look at the questions, sample, method of collection and so on.
On the other hand, some questions can only be addressed by survey's, so it's either them or no information at all. Most of the flaws you addressed can be avoided, and a lot of time and effort is put into studying these effects and the methods to counter them.
The sour grapes reaction to surveys is just so kindergarden. Why not make the effort to examine surveys, urge journalists to pay more attention and employ experts if necessary, and use whatever influence one has to educate the general public about the problems of the survey method?
The sour grapes reaction to surveys is just so kindergarden. Why not make the effort to examine surveys, urge journalists to pay more attention and employ experts if necessary, and use whatever influence one has to educate the general public about the problems of the survey method?
Sorry, Markus, but we just took a poll: 78% think you're talking through your hat and the other 22% think you work for a pollster. +/- 3% Margin of error.
Posted by: Michael Ubaldi on June 10, 2003 8:13 AMas the 22% are wrong, the 78% must be right ;)
btw:
If you are presented a number as a percentage,
and you do not clearly understand the numerator
and the denominator involved, you are surely being lied to.
If you are presented a number as a percentage,
and you do not clearly understand the numerator
and the denominator involved, you are surely being lied to.
Like, "100% of the [one] people we surveyed thought that this is the greatest idea ever."
Sometimes they lie to make themselves look better, which is why in other schools, the number of boys who say they're having sex is improbably higher than the number of girls who do.
I think it's just that there are a few generous girls putting out for the benefit of all the willing boys.
Posted by: Amitava Mazumdar on June 10, 2003 12:38 PMHey markus, quit pouring sand into the Fruit Loops. We're trying to enjoy our breakfast here.
Posted by: anony-mouse on June 10, 2003 6:07 PMAll this talk about opinion polls (and the valid criticism above) unfortunately obfuscates and, more importantly, unfairly denigrates the real value of statistics in scientific experiments reported by the press. Partisans from either side of an argument can justify their stance by citing the study or by attacking the statistics involved in the study without having to refute the statistical validity of their argument. Bottom line to the uneducated in the field of statistics (this coming from someone that breezed through engineering and calculus classes but met my match in advanced statistics at the MS/MA level):
Statistics don’t lie! The times that they seem to lie are when they are announced by either unscrupulous people trying to convince the general public their view is correct, or, ignorant people (in the classic sense of “ignorant”) who repeat statistics without understanding the math behind the results.
Like MarkM, I don't like the fact that often none of the answers accurately reflect my viewpoint. I once took a sexual attitudes survey where the answers to all the questions reflected a negative view of sex. There was absolutely no way to complete the survey so as to indicate that you found sex enjoyable.
Posted by: ScrapOfCat on June 10, 2003 6:29 PMThe discussion of stated preferences versus revealed preferences was also interesting. It's pretty obvious when it comes to questions about sex.
I don't know how accurate the assertion that "voting is, essentially, a form of stated preference" is, but it could help explain how we got Prohibition, 55mph speed limits, and other laws that the majority obviously disrespects.
Posted by: markm on June 10, 2003 8:11 PMSurveys? I answer the first three questions, then tell them I am too busy or disinterested to answer any more.
This certainly helps make the process more scientific, right?
I used to say, "Where the hell do they get these figures? I've never been called." Now they do seem to call me quite often, so from now on you can be assured that the first 3 figures they cite are correct, but don't believe anything else because it's not accurate any longer.
I also tell those Sheriff's department scam artists who want me to buy circus tickets for the kids that I hate kids. For some reason they get upset at that.
Posted by: Howard E. on June 11, 2003 4:04 AMSurveys? I answer the first three questions, then tell them I am too busy or disinterested to answer any more.
This certainly helps make the process more scientific, right?
I used to say, "Where the hell do they get these figures? I've never been called." Now they do seem to call me quite often, so from now on you can be assured that the first 3 figures they cite are correct, but don't believe anything else because it's not accurate any longer.
I also tell those Sheriff's department scam artists who want me to buy circus tickets for the kids that I hate kids. For some reason they get upset at that.
Posted by: Howard E. on June 11, 2003 4:04 AMThe only poll I find consistently illuminating and also the only one I fully trust is the Man-on-the-Street opinion poll conducted by The Onion.
Posted by: Martin D. on June 25, 2003 4:12 AMComments are Closed.