Asymmetrical Information

Nice article, Megan.

Jane,

Congrats on the editorial column. It's fun to read something by someone you "know". (Which is why we all wish we could tell which articles are yours in the Economist.)

I agree with your take that it's best to assume the Detroit Free Press' error is attributable to an honest mistake -- ignorance -- rather than political bias. But, how then do we explain why they choose to compare the ACS data from 2005 with the CPS data from 1999 rather than use the CPS data from both years? They had to know that their approach yielded the most dramatic result available from the two approaches.

I think bias was involved. I think it was most likely the bias of a story teller wanting to tell an interesting story. They may not have known that their choice to use the two surveys led to an invalid result. That's the ignorance you and Stuart discuss in your column. But that ignorance does not explain completely the choice they made. It only explains why they thought they had a choice between to acceptable methods.

Good as always. Back in my undergrad days (now 10 years ago--yikes!), I vividly remember the gulf between the journalism students and the econ, social science, and physical science folks. The journalism kids were taught how to write and be advocates (in fact, they were self-selected into this role), while the kids over in the physical and social sciences were studying statistics and, basically, how to be skeptics. This was at one of the best journalism programs in the country, alongside excellent economics, biology, and chemistry departments (about 25 miles north of Megan's old haunts).

Let's face it--statistics and an open-minded skepticism are hard things to learn even if that's the dominant culture of a field. They're nearly impossible to learn in a culture of advocacy. Journalists really don't seem to know what physical and social scientists do all day.

I think that there's a bias here, even though it's probably not a conscious one. Most journalists tend to favor one side, have friends who all favor that same side, and already have certain beliefs and expectations. If a story or way of looking at the data fits with what they and their friends already believe, then they feel that they already have their 'two sources' and can safely go ahead with the story.

After all, everyone knows that we're much worse off under Bush, so any numbers showing that we're worse off must be right. It's numbers indicating that things are going well that are clearly biased, in some way that we can't quite figure out, but we know it's there. Thus it's good journalism to ignore the numbers (and other details) that we intuitively know must be wrong.

That's not deliberate, and it's based largely in ignorance, as you said, but it's still bias.

Did the Freep ever issue a retraction for the very erroneous map?

One anecdote does not equal evidence for this sweeping statement:"Every year, scores of fledgling journalists pour out of liberal arts programs. Though many will need to pick through mountains of statistics in search of the truth, few have been taught the skills to do it."

How many "scores of fledgling journalists pour out of liberal arts programs" per year? Of those unnumbered "scores" what percentage equals the "many [who] will need to pick through mountains of statistics"? Of that last group who need statistical training, how "few have been taught the skills" necessary?

I assume you have some empirical evidence for these sweeping generalizations.

What are you disputing--that at least 40 aspiring journalists graduate from liberal arts colleges every year; that some of them will need to judge statistics in order to write their articles; or that few of them have a solid grounding in statistics? Working and moving in journalistic circles as I do, my experience is that most journalists were liberal arts majors, very few of them understand statistics to the level I think necessary to write any story that involves an academic or think tank study, or government data; and most disciplines, other than those of "war correspondant", "movie critic", or "fashion reporter", eventually involve writing about academic or think tank studies, or government data. The journalists of my acquaintance who ask me to interpret data for them would probably agree with all three statements.

Though I agree with Jane on the substance, I have to say: touche, Aaron.

Though I'm a liberal arts graduate, and I don't do journalism, I nonetheless am numerate enough to work in finance, so I suppose my experience is instructive: if all these journalists are as smart as they claim they are, then certainly they can learn the relevant statistics.

Quick question: you win the lottery. So your chances of winning any future lottery must be diminished.

Most journalists would agree with that statement.

Wouldn’t the IRS provide the best source for this information? Every taxpayer has a zip code, and presumably the data could be combined for persons living at the same address. Or is there some reason that anonymized data cannot be released by the IRS? I'm certain I'm being naive or ignorant, and someone reading this will put me straight.

Hmm, I suggest McArdle's Law for media bias:

Never attribute to malice what can be ignorance.

This would go a long way to reducing the acrimony in discussion of the the media.

Well, in politics, too, for thant matter.

"Quick question: you win the lottery. So your chances of winning any future lottery must be diminished."

Actually, if you win the lottery, your chances of winning any future lottery still "suck" as badly as they did before you won! You'll not hear many journalists say that, even though it is true. You also won't hear them say that state lotteries are a tax on the numerically challenged; or, that they are a voluntary tax on those least likely to pay income taxes under current law; or, that they are the most regressive tax currently in place; or, that most of these lotteries were passed into law by liberals.

And yet, some woman won a $1 Million scratcher, twice.

I also would like to know about what Christina asked: Did the Freep ever issue a retraction for the very erroneous map? A comment by an editor, or Editor's Note? Ombudsman article? Anything at all? Are they even aware that their reporter screwed up?

Innumeracy was not the cause of that bungled incomes article. The mistake which the journalists made - using income data from two non-comparable sources - has little or nothing to do with statistical knowledge. A careful writer could have picked up on this issue even if he or she can't tell a standard deviation from a chi-square.

One could argue for accidental error if there was a significant number of similar errors on the other side of the political divide. To put it in statistical terms, if media error were a random process, then we'd expect to see some measurable number of such errors that made Democrat Presidential policies look bad. The issue goes beyond innumeracy on the part of liberals arts graduates, which is so common as to have become a cliche. The issue goes to willful malice on the part of alleged dispassionate reporters of truth, to actual malice on the part of the "watchdogs" of government, who actively and aggressively take sides on issue after issue after issue, while wrapped in the Emperor's New Cloak of "impartiality".

Or, to put it a shorter way, in a newsroom that had no obvious tilt, the forged TANG documents never would have made it onto any broadcast. In a newsroom that had no obvious tilt, the Swift Boat vets would have gotten an honest hearing.

To put it really concisely: Bellesiles and "Arming America", need one really say more?

Jane, you bounce from the flipper of one anecdote to the post of your experince without once mentioning any empirical evidence. I'm not disputing anything because there is nothing to dispute. The Freep mixed apples and oranges and you have exeriences. Okay, I get that. Generalizing to your sweeping conclusion from one anecdote and your experiences leaps over a lacuna of missing data however. Your intuition may in fact be correct, but...

Aaron is on target.

OK, now that you have demonstrated that a major(?) newspaper made a mistake how about doing something like comparing the analysis of the Brookings Institute to the Heritage Foundation?

It would be very interesting to find a comparison of the caliber of reserch at the two institutions.

Spencer--even better, how about I compare Brookings to EPI? :-)

Seriously, though, if you send me something from Heritage you think is off, I'll look at it. I didn't pick on the Freep because they're liberal (at least, I assume they are--I don't read them). I picked on them because their self-description of their methodology was appalling.

"Quick question: you win the lottery. So your chances of winning any future lottery must be diminished."

I think that your chances of winning would diminish slightly. Surely most big lottery winners stop buying the tickets! Not buying them doesn't have much of an effect, but there's a tiny decrease in your chance of winning.

Surely most big lottery winners stop buying the tickets!

But how would any of us know that? Easy come, easy go -- perhaps lottery winners blow a few thou more on tickets than they ever would have spent otherwise, because they have cheap money to throw around.

Aaron is on target.

If you mean that he successfully made wine vinegar, then yes. Give it a rest; it was an op-ed piece, not a news report or news analysis item. In the context of an op-ed the writers are quite entitled to draw from their personal experiences in the profession they criticize.

Obviously that means the respondants are equally entitled to provide their own experiential counter-examples in criticizing said op-ed, but the "you didn't use a study for that, so bleah" response is largely unhelpful.

It's very easy for all but the most skilled to miisue or misinterpret statistics. Some years ago, the American Medical Association pronounced that left-handers had a lower life expectancy than right handers. Their methodology was subsequently shown to be flawed. These were not just doctors - they were a professional organization that produced papers subject to the benefit of peer review.

Why would we expect individual journalists to do any better? As previously stated, many of these people are in this for the advocacy - i.e. take a position and do whatever you can to support it.

My favorite misuse of statistics is in sportscasts when you hear things like "That's the first time in 37 years that a goal has been scored in the final 5 minutes by a left footer whose name begins in a vowel.

Perhaps this is one more reason that we flock to blogs like this for reasoned discussion.

Would a course in probability and statistics make more sense than trig in high school curricula? After all, how many people outside of science, engineering, surveying, and maybe some of the building trades use trig on a regular basis?

On the other hand, we are consistently bombarded with statistics, many of which are misleading, misrepresented, or perhaps even spurious.

Comments are Closed.