Asymmetrical Information

And remember Al Qaida has a dog in this fight. A high mortality rate benefits their side. Just as they provide stringers and photographers to the media, they probably provided interviewers for this study.

20 years ago Lancet had a great reputation for well research papers. It is a shame they have abandoned their commitment to reproduceable results.

Because it is impossible in the current security situation for more accurate samples (not all bodies end up in either the morgue or in hospitals), then no effort should be made to estimate actual Iraqi fatalities?

Regarding the pre-invasion to post-invasion numbers:

It seems from this study that they interviewed households, and asked them if a household member had perished. Quite obviously it seems to me that this would miss any household which was totally exterminated. Given that Saddam had a propensity (common among dictators) to kill the wives and children and other relatives (extending even to villages) of his enemies, certainly this could cause some undercounts of the pre-invasion numbers, yes?

In addition, there's the customary complaint that the pre-invasion 14 months were fairly atypical in Saddam's reign, though I'm willing to excuse that somewhat so long as people use the study to argue "we should have just continued the No Fly Zone forever," not "it's worse than Saddam's reign in general." None of the slaughters of Marsh Arabs, Shiite Arabs, or Kurds occurred during the immediate 14 months before war. (It does beg the question of whether a similar slaughter to the one currently would have occurred whenever Saddam's reign ended or at the next revolt against him.)

It also doesn't inspire confidence that the 600k+ number was extrapolated from 547 reported deaths.

Frankly, I've been skpetical of the prior Lancet study for a long time given that a separate study, by the UN Development program, with a much larger, more reliable data set, found that the 2004 Lancet study was off by about a factor of 4. The UN found after the first year that there were 24,000 war-related deaths (18,000-29,000, with a 95% confidence level), which is approximately 4x the number of excess deaths the Lancet found with their survey. Of course, the UN used similar techniques - clusters, etc. - but with a much larger data set. (link)

To me, the hallmark of a good study is if the results are able to be duplicated. The UN basically tried with repsect to the first Lancet study and found the Lancet off by a factor of 4. I'd say that's probably how much this Lancet study is off by too...

Because it is impossible in the current security situation for more accurate samples (not all bodies end up in either the morgue or in hospitals), then no effort should be made to estimate actual Iraqi fatalities?

Typical mendacious mischaracterization there Andy - the old false dichotomy ploy.

First of all, it is hardly an ironclad fact that better samples are impossible. But apart from that, the point here is not that no attempt should be made to get an accurate count. A lot of attempts have been and are being made. The point is that those other efforts tend to cluster around a figure in the 50,000 neighborhood and the methodologies employed have much more statistical rigor than the farcical steps and assumptions described in the Lancet "study."

That being the case, the Lancet numbers certainly constitute an "extraordinary claim" to use the language of my old Logic and Philosophy of Science classes. The rule is that extraordinary claims require extraordinarily good evidence in support. Clearly, the Lancet study fails such a test miserably.

There is an obvious need on the part of many on the Left to believe that America has been brutal and indiscriminate in its conduct of Operation Iraqi Freedom. There being no actual facts available in support of this proposition the usual next resort of the Left is to naked prevarication.

Some of the liars in question happen to hold editorial positions at the Lancet. Others of them are willing to create "studies" by, in essence, pulling numbers out of their asses, then writing down the results and passing them along to the former. Bald-faced lying has always been a favored tactic of the Left. Back during the Cold War they were merely a bit more skillful about it.

Actually Jane the passage you quote qould indicate the study is likely to underestimate the real number.

On a more fundamental note, the methods and techniques used here are widely accepted in the relevant scientific communities. Despite the fact that they are not perfect the experts in this field think that the studies provide useful data. You, without any real methodological critique other than that to claim the numbers are too high, are ready to dismiss all such reports.

Not sure there's much to debate after that.

"Entire households could have been killed, leading to a survivor bias."

I think that could be a biggie. The point of the study is to compare deaths now with deaths pre-invasion--the conclusion is that ~600,000 more people died than would have died if we'd left Saddam in power. To get that figure, they need to find out how many people were dying then; and if they do that by interviewing households that are still around now, they'll miss any cases where Saddam wiped out an entire household (or village).

If Saddam's goons machine-gun a Marsh Arab family, there's nobody left in that family to report it, so as far as the Lancet's concerned, those deaths never happened.

I note that the Lancet study found that the death-rate from violent causes, pre-invasion, was 2% (as compared to 55% post-invasion). Things sure were great under Saddam, weren't they?

It's entirely possible that those fractions are true, though. Where can the police be more effective at keeping down crime than in a police state?

Steve Sailer has some interesting discussion about the numbers--particularly the numbers attributed to car bombs, which seem to imply lots of very powerful car bombs which were never reported. This seems pretty unlikely (how do you keep a big explosion in a populated area secret?)

GT,

I'll grant you that the experts in the field find these studies to be of great worth. It's also true that the experts in the field make much of their income producing and analyzing such studies. Might their dependence on these studies influence their opinion of the studies' worth?

It is also worth pointing out that the people who did this study are not "experts in the field" of analyzing wartime deaths. They primarily do medical research.

I'm surprised they aren't trying to include natural deaths from old age in the 'number of people who died in Iraq' tally. I wouldn't put it past the anti-Americans.

BTW, in America, 3 million people died last year! OH the Humanity!!!!

Victory in Iraq will be achieved in 2008, due to a convergence of many trends at that time. This is a certainty.

Remember, too, that the violence is primarily restricted to just three provinces. That means this would be 600K out of about 10M, or 6%. We would also expect to see about 5 times as many who are merely seriously wounded. That means 3 million, out of 10 million.

Are they joking? Baghdad would be a ghost town with numbers like that. Hospitals would be crammed full and spill into the street. Morgues would be stacking bodies twenty feet high in the sun. The smell alone would drive people out.

Deaths are overreported, the authorities are unreliable and are known to have printed false certificates, and there is a significant population of Sunni Arabs who badly want to make the war effort look bad.

Worst of all, the "scientists" didn't even pretend to be objective, and have now TWICE explicitly published their results right before an elction. Rarely is even the worst agenda science this shamless.

Tood: they would in fact include such deahs in a study of this type (it's "excess deaths").

Robert Cox makes a good point at

http://www.rantsandrayguns.motime.com/post/611902/%22If+They+Liked+It+Once,+They'll+Love+It+Twice,%22+or+%22A+Logical+Lapse%22

According to the study, 80% of the deaths in their sample had assosciated death certificates. So it seems like a well-designed study should come in close to the government counts, since almost all deaths were counted. Instead, it diverges from the direct counts by a factor of five. What gives?

To preemptively answer the Crooked Timber objection, if you wanted to measure deaths accurately, were scared of getting shot, and believed that recorded deaths were somehow not making it into the official statistics, why on earth wouldn't you try to track death certificates on a local level? Go to local coroners, ask for a count of the certificates. You should really do that anyway, even with this sampling methodology, just to get an idea of whether your sampling is totally off the wall. Why not a map of your measured death rates, compared to issued death certificates, broken down by county, or whatever equivalent unit they used for the clusters?

Designing a study like this with no built-in reality checking is asking for trouble. You don't even know what your sources of error are.

Given its acknowledgment of the flawed process, the question is why did the Lancet publish the study in the first place?

While I will not pretend to be qualified to comment on the methodology, I have two cents worth throwing in. It is far too rarely that there are even attempts to quantitatively and rigorously evaluate the impact of a given policy action, and the authors should at least be commended for trying to inform the policy discourse in this manner. Secondly, I find it disturbing the extent to which commenters tend to rely on UN or government statistics as some kind of baseline or "smell test" against which to measure the accuracy of the study's findings. That is, to put it mildly, bunk. WHO and UN numbers are usually self-reported and as such are completely subjective and made up by the government in question, and then massaged a bit to appear somewhat comparable to the equally made-up numbers from other countries. There are often incentives to over- or under- report, and even if there weren't, the national information systems in low and even middle income countries - let alone ones in conflict - are rarely strong enough to be even remotely accurate. Irag is a head and shoulders above most countries for even having a vital registration system for tracking births and deaths in the first place! As a health policy professional, I would much prefer to use numbers generated by an independent academic group or outside institution (rather than one comprised of member states) whenever possible; and in fact, the Demography and Health Surveys, which are (you guessed it!) household surveys, are generally considered the gold standard.

Right. Count the death certificates issued. Add 10% to account for the observed rate of non-issue. Problem solved.

has anyone bothered counting graves?
or headstones? or other funeral markers?
can this be done by satellite survey?

If the goal of the study had been to estimate the percentage of deaths in Iraq for which death certificates had been issued, the data would yield a point estimate of 91%. Applying that to the actual number of death certificates issued you’d end up with a mortality estimate in the neighborhood of 50,000 or 60,000, which matches other studies quite well.

The researchers, on the other hand, observed the sample mean of 91% and concluded that the population percent is highly likely to be less than 10%. Surely even a non-statistician can find the flaw in that.

Re. Al's comments on the larger UN study, and how it undermines the original 2004 Lancet one with its '100,000 dead' suggestion;

Roberts et al have actually cited the UN study as a reference on the first page of the latest report in The Lancet. This implies that they haven't actually read it properly.

I think everyone is missing the point with the UN figure of 24,000. If you look at the actual report, you'll see that the survey asked for any casualties in the previous 24 months to April 2004. It then unilaterally decided that all of these casualties happened after the ground invasion, and that none happened before! This can be seen on page 55 of the report (http://www.iq.undp.org/ILCS/overview.htm).

Granted, Saddam was probably killing less people in the last year of his reign than before, but the combination of an ongoing coalition bombing campaign, state oppression, and the heavy fighting between the kurds and Ansar-al-islam must have still caused some casualties. It seems strange to assign every single one to post March 2003. When we consider that some of the 24,000 are also Iraqi military or insurgents, I see it as even less supportive of the earlier Lancet study.

What it boils down to here is: UN; 24,000 dead over 24 months up to April 2004, and Lancet, 98,000 dead over 18 months from March 2003.

My understanding of the two figures is that the UN figure is a 'total' and the Lancet one is an 'excess over pre-war'. Now if I'm reading that part right, the difference may even be greater still.

Some may claim the difference is due to non-violent deaths being included in the study by Roberts et al. However, although they published that violent deaths and infant mortality had increased post invasion, they also stated near the end of their report that adult non-violent deaths had not increased. To quote page seven of their 2004 findings, referring to non-violent deaths;

"It is suprising that beyond the evidence of elevation in infant mortality and the rate of violent death, mortality in Iraq seems to be otherwise similiar to the period preceding the invasion."

So, to re-inforce what Al said, a much larger UN study couldn't replicate the earlier results. And yet Roberts and his team appear to be quoting it as a reference for their latest work. Why did they do this without explaining and analysing where it contradicts theirs? They certainly think UN figures are accurate enough to help choose their samples.

It seems to me that when you don't know the answer, the correct answer is "I don't know", not "here's my wild-assed guess".

Clearly you are a reporter and not a scientist. Referring to it as a wild assed guess betrays your biases.

Is it possible that one reason they got high numbers is that they had people counting deaths outside of their immediate families. For example if the interviewed people included first cousins in their deaths mentioned that alone would throw off their estimate by 2-3 times.

Given the news reports about the danger in Bagdad I have been wondering why people didn't leave. This report indicates that it is even more dangerous elsewhere. What ever it's accruracy they probable got that right.

Umm... let's imagine, for the sake of argument, that these various objections by non-experts in public health are true. (And to be sure, I'm not one, nor am I a demographer, so I can't evaluate the methodology with any expertise.) So let's use the factor of four critique that people seem to be attaching to. The study overestimates casualties by a factor of four. That means that more than 160,000 civilians have died so far, which still seems like an awfully high number to me, especially when you consider that:
--Things keep getting worse in Iraq--whether or not we want to call it a civil war, people are blowing the hell up out of each other every day.
--We have said we will be there until 2010. That is the army's plan. George Bush has stated that he will not leave Iraq under any circumstances. There's little doubt that the death toll will get much higher than it is now.
--None of the rationale for invading Iraq can be supported with the benefit of hindsight. No WMD, no connection to al qaeda, it has distracted us from more pressing problems. I think, using game theory (or for that matter, the justification many opponents of gun control use to argue against gun control) I could make a reasonable case that North Korea's decision to develop and test a nuclear bomb has something to do with pre-emptive war. We invade countries if we think they threaten us. But we know enough not to invade a nuclear power--even Bush isn't that crazy. So it seems pretty damn rational, from Kim Jong Il's perspective to gain some protection.
--We reacted to fewer than 3,000 deaths on Sept. 11 by treating it as the worst tragedy in the history of man. By that, I do not mean to suggest that that attack was acceptable. It was a horrific, disgusting event. But think about how we reacted when we were attacked one single time. Now imagine how it would feel if Al Qaeda had invaded us and occupied us for three years. Now, some people might argue that we should value our own lives more than the lives of people from other countries. Fine, let's accept that too. We still have to consider how those people will feel and react, don't we? After all, Osama Bin Laden rose to prominence during an ill-advised escapade in the 80s.
How would we react if a foreign army had destroyed our infrastructure and killed hundreds of thousands of people? Angry, I imagine.

I guess what I'm saying is this: Maybe the number is a bit off. Fine. Maybe the death count is only 165,000. Hell, maybe it's only 100,000. But so what? Does that mean Iraqis don't have the right to be pissed off that we destroyed their country? Does that, somehow, make this war acceptable?

This type of methodology is the standard way of making an estimate in these situations. It is the way they got estimates for the congo and sudan.

Sol Vason is right that the high mortality rate in Iraq benefits Al Qaida. That isn't the study's fault.

How would we react if a foreign army had destroyed our infrastructure and killed hundreds of thousands of people? Angry, I imagine.

Which foreign army are you speaking of? There are at least three -- the US regular army, Iran-supported terrorist groups, and Al Quaeda and the broad spectrum of assosciated groups. Don't hog more guilt than America has actually earned.

Grouping together "excess deaths" into one gigantic hero statistic is slightly obtuse for this reason. It suggests you want the guilt more than you want to understand the situation. Lumping everything together and then being so wildly off reinforces that.

This type of methodology is the standard way of making an estimate in these situations.

What situations? Between 80% and 90% of the deaths they recorded were officially registered. Congo and Sudan have nowhere near that level of record keeping. You have available a direct count of the exact number you're interested in, with larger sampling and more comprehensive coverage than the best your study could do. Would you do this kind of study to find the murder rate in New York City, or would you look at police or morgue records?

If you were interested in improving the current best estimate, you'd start from the most accurate measurement -- the direct count -- and systematically study how it errs.

Look at it this way: this study should work as well to estimate the number of deaths recorded by the Iraqi government as it does to estimate the number of deaths overall. The populations might be slightly different, but again, there's an 80-90% overlap, so the difference isn't that large. So if this study is working, their estimate of _recorded_ deaths should reproduce the government tally. But it doesn't.

Good studies should have these kinds of reality checks built into them at the design stage.

TallDave says: "Baghdad would be a ghost town with numbers like that. Hospitals would be crammed full and spill into the street. Morgues would be stacking bodies twenty feet high in the sun. The smell alone would drive people out."

Are you sure about that? The total estimated death rate from the study was about 13 (deaths per thousand, up from 5.5 before the invasion). According to the CIA Factbook, Hungary has a rate of about 13. Are people leaving Hungary because of the smell?

You, without any real methodological critique other than that to claim the numbers are too high,

Read: "You, without meeting the standard that would convince me, which you could not possibly meet anyway this side of an ideological conversion to a reasonably complete set of my existing prejudices..."

In multiple posts Jane has provided a layman's statistical overview of why she doesn't trust the numbers on the basis of the assumptions and the collection methodlology, and just in case that isn't enough to cast reasonable doubt, she maintains an open comment section in which the commentariat have been delivering further objections of their own.

Feel free to criticize any misrepresentations of the study methodology or data you think you see (I, for one, will thoughtfully read those posts), but the false-premised 'gentle chiding' schtick -- like the quote I snipped -- is about as transparent as air, and far less beneficial.

Many statisticians get so wrapped up in their own work that they forget about making a sanity check against reality.

Clayton Cramer has some perspective on this.

The more I think about the mechanics of carrying out the survey on the street without getting killed, the more I suspect that the Iraqi interviewers didn't actually implement the purely random survey design that the American professors from MIT and Johns Hopkins dreamed up for them. It would be nuts to to let luck determine which streets you'd choose, as the report claims they did. You'd want to only go where you knew you'd be safe. Then you'd tell the Americans you did exactly what they told you to do.

Or it could be that the interviewers got in contact ahead of time with neighborhood leaders to see if their presence would be welcome to reduce their chances of being killed. (That's not good random surveying hygiene, but are you going to blame them?) Then, in a neighborhood where the local big shot wanted their presence, he might have passed the word around to aggrieved families to get ready to tell their stories to the interviewers when they showed up. This could cause a bias upward in the number of deaths reported. But it's equally easy to imagine scenarios that would bias the death count downward.

The overall point, however, is that nobody else appears to be doing this kind of study because it is so hideously dangerous, which ought to tell us something.

More analysis is necessary, but, after a few hours of kicking the tires, these numbers don't strike me as obviously implausible. I wouldn't put tremendous confidence in them either, though, due to the savage conditions under which this heroic effort was carried out.

The American military is firing hundreds of thousands of bullets per day in Iraq, perhaps as many as a million per day. Somehow, I suspect some of them hit people.

And the reason it is not confidence inspiring is that the fuzziness prevents comparison with figures known to be relatively reliable, such as those from Baghdad's central morgue

They are not "known to be reliable". They are known to be unreliable. It is also known that they are subject to periodic censorship, and it is not safe to assume that this periodic censorship is uncorrelated with the number of deaths.

The number refers to "excess deaths", which compares the death rate or numbers before and after the invasion.

You'll note the before death rate was asserted to be 5.5 You'll also note an earlier commenter referred to Hungary's death rate of 13 per k.

The key to the figures is the asserted before number of 5.5 per k. And any number of developed countries--such as Hungary referenced above, have death rates considerably more than that. Looks like somebody stood on the before rates until they shrunk to an incredibly low--but far more useful--figure.

Well, at least that tells us that huge numbers didn't die of the EEEVILLL sanctions. Or the before rate would have been higher.

If there have been 650,000 excess deaths, and my understanding is that violence is the predominate cause of this excess, then I wonder about the ratio of wounded to dead. From my reading of history, in war there is about a 3 to 1 or greater ratio of wounded to dead in combat. If we take the study seriously, then we should also have well over 1.5 million wounded. Has anyone checked this out?

Steve Sailer wrote-
"The American military is firing hundreds of thousands of bullets per day in Iraq, perhaps as many as a million per day."

Ummm........no, looking at the supply replacement data, it would appear that they are firing about 5,600 rounds per day on average-and that would include major operations, daily patroling, and training and probably includes Iraqi military and police and US forces combined.

Not sure the "entire household wiped out by Saddam" would show up (for failing to show up) in these numbers, since the structure they dwelled in would have had to remain empty. I'd need to see the interview-disposition sheets to see what the teams reported in terms of empty homes.

I had worried about spatial correlation but now I see that's not really any different from any other strong within-cluster association -- at the level the clusters are defined, nothing else needs to be done to correct for that. I'm still not sure whether the death-certificate lists were unduplicated across households (the residency requirement notwithstanding), and as the authors noted migration away from high-mortality areas would create an upward bias.

I'm amused by those who suggest that an economist, such as Megan, dare not question a study conducted in the medical research field. I'm an astrophysicist, and medical research is in general a subject of much derision over in the hard sciences. Why? They overinterpret their results (we would never publish a 2 sigma result with uncertainties from 100,000 to 800,000 dead, as Lancet has), they gloss over correlation versus causation, and they underestimate systematic errors. Why do you think eggs are vehicles of death one year and then are nature's perfect food?

For this study in particular, two things concern me. First, in the original paper, they had to throw out a datum that randomly landed in Fajullah because it would have resulted in their study returning nonrealistic results. In my field, throwing out a data point in this manner would have raised serious red flags about the entire methodology, but the authors didn't pause, and have repeated the same data-gathering method. Second, in both papers the authors briefly discuss systematic errors (which are not covered in the statistical uncertainties they quote) but in a cursory fashion, discussing only those errors that would cause them to underestimate the death toll. A lack of skepticism toward an explosive result is poor science.

I don't have a dog in this fight in terms of the number of Iraqis killed. I do get very cranky about sloppy science, though, and this paper falls into that catagory for me.

Out of interest, in your field is it normal to say "I have two problems with this research" and for one of them to be a single outlier in a paper written by the same guy two years ago? Astrophysicists must be an odd bunch.

Actually, it is normal. If an author is known to make up data, or has published results with serious flaws, that is taken into account by scientists reading their subsequent work: I would hope it would be taken into account by anyone else doing the same.

Astro is also quite correct, as far as I have observed, about attitudes in science toward medical research publications (I'm a biochemist). I think that among non-scientists, journals such as the Lancet and JAMA, are given more respect than they are given by and among scientists.That doesn't necessarily mean that this study is flawed, of course.

And yes, Astrophysicists are an odd bunch. Sorry, Astro, but it only hurts because it's true.

Steve, the mere fact that these figures disagree with every other estimate by a factor of 13 (655/50=13.1) is more than enough reason on its own to question the results. The methodological issues are just confirmation of the original intuition.

I'm a physicist. Most of the commenters are assuming that the imagined errors made by Burnham et al overcount the number of deaths. This is naive thinking. In the real world, errors can be positive or negative.

Dragging the Lancet's name through the mud is thoroughly disgusting. It's one of the oldest scientific journals. We know this report was subjected to intense scrutiny before publication. Of course, the editors knew it would generate controversy. By implication you smear the editors of the Lancet, the authors of the paper, the Johns Hopkins Bloomberg School of Medicine, and all of the reviewers. Are you conspiracy theorists? What standard of truth do you seek? There is none higher than publication in a peer-reviewed journal. I'm sick and tired of innumerate fools smearing decent, honest scientists. The argument that something was published, once upon a time, that was incorrect, is irrelevant to this recent cluster sampling survey. Criticize the methods or the analysis. Write a letter to the editor. Good luck getting your rigorous arguments published.

Astro, you certainly sound like an arrogant astrophysicist. I agree with Seb. Let's see, you're saying some doctor somewhere made a mistake. Therefore all medical research is suspect. Brilliant deduction. What's with the two sigma comment? Who is we? Are you saying broad peaks don't occur in astrophysics, or in nature? That would be remarkable.

I have read numerous commenters on this thread reference "other studies" that claim 50,000 deaths.

Where can I find these studies. WEB of Science data base does not find them, nor do my efforts at googling them yield results.

The truth is, there are NO official tallies of deaths in Iraq due to the war. The Iraqi government has actually taken steps to prevent such information from being gathered. The US government position has been publicly stated that it will not collect such information. The Lancet article is not in conflict with any official account. It is only in conflict with conventional wisdom.

There is merit to conventional wisdom. It is not all of the "world is flat" variety, but it is not proof.

I've mentioned this before, but folks seem to be too busy hurling clever insults at each other to notice, so I'll try again.

The data in the study appear to provide confirmation for two hypotheses. One is that the random variable “excess deaths” in Iraq has a central tendency of about 650,000 (leaving aside the issue of discarding outliers). The other is that 91% of respondents could produce death certificates for those family members who have died since the invasion. This tends to confirm the validity of studies based upon counting the death certificates issued by hospitals, morgues, etc. Those studies yield estimates closer to the 50,000 level.

So there is a serious inconsistency here. An argument in favor of the method used to arrive at the larger number is that it is “widely accepted”. An argument in favor of the “count the certificates” method is that this set of data suggests it is worthy of wide acceptance, at least in this case.

Either way, some important information is missing. Unsubstantiated assertions about the number of issued but unfiled death certificates can provide hypotheses regarding the nature and source of the gap, but hypotheses are not data. They aren’t even anecdotes!

So it seems that all we know with a high degree of certainty is that the study in Lancet is at best incomplete in that it tends to confirm two contradictory hypotheses regarding the scale of Iraqi deaths since the invasion. I might argue that a paper that confirms two conflicting hypotheses is not worth publishing, but that is beside the point.

By the way, anyone who believes that no standard of truth is higher than publication in a peer-reviewed journal must be living on a different planet. In my doctoral program I took some classes in the philosophy of science, and I learned about induction, deduction, logical positivism and a lot more, but not the peer reviewed journal standard of confirmation.

Working in statistics, I have to say that this paper is not that bad. The methodology used is a good one. The problems with these retrospective studies are fairly well known and well documented. They are not the best type of studies. But nonetheless, they are not useless and often are the only kind you can do. They do offer a picture of what is happening, even if the picture is fuzzy.

One way to know how accurate the estimates are is to look at the confidence interval. The earlier 2004 study had a wide confidence interval because of the small numbers of people they interviewed. Here the CI is also wide but narrower than the previous study. Because they interviewed more people -- about 2000 households. So they are extrapolating from about 13,000 people onto the 27,000,000 people in Iraq. This is not a bad sample size.

There are polls routinely released in the US that sample fewer people and infer onto a larger population (US:300,000,000)

And those polls often affect political decisions.

Additionally, if you want to be conservative in the estimate of deaths, take the lowest part of the 95% CI which suggests that about 400,000 people have died. That is still a lot.

Even if it is half of 400K, it is still a lot. That is where this study is useful. It draws a reference point where we now have an idea of the magnitude of damage the fighting is causing.

As a note to the physicist who said that the Fallujah data point was thrown out because it was an outlier:
Fallujah was an outlier because at that time there were a TON of deaths there. The authors noted this. They felt that including Fallujah would artificially increase the estimates of death. Fallujah was unusual compared to what was happening elsewhere in Iraq. So the authors omitted the data point. BUT they gave estimates of the death rate if they included Fallujah.

Yancey,

Your question about the ratio of wounded to killed led me to crunch some numbers using the of 2.3:1 (World War II) and 3.28:1 (Vietnam) ratios involving U.S. troops as a guide and the 601,027 number alleged killed by violence in the Lancet study as a guide.

I came up with 1,382,363-1,971,369 Iraqis wounded by violence, for a total of 1,983,390-2,572,396 Iraqi civilians that have either killed or wounded since 2003 based upon the Lancet's figure.

The CIA World Factbook estimates the population of Iraq at 26,783,383 as of July.

That's almost 1 in ten Iraqis killed or wounded.

If that is correct, I find the Lancet study a bit hard to swallow.

The study done by counting the death certificates was done by the Iraqi government. They have a strong incentive to underreport the number of deaths.

Families of the victims have less of an incentive to do so.

"I came up with 1,382,363-1,971,369 Iraqis wounded by violence, for a total of 1,983,390-2,572,396 Iraqi civilians that have either killed or wounded since 2003 based upon the Lancet's figure."

That's nonsense.

Attacks on coalition forces, which have escalated to about 500/week rarely result in any deaths and usually result in no casualties at all. Attacks on civilian targets average 2.3 deaths per attack. Significant numbers of deaths have been from abduction and execution. That scenario does not result in any wounded.

No information can be extrapolated with regard to wounding of Iraqi civilians from casualty statistics from other wars. It's apples and oranges.

Details on the Death Certificate study that estimated 50,000:
http://www.commondreams.org/headlines06/0625-03.htm

"Iraqi officials involved in compiling the statistics say violent deaths in some regions have been grossly undercounted, notably in the troubled province of Al Anbar in the west."

"Many more Iraqis are believed to have been killed but not counted because of serious lapses in recording deaths in the chaotic first year after the invasion, when there was no functioning Iraqi government, and continued spotty reporting nationwide since."

"the ministry said its figures exclude the three northern provinces of the semi-autonomous region of Kurdistan because Kurdish officials do not provide death toll figures to the government in Baghdad."

I know the 2006 Lancet study has its flaws. But the flaws of the Morgue/Certificates study is more severe:

1. They didn't take into account the invasion and the early part of the war when the gov't was non-existent.

2. Left out 3 provinces in the Kurdish area

3. Left out the most dangerous areas.

In addition, these study only counted bodies received by the morgue. In many cases, due to Muslim belief requiring a speedy burial, the body may never have been sent to a morgue.

So it is likely that the deaths were underreported.

Some posters’ emotional attachment to the Lancet study is almost touching! But it’s true that love is blind, isn’t it?

Now we learn that you can ignore the Iraqi government estimates because they have an ax to grind. The publicly, vociferously anti-war authors of the Lancet study must be accepted as impartial scientists, however. Or perhaps counting the actual population is no longer considered a reasonable estimation method?

We also learn that "Iraqi officials involved in compiling the statistics say violent deaths in some regions have been grossly undercounted”. Is this a trustworthy Iraqi official? I didn’t know there were any!

And another tidbit: “The flaws of the Morgue/Certificates study is (sic) more severe”. How do you know? Have you been holding the actual number back from us?! That’s cruel.

Also: “No information can be extrapolated with regard to wounding of Iraqi civilians from casualty statistics from other wars.” Really? None at all? So the actual ratio of wounded to killed in Iraqi is just as likely to be 0.001 to 1 or 1,000 to 1 as it is to be near the historically observed ratio of 2 or 3 to 1?

Then there’s: “There are polls routinely released in the US that sample fewer people and infer onto a larger population (US: 300,000,000)”. Which Lancet survey result are you defending here, the 650,000 deaths or the 91% of deaths having associated death certificates? And please spare me any explanations of how this could conceivably have happened unless you have actual evidence that can explain a factor of 10 discrepancy.

And while we’re on that subject, the government study is dismissed as the only survey that turns up a number around 50,000. But what about the publicly, vociferously anti-war Iraqi Body Count, which is publishing a high-end estimate of around 48,000? Should we also ignore Brookings and Reuters? All of these are by degrees anti-war and all generating estimates more than an order of magnitude less than the Lancet study (whose authors won’t release the raw data, by the way).

No doubt about it. The only correct thing to do is to throw out all of the other estimates and latch onto a far outlier (whose unpublicized results, I reiterate, also tend to confirm the lower estimates) and man the barricades.

Is this what they call the “reality-based community”?

the historically observed ratio of 2 or 3 to 1

Is a ratio describing outcomes for soldiers in battle. I have no idea why so many of the insta-experts who have suddenly discovered a lifelong interest in casualty estimation think it might be appropriate to apply this ratio to civilian casualties of a guerrilla war.

But what about the publicly, vociferously anti-war Iraqi Body Count, which is publishing a high-end estimate of around 48,000? Should we also ignore Brookings and Reuters? All of these are by degrees anti-war and all generating estimates more than an order of magnitude less than the Lancet study (whose authors won’t release the raw data, by the way).

None of these are casualty estimates. They are casualty counts. They count every death reported by CNN or by The Iraqi government. Given current conditions in Iraq, even a quick review of the error rate in **counting** deaths by governments involved in communal intrastate violence suggests that governments *always* undercount their own deaths in these situations. Typically by large multiples.

Every death number you hear for every other conflict in the past two decades has been based, at least during the conflict, on ***estimates***. Darfur is an estimate. Kosovo was an estimate. Rwanda is an estimate. Sometimes the estimates are literally guesses. Sometimes they involve, like in this case, sound statistical science. For no obvious understandable reason, this is the first major conflict in the past several decades where we're relying on *counts* instead of estimates.

Funny, ain't it.

You're an idiot.

Two comments:

1) If we've killed 655,000 Iraqis while trying to help them, North Korea will be a push-over. Their 1.5 million man army looks to be an easy target - we'll by TRYING to kill them.

2) Not all deaths are equal. Some poor shmuck going to work getting blown up by a car-bomb is NOT the same thing as soldiers taking out the car-bomb builder. Both are, legitimately, deaths that would not have happened without the invasion. There is absolutely no moral equivalence. Some of those 655,000 deserved to die for killing some of the others.

mrsizer: North Korea would be a push-over. Unfortunately, the domino right behind them is South Korea.

Thanks for the thoughtful comments –especially the idiot part!

Let me try to explain again and see if you can come down to my simple-minded level and consider this bit of pedestrian logic.

In my idiot doctoral program, in which my minor fields were mathematics and statistics, it was generally accepted that a complete count of a population would provide a better estimate of the actual number of items in that population than an estimate derived through the application of inferential statistics. (Note, in case you didn’t take any classes in idiot statistics, that a common definition of a good estimator is one that is; 1. unbiased, and 2. has a low error as measured by things such as mean squared error.)

What we have here is a number of attempts to count the population by various (overlapping but not identical) methods, which come up with numbers well within an order of magnitude of 50,000. Then we have the results of a survey that generates an estimate of 655,000, more than an order of magnitude greater than the attempts to count the population.

So why the discrepancy? One possible explanation is that the attempts to count the population are biased (extremely) low and/or have very, very high errors. This is a reasonable hypothesis that is worth testing. It is not evidence, however.

The problem that this hypothesis faces is that the study that claims 655,000 “excess deaths” also tends to confirm the lower number because around 90% of the sample surveyed had, in their possession, official death certificates that it is reasonable to believe would have been available to the population counters.

Of course, perhaps only a small percentage of those death certificates were available to the population counters. This is a hypothesis worth testing. It is not, however, reasonable to assume that the larger number must be right, evidence or lack of evidence notwithstanding.

That logic seems to run: “The population counters must be way, way low. We know this because the survey returns a value that is 13 or more times higher than the population count. And we know the survey is right because it returns a value that is much higher than the population counters. Q.E.D.”

Do you see any problem with that logic?

I think that the estimate of 600,000 violent Iraqi deaths is a lot less credible than the authors of the Johns Hopkins study think it is.

http://willscommonplacebook.blogspot.com/2006/10/more-than-600000-killed-in-iraq.html

Will McLean

Steve Sailer wrote-
"The American military is firing hundreds of thousands of bullets per day in Iraq, perhaps as many as a million per day."

Ummm........no, looking at the supply replacement data, it would appear that they are firing about 5,600 rounds per day on average-and that would include major operations, daily patroling, and training and probably includes Iraqi military and police and US forces combined.

I'm going with Steve on this one. I read in Lancet.

JL:

What we have here is a number of attempts to count the population by various (overlapping but not identical) methods, which come up with numbers well within an order of magnitude of 50,000. Then we have the results of a survey that generates an estimate of 655,000, more than an order of magnitude greater than the attempts to count the population.

I'm sorry, but what population counts are you referring to?

Iraq Body Count is *not* a population count. It counts

1) only civilians,
2) only killed by violence,
3) only reported by 2 English speaking news sources.

That it lists a "high" count of under 50k does not mean that they think that no more than 50,0000 people have been killed by violence. It means that they don't know of 50k who've been civilians, who died by violence, and were reported by 2 separate news sources.

It's absolutely certain to be an undercount; it's not trying to count military or insurgent deaths, non-violent deaths, and it seems impossible to believe that two English speaking news sources catch every single violent death in Iraq.

As for the death certificates, pieces of paper are not the population; bodies are the population. If you can't find a government office that has more than 50,000 death certificates, that only tells you about how many death certificates that government office can lay its hand on, not how many have been issued... especially when certificates are issued locally, and might not even be gathered where they can be counted. To call a government issued certificate count "the population" is to show a profound misunderstanding of statistics.

Yes, if we walked around the entire country, and did a similar survey, it would be more accurate than merely talking to 12,801 people. *That* would give us a population count.

Even if we could locate every death certificate ever issued (assuming none have been lost, destroyed, etc.), that wouldn't be a population count... unless we were interested in the pieces of paper, not the people they represented.

Longhairedweirdo,

The population that the IBC is attempting to count is civilians killed by violence. Did you notice that I used the word attempt? That means that neither I nor the IBC believe that the number is absolutely accurate. Just that it is an attempt to count the population as defined above. It may be – in fact probably is – low biased. The Lancet study, on the other hand, was an attempt to extrapolate the answer through the use of a sampling technique. As I understand it, both Brookings and Reuters are also attempts to count populations. Again, only attempts.

Note that I very explicitly noted that there are reasons why the attempts to count the population could very well be off, even by very large amounts. They are still attempts to count the population. The number of death certificates issued may or may not map well with the number of deaths, but the people who are using them are still attempting to count the population, not extrapolating death numbers using sampling techniques. Using your implied definition, the US Census is also not an attempt to count the population because census takers go to households and ask whoever answers the door how many people are living there rather than having them all line up on the front lawn to be counted.

Dare I suggest that your inability to recognize this relatively simple distinction suggests a profound misunderstanding of statistics?

By the way, the Iraq Body Count has published a thoroughly withering critique of the Lancet study. You might want to read it.

There is someting funny here, to be sure...
91% of all deaths counted by the Lancet survey had death certificates. This can only mean one of two things:

1. Most Iraqis who are killed or die in Iraq are issued death certificates, in which case a far more accurate count of fatalities can be made by counting death certificates issued; or,

2. A higher than average number of Iraqis surveyed had death certificates for casualties in their families, which indicates that the sample was not random.

No other conclusions are possible - which is it?

Tom

Basically, what the report is suggesting is that the estimates from the Iraqi government, and from IBC through the press, are focussed on Baghdad, while mortality is up in many other regions too, including rural areas.
the following article:
http://news.bbc.co.uk/1/hi/world/middle_east/6045112.stm
mentions that the UN published 6599 killed in Iraq in July and August, of which 5106 in Baghdad. The lancet report would suggest 500 killed per day, that's 30.000 for these two months together.

Now, if the UN numbers (given by Iraq's health department) are correct, than Baghdad, with 20% of the population, is by far the most dangerous area.

The Lancet report however suggests that mortality is high throughout the country, with Baghdad being no special exception. Then the 5106 out of 6599 suggests that deaths outside of Baghdad are seriously underestimated by the Iraqi government.

This is in agreement with the idea that local authorities are giving out death certificates, but with the central authority not keeping good track of them.

It also fits with the fact that the press ( and as a result Iraq Body Count) reports much lower death tolls than would be required for the Lancet figures. Most press is in Baghdad, and large parts of Iraq will have no press at all. From the news papers, my impression was that Baghdad was the centre of the fighting, with the rest of the country calmer. If that's not true, the Lancet numbers are still high, but not unbelievably high.

Many people in the posts above are mentioning a "factor 10" difference between the normal numbers and the Lancet. That is however clearly not the case: Iraqi Body Count already has around 45000 CIVILIANS killed,REPORTED in the press.
If half of the civilian deaths are reported ( which seems high, given that many parts of the country have no journalists), and assuming that as many guerilla are killed (by each other or by American forces)as civilians (which seems again a safe assumption), then we're already on 180,000 death. Far from 600,000, but not an entirely different region. ( the last 50,000 in the Lancet report are disease and nutrition related deaths, which would not be counted in other reports at all )

So, the Lancet report steps over its possible structural errors too easy for my taste, I would have liked to see better calculations on the possible effects of these errors, and they have to provide better arguments for the death certificate gap, but their number is not so unbelievable as it seems at first sight

JL:

Note that I very explicitly noted that there are reasons why the attempts to count the population could very well be off, even by very large amounts. They are still attempts to count the population.

Neither Iraq Body Count, nor the Iraqi government *have* attempted to count the population. Both are counting second order information. IBC is tracking news reports, the Iraqi government claims to be tracking death certificates.

Neither is going out, asking about deaths, and trying to verify those deaths. That would be a population count.

Hence, my question:
I'm sorry, but what population counts are you referring to?

Do you now have a meaningful answer, or do you just want to try to bluff your way through again?

Tom:

You're correct; something is wrong with death certificate tracking, *or* something is very strange with the Lancet report.

Iraq is in a war zone; it had a government, that was tossed out, and another government was imposed, then another government was appointed, then another government elected. Reconstruction of Iraq has moved very slowly, and we don't even know if they had perfect death certificate tracking before the war, but we have reasons to suspect they had problems even then. Last I'd heard (admittedly, a long time back), they couldn't even keep the lights on 24x7... I imagine tracking death certificates isn't their top priority.

Thus, I suspect that the problem lies with the death certificate tracking. If someone showed that there was very strong death certificate tracking (not merely a good system, but one that worked, where any certified death could be quickly verified by the certificate), and the certified number of deaths was much, much lower than the number suggested by the Lancet study, then it would call the study into question.

No one has attempted to prove there is good tracking; they've simply asserted that there is, and that it works well enough that the Lancet study can't be correct.

I'd read an article somewhere (can't remember the source) which discussed that the problem with recording deaths in Iraq was that the local hospitals were very good about issuing death certificates (a death certificate is required for burial or to receive benefits), but providing the records to the Ministry of Health (or whatever it's called) was a problem.

Seems to me a useful study would be to visit the hospitals and do a sampling of their records. If the hospitals are issuing death certificates (as seems to be the case with 90% of the people having a certificate handy) but the certificates aren't reaching the government to be counted, then wouldn't it make sense to just ask the hospitals?

I'm willing to bet such a study would be a lot safer for the researchers since most of the hospitals in Iraq are protected by coalition or Iraqi troops.
Just a thought.

Comments are Closed.