January 18, 2007
From the desk of Jane Galt:Doves are not evil, even ones who endorse the Lancet study
I seem to have been misunderstood slightly on my previous post on the Lancet study by Burnham, et al. Both pro- and anti-war commenters took me to be saying that I thought people who supported the high Iraq casualty figures wanted an extra half million people to be dead. This is not what I was saying, and I apologise for the false implication.
What I was trying to say is that it's easy in an argument, particularly one that has been as vicious and emotional on both sides as the debate over Iraq, to get caught up in winning. Repeated ill-tempered exchanges etch resentments of your opponents deep into your soul; repeatedly defending yourself from attacks that blend personal insults with repetitive iterations of arguments you consider self-evidently ridiculous, combined with an obstinate refusal to consider the irresistable logic of your position, ratchet up our committment to the position you have staked out. Then along comes something like the Lancet study*, and you think "Yes! I was right! Those unreasonable, amoral bastards will have to listen now!"
It's very easy to overlook the fact that "Yes! I was right all along!" means that 500,000 or so extra people are dead. We're not programmed to deal with that sort of thing. We're programmed to live in small bands of perhaps 30 people, and to respond to conflicts with an implacable rage to win.
Look back at the Jenin massacre. Did the Palestinians want hundreds of their countrymen to die? Of course not; whatever you may have read, Palestinians are not amoral animals who don't care about human life. They're normal people, who are quite fond of the other people in their country. They weren't saying to themselves "Yes! Hundreds of people are dead!" They were saying (I imagine): "This will show the world what the Israelis really are!" And they held onto the belief in a massive massacre at Jenin longer than the evidence warranted because it fit with their (very emotional) beliefs about the Israelis.
I'm sure that the people who were defending the Lancet study with every fibre of their being feel just as I do about half a million dead people. I just got the feeling that they weren't thinking about the 500,000 extra corpses when they defended the Lancet study with every fibre of their being.
*Yes, this is a terrible way to describe it, but that's the popular locution now
Posted by Jane Galt at January 18, 2007 12:26 PM | TrackBack | Technorati inbound links"); ?>If that's all you're saying, then who gives a damn?
What makes it remotely interesting, or affects the quality of anyone's arguments in any way, that they may or may not be keeping the human cost in the forefront of their minds while making them? For your personal friends, I suppose you care about whether they're nice people, but when you're arguing about the validity of a study, what's the value of bringing in the emotional decency of the people you're arguing with?
Posted by: LizardBreath on January 18, 2007 12:52 PMThe only thing that bothered me about the study's defenders is that they refused to consider the possibility that the data set was flawed, and this is only irksome because there was a lot of on-the-ground data suggesting that the study's conclusions were anomalous. Instead of taking the reasonable position of writing that the conclusions may not be correct, and that the actual data collection needs to either be carefully examined for errors or completely redone with better controls, they chose to repeatedly point out that the manipulation of the data after collection was proper statistical analysis; they then repeatedly claim the study's critics were simply wrong for this reason- a classic strawman argument if I have ever seen one.
There are a lot of people on both sides of this debate who let their partisan, political passions cloud their judgement. A good referee of this paper would have questioned, strongly, the conclusions and their large variance from actual data. I wonder if the authors themselves even question their own results. I know I would if I had done the study.
Posted by: Yancey Ward on January 18, 2007 1:27 PMBut of course defenders of the study generally didn't refuse to consider the possibility that there was something wrong with the data set -- I was reading, and engaged in, arguments on that very point. Where I came down was that the contradictory data sets being appealed to (the IBC, centralized Iraqi government data) had obvious reasons to be flawed, and there were no persuasive reasons to believe that the Lancet data set was (rather than 'could be') flawed.
It's possible that the methodology of the data collection was perfect, and just through the luck of the draw oversampled deaths -- there's a 1/20 chance for any statistical sample that the true results are outside the confidence interval for the study. But I haven't seen evidence that convinces me that that is the case.
Posted by: LizardBreath on January 18, 2007 1:49 PMIt's amusing that supporters of the war, proven wrong are now chastising opponents of the war for claiming philosophical victory. Well, I won't claim victory if you admit defeat. Don't get me wrong: there should be no celebrating: there's been way too much dying for that, but unless it's universally understood that the beliefs underlying the invasion and occupation were morally and strategically unsound, there stand a good chance of us repeating this godawful mistake.
Posted by: david mizner on January 18, 2007 2:02 PMLizardBreath, there was, for starters, a UN study which used the same methodology as the Burnham studies, but a much, much larger number of clusters, and found many fewer deaths than the Burnham study for its time period (the Burnham study is more recent than the Iraq study, but some parts overlap.) And while even the folks at the IBC would undoubtedly concede that they are undercounting because of deaths that didn't make it into the newspapers, the checks they offered were not against their own work; rather, they were asking, quite reasonably, uestions like "Could the Sunni Triangle really have lost well over 10% of the young males between the ages of 15-45 (which is a far higher casualty rate than America experienced during World War II) without many people noticing this?" and "Could we be having so many mortar attacks causing deaths without causing injuries that would show up in hospital records?"
At any rate, we now have a UNAMI report that purports to have checked all the death certificates in teh country and come up with an appalling figure of 34,000 violent civilian deaths in 2006, which is awful, but many multiples lower than even the lower bound of Burnham et al. If they did in fact check all the certificate issuing authorities, and Iraqis really do virtually all get death certificates because they are required for burial on sacred ground, then the UNAMI figure can be assumed to be a slight undercount, but basically authoritative.
Posted by: Jane Galt on January 18, 2007 2:06 PMThe beliefs underlying this invasion were neither morally nor strategically unsound, and my biggest fear is that if we don't admit that now, we will be faced with a much larger conflict within the next 30 years. Let's continue to fight this morally and strategically sound battle now instead of later.
Posted by: Rex on January 18, 2007 2:09 PMOne more thing: it's ironic that Galt is claiming that opponents of the war sometimes don't care much about the Iraq it dead, given that the last time I debated this issue with her, she said that there was no reason for the United States to keep track of how many people have died. She said, "The U.S. doesn't have any particular reason to record them--that's the Iraqi government's job--much less any ability to do so..."
How ridiculous is this? The American military, with it's billions of bucks, can't conduct or fund a study? It can't do what the U.N just did? And of course the United States has the moral responsibility to try as best it can to measure the impact of the war it started. But Bush and team don't, because they know that the numbers would hurt them politically. It's for this reason that the Iraqi government just blasted the U.N. report. Having expressed clearly that she doesn't think there's any reason to figure out how many Iraqis this war has killed, Jane now chastises opponents of the war for our alleged callousness. Cute.
Posted by: david mizner on January 18, 2007 2:12 PM"It's very easy to overlook the fact that "Yes! I was right all along!" means that 500,000 or so extra people are dead."
Hello? Accepting the study's validity doesn't actually result in actual death? Y'know, like, bombs and stuff do? It just results in accepting a number that doesn't reflect reality. And if the study's right and you hope it's wrong, that hope doesn't actually bring people back to life.
This whole tangent -- and it is a tangent -- is just a "the best defense is a good offense" blameshifting exercise. A bad war is morally repugnant. To avoid looking in the mirror about that and avoiding the terrible consequences of being wrong about both the premise and justifications for advocating this fiasco, best to find moral repugnance among those on the other side of the question who are looking at the consequences of your being wrong. There! It's all even now!
Posted by: g-man on January 18, 2007 2:16 PMHere's a response from Les Roberts, one of the authors of the Lancet study to the criticism that it was inconsistent with the UNDP study:
LR: The UNDP study was much larger, was led by the highly revered Jon Pederson at Fafo in Norway, but was not focused on mortality. His group conducted interviews about living conditions, which averaged about 82 minutes, and recorded many things. Questions about deaths were asked, and if there were any, there were a couple of follow-up questions.
I suspect that Jon's mortality estimate was not complete. I say this because the overall non-violent mortality estimate was, I am told, very low compared to our 5.0 and 5.5/ 1000 /year estimates for the pre-war period which many critics (above) claim seems too low. Jon sent interviewers back after the survey was over to the same interviewed houses and asked just about
I'm not saying that you are compelled to be convinced by this, but it's a reason for accepting the Lancet numbers over the UN numbers.
If they did in fact check all the certificate issuing authorities, and Iraqis really do virtually all get death certificates because they are required for burial on sacred ground, then the UNAMI figure can be assumed to be a slight undercount, but basically authoritative.
This I find completely unconvincing. Iraq is a war zone, and keeping accurate statistics isn't a trivial task anywhere. I don't see any reason to assume that the Iraq government is capable of tracking all of the death certificates issued by local doctors, hospitals, and other issuing authorities at any level of accuracy.
Posted by: LizardBreath on January 18, 2007 2:31 PMSo, to take up dsquared's earlier analogy, people who argue against Holocaust deniers don't necessarily want six million Jews to have died, but they're probably not really thinking about the six million Jews.
Hard charge to defend against. But even assuming it were true, trying to minimize and downplay the deaths with specious quibbles about methodology and datasets and the presumed motives of the researchers would still be the nastier habit. From what I can tell, this is what happened and is still happening with the Lancet study, where the initial thought process seems to have been:
Then along comes something like the Lancet study*, and you think "No! They're lying! Those unreasonable, amoral bastards are just trying to influence the elections!"
This sort of thing was in fact what many of those obsessed with defending denialism with "every fiber of their being" came out and said.
It wouldn't be too much of a stretch to vestiges of this in your attempt to seize on the UNAMI report as a way of minimizing the Lancet study's findings. If the number quoted in the UNAMI report is a count based on Iraqi government sources (as I suspect it is) comparing it to a statistical study is apples and oranges. Maybe a little less haste is in order.
Posted by: Doctor Slack on January 18, 2007 2:41 PMdavid, i would think that the military has more important things to do with its resources that actually involve its core competencies than to run civilian casualty studies. besides, even with its billions of bucks & presuming a moral responsibility, would a perfectly accurate military sponsored report even be credible? wouldn't you hear "instead of funding bogus, politically motivated studies, shouldn't the military be spending its resources doing [some army-like activity]"?
That's why the caveat "IF it is true that they visited all the issuing authorities". They didn't get their numbers from the Iraqi central government; they say they got them from the actual issuers. The Iraqi government, by the way, says the number is too high, but they have clear political incentives to downplay the deaths.
I'm pretty sceptical of any claim that the UN can't get good data because it's a war zone, but a bunch of unarmed doctors can. If things are so fucked up that you can't get into the local hospital, then things are so fucked up that it's seriously throwing off any results: massive population transfers, etc. Any cluster sample done during World War II would have been completely useless.
But it seems to me that Iraq isn't that kind of war zone. It's a place where it's extremely dangerous for unarmed strangers to travel, but it is not a place where a UN armed convoy can't travel to the local hospital because it requires crossing militarised lines where trigger-happy boy soldiers shoot to kill at anything that moves. The UN document is extremely sketchy on the details, so as I say, it may turn out that one of the two assumptions is wrong. But if those assumptions are validated, I'd say it's a nearly definitive rebuttal of Burnham. If Iraqis really still have to have a birth certificate to bury their dead, if those certificates are issued by hospitals, health ministries, and morgues, and if the UN in fact visited every hospital, health ministry, and morgue, then it seems virtually impossible that Burnham's numbers are even close to correct. But those are very big if's, because I just don't know much about the UN mission in Iraq.
Posted by: Jane Galt on January 18, 2007 2:45 PMSure, Will, they've got more important things to do, like, you know, trying to stop the killing. But it's of obvious moral and political importance to know how many people have died, as the furor over Lancet indicates, and the U.S. government should try to find out the truth. It doesn't have to be the military. State could do it. But we all know the reason it hasn't.
All politics all the time.
Posted by: david mizner on January 18, 2007 2:50 PMJane: That's why the caveat "IF it is true that they visited all the issuing authorities". They didn't get their numbers from the Iraqi central government; they say they got them from the actual issuers.
Jane, where does the UNAMI report say that they're getting numbers from the actual issuers? Is this the same report you're referring to? It says (footnote 2, bottom of page 4) that its figures are derived by adding up two numbers, one from the Iraqi ministry of health (based on reports from hospitals) and one from the Medico-Legal Institute (MLI) in Baghdad, which reports the number of unidentified bodies it receives. I didn't see anything about checking with individual issuers of death certificates.
Posted by: Russil Wvong on January 18, 2007 2:53 PMThe UN document is extremely sketchy on the details, so as I say, it may turn out that one of the two assumptions is wrong.... But those are very big if's, because I just don't know much about the UN mission in Iraq.
Couldn't have said it better myself. As between two numbers, one where I know how the data was collected, and the other where I don't, I'm going to regard the first as more reliable. This doesn't mean I'm always going to be right, but it certainly seems like the way to bet.
Posted by: LizardBreath on January 18, 2007 2:55 PMThere will never be a definitive body count from Iraq. It's just like economists to argue the number and not the sin.
Posted by: judson on January 18, 2007 3:03 PMRussell, I'm relying on news reports and a quick read of UNAMI. Will go back and check later.
Posted by: Jane Galt on January 18, 2007 3:07 PMAnd LizardBreath, I'd say its a stretch to claim we know how the data was collected; Burnham et al have been rather sketchy on the details, and sending amateur surveyors, even well-meaning ones, unsupervised into a dangerous area does not fill one with greatest confidence in the results.
But I'll reverse Daniel Davies' question here: given that Burnham claims the majority of deaths have occurred in a few cities (Iraq's population being highly concentrated in urban areas), what are the odds of getting this result from the government figures if the Lancet figure is correct? This would imply either massive, massive fraud on the part of the Iraqi authorities (certainly not impossible), or that only about 10% of the bodies are making it into morgues in the major cities.
Posted by: Jane Galt on January 18, 2007 3:13 PMBurnham et al have been rather sketchy on the details
This is simply untrue. What would you want to know about the methodology that you don't? I read the initial Burnham paper, and the methodology seemed to be fully described.
sending amateur surveyors, even well-meaning ones, unsupervised into a dangerous area does not fill one with greatest confidence in the results.
How so -- what about the danger of the area is likely to introduce error into the surveys?
This would imply either massive, massive fraud on the part of the Iraqi authorities (certainly not impossible), or that only about 10% of the bodies are making it into morgues in the major cities.
Or, you know, that they aren't successfully compiling accurate statistics on the bodies that do make it in.
Yes, right, it could also imply some sort of massive incompetence on teh part of IRaqi authorities, but Iraq is not Tanzania; Arab countries tend to have relatively decent recordkeeping on that sort of thing.
Burnham haven't released data on things like how the interviews were conducted: things like one researcher or two? (Two-one male, one female would be standard, but by my count this makes it mathematically impossible for them to have conducted the number of interviews claimed even if every single interviewee lived in an apartment building). Nor, AFAIK, are they willing to release the underlying dataset even with the names stripped off. This is claimed to be standard in public health becuase the results are often controversial and people nitpick; all I can say is that if this is true, I am even less impressed by public health surveys than I was previously; what part of "scientific method" did they not understand.
Lots of things about the danger are likely to introduce error. If (for example) you skip houses or streets because there seem to be gun waving lunatics in the neighbourhood, but do not know how important it is to record these things, you could wildly alter the results. If you decide that it would be safer to do non-random interviews by, say, asking the village head who you should talk to, and do not realise that this completely invalidates any results you get, this could wildly screw up the sample. If you cut short an interview, and don't realise that this invalidates it, that would be a huge problem. If you're missing a big chunk of population which has fled the violence, and do not know it, your estimates of current populations will be wrong, screwing up your analysis.
The biggest problem is the lack of supervision: the people who designed the study could not oversee the research. If you've ever worked with groups where amateurs are involved in, say, a street canvass or petition signing or a study, you know that the actual procedures quickly vary quite a bit from what is supposed to happen. Even assuming no malice on the part of the doctors who did the surveying, they could have gotten bad data; and with unsupervised researchers, especially working alone as the Burnham reasearchers pretty much had to have been to have completed the claimed number of interviews, there is an obvious danger that someone with an agenda is throwing off your results. One of the reasons that people wanted to see the underlying data with the identifiers stripped off was to see whether certain researchers got anomalously high results.
Posted by: Jane Galt on January 18, 2007 3:31 PMLizardbreath,
Burnham has not, to the best of my knowledge (correct me if I am wrong), the actual data so that it can be checked to see if it is actually appropriate for the analyses performed. Until this is done, I can certainly not put any more faith in their results than I can any other numbers being reported out of Iraq.
This is why I am irked (actually, amused is probably a better description) by the study's supporters-they readily dismiss any and all contradictory findings while asserting that the Burnham findings are the best count of mortality.
Posted by: Yancey Ward on January 18, 2007 3:34 PMThat first sentence was missing the word "released".
Posted by: Yancey Ward on January 18, 2007 3:38 PMbut Iraq is not Tanzania; Arab countries tend to have relatively decent recordkeeping on that sort of thing.
Are you on crack? It's a freaking war zone. While it's possible that the statistics-compiling functions of the Iraqi government are ticking along just fine, not much of the rest of it is.
Posted by: LizardBreath on January 18, 2007 3:58 PMPlaestinian mothers hand out sweets on being informed that their sons have blown themselves up. Many leaders of Jihadist groups brag that "they love death" more than we love life. They are immoral animals that glorify in death and want their fellows to die, as long as they kill the Jews. Listen to Friday prayers, listen to Al-Jazeera, listen to their videos. They are inhuman animals who will do anything and everythin to kill every Jew in the world, every idolator in the world, every heathen in the world, and every apostate in the world (for the Sunnis, this means every Shia and Ismaili).
The Palestinians voted for Hamas, a party with an explicitly genocidal aim. They deserve to eat plutonium.
Posted by: Hey on January 18, 2007 4:00 PMThis random moment of cherry-picking and unhinged genocidal hatred brought to you by Greenstripe. Who may or may not be craven little time buyers.
God, but it's good to see civility in action.
Posted by: Doctor Slack on January 18, 2007 4:10 PMIt's mathematically impossible for two researchers to interview thirty households in a day? Ten minutes an interview, five to walk next door, and bob's your uncle. I could see thinking that they were working hard, but calling it impossible seems poorly supported.
Posted by: LizardBreath on January 18, 2007 4:12 PMJane Galt correctly points out that doves have a bias towards believing war is more costly and destructive than it actually is, the better to argue against it. Hence it is reasonable to somewhat discount dovish estimates of US casulties and the like.
However if Galt was really being even-handed she would also point out that hawks have a bias towards believing our potential enemies are more evil and dangerous than they actually are, the better to argue for war. Hence it is reasonable to somewhat discount hawkish estimates of past atrocities and the like.
Everybody likes to be proved correct even if it means bad things have happened. Galt criticizes doves for being happy with high estimates of war casulties. Why doesn't she similarly criticize hawks for being happy with the discovery of mass graves and other evidence of prewar atrocities?
Posted by: James B. Shearer on January 18, 2007 4:19 PMRemember, you have to include travel time, explaining the purpose of the survey, stops for nature's calls, waiting for someone to answer the door, having a stone in your shoe that needs removing, etc. On a very, very, very good night as a canvasser, in a tightly packed suburb, I would cover 30-45 houses, but that includes the 50% or so who weren't home, and the much larger number who said "not interested" and slammed the door in my face. Then you have to factor in that they reported getting death certificates from virtually everyone they talked to. Even if our researchers are trained speedwalkers who don't need to eat, never go to the bathroom and breeze their way through roadblocks, it is not credible even with a short list of questions that they made their way into the door, explained why they were there, asked all their questions, and had the family dig up death certificates for everyone in the household who had died in the last five years, in under ten minutes. Especially since arabs are chatty.
Posted by: Jane Galt on January 18, 2007 4:22 PMThen you have to factor in that they reported getting death certificates from virtually everyone they talked to.
No. The reported getting death certificates from 80% of the people who reported a death. That's nowhere near everyone they talked to. The modal response to the questions was: "No, no one in this household died over in the relevant time period." Which shouldn't take all that much time.
I too have canvassed door-to-door (Sane/Freeze, Boston, 1990), and in a densely packed area this doesn't sound improbable at all.
Posted by: LizardBreath on January 18, 2007 4:45 PM1) You and I didn't have to contend with roadblocks
2) We didn't have to write anything down except their address
3) We didn't have to get official certification of anything
4) We didn't care about accuracy
5) We didn't have to ask about sensitive subjects requiring elaborate explanation of why we wanted the data
6) We didn't canvass any low-density areas
7) We didn't have to explain what counts as a "household", wait while people remembered whether Cousin James lived here in 2001, etc. etc.
And I *still* didn't talk to forty people in one day. All the experienced social science researchers I have seen discuss this find a ten minute interview extremely implausible. I've done that kind of interview, where you have to take down demographic information about families, and it takes a long damn time. People talk about their children, and it's not productive to cut them off. I'd say on an average interview where I ask people about their families (I've just done quite a lot of these in Upstate New York), ten minutes is practically the bare minimum to get through glowing reports of Junior's baseball prowess and Sis's terrific report card. And from what I understand, Iraqi families are rather larger than the ones I was talking about.
Estimating a single interview as, minimally, the time required to issue a complete rap and get a signature on a postcard and a petition, and an "interview with death" as doing aforementioned plus getting a check (rather conservative, since I would imagine that even in Iraq, death certificates are not kept as handy as checkbooks), I still find this a completely incredible number. I can't get out of a basic interview for a soundbite with an experienced interviewee in under ten minutes. Maybe people move faster in Boston, but if I had managed to do a complete rap with postcard and petition signature on 25 people in one night, I would have nominated myself for "Best canvasser in history". As I recall, it was a banner night when I went over a single sheet with ten slots.
Posted by: Jane Galt on January 18, 2007 5:09 PMAt this point we're playing dueling anecdotes. But do note that you're not talking about anything like 'mathematically impossible'. You're talking about, "Seems really unlikely to me, given my assumptions about how chatty Arabs are, how far apart Iraqi houses tend to be built, what the odds are that anyone's home in a given household when you arrive, etc." I figure anyone reading should take your qualms about the practicality of the study for what they're worth.
Posted by: LizardBreath on January 18, 2007 5:17 PMWhile I don't want to debate the Lancet study I would like to point out a couple of things. Jane you earlier said "Could the Sunni Triangle really have lost well over 10% of the young males between the ages of 15-45 (which is a far higher casualty rate than America experienced during World War II)"
As far as I remember the United States was never occupied, so comparing the US casuality rate with the Sunni males isn't a valid comparison. I notice you didn't trot out the Soviet Union (which might be a better comparison at 13.77%) or Poland(at 16%) or Germany (10%). So in light of these figures 10% of MAMs in the Sunni Triangle doesn't sound impossible - implausible but not impossible.
Additionally people seem to forget (or least those not familar with Islamic law) that after death, Islamic law requires burial within 24 hours of death. Most people I have seen commenting on the Lancet study never seem to realize this. They assume the process is identical to here in the United States that you take the body to the hospital, get a death certificate and bury someone a week later. It's 24 hours and you get the death certificate later if possible.
Posted by: Brian Despain on January 18, 2007 5:31 PMPlease note those death figures I cited are for the overall population both male and females. The death rates among military age males is certainly higher. For some reason young males seem to die more in wars and their death rates very easily could be 10%
Posted by: Brian Despain on January 18, 2007 5:34 PMBrian, I think you're missing my point. It is not that such casualty rates are impossible; the British aristocracy lost nearly 20% of its young men in WWI. But such losses are experienced as completely devastating, to the extent that they dominate the public mind. My point, rather, was that the American public lost a comparitively trivial number of young men, far less than 1% of the population, and that subject was experienced as a devastating demographic impact; it seems unlikely that Iraq could have lost five times as many without (IBC claims) public communications agreeing on that fact. When you lose 10% of your population of marriageable males, the impacts are immediate and obvious. They don't stay hidden.
Now, since I don't read Arabic, it's entirely possible that all anyone in Iraq is talking about is the fact that every family they know has lost at least one son in this conflict. But the IBC implies that this is not the case.
Posted by: Jane Galt on January 18, 2007 5:38 PMAs for Islamic law, the doctors at Healing Iraq claim that there should be death certificates somewhere for all of the deaths. The issue was whether they were getting lost or not counted somehow, not whether they were being issued; all the sources I've seen from Iraq seem to agree that you still need a death certificate to get buried.
Posted by: Jane Galt on January 18, 2007 5:39 PMOk Jane I understand what you were trying to say. Using the example of the United States was bad idea, especially in the context of casuality rates.
What you are saying now is a bit clearer - if the death rates were this high, we would be seeing other effects in the society.
Posted by: Brian Despain on January 18, 2007 5:45 PMBut then I'm wondering what your basis is for saying that we aren't seeing those effects? Your extensive Iraqi acquaintance? Your habit of reading the Iraqi news in Arabic?
I don't have any first hand knowledge of what's going on in Iraq either, but I'm not basing anything on it.
Posted by: LizardBreath on January 18, 2007 5:47 PMFrom the previous link:
13. Madelyn Hicks, a psychiatrist and public health researcher at King's College London in the U.K., says she "simply cannot believe" the paper's claim that 40 consecutive houses were surveyed in a single day. Can you comment on this?
[Roberts]: During my DRC surveys I planned on interviewers each interviewing 20 houses a day, and taking about 7 minutes per house. Most of the time in a day was spent on travel and finding the randomly selected household. In Iraq in 2004, the surveys took about twice as long and it usually took a two person team about 3 hours to interview a 30 house cluster. I remember one rural cluster that took about 6 hours and we got back after dark. Nonetheless, Dr. Hicks concerns are not valid as many days one team interviewed two clusters in 2004.
Analyzing this:
In Iraq in 2004, the surveys took about twice as long and it usually took a two person team about 3 hours to interview a 30 house cluster.
This implies that each interview was conducted by only one interviewer, each interview taking 12 minutes (five per hour).
With a four-person team (in the 2006 study), five interviews per hour, that's 20 households per hour. In the 2006 study, each cluster was 40 adjacent households. So that'd take about two hours.
Posted by: Russil Wvong on January 18, 2007 5:51 PMLizardBreath: But then I'm wondering what your basis is for saying that we aren't seeing those effects?
Presumably if casualties were that high, Iraqi public opinion would have turned against the US, despite the US having overthrown Saddam Hussein's brutal government.
Posted by: Russil Wvong on January 18, 2007 6:01 PMWhen you lose 10% of your population of marriageable males, the impacts are immediate and obvious. They don't stay hidden.
For example, you might expect Iraqis to be saying things like: "A day in the life of the average Iraqi has been reduced to identifying corpses, avoiding car bombs and attempting to keep track of which family members have been detained, which ones have been exiled and which ones have been abducted."
Those of us who can't read Arabic are working with something of a handicap, but I don't see any difficult squaring the statements of Iraqis that we do have with the mortality rates reported by Lancet. If anything, it's the lower numbers that look suspect given the context.
Posted by: Doctor Slack on January 18, 2007 6:03 PMRussell, Les Roberts' response to critics who say you can't do good interviews that fast seems to be "Well, we did our interviews that fast", which rather begs the question. He doesn't have any explanation for why his interviews go so much faster and better than everyone else's. In a similar vein, his response about the UNDP survey seems frankly bizarre. If he's really doing his interviews that fast, then "focussing on deaths" is irrelevant, because he can't really be asking more, or better, questions than the UNDP did. He then says that if you go back and ask those questions again, without asking other questions, you get wildly different responses. This is supposed to reassure me about the quality of his data? This is, as I understand it, why researchers take a little time with their interviews; one of the things they do is ask the same question in slightly different ways, so that you uncover those kinds of differences in how the question is asked, and reconcile them.
Again, obviously things are very fucked up in Iraq. And I've no doubt that the UNAMI number is understated. But understated by a *factor of ten or more*? Iraq isn't Dresden, or even Liberia. The hospitals haven't been shelled, and they aren't shovelling unidentified corpses into mass graves. There is a government recording infrastructure, which Iraqi doctors in urban areas allege is still working.
Posted by: Jane Galt on January 18, 2007 6:33 PMI've no doubt that the UNAMI number is understated. But understated by a *factor of ten or more*?
Again, you seem untroubled by the possibility that you're comparing apples and oranges. The Lancet study is a statistical estimate of both violent and nonviolent excess deaths. The UNAMI number is a count of violent civilian deaths. I don't see why it's so difficult to understand that the numbers differ -- it doesn't mean one of them has to be false or a "rebuttal" to the other.
Posted by: Doctor Slack on January 18, 2007 6:47 PMAskdocslack, read the study. Violent deaths comprise something like 90-95% of the excess deaths.
Posted by: Jane Galt on January 18, 2007 7:04 PMAround 90%, looks like. Right you are; there would need to be an explanation for the discrepancy.
Posted by: Doctor Slack on January 18, 2007 7:11 PMActually, on further review, LizardBreath's point about incomplete reporting is probably correct assuming the Lancet study holds (as it has thus far).
Posted by: Doctor Slack on January 18, 2007 7:24 PMJane: Russell, Les Roberts' response to critics who say you can't do good interviews that fast seems to be "Well, we did our interviews that fast", which rather begs the question. He doesn't have any explanation for why his interviews go so much faster and better than everyone else's.
Hicks' initial comment was assuming that the entire interviewing team ("one intact team") was conducting each interview, and therefore it would take at least 10 hours to interview 40 households, assuming 15 minutes per interview. This wouldn't allow for any travel time, hence Hicks' criticism that this was physically impossible.
From Roberts' response, each interview was actually conducted by a single interviewer. So it would take only 2.5 hours to interview 40 households (10 households per interviewer), assuming 15 minutes per interview.
Posted by: Russil Wvong on January 18, 2007 7:30 PM"what about the danger of the area is likely to introduce error into the surveys?" Mainly that rather than going there, poorly supervised surveyors are likely to either survey somewhere else, or just make something up. The latter would also explain the amazing speed of the surveys. About that, I did census work once. When I found someone at home, it usually took ten minutes just to get started on the forms. I'd think Arabs, like nearly anyone else, would find someone who asked them about their children, wrote a few things down on their form, and rushed off in 10 minutes or less rather impolite - and I really, really wouldn't want to seem impolite in a country where everyone is on edge and owns an AK47!
Another thing that would reduce the danger and increase the speed would be if the surveyors made arrangements with the local chief for a conducted tour - but rather than a random sample, they'd get shown around to people who had been selected and primed by the chief to tell a particular story.
There's one statistic I haven't heard: how many of the surveyors got killed or wounded? Going into strange neighborhoods and knocking on stranger's doors is much, much more dangerous than living in those neighborhoods. If the violent death rate in Iraq was really as high as this study claims, I rather doubt that they could have done the study properly without taking casualties.
Posted by: markm on January 18, 2007 7:39 PMJane: In a similar vein, his response about the UNDP survey seems frankly bizarre. If he's really doing his interviews that fast, then "focussing on deaths" is irrelevant, because [the interviewers] can't really be asking more, or better, questions than the UNDP did.
How long would be typical for a survey? Is 12 or 15 minutes really that short? (The press release for the UNDP survey: Compared to many surveys, the questionnaires were quite long, with a median interviewing time of 83 minutes.) Thinking of the kind of phone survey calls I get, 10 or 15 minutes doesn't seem so short.
Here's the questionnaires used in the UNDP survey. The household questionnaire is 60 pages; the questions regarding deaths are all on page 48. I would venture that the Lancet interviewer would have had considerably more time to ask about deaths than the UNDP interviewers (one or two minutes?).
He then says that if you go back and ask those questions again, without asking other questions, you get wildly different responses.
He was referring to the UNDP survey, not the Lancet survey.
Posted by: Russil Wvong on January 18, 2007 7:51 PMmarkm: I'd think Arabs, like nearly anyone else, would find someone who asked them about their children, wrote a few things down on their form, and rushed off in 10 minutes or less rather impolite--
I'm not so sure. The first thing phone surveyors usually tell me is how much of my time they're going to take ("this will take five minutes or so"), because answering somebody's survey questions is less important than other stuff I have to do. So a 10- or 15-minute interview time doesn't seem so unreasonable to me.
There's one statistic I haven't heard: how many of the surveyors got killed or wounded?
It's in the article: none. This wasn't a large-scale census. There were two survey teams, each with four interviewers and a supervisor. They interviewed 47 clusters of households.
Posted by: Russil Wvong on January 18, 2007 7:56 PMJane: Again, obviously things are very fucked up in Iraq. And I've no doubt that the UNAMI number is understated. But understated by a *factor of ten or more*?
Again, it appears that the UNAMI number is just the number from the Iraqi ministry of health, plus a count of unidentified bodies from the Medico-Legal Institute in Baghdad. So it'd be pretty dependent on the number from the ministry of health. It doesn't seem so unlikely to me that it'd be inaccurate.
Posted by: Russil Wvong on January 18, 2007 8:00 PMBruce Rolston's assessment of the 2006 Lancet study.
Posted by: Russil Wvong on January 18, 2007 8:05 PMI appreciate Jane's explanation. I did read her originally as believing that Lancet supporters wanted hundreds of thousands to be dead. I am glad to hear that is not the case.
Tom
Posted by: Tom G. on January 18, 2007 9:13 PMThe problem with many Iraq doves was that we didn't predict how wayward the US would be - how bad it could get. We merely knew it was wrong, and why.
The lesson for us is that we need to LOSE our ideological blinkers. Too many of us thought that nd athe US was a functional democracy with enduring institutions that would correct with relatively untragic expedition for insane administrations.
The genius of the US right has been to subvert those institutions and facilitate the insanity - if not exhibit it, almost invariably at personal profit. No losers, them. At least they're not liberals though. Hold to that thought.
And Lancent schmancet. What's 100,000 arabs, either way?
Posted by: AlanDownunder on January 19, 2007 4:06 AMFrom Doctor Slack: assuming the Lancet study holds (as it has thus far)
What are you talking about? The study hasn't held at all. It is the extreme outlier of every survey done to date. Every other piece of data contradicts it.
A few comments:
On death certificates:
The certificates are issued but not tabulated. This is not a matter of either incompetence or fraud, but simply that statistical record keeping is never anyone's top priority, and even well-funded and stable national statistics operations get things wrong. That's why we have a census every ten years. Note as well that for most of the first year after the war, you could have been as conscientious as you like as an Iraqi doctor and your death certificates would still not have been collated because there was no government to send them to
On the time taken in interviews:
People seem to be forgetting that the most common interview would have consisted of two questions:
Q: Did anyone in your household die in the eighteen months before the war
A: No.
Q: Has anyone in your household died since the war
A: No.
Q: Thank you very much and off we go to the next interview.
Death is (still) the exception rather than the rule.
Germane to the general issue.
My actual, contemporaneous response.
In general, people who understand a bit about science tend to be rather cross when they see people who don't know what they're talking about start weighing in on scientific debates. The defenders of evolution aren't motivated by a desire for "victory" and nor were the detractors of Dr Andrew Wakefield.
Posted by: dsquared on January 19, 2007 11:09 AMWhat are you talking about? The study hasn't held at all.
... and the WMD's are really in Syria! And Saddam did so have ties with al-Qaeda!
Posted by: Doctor Slack on January 19, 2007 11:17 AMDoctor Slack,
I take it you concede the point, then.
Posted by: Yancey Ward on January 19, 2007 11:31 AMIn general, people who understand a bit about science tend to be rather cross when they see people who don't know what they're talking about start weighing in on scientific debates.
Indeed they do--but a retreat into personal insult, accusations of bad faith, and arguments from authority do nothing to burnish their supposedly scientific credentials.
And indeed if you believe that scientists can't be blinded by the need for "victory," especially in public policy debates, then I must suspect you of rejecting a fair amount of (scientific!) psychological research without adequate justification.
(I acknowlege that these sorts of failings are widely present across the political and scientific spectrum, but they are univerally the mark of weak arguments, not robust science)
Posted by: Rob Lyman on January 19, 2007 11:53 AMI understand that the ties between Saddam and al Qaeda are now well documented (although there are no ties between Saddam and 9/11 AND THE BUSH ADMINISTRATION NEVER SAID THERE WERE), but I didn't know that WMD's have been found in Syria.
Posted by: Rex on January 19, 2007 12:12 PMRex: I was making a nasty joke. The denialism about the Lancet study reminds me of other forms of denialism that persist in discrediting the pro-war-o-sphere. Basically, in order to reject the Lancet study as an "outlier" you have to reject all context and throw out the discipline of statistics to boot.
Posted by: Doctor Slack on January 19, 2007 12:51 PMDoctor Slack,
It is you that is in denial. If the Lancet study were the truth of the situation, then every other study (some also statistical studies) would have to be wrong. Also, every on-the-ground count of violent deaths would also be off by a factor of at least 5. Indeed, we are asked to accept the Lancet study without being able to study the data collected to verify if it is, in fact, valid data. It is a simple matter to take the data collected and have a check of its validity- for example, actually verifying if the correct households were interviewed at all. That the study's authors have thus far refused to release any of the data is enough reason to suspect its validity, even without having to point out how much of an outlier the conclusions are. There is something very wrong here, and no amount of hand-waving from the study's supporters can keep a large amount of the suspicion from falling on the study itself. The authors need to release the data in its entirety for verification, otherwise, the obvious conclusion must be that it was flawed.
Posted by: Yancey Ward on January 19, 2007 1:36 PMIf the Lancet study were the truth of the situation, then every other study (some also statistical studies) would have to be wrong.
There are several links in this very thread explaining very patiently why that isn't so. None of the Lancet denialists has ever managed any convincing refutation of those explanations that I've seen -- it's usually mostly in the line of "that number looks too big! Hey, why are you laughing?" or "Iraq is not [insert name of African hellhole]" or "the Lancet researchers haven't provided [insert something they have in fact provided]."
Posted by: Doctor Slack on January 19, 2007 1:42 PMdsquared: People seem to be forgetting that the most common interview would have consisted of two questions:
Q: Did anyone in your household die in the eighteen months before the war
A: No.
Q: Has anyone in your household died since the war
A: No.
Q: Thank you very much and off we go to the next interview.
Death is (still) the exception rather than the rule.
Good point. There were 1840 households surveyed (total population 12,801); there were a total of 629 deaths. So at least two thirds of the households would have had no deaths to report.
For households with no deaths, the interviews would have been a bit more elaborate, but not by much:
The survey purpose was explained to the head of household or spouse, and oral consent was obtained. Participants were assured that no unique identifiers would be gathered. No incentives were provided. The survey listed current household members by sex, and asked who had lived in this household on January 1, 2002. The interviewers then asked about births, deaths, and in-migration and out-migration, and confirmed that the reported inflow and exit of residents explained the differences in composition between the start and end of the recall period.
Posted by: Russil Wvong on January 19, 2007 1:59 PMDoctor Slack,
A simple question. Why haven't the study's authors released the full data set from which they draw their conclusions?
Every piece of evidence suggests that the study is, in fact, wrong, except for the data the study is relying on for its conclusions. Any scientist in this situation would want to examine the underlying data for mistakes. What is so amusing to me is that the supporters of study's conclusions seem so disinterested in such an examination. Why is that? At least some of the study's detractors are looking for data that explains how many violent deaths could have been missed, and, so far, no explanation has been found.
Posted by: Yancey Ward on January 19, 2007 2:07 PMYancey Ward: Every piece of evidence suggests that the study is, in fact, wrong--
Again, see Bruce Rolston's comments.
If one had had to do a thumbnail estimate before the Lancet study came out, it probably would have gone like this. The highly reputable Carl Conetta study on total Iraqi fatalities, combat and non-combat in April-May 2003 had an upper limit of 15,000 dead, so start there. Before they were ordered by the government to stop publishing figures, the Iraq health ministry was documenting 400 civilian deaths from coalition forces alone per month, or another 15,000 in this three-year period. The Brookings Institute said in the middle of last year that it estimated 27,000 insurgents had been killed or imprisoned in the first two years of the war: it is unlikely that the total number of non-foreign insurgents killed over the last three years is much less than 40,000. Finally, Iraq's own security forces and police have certainly incurred somewhere close to 10,000 fatal casualties to date (see Donald Rumsfeld's frequent remark that they have been incurring fatalities at "twice the rate" of coalition forces).
So, even without counting all the other kinds of violent death (Shia-on-Sunni, terrorist attack, etc.), there was no way that we were looking at less than 80,000 fatal casualties in Iraq in this period. Add, say, another entirely plausible 40,000 per year for the ongoing Shia-Sunni strife, terrorism, extrajudicial killings, murders, car accidents, and the like (Baghdad morgue admissions alone have never been much less than 1,000 per month, and have recently ramped up quite sharply), and you're already up to the 200,000 fatality number I've been talking about, and beginning to approach the 400,000 lower limit of the actual Lancet study.
At the very most, I would have said the Lancet study's result could only have ever been reasonably off by a factor of two or three, which isn't bad for epidemiological work.
Posted by: Russil Wvong on January 19, 2007 2:09 PMWhat is so amusing to me is that the supporters of study's conclusions seem so disinterested in such an examination.
I'm curious. Have the various other studies to which you've tried to call the Lancet an "outlier" released all of their raw data online?
Posted by: Doctor Slack on January 19, 2007 2:41 PMFrom the first study, there is a very suspicious discrepancy:
2 Non-Response – Interviewers skipped a lot of houses where no one answered. There is every reason to believe that the households skipped are different than the ones who answered. No attempt was made to characterize the non-responders.
3 Subject Bias – Households who answered the door were almost uniformly responders. This implies that they knew what the study was about ahead of time and had already assented when they opened the door. Therefore we have to assume that this was essentially a self-selected sample. It’s no better than walking through a mall interviewing only people who ask to be interviewed.
This does not necessarily imply that the study was done in bad faith, or even that it is wrong. However, I do believe, for the reasons that Jane indicates, that it is wrong. 655 people a day for 1,000 days would show up a lot more than it does. For one thing, no one would be going out to pick up the bodies any more and you could count them from satellites. Has anyone reported a sudden increase in the vulture population?