Asymmetrical Information

Legitimate questions on the Lancet survey

The good thing about the study is that people are having some very interesting discussions of the data. There are some good objections to it, and it makes interesting reading.

John Quiggin examines airstrikes, comparing them to the number of deaths by airstrike in the survey. His comments explode.

• One commenter says that Mr Quiggins' estimate of the deadliness of airstrikes is way overblown:

  In the recent Israel-Lebanon War, “The IAF flew some 15,500 sorties, including some 10,000 fighter sorties, and attacked a total of around 7,000 targets.” Since Israel killed about 1,000 people by all means, I conclude that on average it takes 10 air attacks to kill 1 person, and that John Quiggin’s figures are inflated by a factor of 100.

  On the other hand, if the strikes are happening at night, people won't really know whether they got hit by an insurgent mortar or a US plane. Still, it does give pause. Can any of my many militarily knowlegeable people comment?

• Others point out that while the numbers of deaths in the two studies (Lancet I and Lancet II) agree, the causes of death do not; the second study says that there's almost no excess mortality from non-violent causes. This is definitely disturbing. It's hard to confuse a stroke with getting blown up. Many of the commenters on Crooked Timber purport to believe that this does not matter. But the authors are claiming corroboration of Lancet I via Lancet II, when in fact the samples say something very, very different. Think of it as a survey of "excess violent deaths" in Iraq right after the invasion: Lancet I discovered approximately 60,000, while Lancet II discovered double that number, and no excess non-violent deaths.

  This is not agreement. Deaths from various causes like heart attack and stroke may vary randomly, because old people who die tend to have multiple problems; if one doesn't kill you, the other will. Also, the cause of death isn't always clear--did you die of the cancer, or the stroke it caused?--so there will be random variance just because doctors put deaths in different baskets. This is not true of "deaths by heart attack" and "deaths by gunshot", which strike different populations and thus are at best, weakly correlated, less so over longer periods of time. One expects random variance between, say, mortar and rpg deaths, but if a survey can't distinguish between "violently killed" and "natural causes", you've got a problem. To me this suggests a couple of things: either people's memories are failing them more than even critics think, or there is a substantial difference in the two samples, which is worrying.

• The Iraqi researchers broke randomness. At first blush, this seems most likely to have surpressed the body count downwards, since they probably tried to go to less dangerous areas. On the other hand, if they *really* broke randomness--by, say, asking the locals who wanted to talk to them--it could have gone the other way, since presumably the households with high body counts would swarm to the researchers with their stories. This would also explain the sudden decline in excess mortality from non-violent causes, since people are less likely to blame Grandpa's heart attack on the Americans.
• Interesting tidbit: one commenter alleges that if they hadn't thrown out the Fallujah sample from Lancet I, the first study would have shown something like 40% of the population of Fallujah dying in the attack. The problem with throwing out obviously erroneous outliers like this is, of course, that they may be pointing to a problem with your sample. They also may not, mind you; this is not some dispositive refutation, just an interesting item.
• At least some people report that Roberts et. al. will not release raw data, even scrubbed of identifying information. The same people defending the study (rightly) found this sort of behaviour very suspicious when John Lott did it; that's certainly why I no longer trust anything the man says. But if there are people who have succeeded in getting the raw data, please tell me. Also let me know if it truly is standard practice in public health never to share your deidentified data with anyone except collaborators. If that is true, can you explain why? That seems totally unscientific to me; no one would ever be able to replicate anyone's results or check sampling problems.
• Let me preface this by saying that I am completely, 100%, not in any way implying that Roberts et al. made the data up. But a commenter in John Quiggins' post asks "how could anyone tell if they just made it up?" My sense is that it's actually pretty easy to tell things like that; researchers get sloppy and tend to repeat patterns that you can pick up. That's how Steven Levitt caught teachers cheating on standardized assessment tests of their charges in Chicago. This is just a hypothetical, mind you, not any sort of accusation at all. I have absolutely no reason to believe they are making anything up, and if any one cites me in support of the proposition that they are, I will sue them. Or at least be very, very angry.

  A more interesting question is: could you tell if the Iraqi surveyors made stuff up? I find it much easier to imagine a survey team deciding that discretion was the better part of valor and faking entries rather than risk their lives. Hey, back when I was into political action, I saw people make up signatures on petitions so they could go get high. Again, I have no reason to think that this happened; I'm just curious whether it would be easier or harder to pick out researchers fudging the data.

• A knowlegable commenter with extensive survey research experience says this:

  . . . I’m also very worried about the fieldwork itself. I believe the reported refusal rate was 0.8% (I can’t find this in the report itself, so feel free to correct me). This is simply not believable. I have never conducted a survey with anything like a refusal rate that low, and before anyone talks about cultural differences, there are many non-cultural reasons for people to refuse to participate. If my survey was in a war-zone, I would expect refusal rates to be higher than normal.

  . . . I understand that each of the clusters (40 interviews) were conducted in one day (again, I don’t have a direct cite). The paper itself is ambiguous as to how many interviewers were deployed at each cluster, and whether they worked in pairs or solo. Assuming it means 4 interviewers each interviewing 10 people, that’s a very impressive work-rate in the context of the difficulties. See Appendix B of the report where they report being held up for hours at roadblocks and the extreme suspicion of initial respondents, requiring lengthy explanations. Apparently these “lengthy explanations” migrated quickly from household to household.

  If the interviewers worked in pairs (as one would have assumed – one male and one female) then the interview rates are unbelievable.)

  . . . The report states that in 92% of when it was requested, a death certificate was produced. I also find this difficult to believe, although cultural practices might come into play.

  I don’t attribute improper motives to the John Hopkins team. As I understand they remained in Jordan and relied on their intrepid field team. I find it difficult to believe that field team did what they say they did.

  Now that it is pointed out, I also find the response/not home rates rather unbelievable. Admittedly, phone surveys are easier to opt out of, Iraqis probably have more stay-at-home Moms, and Iraqis almost certainly don't go out much these days. But if Iraq really is that violent, people will be fleeing, which means they should have knocked on more doors where no one answered. As one blogger asked:
  

  
  Don’t heads of households and their spouses in urban Iraq have jobs? Don't they go out to meet friends? Do they never visit relatives in other neighbourhoods or towns? Do they not engage in any activities outside their homes? Are they never in the middle of a family meal and don’t want to be interrupted by unknown visitors asking intrusive personal questions? Never out shopping for groceries or passing the time of day at a local coffee shop or dropping off the family car at the mechanic’s? Do they just stay around the house all day every day? In short, do those folks living in urban Iraq have any semblance of normal lives?

  I realise that armed conflict would impel most people to huddle in their homes behind locked doors (in which case they would be unlikely to open the door to strangers), but that possibility doesn’t enter into it because the locations selected for interview were altered if they appeared unsafe.

  And 92% produced death certificates? Maybe households in poorer countries keep that stuff in a special place forever, where they can instantly put their hands on it . . . but I certainly wouldn't want to bet that my family of pack rats could immediately produce a death certificate when asked for any of my dead relatives, much less all of them. And the average household had six people, yet the interviews supposedly took only 15 minutes. Have you ever sat around with a family of Arabs--or for that matter any sort of family--talking about recent family history?

  "Hussein moved to Basra in November 2003."

  "No, Ali, it was 2002. It was a month after Aunt Maryam died. He used her suitcases."

  "Aunt Maryam died in 2001. Remember? She choked on a date at Noor's wedding, and at the funeral Grandma forgot where she was and tried to do the wedding dance." (To interviewer) "My grandmother hasn't been the same since Uncle Samir was taken by Saddam's police."

  "So you're saying Hussein moved to Basra in 2001? That's ridiculous! He was still in school. I have his diploma right here. June 18th, 2002." (To interviewer, fondly) "His math teacher said he was the best student she ever had."

  "I'm not saying he moved in 2001, I'm saying he didn't go a month after Aunt Maryam died."

  "You don't have to shout, Karim." (To interviewer) "We're all so tense with the new curfew rule. Karim hasn't been to the cafe in four days. Anyway, it couldn't have been 2003, because he drove down with Cousin Mohammed, and Cousin Mohammed fled to Amman before the bombing started. His mother should die of shame, he's such a coward."

  "Don't start that again. He's been very nice about sending money."

  "He's been very nice about sending money because he's in Amman sitting in cafes instead of home taking care of his mother where he ought to be."

  Fifteen minutes my Aunt Fanny.

• Given the description of the data collection methods--places where the researchers can walk to at least 40 houses in one day--the clusters selected are almost certainly fairly urban. Since that's where the violence is concentrated, that would bias the sample; something like 1/4-1/3 of Iraqis live in rural areas
• A commenter at Marginal Revolution points out that if the violence really was as bad as the Lancet studies say, there should be massive refugee flows outward, but the UNHCR numbers do not show this.
• At Blogcritics, David Nalle makes what seems to me like a good point, though he's rather more pugnacious than I think is wonted:

  With a population of 27 million people in Iraq and 50 data clusters, they allocate one cluster per 540,000 people. That's fine. The problem is that they then distribute the data clusters based on the governorates, which are political divisions and do not have evenly distributed populations. This means that when a governorate's population is not evenly divisible by 540,000 they round up or down to determine how many clusters to locate there.

  This creates an instant problem because the areas where they round down the number of clusters will be underrepresented and the areas where they round up will be overrepresented in the final numbers. As it works out, the most overrepresented governorates are two of the most violent and most populous, Diyala (+28%) and Anbar (+36%) and the most underrepresented are some of the most peaceful, Wassit (-45%), Qadissiya (-41%) and Tameem (-37%). The pattern is similar but less dramatic with the other governorates, plus two entire governorates in the more peaceful regions failed to return results at all. Only about 3 of the total of 20 regions are at all accurately represented. This leads to a cumulative effect of more violent areas likely being overrepresented by a rough figure of at least 20%.

  The math is off, but the point is valid: very violent areas were probably oversampled. Since their deaths (calculated as deaths per thousand people per year) were 5 or more times higher than those in the low-violence areas, and probably around twice that in the medium-violence areas, that would tend to skew things.

Posted by Jane Galt at October 17, 2006 10:14 AM | TrackBack | Technorati inbound links"); ?>