October 17, 2006
From the desk of Jane Galt:Illegitimate arguments about the Lancet study
Sorry I've been away from my computer lately. Life, y'know.
Anyway, I've been catching up on the Iraqi death count commentary. There are a lot of arguments out there on both sides, many of them bad. I thought I'd highlight some of the problems, starting with my own side:
- The study doesn't agree with the Iraq Body Count This is not by itself convincing. The Iraq body count is, +/- 50% to allow for reporting errors and double counting, the bottom bound of the number of dead civilians. I, and others, have argued that the ratio of alleged dead in the Lancet study to the number of dead people counted by the IBC is too high. But the number of dead is, in my opinion, almost certainly substantially higher than the number being recorded by the IBC.
- There is something wrong with the statistical method There doesn't seem to be much problem with the analysis. If there is a problem (and obviously I think there is) it's in the data collection.
- The death rate for pre-war Iraq is lower than that of the US or other developed countries. The median age of Iraqis is very young; half the population is under the age of 20. Young people who do not have access to drugs and cars die at a low rate. Infants don't, but infant mortality numbers are undoubtedly at least slightly depressed by counting births we would count as live births as stillbirths.
- The sample is too small. It doesn't seem particularly small to me. It would be nice if we could interview everyone in Iraq, but not practical. For many of the constituent death rates, the samples are too small--it's unlikely that we could really extrapolate, say, cancer deaths from the Lancet figures. But for the big causes--violent deaths, non-violent deaths, total deaths--the sample seems perfectly adequate.
- It's entirely possible all these people are dying without death certificates, which is why there's no official record of them. The Lancet study alleges there were death certificates for 92% of the people it interviewed. So do other sources from Iraq, which say it's very hard to bury someone in a cemetary without a death certificate. In urban areas, people aren't just tossing Grandpa in the backyard and planting some ornamental hedges over the body.
- There is absolutely no passive count which is at all useful in checking this survey I just don't buy it. If 92% of responders produced death certificates, check the government figures. If the central government doesn't have them, the provincial government will, at least according to this Iraqi doctor:
And by the way, most cemeteries in Iraq would not accept a body without a death certificate, unless the bodies are buried in mass graves or backyards without reporting them to health authorities (look at this to understand why), which in this case the government would regard them as ‘missing.’ While working in hospitals and health centres in Iraq, it was sometimes my responsibility (when the late-night doctor was unavailable or, in some cases, sleeping) to oversee the checking in of corpses at the hospital and to issue a death certificate indicating the cause of the death. No certificate is issued without a body, and it is required that several copies are kept. IDs of dead people are shredded at the spot and their names are removed from their family’s food ration cards. The Ministry of Health should have access to certificates issued throughout the country over the last 3 years. And both the Defense and Interior ministries have their own counts. Now why isn’t any independent body looking into that information?Even if the provincial government doesn't, check the cemetaries. And the hospitals and doctor's offices. If you can't get a good population for the hospital or cemetary (unlikely; this is not London, where you can't swing a cat without hitting a hospital), you can check the ratio of current deaths to pre-war ones, which should display roughly the same pattern as the study.
- None of the people questioning the Lancet study have made a serious question about the study; they're just using "gut check incredulity." This betrays the watcher's ignorance of what a "serious question" is. When an active study implies unrealistic levels of error in passive measures, that is a serious question. When someone asks how you could get a good sample if the country really is that chaotic, this is a serious question. When someone says that the numbers disagree with a study performed by the UNDP using similar cluster analysis, which found less than 1/4 the number of deaths as the Lancet surveys, that is a serious question. When someone asks how reliable a non-random sample can be, that is a serious question. For more serious questions see this post
- The official Iraqi death counts are wildly off; this has been known for a long time. Not according to the Iraq Body Count, whose data show that the central government's data is pretty close to dead on.
- The same methodology is used to estimate deaths in Darfur and the Congo. And your point is . . . ? Most of the people criticising the study, including me, have never cited death figures for the Congo or Darfur, and as with Iraq, I don't think such things can ever be known with any reasonable degree of precision. The eyewitness reports are enough to tell me that something very bad is going on, and the data collection may be useful to indicate the various kinds of bad things that are going on, but in a country with poor recordkeeping where a lot of people have been fleeing violence for a long time, statistics give false precision. Moreover, Congo and Darfur are both much, much, much poorer than Iraq--Sudan has about half the GDP per capita of Iraq, and Congo has about 20% of the per capita income. That means that even before the civil wars screwed everything up, the institutional structures for counting deaths, etc. probably weren't there. This is less of a problem in Iraq, which should make its death-counting apparatus more effective now, leaving a smaller gap between formal and informal counts. Interestingly, relatively wealthy Bosnia (per capita GDP $6,800) is cited by the authors as the exception to the rule that official counts understate deaths.
- Why are critics paying more attention to this than to the Darfur figures? For the same reason the commenters are so emotionally invested in refuting them, when they probably wouldn't bother reading a similar study about Darfur: the outcome of Iraq is politically important in America and Europe. The outcome of Darfur (sadly) isn't.
- The same methodology is used in polling Polls aren't very accurate, even though pollsters have much, much, much, much, much, much better data on the people they are calling than the Iraqi surveyors. Their saving grace is that we do a lot of them, so that the errors tend to cancel out overt time. Polls also tend to produce the answers that the pollsters would like, which is one of the many reasons that one never puts too much of a weight on any one poll. Perhaps most importantly, polls don't just extrapolate upwards from raw data and gross demographics of age and sex; they massage their raw data considerably using historical data, of which they have a lot. They also have an accurate passive measure--the actual vote--against which to check their predictions frequently, and refine their method. Obviously, the Iraqi surveyors have none of this.
- ". . . given all the attention that the study has received—much of it hostile, and much of it from people with agendas—the authors just don’t want to mess with what would happen if they started sharing their data with strangers just now. I certainly can’t blame them for wanting to keep a few things close to their chests for the time being." The idea that scientists are somehow immune from sharing their data because . . . there are a lot of people interested in seeing their data, and some of those people want to challenge their conclusions, is ludicrous. That's how science works.
Great post.
I still don't understand, though, why you seem to be so confident that the Iraqi goverment (central or provincial) has good data. You might be right of course, but how do you know? I have no idea how well these systems are functioning. I tend to be skeptical of isolated quotes such as the one from the "Iraqi doctor," since I don't know anything about their motives or knowledge.
Jane wrote:
"The idea that scientists are somehow immune from sharing their data because . . . there are a lot of people interested in seeing their data, and some of those people want to challenge their conclusions, is ludicrous. That's how science works."
Yes, that IS how science works. Thank you! Now can you please explain that to the masses who subscribe to the catastrophic man-made gobal warming cult?
Posted by: Smoov on October 17, 2006 8:07 PMJane, as far as the study alleging that "there were death certificates for 92% of the people it interviewed," the short paper (page four) says, "Survey teams asked for death certificates in 545 (87%) reported deaths and these were present in 501 cases." So they got death certificates for about 80 percent of the deaths (there were 629 total). Just sayin.
Posted by: Dave on October 17, 2006 8:29 PMSmoov,
Regarding the warming-mongers, I guess we could call this the "Iraqi Hockey Stick of Death". There is a difference though: Mann et al refused to release their algorithm code while the Lancet study won't release their data.
Posted by: G. Hamid on October 17, 2006 10:05 PMSo they got death certificates for about 80 percent of the deaths (there were 629 total). Just sayin
I'm not sure where Jane got the 92% figure from, but remember that not all of the deaths were post-war -- if the 501 death certificates were concentrated among the more recent deaths (which is possible, since the longer ago a death happened the less likely people are to still have the paperwork handy) then the postwar deaths might be 92% documented even if overall deaths aren't.
That's one possibility, anyway.
Posted by: Dan on October 17, 2006 10:45 PMIt was 91-92% of those they asked for certificates, 80% of the whole sample.
Posted by: Jane Galt on October 18, 2006 12:34 AMA great article today in the Wall Street Journal, reproduced below to foster comment and discussion. This guy sounds like he knows what he is talking about AND he has the credentials:
655,000 War Dead?
By STEVEN E. MOORE
October 18, 2006; Page A20
After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don't get me wrong, there have been far too many deaths in Iraq by anyone's measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5% -- not 1200%.
The group -- associated with the Johns Hopkins Bloomberg School of Public Health -- employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in "clusters" within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.
However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points.
Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.'s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711 -- almost three times that of the Johns Hopkins team for 93% of the sample size.
What happens when you don't use enough cluster points in a survey? You get crazy results when compared to a known quantity, or a survey with more cluster points. There was a perfect example of this two years ago. The UNDP's survey, in April and May 2004, estimated between 18,000 and 29,000 Iraqi civilian deaths due to the war. This survey was conducted four months prior to another, earlier study by the Johns Hopkins team, which used 33 cluster points and estimated between 69,000 and 155,000 civilian deaths -- four to five times as high as the UNDP survey, which used 66 times the cluster points.
The 2004 survey by the Johns Hopkins group was itself methodologically suspect -- and the one they just published even more so.
Curious about the kind of people who would have the chutzpah to claim to a national audience that this kind of research was methodologically sound, I contacted Johns Hopkins University and was referred to Les Roberts, one of the primary authors of the study. Dr. Roberts defended his 47 cluster points, saying that this was standard. I'm not sure whose standards these are.
Appendix A of the Johns Hopkins survey, for example, cites several other studies of mortality in war zones, and uses the citations to validate the group's use of cluster sampling. One study is by the International Rescue Committee in the Democratic Republic of Congo, which used 750 cluster points. Harvard's School of Public Health, in a 1992 survey of Iraq, used 271 cluster points. Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million.
When I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored. Which led me to wonder what other sections of the survey should be ignored.
With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census.
Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.
And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it -- try using Google to find a survey that does not ask demographic questions.
Without demographic information to assure a representative sample, there is no way anyone can prove -- or disprove -- that the Johns Hopkins estimate of Iraqi civilian deaths is accurate.
Public-policy decisions based on this survey will impact millions of Iraqis and hundreds of thousands of Americans. It's important that voters and policy makers have accurate information. When the question matters this much, it is worth taking the time to get the answer right.
Mr. Moore, a political consultant with Gorton Moore International, trained Iraqi researchers for the International Republican Institute from 2003 to 2004 and conducted survey research for the Coalition Forces from 2005 to 2006.
The doctor's quote might indicate a reason that Grandpa might just be buried under the hedges--Grandpa gets removed from the family's food ration card when Grandpa dies. Presumably, by not presenting Grandpa to the morgue, the family keeps his food ration coming in--until he is declared missing instead.
Posted by: Dave on October 18, 2006 9:07 AMWhen I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored.
Wow ... just ... wow. You put your name on something and dismiss a part of it because, oh, just some dumb student wrote it, I don't have to take responsibility for defending that part.
I wouldn't expect that kind of behavior from someone at Johns Hopkins. Maybe the School can defend its name?
Posted by: Person on October 18, 2006 9:10 AMI'm still waiting for a statistics expert to tell me:
1) If cluster-sampling has been used for death counts in peaceful, advanced countries (like the US, or UK, or Japan) where the official records are likely to be pretty good
and
2) how close those counts came to the official counts.
Forgive my empiricism. But unless someone has tried this method for this purpose to make sure it works, the fact that it is mathematically impeccable and "widely accepted" doesn't mean anything.
As we used to say in the lab: in theory, theory is no different from practice. In practice, it usually is. (Or: what's the difference between theory and experiment? About 4 standard deviations)
Posted by: Rob Lyman on October 18, 2006 9:18 AMIt's entirely possible all these people are dying without death certificates, which is why there's no official record of them.
I haven't seen people making this argument -- as you say, right there in the study it says that 80% of those surveyed, 92% of those asked for death certificates, had death certificates. The argument that I've seen made is that death certificates are issued by local, decentralized authorities (doctors, hospitals, etc.), and then aren't successfully collected or compiled by any central authority that could accurately report the total number of certificates issued.
Posted by: LizardBreath on October 18, 2006 10:10 AMRob: good question. That's a great case of where you can compare a methodology to a case where you "know" the answer.
Here's another one: Has anyone taken any historical item well-documented to be over 500 years old, and given an un identifiable sample of it to an isolated radiometric dating team to administer a triple-blind test to see if their methods yield the right age?
Posted by: Person on October 18, 2006 10:23 AMJane writes: The sample is too small. It doesn't seem particularly small to me. It would be nice if we could interview everyone in Iraq, but not practical. For many of the constituent death rates, the samples are too small--it's unlikely that we could really extrapolate, say, cancer deaths from the Lancet figures. But for the big causes--violent deaths, non-violent deaths, total deaths--the sample seems perfectly adequate.
Do you mean just that the you think that the sample is adequate, or also that you think the number of clusters is adequate? They're not the same thing.
As pointed out above, the UNDP survey used 10x the number of clusters as did the Lancet I survey. And (as you note) UNDP came out with a figure about 1/4th the size of the Lancet I.
Don't you think that the size of the sample and/or the number of clusters may have affected the results in these two surveys? That seems to me to be the principal difference between the two.
The way I look at it is that UNDP showed that Lancet I overstated the deaths by about a factor of 4. My rule of thumb is that Lancet II probably overstates the number of deaths by a factor of 4 also.
Posted by: A.S. on October 18, 2006 11:05 AMWell, thought I had a point to make, but Smoov and Hamid got here first.
Posted by: MikeinAppalachia on October 18, 2006 11:40 AMJane also writes: The death rate for pre-war Iraq is lower than that of the US or other developed countries. The median age of Iraqis is very young; half the population is under the age of 20. Young people who do not have access to drugs and cars die at a low rate. Infants don't, but infant mortality numbers are undoubtedly at least slightly depressed by counting births we would count as live births as stillbirths.
Those are perhaps good speculative reasons why Iraq might have a lower crude death rate.
But the evidence is that Iraq in fact did have a higher death rate than the US pre-war. See, e.g., the World Health Organization, which estimated the crude death rate for Iraq in 2001 to be 9.0 per 1000. (And, no, the 2003 CIA factbook is not very good evidence to the contrary, given that the 2006 CIA factbook estimates the Iraq death rate as about 5 per 1,000.
Posted by: A.S. on October 18, 2006 12:05 PMHere's another one: Has anyone taken any historical item well-documented to be over 500 years old, and given an un identifiable sample of it to an isolated radiometric dating team to administer a triple-blind test to see if their methods yield the right age?
Well, that's off-topic, but the fact is that this setup has happened several times on both sides of your 500 year mark, and answers have varied widely in both cases.
Consequently, it is more desirable to try and establish ages of things by an isochronal method analysis of surrounding rocks that the object was buried in, although this approach is not freed from the risk of systematic errors.
Posted by: anony-mouse on October 18, 2006 1:01 PMMegan,
On the one hand you claim that the statistical methods were sound and then you claim the sample was not random. Typically sound statistics requires a random sample. I think the first claim is correct and the second is mostly wrong.
The medodology was designed to chose the location of the sample clusters randomly.
There was one instance, mentioned in the study, when the chosen location was judged too dangerous and a substitute location was chosen. This single instance was not shown to have been biased, and even if it had it would have been too small to invalidate the results of the study.
The over the top stat guy that you link to claims no rural areas were surveyed with almost zero evidence to support his claim.
However, this stats girl says this "It is certainly true that densely populated areas are more likely to be sampled – but only proportional to their population."
As to low number of clusters sampled. This will increase the variance of the estimate and widen the confidence intervals, but it should not bias the results. The confidence intervals were printed in the study as between 400k and 900k excess deaths.
They kill each other off at the official rate of about 50 per day, or 15,000 per year. If the study is correct, then the unofficial rate would be 500 per day. It should be easy to confirm that Iraqis secretly sequester about nine dead for each one reported.
Regarding the claim that the estimated death rate pre-war wss too low.
If the study methodology led to a low estimate pre-war, it also should have led to a low estimate post war. But the study was measuring the difference between the death rates so this shouldn't present that much of a problem.
As to whether the 5.5% for 2002-2003 was in line with other estimates at the time. We apparently have the CIA and U.S. Census estimaes coming in at around 5-6% for 2000 and the UN around 9-10%. But we don't know the methodology of these other estimates nor do we know the years for which they were calculated. They undoubtedly contain dated estimates, since there has been no on the ground surveys done until 2004. For example, I've read that the UN estimates were extrapolations from 1993 through 1999 surveys during the worse times of the sanction regime. By 2002 the death rates had decreased because of the Oil For Food program.
The US has lots of very, very, high resolution data for cemetaries in urban areas in Iraq. This is much higher resolution than you can get with Google Earth. The US government for some reason hasn't seen fit to share their photos with us. Perhaps they think we don't know that their satellites are better than Google Earth?
For those of you who haven't thought much about this, the back data on satellites is kept and prized because you want to know not only what is there, but what is there that's new.
The issues raised by IraqiBodyCount, an organization with actual motive to inflate mortality numbers (and I am not implying they do this), finds the Lancet study to be unrealistic by a pretty wide margin. For me, this really closes the issue- the Lancet study was flawed, for whatever reason.
Chew2,
I find 40 clusters for a country the size of Iraq to be way too low. I seriously doubt that Iraq is homogenous enough to get away with such a small number of clusters.
Posted by: Yancey Ward on October 18, 2006 2:45 PM'IDs of dead people are shredded at the spot and their names are removed from their family’s food ration cards. '
No wonder there's violence there. If they ration food, then politics is a matter of life and death.
As to the Lancet study, it's a joke. There's nothing tangible to support their numbers. No hundreds of thousands of wounded, no large amounts of damage to buildings and property, no refugee crisis. No cemeteries bulging with graves.
Posted by: Patrick R. Sullivan on October 18, 2006 2:50 PMWK
The US has lots of very, very, high resolution data for cemetaries in urban areas in Iraq.
And you know this, how?
easy to speculate that they do have recon photo's of many of the urban areas but are you saying that they were tasked to observe the cemetaries? Doubtful, but maybe such views would be incidental to other motives. I suppose that the failure to publish such indicates that there are such pictures and "they" do not want them revealed? Yeah, right.
"Here's another one: Has anyone taken any historical item well-documented to be over 500 years old, and given an un identifiable sample of it to an isolated radiometric dating team to administer a triple-blind test to see if their methods yield the right age?"
See for example here for some general ideas. However, some radiometric dating methods will be wildly inappropriate for historical artifacts.
Anony-mouse, I don't understand your suggestion - how will this help for historical objects?
Anyway, it's interesting that what sounds like a evolution-denial feeler and global warming-denial comments are also popping up in this thread.
Patrick, is that sarcasm? Nowadays I can't tell anymore.
Posted by: Dan S. on October 18, 2006 8:27 PMDan S., I usually don't let people get away with "this link will do my thinking for me", but I'll make an exception since it's short.
How does the content of that link answer my question? It says that the method is okay, but people misuse it. But if they get the wrong results when we know the right answer, how will it be reliable when we don't have such luxuries. It says the tree rings are its confirmation, but that's just circular reasoning. How do we know the tree ring correlation holds too? Did someone sit down for a thousand years and check off, "yep, tree rings accumlate at that rate consistently over a thousand years"?
Anyway, it's interesting that what sounds like a evolution-denial feeler and global warming-denial comments are also popping up in this thread.
Weak. The fact is, you cannot use the same scientific methods to validate global warming and evolution theories as you can to validate Newton's laws of motion. You cannot, for example, create a thousand copies of the earth, causally separate them, randomly vary industrial emissions in them, wait a hundred years, and run a regression. Any pretense that the models carry such validity is a lie, and would make the same error that the Lancet study does, which is to equate "best" with "good" or "valid".
Posted by: Person on October 18, 2006 9:49 PMI have a more basic question about Lancet's conclusions.
From Page 3:
Researchers interviewed 12,801 people in 1,849 households and recorded 629 deaths in the past 54 months. Of these, 82 (13%) occurred prior to the invasion, and 547 (87%) occurred after the invasion. Of the post-invasion deaths, the raw data says that 247 (46%) were non-violent (e.g., heart disease, cancer) and 300 (55%) were violent (e.g., gunfire, explosive device, car bomb).
Now, if the 629 deaths, extrapolated across all of Iraq, comes to an estimated 654,965, then the number of post invasion, violent deaths should be 654,965 x 87% x 55%, which is 313,400 (as opposed to the 601,027 that is mentioned in the oft-quoted report summary). Either I'm missing something big here, or something is drastically wrong.
Thoughts?
Posted by: Brian Greenberg on October 19, 2006 3:39 AMBrian: I think the Lancet is claiming that nonviolent deaths also increased as a result of the war. That's not unusual. Death by starvation and exposure is nonviolent, but many wars caused more such deaths than occurred in combat when the fighting drove people out of their homes and interrupted food shipments.(That hasn't been the case in Iraq, but "liberals" like Lancet's staff don't seem to like to point out things that are to the USA's credit.) War increases stress, which probably will bring on heart attacks and other illnesses sooner, plus making it difficult or impossible to bring the victims to a hospital in time. War can interrupt medical treatment. War can mean you get a choice between death by dehydration and drinking polluted water, with no assurance that you can get medical treatment if you get sick. War will almost certainly get your electricity turned off for quite a while; I don't know how many Iraqis had air conditioning in the first place, but if you blew up the power lines in, say, Phoenix in the summer and kept linesmen from repairing them for a couple of weeks, while roadblocks and shootouts on the roads also discouraged people from loading up the SUVs and heading for the mountains, you'd see a great increase in deaths from heat stress.
However, the deaths due to *us* bombing power plants, blocking the roads, and interrupting water supplies stopped three years ago. Whatever the continuing excess death rate (and I agree with JG that the Lancet's numbers strain belief), the primary culprits are not us, but the terrorists.
Posted by: markm on October 19, 2006 12:06 PMBrian,
You ask why, if 45% of the post invasion deaths were non-violent, aren't 45% of the excess deaths non-violent.
The reason is that those 240 non-violent deaths are what you would have expected if the pre-war non-violent death rate had continued in the post invasion period. They are not excess deaths. Whereas there was a huge increase in violent deaths.
BTW Tim Lambert explains why 50 clusters were enough, and Les Roberts explains that they did collect demographic info from their subjects. So much for the WSJ "critique".
Pre-war there were 2? violent deaths and about 80 non violent deaths for a 14 month period. Post war there were 240 non-violent deaths and 300 violent ones for a 40 month period. The post wr period was 2.86 times longer than prewar. You do the math.
Chew2,
I checked Lambert's site and did not find an explanation. What I did find is one of the authors saying that it was enough, but really, how do you know? It seems to me that, in order to be sure the numbers and locations of the clusters is statistically valid for extrapolation, there has to be some data to verify against. Exactly how did the researchers determine that these 47 clusters could be used to predict the excess deaths over the whole of Iraq with any confidence?
My only experience with sample clustering is with exit polls in political races, and the sample precincts used to produce such results (an actual cluster) are carefully selected because it is known from historical election data how those precincts behave relative to aggregate totals you are trying to predict (both state and national total vote). In 2004, the NEP used 1400 clusters, or one precinct for every 86,000 voters-or 28 per state- and the average sized state would have 5.8 million residents. And they use this sort of cluster coverage trying to predict a statistic that has an extensive historical background for design.
And even if I accept that the John Hopkins study's methodology is sound, groups like IBC, with no reason to obfuscate the matter, are stating in no uncertain terms that it is nearly impossible to make the results and the known evidence match up.
Posted by: Yancey Ward on October 19, 2006 5:33 PMAs, I think, the author of one line quoted above "". . . given all the attention that the study has received—much of it hostile, and much of it from people with agendas—the authors just don’t want to mess with what would happen if they started sharing their data with strangers just now. I certainly can’t blame them for wanting to keep a few things close to their chests for the time being."
let me say a bit to defend myself. As you yourself have noted, there are _lots_ of really, really bad argument floating around out there about this stuff, and much of it from semi- or innumerate people twisting stuff that they just don't have the background to say anything sensible about. I'm not saying that all the commentary is of that sort, but lots -- indeed, it seems to me, the vast majority of it is. So I do think it is entirely reasonable of the authors to want to keep their data out of _those_ hands altogether, _for now_. Does this mean that they shouldn't be willing to release their data to members of their profession who they can trust not to disseminate it widely? No, it does not mean that. Does it mean that they shouldn't down the line, when the partisan hubbub has died down a bit, make their data more generally available? No, it does not mean that either. I fully support the scientific norm that you take yourself to be defending in your reply to my earlier comment. But fruitful compliance with that norm does not require that one immediately subject one's data to the vagaries of the web. In particular, a commentator on the CT site that my comment appeared in was complaining that he had not been granted access to the data, in a tone that suggested that something might be amiss because he had not been granted that access. And I was suggesting there that _right now_ is obviously a bad time _for a person unknown to them_ to be asking them for it.
Yancy,
The fewer the clusters the more error range in the estimate. The study authors were satisfied with the error range of 400k to 900k excess deaths.
The WSJ guy, who doesn't appear to have much stats training, just claims you need more clusters but doesn't explain why. He was doing opinion surveys for the US in Iraq and needed a much higher level of accuracy so a higher number of clusters, but even he got by with only 75 clusters.
Check out this:
Did Wall Street Journal Find Fatal Flaw in Lancet Iraq Study?
October 18, 2006
Rebecca Goldin Ph.D. and Trevor Butterworth
http://www.stats.org/stories/did_wsj_flaw_iraq_oct18_06.htm
"On the face of it, this sounds like a fatal flaw. But unless the sample is actually biased, a smaller number of cluster points only has the effect of widening the confidence interval. Polls don't like large confidence intervals, but for the purposes of estimating large numbers of people, even the wide confidence interval of the Lancet study is informative."
The study made every effort to select the clusters randomly. Cluster sampling actually tends to under-estimte the probability of episodic and relatively rare events like violent death. It's more likely than not that a cluster will miss a violent event which may tend to be localized.
As to exit polls. They are trying to get a fine degree of accuracy (2- 3%?) and have to take into account many demographic factors race, sex, age, party affiliation, and location, so I'm not surpised they need to sample many more clusters to be representative, especially if they have to predict who will win each state.
Caveat: I have some stats training, but am not a polling expert by any means.
Posted by: chew2 on October 19, 2006 7:52 PMphilosopher,
Sorry, but in this case there really is no defense for not releasing the data widely. It is not like the data can be corrupted or altered by people who have agendas. The longer the authors refuse this, the more I suspect some serious error or outright fraud.
Chew2,
Yes, with fewer clusters the interval widens, but you and others seem to be implying that the midpoint does not change. I find this impossible to believe, and think it is simply untrue.
Posted by: Yancey Ward on October 20, 2006 9:33 AMSample size does not affect the sample midpoint. If this is not clear, you may want to look up the difference between precision and accuacy. Higher sample sizes make the result more precise, not more accurate. Also, the necessary sample size rarely has any relationship to the population size; a poll needs the same number of responses to estimate election results in a town, state, or country.
These are not really points of debate, so much as basic statistics.
Posted by: Zubon on October 20, 2006 2:06 PMZubon,
So if the numbers of clusters, randomly selected, were to be, oh lets say, 4, then the midpoint of death estimate would still have been 650,000, but the confidence interval would be, just for the sake of argument, 50,000 to 1,250,000?
Posted by: Yancey Ward on October 20, 2006 2:21 PM"It is not like the data can be corrupted or altered by people who have agendas." No, but what one does with that data can be fraudulent, or just plain loud & innumerate (as is the case with about 98% of the complaints against the study that have shown up here, or on CT, and elsewhere). It's just not part of general scientific practice to release one's data to the public at large, because the public at large tends not to know what the hell to do with data when it gets it. For example, sometimes people believe things that just don't make any sense if you actually know much about scientific practice; e.g., they say things like "the longer the authors refuse this, the more I suspect some serious error or outright fraud."
Posted by: philosopher on October 20, 2006 7:27 PMThere was a significant interview by pajamasmedia where they questioned the validity of the Lancet's arguments. Plenty of holes to be found, like the baseline mortality rate they used for Iraq, which is half that of modern, western countries.
http://pajamasmedia.com/2006/10/joisting_with_the_lancet_the_p.php
It seems more and more like a hack job where they tried to carry out a study with pre-determined conclusions, and used a variety of tricks to make the data and methodology fit their aims.
Posted by: The Wobbly Guy on October 22, 2006 2:39 PMSample size doesn't affect midpoint? So, you're saying we know the midpoint exactly?
You're right, in a way: we do know the midpoint of the sample exactly, but we know rather less well the actual midpoint of the total population.
Posted by: Slartibartfast on October 24, 2006 3:57 PM