Asymmetrical Information

Regarding: "those of us who want the gun rights movement to win on the merits"

Actually, I think for many of us, and I'm expecting you are one of them, we'd want the merits to win. Many of us have adopted a gun rights stance because of the evidence it makes people safer -- we follow the truth, as best as we can find it, wherever it goes.

I'm sure we agree here. I'm probably just being picky about phrasing. Forgive me.

Speaking of following the truth wherever it leads, know of any refutations of Tim Lambert's assertions regarding the predictive nature of Lott's model, as detailed on John Quiggen's blog?:

http://johnquiggin.blogspot.com/2003_01_19_johnquiggin_archive.html#90215291

I don't have a source but in general I would dare say that economists would trust the empirical work of Mark Duggan or Ian Ayres over the empirical work of John Lott any day.

It is not just this former assistant prosecutor who has come forward, but academics like Professor David Mustard at the University of Georgia who worked to replace the lost data from the hard drive confirm that Lott lost the survey data. According to Julian, a former editor at the University of Chicago also confirms other important aspects of what happened with Lott's book.

Ironically, just after this one was knocked down, turns out he's engaging in some argumentative virtual crossdressing. Te he.

As someone who works for a cable based internet service provider--not in Pennsylvania, so this isn't definitive--I can tell you that they rarely use static IP addresses. Indeed, in this industry we tend to go out of our way to change your IP address. Also, many large-scale ISPs use proxy servers in some parts of the country, and if that's the case, then it's quite possible that there are thousands of users with the same IP address as Mary Rosh.

I'm not saying it's so. If Comcast doesn't use Proxy servers, then the odds of Rosh and Lott sharing an IP address that reliably are, well, not impossible but fairly low.

Jesus, stop it, guys. After all that Dr. Lott has gone through in his honorable work for the freedom of our society, only with the best intentions for mankind in, well, in the history of ... history itself, I dare say, I simply can´t understand why you guys keep picking at him.

Don´t you understand that this is all mere coincidence ? Nothing more than coincidences it is, believe it or not.

- Is it his fault if all the other surveys find other percentages ?

- Is it his fault that he had a computer crash but only discovered his pity when pressed for a source of his seemingly wrong figures ?

- Can he be blamed when some studies don´t relate to the question at hand but he keeps citing them anyway ?

- Is he responsible for someone named Mary Rosh defending him and using his IP no. ?

- Is it his fault that all his answers in his e-mails seem to elude the questions brought up by Lambert and others ?

- Is it his fault that we all can´t correctly count to thirteen, let alone fifteen as happens with that survey-table ?

Heck, no. So you should stop picking at him. After all, you also cash in the benefit of his work: a peaceful, loving society he brings to us But I understand you somehow. Altogether, the coincidences keep coming and coming and coming...

Just my 02 cents,

Chief Pedro

Lott's reputation as an econometrician is very high. My Go-To-Guy, U of Calgary's Chris Auld, had this to say about him (on usenet's sci.econ:

Message 19 in thread
From: Christopher Auld (auld@acs.ucalgary.ca)
Subject: Re: John Lott-
Newsgroups: sci.econ
Date: 2003-01-19 14:10:07 PST

Tim Lambert wrote:
>> From: Christopher Auld (auld@acs.ucalgary.ca)

>> This accusation is incorrect.

[Lambert:]

>You are mistaken. The graphs are labelled as Ehrlich says,

[Auld's response]

The accusation was that Lott deliberately misleads
the reader. The accusation is incorrect because
in context it is obvious what Lott is graphing, so
there is no intent to deceive.

[Lambert had written:]

>It is very obvious to you, but you are probably one of
>the few people that it is obvious to.

[Auld's response:]

Anyone familiar with multivariate statistical inference
would have no difficulty interpreting Lott's results.
Anyone who lacks that familiarity should probably not
be arguing about Lott's quantitative methods or results
in any case.

[Lambert:]

>I don't think Lott is being deliberately misleading with the
>graphs. In fact he has misled himself:

[Auld:]

I don't think so, and I don't think your quote proves
your point. Most of the results in the book were
previously published in prestigious journals, and Lott
is a capable econometrician. I don't have the book
handy, but if I recall correctly the key results were
based on dummies for years before and after the
introduction of laws. I don't think your comments on
the statistics are correct, but perhaps you could
present in more detail what you think the problem is,

None of this has anything to do with the recent
allegations that Lott simply fabricated data.

--
Chris Auld
Department of Economics
University of Calgary
auld@ucalgary.ca
----------------------------->>

Patrick, Chris Auld says that Lott is a "capable econometrician". That's quite a bit different than saying his reputation as an econometrician is very high.

Jonathan,

If you know Chris Auld, it's high praise for him to call someone "a capable econometrician". In the four or so years I've been reading his sci.econ posts, and occasionally corresponding with him, I've yet to see a clue as to his politics. He's a verrrry straight shooter.

Here's part of another response from him to Tim Lambert, on sci.econ:

>> Lott's models are perfectly standard, as is his interpretation.
Ignoring quadratic terms, the models in chapter 4 [of More Guns, Less Crime] take the form

y = Xb + ad + b(dt) + noise,

where d is a dummy indicating a policy change and t is a
time trend. If one generates a "sequence of random numbers"
with the property that nothing happens in response to the
policy, one will not "almost always [find] the curves show
that something happened at the junction." One will in fact
find, with reasonably large samples, that the coefficient on
the interaction term is statistically insignificant at a
rate equal to the size of the test, conventionally 5%. What
is more, most of Lott's models are less restrictive, treating
the impact of a policy change in a semiparametric manner by
using sets of dummies capturing time to/since the policy
change. Again, this is all very standard and uncontroversial.

From the orginal page:

Update 3: Dr. Lott has just done something I can only imagine must have been rather difficult, and written confirming the above; "MaRyRoSh" are the first letters of his sons' names. The account was set up for his children years ago, and kept around as a way to respond to points in online discussions, he said, without the time commitment posts under his real name might have required. Neither Lott nor Rosh, though, subscribed to the firearmsregprof list where this little tempest originated.

Ok -- wrap up. None of this really affects the core issue of the survey, which as far as any of us can tell at present, has still been confirmed by someone prof. Lindgren found to be credible. Voluminously attacking your critics under a nom de Net isn't academic misconduct, it's just kind of weird. But on that point, I'm hardly inclined to throw stones.

You win, Jason. You've embarassed him. Feel better?

"You embarassed him" carries the connotation that somehow I'm at fault here for something.

Gee, Jason, how else are we to interpret the "Te he" with which you closed your post?

Actually, I don't find the use of pseudonym emails for discussion in online forums at all weird. There are often times when people will want to participate in a discussion but if their actual identities were known then their identity would become the topic of conversation rather than the idea. Further, there are so many online stalkers that a pseudonym can deflect. I know of several cases where people do this in different forums such as usenet and even at least one BBS where I moderate ( and in fact moderate under a pseudonym ).

I think the Lott story goes a little beyond using a pseudonym to post on online forums, but I am inclined to agree with Jane that I think its piling on at this point to pick on him.

Gee, Jason, how else are we to interpret the "Te he" with which you closed your post?

Schadenfreude. Which isn't "wrong," as far as I know.

.....and as if you guys would think it was no biggie for Paul Krugman to post under aliases.

You can't possibly think that, Jason; you were on this site to read what I wrote about Bellesiles. I don't think there's any reason to laugh about people who are embarassed just because they disagree with me. I didn't like it when people danced on Bellesiles' grave, and I don't like it when people triumph over an opponent's embarassing moment when they haven't done anything wrong. If Paul Krugman embarassed himself -- in a way other than the current embarrassingly inflated way he talks about himself in the media these days -- I wouldn't laugh at him because I don't relish the humiliation of others.

An accusation of prospective hypocrisy is amusing. The issue of using pseudonyms for online discussions is a topic to which I've actually applied a lot of this thing called "thought". While I have a very low opinion of Paul Krugman, I'd say the same thing about him were it to come out that he participated in online discussions under a pseudonym. In fact, I would encourage him to do so.

Lott did more than 'post online under a pseudonym'. He has barefacedly denied doing so, even when nobody asked him. He has reviewed his own book at Amazon.com under the pseudonym (and found it a must buy), which is not only morally objectionable but also in violation of Amazon's stated policy, as if the ethical problems with that kind of activity had to be drilled into the head of a criminology expert.
For an academic with a good reputation, one has to wonder why he has never found tenure at any of the prestigious, or less prestigious, institutions where he has worked, and winds up captive house scholar for a partisan publicity mill, the kind of position his academic cronies must have belittled constantly throughout his academic career. Something is lacking, whether his competence, his private reputation, or something else; it's certainly not his prestige or public visibility. If folks at Yale perhaps did not find him up to snuff or even politically correct enough, certainly one would normally expect many a small or medium sized pond would be anxious to enhance its reputation by providing a home for such a large fish.

I would also encourage someone as famous as Paul Krugman to think twice before using his real name. Opting for a pseudonym in some situations makes perfect sense to me.

Yes, but as Mick said, this isn't a matter of using a pseudonym at all. It's the deliberate use of pseudonym to promote his own materials- to say that his college class is excellent, to promote his book, to even talk about his facial scars that's downright creepy. Rather than the Bellesiles affair, I see this as closer to the David Manning scandal, where a studio used fake reviewers to pump up a Rob Schneider movie.

I wouldn't laugh at him because I don't relish the humiliation of others.

If you say so.

Are you sure that John Lott is guilty of promoting his own book on the Amazon.com community board? Indeed, if true, such behavior is immoral and outrageous. Is there any convincing proof?

David,

Here's some Pretty Good Proof:

5 out of 5 stars SAVE YOUR LIFE, READ THIS BOOK -- GREAT BUY!!!!, August 18, 1999
Reviewer: maryrosh (see more about me) from Philadelphia
"If you want to learn about what can stop crime or if you want to learn about many of the myths involving crime that endanger people's lives, this is the book to get. It was very interesting reading and Lott writes very well. He explains things in an understandable commonsense way. I have loaned out my copy a dozen times and while it may have taken some effort to get people started on the book, once they read it no one was disappointed."

See http://www.amazon.com/exec/obidos/tg/stores/detail/-/books/0226493644/customer-reviews/9/ref=cm_rev_next/104-7814677-3291129?show=-submittime

I think that Jason was right to point out who was really behind "Mary's" post and what it may suggest about Lott's character, but I agree with Jane that there's no need to be malicious about doing so. Lott will suffer more than enough humiliation if it turns out that his work is fraudulent or that he used dishonest means to promote it.

Jane,

I'm going to ask politely if you can read CalPundit's post, Kieran Healy's post, and my post, and still feel comfortable with the conclusion that John Lott has been vindicated.

In order for John Lott to be telling the truth, you have to believe that he conducted a survey of 2424 people

- without notifying the Human Subjects committee
- paying thousands and thousands of dollars out of his own pocket (of which he has no record) all for one line in his book
- with callers making thousands of long-distance calls out of their own rooms (for whom he has no names)
- without creating a single piece of paper that relates to the survey
- even though he's changed his story about where the 98% figure came from
- even though this survey couldn't possibly have been completed by the time his book came out
- even though a perfectly sampled, perfectly weighted study would have such a small relevant sample that he'd have a 20% margin of error
- even though no other survey has found results anywhere close to his numbers

You're willing to believe all of that because one pro-gun activist has stepped forward and said that he was interviewed?

For the record, I've never said a kind word about Bellesiles, and I'm pretty agnostic about guns. I lose my top, however, when I see intelligent people excusing or dismissing obvious academic fraud because it's useful for their cause. If this is the way we're going to evaluate evidence (useful to my side= correct), what's the point of performing studies? We should just watch Crossfire and see who gets the loudest cheers.

Aaaaand it turns out the "guy who came forward about the survey" is a gun rights guy, and not a particularly hands-off kind: he (hilariously!) managed to get the legal rights to the names of some MN gun control groups when they accidentally let them expire. In other words, not even remotely a reliable source to a disinterested third party.

For reference: the only gun control I want is to see handguns banned in favor of rifles. That's it. I threw away Bellesiles's book when it turned out (and Josh Marshall confirmed) he was full of it.

I couldn't believe conservatives took Lott seriously after reading Lambert's criticisms of his methodology, mind you, but now? After it turns out he almost certainly made up that survey, and has been using an internet sock puppet to review his own books?

What on earth more do you need?

“David,

Here's some Pretty Good Proof:

5 out of 5 stars SAVE YOUR LIFE, READ THIS BOOK -- GREAT BUY!!!!, August 18, 1999
Reviewer: maryrosh (see more about me) from Philadelphia
"If you want to learn about what can stop crime or if you want to learn about many of the myths involving crime that endanger people's lives, “

What in heaven’s name are you talking about? Where is the evidence that Mary Rosh is John Lott? Trust me on this, Amazon.com would have removed the review if they had evidence that Lott is the actual author.

Lott has written to Julian Sanchez and Kevin Drum admitting that he was Mary Rosh. Here's the text of the email:

From: MaRyRoSh@aol.com
To: CalPundit@cox.net
Sent: Wednesday, January 22, 2003 5:39 AM
Subject: Sorry

You are correct. The MaRyRoSh pen name account was created years ago for an account for my children, using the first two letters of the names of my four sons. (They later got their own accounts but this one was never erased.) I shouldn’t have used it, but I didn’t want to get directly involved with my real name because I could not commit large blocks of time to discussions. (However, I never subscribed to the firearmsregprof posting hosted by Volokh.)

I based my assessment on Lindgren, who I find credible. I haven't talked to the guy, and Lindgren has, so I'm going to have to take his word for it. Now, I don't say that this is definitive proof -- as I said, I would like to see another person come forward. But Lambert, who is one of Lott's most ardent critics, is also convinced. And as Lott said, there are also tax returns showing out-of-pocket research expenses, tenative contemporary confirmations of events like the loss, etc. I could still be convinced the other way, but for now I'll trust Lambert and Lindgren.

None of which goes to any question of methodology -- the question is falsification of data, which is rather more serious than arguments about methodology, however germane. (And to be fair to Lott, I'm astounded at how neatly the conviction that Lambert has completely disproven his thesis dovetails with the believer's position on gun control -- particularly among people who apparently have no knowlege of statistics or research methodology upon which they might base an objective conclusion.) None of this supports or vindicates Lott's work -- only questions of fraud. I'm more than happy to see continuing investigation of the matter, however. But unless you can prove collusion, the burden of proof is on his accusers to prove a negative, which is an uphill charge -- the only thing that hung Bellesiles was the truly massive scale of the fraud. Until then, I thought the standard was supposed to be "innocent until proven guilty"?

Having an internet sock puppet strikes me as a trivial, if embarassing pecadillo, and I don't think it reflects well on those who pounded it. It's certainly valid to point it out -- but I find the glee as distasteful as I did among gun rights supporters when Bellesiles was fired. I don't like what I believe y'all once termed "the politics of personal destruction".

Oh boy, this really does look bad. John Lott has indeed severely damaged his credibility. I am now forced to suspect the worse regarding his gun study. This is, of course, the difference between people like myself and many Liberals. I refuse to mealy mouth while there are still Liberals making a half hearted attempt to defend Michael Bellesiles' nonsense.

In many respects John Lott's situation is comparable to "Founding Brothers" author, Joseph J. Ellis. The latter was caught being deceitful about his personal background---but so far nobody (that I'm aware of) has been able to cast aspersions upon Ellis' book. The jury is still out concerning Lott's work. Not so with Michael Bellesiles. The former Emory history professor has thoroughly destroyed his reputation.

And to be fair to Lott, I'm astounded at how neatly the conviction that Lambert has completely disproven his thesis dovetails with the believer's position on gun control -- particularly among people who apparently have no knowlege of statistics or research methodology upon which they might base an objective conclusion.

Not quite; BS in math here.

And to be fair to Lott, I'm astounded at how neatly the conviction that Lambert has completely disproven his thesis dovetails with the believer's position on gun control -- particularly among people who apparently have no knowlege of statistics or research methodology upon which they might base an objective conclusion.

Also not quite in my case -PhD in economics, 100+ journal articles, taught numerous courses in econometrics and methodology (even used my real name!)

I didn't say that everyone who thought Lott was wrong lacked mathematical background; only that

a) The belief seems to coincide with the believer's opinion on gun control far too neatly for me to say that the weight of opinion swings one way or another.

b) I have an astonishing number of people writing to tell me that Lambert et. al. have just completely disproven Lott who clearly have absolutely no way of judging whether that is true or not, because they don't actually understand the statistics.

My copy of More Guns, Less Crime is in storage, so I am officially agnostic until I can look at it.

Have we had a low-expectations bait-and-switch here? Apparently now, as long as it can be shown that he rely did do some kind of survey and wasn't lying completely about its very existence, then everything's OK. Because at this point the scientific value of the survey which he may indeed have actually done, and which he has vigorously and confidently broadcast everywhere, seems highly questionable to say the least, and entirely impossible to verify or check.

Yeah, I'm biased. I have a bigoted Yankee screed on the topic up on my site. I don't think that the "More Guns, Less Crime" holds up very well except when you cherry-pick your data.

Yes, there are folks who know something about statistics who weigh in on gun control

Interestingly enough, one often finds that they make seemingly reasonable demands of one side that they don't make of the other.

One interesting behavior that can be observed is the juxtaposition of statements like "My position is {whatever}" and "The other side's analysis of certain data has {problems}".

To me, the deciding point is that one side wants to put the other side in jail. To justify that, I think that there has to be extremely strong supporting evidence, and it's pretty clear that such evidence doesn't exist.

My copy of More Guns, Less Crime is in storage, so I am officially agnostic until I can look at it.

Lott's main claims in MGLC aren't the issue here. There are three simple sources of controversy.

1. Lott repeatedly made claims on the basis of data he knew he didn't have. That's wrong, period. In this respect, it doesn't matter if Lott did the survey or not: if your data and all records of it are lost, you can't make claims based on it. It's just basic scientific practice. I would have gotten my Ph.D a lot faster if my advisers had been forced to just take my word about the results I was showing them.

2. Assume Lott really did do his survey as he described. In that case, the data were collected in an appallingly shoddy fashion. The litany of problems with the sample method he claims to have followed and the analysis he says he did means we can't have any confidence the estimates he produced and repeatedly cited.

3. However, Lott's story about why there are no records of any kind of his alleged survey is not credible, for reasons already exhaustively documented by his critics. To begin with, he only mentioned the existence of the survey after the 'national surveys' he cited as sources failed to back him up. It goes downhill from there, into a frankly laughable tale of obliterated data, no backups, no paper records, no codebooks, no survey instrument, no IRB approval, cash payments to unknown RAs making thousands of calls from their home phones, using mysterious marketing CDs of unknown origin to generate the numbers. It stinks to high heaven.

For these reasons, many scholars, including me, do not think Lott is credible. People who want to believe his overall argument would do well not to get sucked into the pit he seems to have dug for himself over this issue.

None of this has anything to do with the main arguments and data analyses presented in MGLC. If you want you want an extensive reanalysis of the central claims of MGLC, go read the recent Ayers and Donovan piece in the Stanford Law Review (text, figures).

The attempt to paint Ayers and Donovan as dispassonate and reliable observers would be somewhat more compelling if A&D weren't so prone to using unsubstantiated smears as "evidence".

Take the section on the intimidation that might be due to CCWs. One of the highlights is A&D's assumption that a "keep honking, I'm reloading" bumpersticker is intimidating. If A&D actually believe that bumperstickers work that way, I'm sure that A&D have been turning in women whose cars sport a "dead husband in trunk" bumpersticker.

And, just to show that their hearts are in the right place, A&D assume that such bumperstickers are somehow associated with NRA membership.

I’d like to make 2 points:

1. John Lott implies that his 1997 tax returns support his claims of having done the 1997 survey. They do not. I have not opened his main return (since no one has suggested that it has any relevant information). I did look at Lott’s 1997 Business expense schedule and there are no reimbursements for phone calls listed. Some commentators have also implied that the return shows payments of wages to research assts, which it does not, nor should it for the 1997 survey because Lott says that they were unpaid volunteers. Of course, Lott might have forgotten about the supposed phone research charges when he filed his return, but there are none listed, which tends to undercut his claim to have done the 1997 survey rather than support it. The return does support Lott’s contention that he spends large amounts of his own money on business travel, computer expenses, journal subscriptions, etc., so it is not hard to believe that he would be willing to fund a large project himself. But there is no evidence in the business deduction schedule on his 1997 return that he did the 1997 survey unless the expenses are mislabeled.

2. I think people discussing the Lott case are starting to cross the line into “hypermorality” in two respects that border sometimes on the hypocritical:

a. Some ANONYMOUS posters have criticized Lott for posting under a pseudonym and I think that an anonymous blogger or two might have made fun of his use of that pseudonym. The use of pseudonyms is common on the internet even in some fora in which they violate the rules. The criticism would seem to be unfair unless it is focused more on what Lott said than on the fact of his posting under an assumed persona. Most, but not all, critics of Lott have of course recognized this distinction, but I don’t think it has been emphasized enough.

b. I find the discussion of IRB approval more troubling and mostly unfair to Lott. If Tim Lambert (who has handled this IRB issue correctly, I believe) were at an American university rather than at an Australian one and in a scholarly article he referred to what he found out about Lott by contacting people, he would violate IRB rules. If I were to mention what I discovered in calling people in this affair in a scholarly article or speech, I would violate IRB rules. Or imagine that you are using a database of data collected by someone else. If you call the person who collected the data to ask questions about them, you certainly need to get permission or waiver from IRB before making the call.

The rules are clear: if it’s research for scholarship to be published or presented at a scholarly meeting, you must get PRIOR IRB approval to make the phone calls. This can take months. There is no de minimus exception. The rules, however, do not apply to journalism or class work not leading to publication or presentation at scholarly meetings. As a Northwestern colleague of mine admitted yesterday, that means that he commits IRB violations every year and mostly every month.

IRB violations are like speeding—almost everybody does it whether they realize it or not. And IRB violations are less dangerous than speeding. Departmental cultures vary a lot on how scrupulous people are on getting IRB approval. I’m almost certain that a survey was done by a chaired professor at the law school at Chicago in about 1998 with no IRB approval. So Lott not getting prior IRB approval was not unusual at U. Chicago law school or at most other law schools. The Chicago Sociology Dept. has been (incorrectly) telling its graduate students for years that you don’t need IRB approval for course work, a position that I had them correct a few months ago since it can lead people to think incorrectly that they don’t need approval for course work they intend to publish.

Last, in one expression of the culture at the U. of Chicago Law School, one professor there considers the IRB system (which is federally mandated) a violation of academic freedom and unconstitutional. It is overtly a “prior restraint” on speech; its only punishment is “censorship” and suppression of speech (the sanction is not being able to publish). It would be very hard to claim that the IRB system was narrowly tailored to a legitimate interest, when making one unauthorized phone call is explicitly listed as a violation of IRB rules. I am not predicting what a court would do (and I expect that IRB systems have probably survived constitutional challenges), but I would hope that people would focus on issues that at least violate the actual norms of the school where Lott was a fellow at the time. Saying that Lott should have gone through IRB is like your mother telling you not to speed when driving. It would be awfully hypocritical to criticize Lott for behavior that is so common among academics.

James Lindgren
Northwestern University

I'm an pseudonymous poster, but I don't make pseudonymous posts reviewing my own books or telling people what a great teacher I am.

Criticisms regarding IRB were not of the failure to go through IRB per se; the lack of IRB approval was brought forward as one piece of evidence that the work wasn't done.

At this point it doesn't seem that Lott's level of misconduct approached Bellesiles', though I'm not an authority. I think that Lott's credibility as a researcher as been pretty badly damaged, though not specifically his ethics. There seem to be an incredible number of loose ends and loads of sloppiness.

The large number of supposedly-authoritative reports he made of his very weak research, and confusion about what research he was reporting and when it had been done also reduce confidence in his work as a resource for the general public.

As to his business tax schedule, John Lott just posted to a discussion list that amounts he paid research assts in 1997 were listed on his schedule C under "Legal and professional services." Of course, research assts should have been paid under "wages," which lists no deduction, not "Legal and professional services," which is for the professional services of firms and independent contractors like lawyers, book agents, and brokers.

But this is a moot point since he has made clear that he did not pay his volunteers on his 1997 survey.

Response to Zizka:

I first raised the IRB in my original letter to FireArmsRegProf on September 14, 2002 for precisely the reason you urge. It would be one more possible source of verification that the study was done. Perhaps some are still using that in the sense I did and that you defend, but that is not the impression I've been getting from reading the blogs linked at Tim Lambert's site.

As to the pseudonym point, I was only pointing out that the criticisms on this point should be focused on any abuses of the pseudonym, not just on having used it. Further, I said that "Most, but not all, critics of Lott have of course recognized this distinction."

I'm not sure that we really are that far apart.

James -

You say,

I find the discussion of IRB approval more troubling and mostly unfair to Lott.

I raised the IRB issue in this post, which got picked up by Kevin Drum and others. I said

One point came up which I haven't seen mentioned before: if Lott did the survey while at the University of Chicago, why didn't he go through their Institutional Review Board? Federal Law says you can't conduct any research involving human subjects without first obtaining IRB approval. Does the Chicago IRB have any record of Lott going through Human Subjects review? Has he given any reason why he didn't?

You're right that people often ignore IRB rules (though you can usually get pro-forma fast-track approval for surveys).

Again, I think the issue breaks into two parts. (1) Did Lott do the survey? You (James) look like the person most likely to give the best-informed answer to that question. But whether the answer is 'Yes' or 'No', there remains problem (2): Lott repeatedly made controversial claims based on data that he knew he didn't have to hand --- and knew that he had hardly any record of ever having. That can't be good practice. When the topic is as incendiary as gun control, it's bound to come back and hurt your credibility.

Andy,

Ayres and Donohue write a 55 page paper that uses a range of regressions and performs a wide variety of robustness checks to provide detailed explanations of the econometric problems associated with Lott's work as well as the exact circumstances under which his results either break down or become implausible.

You, however, ignore all that mountain of evidence and conclude that Ayres and Donahue were "not dispassionate observers" who were using "unsubstantiated smears" as evidence based on a single footnote on page 45 that refers to a DIFFERENT paper by David Mustard??? Amazing. I guess if you don't like the results and can't refute them you may as well clutch onto straws.

Look, John Lott has done a hell of a lot of good work on this topic and I admire him for sharing his data and making his work accessible to others in a way that most economists would not do. I also think that most empirical economics papers will be found to be flawed if subject to the same rigorous going over as Lott's work has been, Senor Barro's growth work being the most notable example that comes to mind. However, given how accomplished researchers like Ayres, Duggan and Donohue and how little their reputations depend on finding the right sign in the link between guns and crime as opposed to how much of Lott's reputation depends on being able to find the right sign in the link between guns and crime, I am amazed at those who continue to insist that Ayres, Duggan et al. are partisan hacks whose work does NOT discredit the results that they and Lott would like to see.

I didn't ignore A&D's analysis at all.

However, I get to start somewhere, so I decided to point out that they decided to accompany it with some unsubstantiated smears.

I note that if the supposed problem is actually real, there are completely adequate solutions. The fact that A&D don't bother is "interesting".

BTW - The "police don't have to check, they can just take people to jail" policy can be used for many different situations. Again, A&D weren't all that rigorous in their analysis.

I'm looking forward to A&D's reanalysis of some of the studies supporting gun control. Only partisan hacks use a different standard for different sides.

Kieran:

Fine. I understand your point and appreciate your repeating it.

I was getting the feeling that people were making more of it than warranted. If IRB rules are often ignored at Chicago, then one should view Lott avoiding IRB as a lost opportunity to prove that a survey was done, but one shouldn't view this avoidance as either a mark on his character or even as significant evidence that the study was not done. It's more of a missed opportunity.

I have an astonishing number of people writing to tell me that Lambert et. al. have just completely disproven Lott who clearly have absolutely no way of judging whether that is true or not, because they don't actually understand the statistics.

Point taken. How many emails do you get a day, by the way?

- without notifying the Human Subjects committee

This requirement is routinely ignored at U. of Chicago Law School...no big deal there.

- paying thousands and thousands of dollars out of his own pocket (of which he has no record) all for one line in his book

Well, personally I'd like to see if he could get copies of the checks from his bank, but he has released some tax returns which do list thousands of dollars in expenditures for such research. Granted it doesn't say it was for this phone survey.

- with callers making thousands of long-distance calls out of their own rooms (for whom he has no names)

I had a professor I used to talk to fairly often. Several years latter I stopped by and he couldn't remember my name. Not a big deal, IMO.

- without creating a single piece of paper that relates to the survey

Actually do we know if Lott collected any of the paper from the students? I remember reading something about the student e-mailing him the data.

- even though he's changed his story about where the 98% figure came from

Two possible explanations. 1. Lott misunderstood the Kleck survey which several others did. 2. Sloppy editing. (Or both)

- even though this survey couldn't possibly have been completed by the time his book came out

Really? You think Lindgren is lying then?

- even though a perfectly sampled, perfectly weighted study would have such a small relevant sample that he'd have a 20% margin of error

Has nothing to do with whether the study was done or not. This does point out that the result is likely wrong.

- even though no other survey has found results anywhere close to his numbers

Given the small sample this surprises you? And this has what to do with whether he did the survey or not.

Sorry Ted, your list isn't all that impressive. Granted, overall Lott has lost credibility here and probably deservedly so, but even Tim Lambert, a staunch Lott opponent, has gotten off the "He didn't do the survey" wagon.

Jason. . . astonishing was hyperbole, you understand.

I get about 50-100 emails a day in general, and have probably had 30 or so on this subject from people who didn't really seem to understand the evidence they were offering. I didn't count, and my hotmail box tends to overflow once a day, so I can't go back & do so now. . .

I hope you'll forgive me the lost data? Since I freely confess that my results are not statistically significant in the 95% range?

Geez, I was honestly asking. That's a lot of email. :D

Comments are Closed.