Lindgren on Lott

Jim Lindgren has two posts about John Lott. First, on Lott's lawsuit against Levitt he concludes:

I think that Freakonomics is misleading in its juxtaposition of different studies, a juxtaposition that might bring one to conclude that the reason that the main More Guns, Less Crime research was not usually replicated in other studies is that there was some chance that a study was never done. Yet, to the extent that Lott's lawsuit is based on a failure to replicate being per se defamatory, I would think he has a difficult chance of winning.

I don't see the juxtaposition as particularly misleading. A reader might form the opinion that Lott's More Guns, Less Crime research was not conducted honestly, but there is certainly evidence that this is true. One of the interesting findings in Ayres and Donohue's exhaustive reanalysis of Lott's data was that the results were much more fragile than Lott reported. There were lots of ways of doing the analysis; some found reductions, others did not. But if you read Lott's paper you don't find this: he reports lots of different ways of doing the analysis and they all show significant reductions in crime. This suggests that he just reported the results that supported his thesis.

Lindgren's second post is about Lott's mysterious survey. The only piece of evidence supporting Lott's claim that he conducted a survey in 1997 is David Gross's belief that he was surveyed by Lott in 1997. Because of differences between the questions Lott says he asked and Gross's recollections Lindgren thought Gross might have been surveyed in a Hemenway survey. Now, after looking at Hemenway's data Lindgren concludes:

Since neither the demographics nor the descriptive accounts matched Gross to any Minnesota respondent, it seems clear that Gross was not surveyed by Hemenway in 1996.

Interestingly, even in 2003 neither Tim Lambert nor David Hemenway thought it likely that Gross had been surveyed in the 1996 Harvard study. Since Gross's account of the questions asked fit Lott's claims about his 1997 study in some respects but not others, we are left with several possibilities. If Gross is not lying (and despite his strong pro-gun orientation and some minor changes in his story, I still think that it is more likely than not that Gross is basically telling the truth as he remembers it), then either Gross or Lott misremembered some of the questions asked or Gross was surveyed for yet another survey.

I do find one of the minor changes rather troubling. Gross wrote:

when I said "a couple of years before" to John Lott, I meant "a couple of years" and not 1 3/4 years or 2 1/4 years. THAT, alone, excludes the 1996 Harvard study, because it WAS early 1997 when I answered the short survey, NOT 1996.

To me "a couple of years" would cover 1, 2 or 3 years. I don't think I've ever heard anyone using it to mean "exactly two years to the nearest month". And why would you want to tell Lott exactly (rather than approximately) when he had been surveyed?

In any case the story of Lott's missing survey has appeared in many major papers such as the Chicago Tribune and the New York Times as well as in Freakonomics, which has sold over a million copies. Supposedly eight or so Chicago students made many thousands of phone calls conducting a this survey for Lott. It's not something you would forget. But apparently none of them have heard of the affair and come forward. You'd think students who studied at Chicago would be particularly likely to read Levitt's book, since he is a professor there.