Freakonomics: What Went Wrong?

[salvorhardin]

We and others have noted a discouraging tendency in the Freakonomics body of work to present speculative or even erroneous claims with an air of certainty...

In our analysis of the Freakonomics approach, we encountered a range of avoidable mistakes, from back-of-the-envelope analyses gone wrong to unexamined assumptions to an uncritical reliance on the work of Levitt’s friends and colleagues. This turns accessibility on its head: Readers must work to discern which conclusions are fully quantitative, which are somewhat data driven and which are purely speculative.

...

Predicting terrorists: In SuperFreakonomics, Levitt and Dubner introduce a British man, pseudonym Ian Horsley, who created an algorithm that used people’s banking activities to sniff out suspected terrorists. They rely on a napkin-simple computation to show the algorithm’s “great predictive power”:

Starting with a database of millions of bank customers, Horsley was able to generate a list of about 30 highly suspicious individuals. According to his rather conservative estimate, at least 5 of those 30 are almost certainly involved in terrorist activities. Five out of 30 isn’t perfect—the algorithm misses many terrorists and still falsely identified some innocents—but it sure beats 495 out of 500,495.

The straw man they employ—a hypothetical algorithm boasting 99-percent accuracy—would indeed, if it exists, wrongfully accuse half a million people out of the 50 million adults in the United Kingdom. So the conventional wisdom that 99-percent accuracy is sufficient for terrorist prediction is folly, as has been pointed out by others such as security expert Bruce Schneier.

But in the course of this absorbing narrative, readers may well miss the spot where Horsley’s algorithm also strikes out. The casual computation keeps under wraps the rate at which it fails at catching terrorists: With 500 terrorists at large (the authors’ supposition), the “great” algorithm finds only five of them. Levitt and Dubner acknowledge that “five out of 30 isn’t perfect,” but had they noticed the magnitude of false negatives generated by Horsley’s secret recipe, and the grave consequences of such errors, they might have stopped short of hailing his story. The maligned straw-man algorithm, by contrast, would have correctly identified 495 of 500 terrorists.
Full article: http://www.americanscientist.org/issues/id.14344,y.0,no.,content.true,page.3,css.print/issue.aspx

[Scuba]

1. I read Freakanomics and found it full of holes....

... any real economists willing to comment?

[salvorhardin]

2. I did too

Here's what I had to say about it right after I finished it: http://www.goodreads.com/review/edit/49174

Also, see John DiNardo's detailed review and critique: http://docs.google.com/viewer?url=http://www.noapparentmotive.org/papers/DiNardo_on_Freakonomics.pdf

[bananas]

3. Romm, Krugman, ...

http://thinkprogress.org/romm/2009/10/16/204815/science-error-superfreakonomics-why-stop-amazon-search/

[salvorhardin]

4. Yeah, the climate change part of Super Freakonomics was an absolute mess

The article I posted goes into that a little bit, but it caused a huge stir in the science blogosphere when it was published. I'm not willing to go quite so far as to say Levitt and Dubner were deliberately misrepresenting the science. I think Levitt was just far afield of his expertise and Dubner is just more interested in being contrarian and sensationalist than he is in good science writing. I'm amazed that their editor didn't send that chapter out for fact checking though when s/he saw it was counter to scientific consensus.

[Boston_Chemist]

8. Economists rarely utilize actual scientific methodology in their work.

It is a field that is remarkably crowded with pundits and public figures, each working a political angle of some sort. That alone ought to give anyone reading books on economics pause for thought.

Caveat emptor.

[Taverner]

11. Well there was this one economist who used scientific methodology...

Went by the name of "Karl"

[Lithos]

9. Your review on GoodReads did not come up

It only comes up to the book itself. (I'm a member of Goodreads as well)

[salvorhardin]

5. A brief followup from one of the authors of the article in the OP

We kept our article short because of space restrictions at American Scientist magazine. Now I want to do a follow-up with all the good stories that we had to cut.

P.S. Let me remind everyone once again that Freakonomics (the book and the blog) has some great stuff. Kaiser and I are only picking on Levitt & co. because we know they could do so much better.

P.P.S. Just to emphasize: our point that Freakonomics has mistakes is nothing new—see, for example, the articles and blogs by Felix Salmon, Ariel Rubenstein, John DiNardo, and Daniel Davies. The contribution of our new article is explore how it was that all these mistakes happened, to juxtapose the many strengths of the Freakonomics franchise (much of the work described in the first book but also a lot of what appears on their blog) with its failings. In some ways these contrasts are characteristic of social science research in general: a mix of careful assessment of assumptions (for example, in the use of instrumental variables analysis rather than simple comparisons to estimate causal effects) and casual storytelling.
Link: http://andrewgelman.com/2011/12/freakonomics-what-went-wrong

[Nevernose]

6. They're little better than numerologist bible-code hacks

"We don't actually have to understand anything, we merely have to understand the hidden mathematical formulae secretly controlling the universe!"

I remember at first being blown away, and then, upon reflection, finding it to be a tad ridiculous. Much like I felt when I first read Von Daniken's ancient alien stuff when I was a high school freshman. Of course, it's been a few years and a few beers since I read Freakonomics, so I'm probably not the best reviewer, but those are my memories.

[laconicsax]

7. I think that's a nice summary of economics.

[Nevernose]

10. "The science of explaining why the predictions you made yesterday didn't come true today"