Super Crunchers Read Online Free Page A

cook myself in the data-mining café. Like Ashenfelter, I am the editor of a serious journal, the
Journal of Law, Economics, and Organization,
where I have to evaluate the quality of statistical papers all the time. I’m well placed to explore the rise of data-based decision-making because I have been both a participant and an observer. I know where the bodies are buried.
    Plan of Attack
    The next five chapters will detail the rise of Super Crunching across society. The first three chapters will introduce you to two fundamental statistical techniques—regressions and randomized trials—and show how the art of quantitative prediction is reshaping business and government. We’ll explore the debate over “evidence-based” medicine in Chapter 4. And Chapter 5 will look at hundreds of tests evaluating how data-based decision making fares in comparison with experience-and intuition-based decisions.
    The second part of the book will step back and assess the significance of this trend. We’ll explore why it’s happening now and whether we should be happy about it. Chapter 7 will look at who’s losing out—in terms of both status and discretion. And finally, Chapter 8 will look to the future. The rise of Super Crunching doesn’t mean the end of intuition or the unimportance of on-the-job experience. Rather, we are likely to see a new era where the best and the brightest are comfortable with both statistics and ideas.
    In the end, this book will not try to bury intuition or experiential expertise as norms of decision making, but will show how intuition and experience are evolving to interact with data-based decision making. In fact, there is a new breed of innovative Super Crunchers—people like Steve Levitt—who toggle between their intuitions and number crunching to see farther than either intuitivists or gearheads ever could before.

CHAPTER 1
    Who’s Doing Your Thinking for You?

    Recommendations make life a lot easier. Want to know what movie to rent? The traditional way was to ask a friend or to see whether reviewers gave it a thumbs-up.
    Nowadays people are looking for Internet guidance drawn from the behavior of the masses. Some of these “preference engines” are simple lists of what’s most popular. The
New York Times
lists the “most emailed articles.” iTunes lists the top downloaded songs. Del.icio.us lists the most popular Internet bookmarks. These simple filters often let surfers zero in on the greatest hits.
    Some recommendation software goes a step further and tries to tell you what people like you enjoyed. Amazon.com tells you that people who bought
The Da Vinci Code
also bought
Holy Blood, Holy Grail
. Netflix gives you recommendations that are contingent on the movies that you yourself have recommended in the past. This is truly “collaborative filtering,” because your ratings of movies help Netflix make better recommendations to others and their ratings help Netflix make better recommendations to you. The Internet is a perfect vehicle for this service because it’s really cheap for an Internet retailer to keep track of customer behavior and to automatically aggregate, analyze, and display this information for subsequent customers.
    Of course, these algorithms aren’t perfect. A bachelor buying a one-time gift for a baby could, for example, trigger the program into recommending more baby products in the future. Wal-Mart had to apologize when people who searched for
Martin Luther King
:
I Have a Dream
were told they might also appreciate a
Planet of the Apes
DVD collection. Amazon.com similarly offended some customers who searched for “abortion” and were asked “Did you mean adoption?” The adoption question was generated automatically simply because many past customers who searched for abortion had also searched for adoption.
    Still, on net, collaborative filters have been a huge boon for both consumers