Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart

In this book, Ian Ayres argues that data crunching is becoming increasingly prevalent in all areas of life - retail, government, medicine, advertising, etc - and that statistical illiteracy will soon join other major modern illiteracies as a stumbling block to career success.

He gives copious examples of how data crunching is taking over in many industries. Walmart looks for buying patterns to determine where to rush their goods to and when. Advertisers test different ads in randomised small-scale tests to assess their efficacy. Coupons are becoming increasingly targeted to specific consumers.

Perhaps the most interesting parts of the book are where he discusses the application of data crunching in government and medicine. The US government now performs randomised trials for social and educational programmes to see which to roll out on a state or national level. In medicine, a computer program called Isabel points out diagnoses that a doctor might miss when given a list of the patient's symptoms.

So where is the role of humans in all this? Ayres notes that not everyone is very happy with the new order. Wine experts were scandalised when they were out-predicted by a simple regression involving a mere three climatic variables. Doctors are not pleased that their brainwork can be imitated by a computer. Many so-called experts may feel that their livelihoods are threatened by computers armed with data and sophisticated algorithms.

In fact, Ayres says that there is still room for human intuition - in looking for the data (it's not always obvious where you can find evidence for particularly tricky hypotheses), determining what features to look at, running and analysing the experiments. In fact statistical intuition may become very important in the future.

And there will always be areas that cannot be reduced to mere numbers. Sure, you could assess students on a bunch of criteria to determine who is admitted to your college. But what number are you trying to maximise? Their salary 20 years after leaving? That's not the whole point to human life. So you can't resort to an equation to admit students.

All in all, a thought-provoking book. If I have one complaint, it is that it doesn't include enough technical books in its very short recommended reading lists (it recommends reading books like _Freakonomics_). I'd like to see a suggested beginning textbook to learn data-crunching techniques. Maybe I should email the author! (read less)