Silver Linings and the Statistical Playbook

We asked historian of science Christopher J. Phillips, an expert on quantification in American public life, to reflect on the role of statistics—and Nate Silver—in the coverage of the 2012 election. He was kind enough to write us the following guest post; you can find out more about his work here.

The 2012 election was a "Moneyball Election" and Nate Silver its big winner. Or so proclaimed the New Yorker's Adam Gopnik.  He was certainly not alone. Deadspin's David Roher lamented the "braying idiots" detracting from Silver's well-deserved limelight; President Obama jokingly praised Silver for having "nailed" the prediction of this year's Thanksgiving Turkey; and Wired's Angela Watercutter perhaps gave the ultimate compliment by calling Silver a "Nerdy Chuck Norris." 

Silver, for anyone who has spent the last few years under a rock, is the creator of the (mostly) political blog FiveThirtyEight. Picked up by the "New York Times" just before the 2010 midterm elections, FiveThirtyEight has become one of the go-to sites for political junkies.

Source: https://pbs.twimg.com/media/A7Cbu_ZCQAE0m6o.jpg:large

An unlikely fate, to be sure, for an unknown consultant at the accounting firm KPMG a decade earlier. Silver tells it as an ersatz rags to riches story, a bored employee and mediocre online poker player who designed a model for evaluating baseball players and then took on the pundits—mainly because in both baseball and politics the majority of "experts" knew close to nothing. 

This account is too modest on the one hand—his evaluative system for baseball, PECOTA, became one of the central predictive tools of the respected Baseball Prospectus operation. On the other, it overstates his statistical creativity—his overwhelming contribution has been to introduce clearer measures of confidence to poll predictions. As he explained to new readers in 2010, his blog was "devoted to...rational analysis" and "prioritize[s] objective information over subjective information." Hardly a revolutionary statistical approach.

Notwithstanding these pedestrian goals, his message has certainly struck a chord. His 2012 book, The Signal and the Noise: Why So Many Predictions Fail—But Some Don't, was perfectly timed to capture the election-season hype and indeed is still on the "New York Times" bestseller list. Few books could possibly be blurbed by both Bill James (of "Moneyball" fame) and Peter Orszag (of perhaps lesser O.M.B. fame). Silver has, in effect, become a hero to thinking folks everywhere (particularly those with only passing statistical knowledge), who happily point to his columns as if to say to the unwashed masses, "I told you so."

But what does this ascendance of "rational" thinking represent? For one thing, it is not about the spread of statistical knowledge. Silver himself almost never fully explains the mathematics behind the models he uses. While he is footnote-happy in his book, he doesn't include any notes or appendices which would begin to teach others how to use the basic statistical concepts he deploys like regression to the mean, probability distributions, or the "nearest neighbor" algorithm. 

That's hardly criticism—each equation in a book is rumored to reduce readership, although to my knowledge no one's ever done the formal regression analysis—but it is telling that Silver is certainly not trying to get more people to understand the models themselves. 

Rather, he seems to be saying, "Trust me."

Source: http://www.newyorker.com/online/blogs/books/nate-silver.jpg

Like most in the futurology business, however, he is also saying, "Distrust others." Indeed, one lesson of his book, despite Silver's avowed optimism, is that predictive models are often worthless and almost always fail to live up to their promoters' claims. Even well-defined rule-bound games like chess and poker turn out to be hard to predict in practice, a fact Silver acknowledges by going Zen: "The closest approximation to a solution is to achieve a state of equanimity with the noise and the signal, recognizing that both are an irreducible part of our universe, and devote ourselves to appreciating each for what it is."

Such concessions are not uncommon. Like nearly all modelers, Silver readily admits that models require active tinkering. He happily combines quantitative and qualitative information in his political predictions, using the Cook Political Report to arbitrarily assign a code of +1 or 0 within his models, for example. And he emphasizes that Bayes's Theorem and conditional probability generally (the probability of certain data given a hypothesis is not usually the same as the probability of a hypothesis given the same data but the probabilities are mathematically related) suggest the importance of context and assumptions, although he perhaps overstates the theorem's importance.

Nevertheless, Silver doesn't really analyze the feedback loops of modeling that often interest historians of science. Following the language of Donald MacKenzie, models can act as a camera, giving a snapshot of a particular process, but can also act as an engine, driving changes in the very thing being modeled. 

Silver admits such loops occur in finance and fashion, and even in disease reporting, but does not address the possibility that all non-trivial models might work like this: more models mean more noise, more noise will require more adjustments. Models are, after all, irreducibly human creations. And even after years of computer models analyzing "Big Data," the list of triumphs is strikingly short.

Source: http://mitpress.mit.edu/covers/9780262633673.jpg

Silver's rapid rise may ultimately represent a fear of declining discourse more than a triumph of the nerds. After all, this is an era in which Rep. Paul Ryan is considered a budget finance guru for, apparently, basic arithmetic calculations. Silver's reputation has grown in no small part because he makes predictions in areas—sports, poker, politics—in which the loudest, wildest, crassest "expert" opinions normally take center stage. 

Maybe members of the "reality-based community" embraced Silver because they're tired of being told, "that's not the way the world really works" in the twenty-first century. 

On the other hand, Silver predicted a Patriots-Seahawks Super Bowl in 2013. Oh well.

—by Christopher J. Phillips, Harvard University