Nate Silver has gone after me twice this election cycle: the first time because I did not think it was toss-up in late September and the second because I do not think the election is won by Clinton 17 days later, or 26 days before the election.
The first time Nate went after me was in the days prior to the first debate. Nate declared on September 26, 2016 “It’s a dead heat.” Then, accused me of being in denial, because, unlike his FiveThirtyEight prediction that bottomed out at 51.5 percent for Clinton that day, PredictWise bottomed out at 69 percent. This led to a robust back and forth where I pointed out a list of concerns about Nate Silver’s methodology, he: uses bad polls, makes unfounded adjustments to the polls, assumes massive errors in state-by-state polling aggregation, and then assumes massive correlated errors in creating a national estimate. In short, the level of his model has been off and it has been too volatile: it does not forecast the election, but is more like a trailing indicator of polling.
The second time Nate went after me (this time implicitly) was today when he attacked prediction markets as being in-sufficiently bullish for Clinton: “don’t get why prediction markets still have Trump at a ~15% chance to win.” Which, of course, is precisely where his forecast was when he wrote it. Nate has stopped linking to PredictWise’s prediction market-based forecast and now points directly to the Betfair‘s raw prediction market price, which is really convenient if you want to (1) ensure that no one can read the data (do you know what a Back of 1.18 means?) (2) make predictions markets look bad (3) get back at me for tearing down your model!
1) Forecasts v. Raw Data: the prediction market price is a piece of raw data, not a polished forecast. Nate knows this, because I know he is familiar with the paper I wrote comparing poll-based models and prediction market-based models. The paper shows that raw prediction market prices are not as accurate than poll-base forecasts, but that prediction market-based forecasts are more accurate than poll-based forecasts. He has cited this paper in his blog posts. Like raw polls, raw prediction market prices have some issues that need to be corrected. They can over-state the long-shot due to: transaction and opportunity costs. They may need to be normalized. They should certainly be aggregated. That is why PredictWise does not show the raw prediction market price, but de-biases it, normalizes it, and then aggregates it with the state-by-state forecasts. This attack by him would be no different than if I wrote polls are silly because the Rasmussen Poll from 10/10-10/12 shows Trump winning. He is cherry picking a raw data point, rather than a polished forecast.
2) Market Advantage: market-based predictions beat poll-based predictions in accuracy early in the cycle, when information breaks, and in coverage. First, early in the cycle there is more idiosyncratic data in the world, stuff that cannot be modeled. This includes the expectation that Trump would do poorly in the debate or have some sort of scandal. The markets do not know the details, but they can price in this expectation that would be impossible to model without historical data. Second, when the information does break, such as Friday’s release of the Access Hollywood tapes, markets could respond immediately, while polls take a few days. Third, markets can cover all 50 states easily, while it is not cost-effective for polling to cover all states every day.
3) Extreme Probabilities: as Trump falls to 10 percent or lower all forecasts have an identification issue, in that we are moving into a space of forecasts our models are not used to seeing, and thus are not well calibrated for. Prediction markets are especially prone to issues there, in that transaction costs alone prevent punters from moving prices much past $0.93-95 per $1.00. PredictWise does its best to interpret the prices as probabilities, but again, since elections do not normally reach this space, it is not ideal for any forecast to provide discretion between, for example, 3 and 4 percent.
Market-based predictions are having a great year. In the primary they were more accurate and covered more election than poll-based predictions from FiveThirtyEight. They appear stable and predictive so far this election cycle in that they do a great job in predicting where FiveThirtyEight will be in a 1-2 weeks out! You can see that in the chart above and from analysis of the data streams. And, from that data along with a review of the market’s forums, I can see the inverse is not true: markets ignore FiveThirtyEight as a meaningful signal of the election (above and beyond the raw polling it aggregates). That seems well-founded.