WE'VE MOVED! Democratic Convention Watch is now at http://www.DemocraticConventionWatch.com
There's been a lot of discussion on the web about the accuracy of the various presidential forecast algorithms that are out there. Chris at Open Left has put forward a few theories:
1. "No Special Sauce Needed for Electoral Projections": Basically use only recent state polls, and nothing else. Sites that use any "special sauce", such as NBC or CNN, or even 538.com, won't be as accurate.
2. Among projections that are mostly polling based, there "A Thousand Ways to Reach the Same Conclusion". Everyone has basically come to the same result.
3. The big media site projections are a joke.
Well, we've been tracking many of the key indexes since June, and it seemed the right time to look at the numbers and see if any patterns emerged.
In the graph below, we've plotted the EV estimates since early June for the following projections: 538.com, Open Left, and a average of 3 "special sauce" estimates: NBC, CNN and Rasmussen. We've also included a graph for the Pollster.Com Sensitive Tracker. For this number, we've assumed each 1 point lead in the poll is worth a gain of 12.5 EVs, according to an analysis done by 538:
(See note at the bottom for how I convert each of the projections to one overall number).
What do we see? 538 is clearly the most volatile, and lags the national numbers by just a few days. This should not be surprising, as 538 uses national polls as part of their algorithm. (538 also showed a bit of a GOP bias during mid-summer). OpenLeft, which uses only state polls, is a bit less volatile, but tends to not catch up to trends as quickly. Note that this occurred more over the summer, when state polls were less frequent - the gap is narrower in the fall, when we're getting a flood of state polls. (Also, 538 updates every day - Open Left does not, which also contributes to the lag - this will not be an issue for the final projections). Finally, our "special sauce" projections are clearly lagging in their response to trends.
So let's look at Chris' theories: Number 3: "The big media site projections are a joke." Part of the theory of them being a joke is that they're purposely biased towards McCain, either for political reasons, or for reasons of just wanting to keep it close. I don't think either is true. I just think they're too wedded to historical trends to deal quickly with an election that, right now, just isn't close, and has a different map than 2004. Looking at our latest forecast summary, NBC still has MO as McCain-Lean, and, as Chris noted earlier this evening, states such as PA, NH and NJ are still being called Obama-Lean in spite of double-digit leads. They just react very slowly to current events. And that hurts their accuracy.
Theory 2, all the projections have the same result, is hard to really analyze when the election isn't close. There were a lot more noticeable differences when the election was about tied before and a bit after the GOP convention.
But the real question is, which projection is more accurate in predicting the final outcome? In some sense we won't know until election day, and, even then, a big Tossup state falling one way or another could affect the accuracy disproportionately. But I think there's one important point to make here. 538 takes the state and national polls, and uses the data to figure out where each state should be right now, even if there was not a recent state poll. Open Left only looks at completed state polls. But when we get to election day, Open Left's methodology will fail to pick up any late movement in states that won't have recent state polls. 538 will look at the latest poll changes and apply them to all states. We will only see this if the polls move a bit over the last weekend, but it seems to me that projecting movement in states even when there are no recent polls is likely to be more accurate.
Below is a graph will all 9 of our projections plotted together:
For each projection other than FiveThirtyEight, we give Obama 100% of the EVs in a state that is solid for him, 80% of the EVs for a leaner, 50% of the EVs for a Tossup, 20% of the EVs for state that is McCain-Lean, and 0% of the Solid McCain states. Exact opposite for McCain. For FiveThirtyEight, we use their overall estimate of Obama's EVs, not the state-by-state categories.
del.icio.us
