Earlier this year, we had evaluated the performance of the “major” pollsters during the 2010 elections.  Now that the 2012 elections are largely behind us, we thought we would revisit that analysis.

Criteria

(1) In the 2012 election cycle, we accumulated a significant amount of polling data. However, we are only interested in polling that was conducted during the week before Election Day. This means we looked at all polling that was publicly released between Monday, October 29 and Monday, November 5 of the 2012 elections (Election Day was Tuesday November 6);

(2) We only included polling conducted on statewide races for President, the U.S. Senate or Governor;

(3) We excluded those pollsters who released less than five polls during that final week – we are only interested in the “majors.”

(4) This analysis is being performed on election results available as of November 11 – there are still states where absentee votes are being counted, but we do not believe that uncounted absentee ballots will materially change the statewide results.

Using this screening criteria, there were 11 pollsters, although two of those pollsters only polled the Presidential race, so they should be excluded. The nine finalists were CBS, CNN/Opinion Research, Gravis Marketing, Mason Dixon, NBC, PPP, Rasmussen, Survey USA (SUSA), and We Ask America.

We are using two tests to evaluate the effectiveness of those nine pollsters: (1) the percentage of races they accurately polled (i.e, how often did they correctly predict the winner ?), and (2) for the races a pollster called correctly, how close to the actual numbers they were.

Test One: Win/Loss ratio

In this test, we counted all of the races the nine “major” pollsters were involved in, and tallied the number of races they correctly called. Using this method, Survey USA (SUSA) correctly called all 10 of the races they were involved in. They were joined by CBS, NBC, and Public Policy Polling (PPP). Mason Dixon was the least accurate – they predicted the winner in only 3 of the 6 races they polled (Incidentally, SUSA had a perfect record in 2010 as well).

Test Two: Accuracy

Those who use polling data to make decisions/evaluate a given race expect two things: (1) that the poll correctly predicts the winner, and (2) that the poll is reasonably accurate. We have already discussed the accuracy (in terms of “win/loss”) of the major pollsters. Now we would like to look at how closely the poll results were to the actual election results – in this situation, CBS and SUSA were the most accurate, while Rasmussen was, on average, the least accurate:

Conclusion

As with our 2010 analysis, we noted that those pollsters who correctly predicted the winners/losers the most often weren’t necessarily those who predicted the winning candidate’s percentage the most accurately. Here’s what we found in 2010: 

• Rasmussen was the only pollster in the “Final Four” both for win/loss ratio and for accuracy;
• Fox was in the “Bottom Four” for both metrics;
• The other pollsters (Survey USA, Quinnipiac, Mason-Dixon, McClatchy/Marist, Public Policy Polling, and CNN) were in the middle of the pack.

For 2012, the “winners” and “losers” changed:

• CBS and Survey USA (SUSA) were the only pollsters in the “Final Four” both for win/loss ratio and for accuracy;
• Rasmussen, Gravis Marketing, and Mason Dixon were below average both for win/loss ratio and for accuracy;
• The other pollsters (NBC, Public Policy Polling (PPP), CNN/Opinion Research, and We Ask America) were in the middle of the pack, meaning that they either were in the “final four” for win/loss ratio or accuracy, but not both.