Nate Silver Whiffs on Montana’s US Senate Race & Other Stuff

by Jay Stevens

Hey, everybody! I thought I’d drop in and say hello and give shouts out to old friends that did well in their election bids yesterday. Jhwygirl and I chatted on the phone the other day, and she said I should post every once in a while, and why the heck not? I miss Montana politics and the hubub of election day in Missoula.

So, anyhoo. Congrats go out to Jon Tester and Steve Bullock, who won very close and extremely important races. Congrats, too, to Linda McCulloch for destroying Brad Johnson, again. (Who will ever forget Johnson botching the 2006 election? Not Montana, apparently.) And to Monica Lindeen in defeating the reality-challenged Derek Skees. (Who won 46 percent of the vote?) Tim Fox‘ win in the attorney general race, and the nearly neck-and-neck OPI race should remind us how nuts and frustrating Montana election results can be. Do folks really like Fox’ dirty politicking? And why would anyone support Juneau’s excellence in public office with a vote for Welch? (Please speculate freely in the comments!)

Congrats, too, to old friends JP Pomnichowski, Bryce Bennett, and Ellie Hill for winning their races. And my sympathies for Richard Turner — a great guy and good friend who deserves a seat in Helena, even if his neighbors don’t see it. I wish someone had written more about the state races this year — I used to do it, and enjoyed it. Anybody want to analyze this year’s results?

What I really came here to talk about, though — me and everyone else, apparently — is Nate Silver. Yes, we all know about the punditry backlash, the dust-up with Joe Scarborough, and the fact that Silver nailed it. (I’m with Conor Friedersdorf: I trust Silver more because of the bet.) Okay, Silver might have destroyed punditry (um, no), but he’s not perfect: He muffed Montana.

Read it again: Nate Silver got Montana’s US Senate race wrong. 

Actually, he missed it by quite a bit. Silver projected that Dennis Rehberg would win the race, 49.9 percent to 48.4. Tester, according to the unofficial results, won 48.5 percent to 44.9. That’s a swing from a +1.5 Rehberg win to a -3.6 Rehberg loss, over five percentage points off from his projection. The odd thing is that recent polls showed Tester with a small lead — even Rasmussen, which tended to overestimate Republican support. How did Silver interpret those results with a “lean Republican” projection?

The big factor in his analysis was an adjustment he called “state fundamentals,” which, according to the blog, is “an alternative forecast of the outcome that avoids polls and instead looks at the partisan environment of a state, public fundraising totals, statistical measures of left-right ideology and candidate quality, and other quantifiable factors.” According to that measure, Rehberg had a 50.7 to 42.2 percent lead. 

That was egregiously wrong.

Why? For starters — and I’d need to check other states’ election results over the years to confirm this — Montana is notorious for splitting its ballots. Montana’s perfectly comfortable, for example, in voting for a Republican president, whisking in Democrats to all the statewide seats, and increasing the number of seats Republicans hold in the state legislature — all in the same election, which happened in 2008. This year, Montanans went for a Republican president, Democratic governor, and Democratic Senator, while dividing the statewide seats.

For another, Silver apparently didn’t calculate the effect of a third-party candidate. This election Libertarian Dan Cox won a whopping 29,979 votes, good for 6.52 percent of the vote, which is nearly double Tester’s margin of victory. That’s reminiscent of 2006, when Libertarian Stan Jones’ vote haul (10,377) was more than Tester’s margin of victory over Conrad Burns (~3,500). Tester, after all, won a smaller percentage of the electorate in 2012 than he did in 2006. It’s just that Montanans apparently dislike Dennis Rehberg even more than they did Conrad Burns — after his disgrace for his involvement in the Abramoff corruption scandals.

Either way, Silver’s election projection model is good, but it ain’t perfect.

About these ads

  1. lizard19

    Jay!

    • Lizard!

      • lizard19

        great post, Jay, welcome back!

        • Rob Kailey

          Second!

          • Whee! I’d move back to Montana in flash. Anybody hiring?

  2. Silver missed the US Senate race in North Dakota as well. He was on the Daily Show tonight to talk about it.

  3. JC

    “Silver’s election projection model is good, but it ain’t perfect.”

    But is it good enough to get him an appointment in the O-house?

    Nice to see your still paying attention to Montana Jay, and not just lost out on the downrigger. ;-)

    • Ha! That’s hilarious.

      Seriously, though, Silver wasn’t the only one this election who did very accurate poll analysis. Just the most famous — thanks to the dumb pundits who challenged him…

  4. Tsk

    Talk about some great Monday morning quarter backing.

    • Wait! You’re Monday morning quarterbacking the Monday morning quarterback! My head hurts…

      In all seriousness, I’m sure Silver is looking at the data and trying to figure out what he got wrong and why. If you read the link to Fangraphs, you’d see that accurate projections from imperfect data don’t equal perfection. Silver obviously screwed up in his attempt to quantify political culture — “state fundamentals” — and needs to look at the data he used to measure it.

      • Tsk

        And it was a great post. I was just being a bit snarky, Jay. Love you’re writing and happy to have you back writing in Montana.

  5. Ryan Morton

    Saying Silver is wrong is akin to saying the weather forecast of 40% chance of precipation is ‘wrong’ when it rains and ‘right’ when it doesn’t rain. As Silver knows, days with a 40% chance of rain have rainy days about 40% of the time. So if Tester had a 40% chance of winning, he should actually win 40% of the time. Unfortunately, we can’t verify that as we only get one election. So, I think you need to change your paradigm of thinking about forecasts.

    Also, Silver didn’t publish all his math so no one can verify that either. What he did provide is something most statisticians could follow and probably reproduce somewhat closely but not precisely.

    • Well….yes, and I get it, and thanks for pointing it out. Just because the US Montana Senate race fell on the small side of Silver’s projection doesn’t mean the model was wrong. Yes. Especially when compared to “gut” feeling.

      Yet. Let me bounce a baseball metaphor off you. If a guy is a .280 hitter, we say in any given at-bat he has a 28% chance of a hit, right? Say he gets a hit. That doesn’t mean the batting average model was flat-out wrong, but it also doesn’t preclude that better projection models can exist that could have more accurately predicted the outcome. We could use a model that included batter side and pitcher handedness (if the batter’s a rightie and the pitcher a lefty, that would increase the batter’s likelihood of getting hit), the quality of the pitcher, maybe even a projection that factored in the pitcher’s repertoire and velocity and the batter’s ability to hit those pitches.

      In fact, if we could know all of the conditions that existed at the time of the hit — the pitcher was a mediocre lefty reliever, the batter a rightie; the pitcher was going to hang a curveball on the upper half of the strike zone, and the batter was a notorious mistake-hitter and his “hot zone” is in the upper half, our projection for a hit would be a helluva lot higher than 28%.

      Even a squibber under the second baseman’s glove could be more accurately predicted if we had data like the angle of the bat when it made contact with the ball, the velocity and direction of the ball after contact, and the positioning of the players.

      So…I’m not saying Silver’s model was wrong, I”m saying he didn’t have enough data, or a calculation he made in the interpretation of the data didn’t account for the state’s Senate race. The polls, after all, showed Tester with a growing lead heading into the election, but Silver still had the race leaning Republican because of quantitative measurements of the state’s political climate. It could be his model didn’t account for third-party candidates. It could me his measurement of states’ political climates weighed something too heavily, or didn’t account for something else.

      • Ryan Morton

        I’m not really sure what Silver really did for Montana without seeing his code. Also, he states in his methodology that variables were dropped if not statistically significant at 90%. That’s a possibility that could also fit your criticism.

        On baseball, I don’t know enough about those models or what is considered a useful variable for predictive models. But I don’t think a batting average is itself a prediction; it’s more of a historical average. Though I suppose you’re right that some people consider the average to be predictive.

        • Yeah, you’re right about the batting average. I was trying to tap into Silver’s past: he created Baseball Prospectus’ PECOTA projection model… So there you go.

  6. One thing that may have thrown off the predictions: How many people actually answered all those political surveys?
    I didn’t. I checked my caller i.d. when I got a call, and if it was from a survey, I let the machine pick it up.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


  • Pages

  • Recent Comments

    NamelessRange on Remember That Plane Those Rebe…
    Mark Tokarski on Remember That Plane Those Rebe…
    steve kelly on Amanda Curtis Should Attack Ob…
    Mark Tokarski on Amanda Curtis Should Attack Ob…
    Lois on Remember That Plane Those Rebe…
    Remember That Plane… on Save Yourselves from the Black…
    Big Swede on Amanda Curtis Should Attack Ob…
  • Recent Posts

  • Blog Stats

    • 1,506,564 hits
  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,271 other followers

  • November 2012
    S M T W T F S
    « Oct   Dec »
     123
    45678910
    11121314151617
    18192021222324
    252627282930  
  • Categories


Follow

Get every new post delivered to your Inbox.

Join 2,271 other followers

%d bloggers like this: