FiveThirtyEight vs. the Data Detractors - OTM Podcast

[quade 4]

Quote

Last Tuesday Donald Trump won the Indiana primary and became the presumptive nominee of the Republican party. In the days that followed, hands were wrung over the question “how did we get this so wrong?” New York Times columnist Jim Rutenberg was particularly critical of data journalism, which one election cycle ago seemed so heroic but in Trumpworld turned out to have feet of clay. Singling out Nate Silver’s FiveThirtyEight (our partner this election cycle), Rutenberg wrote that in relying on polling data that gave Trump a 2% chance of winning the nomination 6 months ago, FiveThirtyEight “sapped the journalistic will to scour his record as aggressively as those of his supposedly more serious rivals. In other words, predictions can have consequences.”

Nate Silver on his podcast this week had a response to Rutenberg (and all the other data detractors). Here is an excerpt from that episode in which you’ll also hear Silver’s FiveThirtyEight colleagues Harry Enten, Clare Malone, and Jody Avirgan.

Source and full podcast:http://www.wnyc.org/story/fivethirtyeight-vs-data-detractors/?utm_source=feedburner&utm_medium=%24%7Bfeed%7D&utm_campaign=Feed%3A+%24%7Botm%7D+%28%24%7BOn+the+Media%7D%29

Listening to this now...

Reactions after I've heard it all.

Edited to add:

Interesting.

Because it's a podcast (and I don't have time to write out a transcript), I'm not going to quote from it.

My impression is one I've always had and I think Nate Silver comes pretty close to saying, political science and polling in particular is not actually a science. Oh sure, there are better and worse ways to go about it, and there are better and worse models, but as William Goldman once famously said about Hollywood probably also applies to it as well;

"Nobody knows anything." -- William Goldman.

Oh sure, you can have good hunches based on past experience, but you don't KNOW until the final count.

It was kind of refreshing to hear Nate Silver sort of say that out loud. Not in those exact words, but certainly in sentiment.

The other thing is the really interesting sort of in fighting, between the "traditional journalists" and the "data journalists." Interesting to hear about it from his perspective.

quade -
The World's Most Boring Skydiver

[wmw999 2,588]

Late last summer or so, there was an episode of "On the Media" on NPR where the commentator said that if a listener began to hear some analysis of the upcoming election based on polls, and it wasn't 2nd quarter of 2016 yet, they should just turn off the radio or TV.

Because the numbers are based on the number of people who have already formulated an opinion, or who are willing to just make shit up that early. Both are small numbers of people before the first run of primaries (probably).

So all the stations are comparing their shit to see whose smells the best, and now they wonder how they got it wrong

Wendy P.

There is nothing more dangerous than breaking a basic safety rule and getting away with it. It removes fear of the consequences and builds false confidence. (tbrown)

[jcd11235 0]

People often make the mistake of concluding that if a model gives a 90 percent probability of an event occurring, then if that event doesn't occur, then the model is wrong. That isn't the case.

There are many algorithms that can be used for predicting classifications, that is predicting whether a future observation belongs to class A or class B. (We're not actually limited to two classes.) For example, we might want to develop a model that can automatically determine whether a baseball team will win (class A) or lose (class B), given the opponent, park, pitcher's ERA, etc.

Some algorithms make predictions by first calculating a probability of an observation (a game, in our example) belonging to a particular class. So, if the model determines there is a 75 percent probability of a win, then a win is predicted.

When we are performing diagnostics on the model, if we find that predictions based on a 75 percent probability are correct 90 percent of the time, that's a problem. Likewise, if they're correct only 60 percent of the time, that's a problem. Predictions based on a 75 percent probability of of an observation belonging to a particular class should be correct about 75 percent of the time.

If an accurate, unbiased model gives a 2 percent probability of an event occurring, we still expect that event to occur about one time in fifty. Those are long, but nowhere near impossible odds. The question we should be asking is, "how often do candidates who are given a 2 percent probability of securing the nomination actually secure the nomination?" Unless that number is statistically significantly different from 2 percent, then concluding the model is wrong is not justified.

Math tutoring available. Only $6! per hour! First lesson: Factorials!

[winsor 236]

jcd11235

People often make the mistake of concluding that if a model gives a 90 percent probability of an event occurring, then if that event doesn't occur, then the model is wrong. That isn't the case.

There are many algorithms that can be used for predicting classifications, that is predicting whether a future observation belongs to class A or class B. (We're not actually limited to two classes.) For example, we might want to develop a model that can automatically determine whether a baseball team will win (class A) or lose (class B), given the opponent, park, pitcher's ERA, etc.

Some algorithms make predictions by first calculating a probability of an observation (a game, in our example) belonging to a particular class. So, if the model determines there is a 75 percent probability of a win, then a win is predicted.

When we are performing diagnostics on the model, if we find that predictions based on a 75 percent probability are correct 90 percent of the time, that's a problem. Likewise, if they're correct only 60 percent of the time, that's a problem. Predictions based on a 75 percent probability of of an observation belonging to a particular class should be correct about 75 percent of the time.

If an accurate, unbiased model gives a 2 percent probability of an event occurring, we still expect that event to occur about one time in fifty. Those are long, but nowhere near impossible odds. The question we should be asking is, "how often do candidates who are given a 2 percent probability of securing the nomination actually secure the nomination?" Unless that number is statistically significantly different from 2 percent, then concluding the model is wrong is not justified.

Put another way:

"The race is not always to the swift, nor the battle to the strong. That is, however, the way to bet."

[Boomerdog 0]

The prognosticators on either side can write all they want. It's what they do and it's what they get paid for. Fifteen minutes is a lifetime in politics. Between now and November 8, 2016, there's a lot of 15 minute intervals left. Carter was leading Reagan in 1980 up until the last week of the election; then it turned decidedly against Carter. Where this election goes I have no idea but so far it's been a lot of fun to watch.

[winsor 236]

Boomerdog

The prognosticators on either side can write all they want. It's what they do and it's what they get paid for. Fifteen minutes is a lifetime in politics. Between now and November 8, 2016, there's a lot of 15 minute intervals left. Carter was leading Reagan in 1980 up until the last week of the election; then it turned decidedly against Carter. Where this election goes I have no idea but so far it's been a lot of fun to watch.

It would be a lot more fun to watch a number of people with viable policies and programs vie for the top slot.

The assemblage of cretins, morons and second-raters that has floated to the top like rotting sewage makes it tough to enjoy the process this time around.

Hillary is basically Gilbert Gottfried in drag.

Trump is not an idiot bully, but he plays one on TV.

Sanders is very appealing to anyone who has never been much of anywhere, and never had to balance a checkbook, much less a budget.

Since we're beyond the tipping point, we might as well have the most despicable among us at the helm as we slip under.

It was fun while it lasted.