What's new
Fantasy Football - Footballguys Forums

This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

FiveThirtyEight.com - Nate Silver's site (1 Viewer)

The long knives are out for Silver right now. He's getting destroyed by the reviews.

Some of it is predictable old-media backlash against a young guy who turned his back on the NYT after making big demands, and some of it is from people who probably never liked him in the first place (FOX News types), but the criticism is mostly pretty fair so far IMO.

He's off to a bad start.
It's not really fair to attribute the bad reviews entirely to politics. Paul Krugman isn't exactly my go-to guy for political analysis, but I agree with his take on 538 completely. Silver's approach to politics worked really well because he was looking at data while "traditional" reporters were going on intuition. Once you branch out into other areas, there are actually people who know what they're doing and who know the data better than you do. That makes it tough for a generalist like Silver (and his colleagues at his new venture).

 
I also think certain growing pains are to be expected. I don't love Grantland, but I like it a lot more now than when it debuted. It figured out what it was (which is increasingly less of the long-form journalism it thought it was going to do). 538 will figure out what they do well and what they don't do well. And if their Sports content sucks, they'll just hire some guy who has does that stuff better. Few outlets are going to pay better for that type of content.

 
The long knives are out for Silver right now. He's getting destroyed by the reviews.

Some of it is predictable old-media backlash against a young guy who turned his back on the NYT after making big demands, and some of it is from people who probably never liked him in the first place (FOX News types), but the criticism is mostly pretty fair so far IMO.

He's off to a bad start.
It's not really fair to attribute the bad reviews entirely to politics. Paul Krugman isn't exactly my go-to guy for political analysis, but I agree with his take on 538 completely. Silver's approach to politics worked really well because he was looking at data while "traditional" reporters were going on intuition. Once you branch out into other areas, there are actually people who know what they're doing and who know the data better than you do. That makes it tough for a generalist like Silver (and his colleagues at his new venture).
This is an interesting point. I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.

His work on the 2012 presidential race was great because there was such a huge amount of data available. I bet he will do a good job with the mid-term elections as well, though not as well as he did in 2012 because media outlets tend to do a lot less polling in non-Presidential election years. But on the other stuff, he is really kind of floundering right now.

 
The long knives are out for Silver right now. He's getting destroyed by the reviews.

Some of it is predictable old-media backlash against a young guy who turned his back on the NYT after making big demands, and some of it is from people who probably never liked him in the first place (FOX News types), but the criticism is mostly pretty fair so far IMO.

He's off to a bad start.
It's not really fair to attribute the bad reviews entirely to politics. Paul Krugman isn't exactly my go-to guy for political analysis, but I agree with his take on 538 completely. Silver's approach to politics worked really well because he was looking at data while "traditional" reporters were going on intuition. Once you branch out into other areas, there are actually people who know what they're doing and who know the data better than you do. That makes it tough for a generalist like Silver (and his colleagues at his new venture).
Not sure if I wasn't clear or you didn't get to the end of my post, but I after saying some criticism was inevitable given the lay of the land and that he'd rubbed some folks the wrong way I ended by suggesting the criticism was warranted on its own. Totally agree with you.

 
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.

 
The long knives are out for Silver right now. He's getting destroyed by the reviews.

Some of it is predictable old-media backlash against a young guy who turned his back on the NYT after making big demands, and some of it is from people who probably never liked him in the first place (FOX News types), but the criticism is mostly pretty fair so far IMO.

He's off to a bad start.
It's not really fair to attribute the bad reviews entirely to politics. Paul Krugman isn't exactly my go-to guy for political analysis, but I agree with his take on 538 completely. Silver's approach to politics worked really well because he was looking at data while "traditional" reporters were going on intuition. Once you branch out into other areas, there are actually people who know what they're doing and who know the data better than you do. That makes it tough for a generalist like Silver (and his colleagues at his new venture).
I agree with this 100%.
 
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.
http://blog.seattlepi.com/football/2013/01/10/statistician-nate-silver-predicts-seahawks-patriots-rematch-in-super-bowl/

 
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.
http://blog.seattlepi.com/football/2013/01/10/statistician-nate-silver-predicts-seahawks-patriots-rematch-in-super-bowl/
Thanks, but that article (not written by Silver) seems to refute your previous statement. It says that FootballOutsiders' DVOA was picking Denver and San Francisco as the teams most likely to make the Super Bowl, while Silver was picking New England and Seattle.

 
Last edited by a moderator:
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.
http://blog.seattlepi.com/football/2013/01/10/statistician-nate-silver-predicts-seahawks-patriots-rematch-in-super-bowl/
Thanks, but that article (not written by Silver) seems to refute your previous statement. It says that FootballOutsiders' DVOA was picking Denver and San Francisco as the teams most likely to make the Super Bowl, while Silver was picking New England and Seattle.
Good point - so he was even worse than Football Outsiders, as at least they got San Francisco right (this was for the 2012 season).

 
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.
http://blog.seattlepi.com/football/2013/01/10/statistician-nate-silver-predicts-seahawks-patriots-rematch-in-super-bowl/
Thanks, but that article (not written by Silver) seems to refute your previous statement. It says that FootballOutsiders' DVOA was picking Denver and San Francisco as the teams most likely to make the Super Bowl, while Silver was picking New England and Seattle.
Good point - so he was even worse than Football Outsiders, as at least they got San Francisco right (this was for the 2012 season).
You seem to be missing the point of the work of FootballOutsiders' DVOA metric (and by extension Silver citing to it). There's no "right" or "wrong." They're tools to determine the most likely outcome. Just because it doesn't happen doesn't mean it's flawed or bad. They'll be the first people to tell you that the teams they "pick" to make the Super Bowl are more likely than not to miss it.

 
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.
http://blog.seattlepi.com/football/2013/01/10/statistician-nate-silver-predicts-seahawks-patriots-rematch-in-super-bowl/
Thanks, but that article (not written by Silver) seems to refute your previous statement. It says that FootballOutsiders' DVOA was picking Denver and San Francisco as the teams most likely to make the Super Bowl, while Silver was picking New England and Seattle.
Good point - so he was even worse than Football Outsiders, as at least they got San Francisco right (this was for the 2012 season).
You seem to be missing the point of the work of FootballOutsiders' DVOA metric (and by extension Silver citing to it). There's no "right" or "wrong." They're tools to determine the most likely outcome. Just because it doesn't happen doesn't mean it's flawed or bad. They'll be the first people to tell you that the teams they "pick" to make the Super Bowl are more likely than not to miss it.
I think the descriptive value of Football Outsiders' work is considerable. I think there are some flaws in their "secret formula" that tend to overvalue passing offenses with high completion percentages and undervalue "big play" offenses, but they provide genuine insight into what teams are playing well. I also think they tend to overvalue their own insights, almost as if they think their metrics should be prescriptive of what happens in the NFL - and if a lower-rated team somehow pulls off a bunch of big plays to upset a more highly-rated team, the game's outcome was in some way flawed. That's just my sense.

More broadly, I agree with you. Their rankings do a good job of exposing teams whose Win/Loss records are at significant variance with how well they are actually playing. But they have very limited value in picking outcomes of games between two closely matched teams. Which gets back to my original point, that Silver seemed very amateurish by touting their statistics so highly in making his picks in the 2012 playoffs. In just over a month, he went from being the guy who called all 50 states in November to just another talking head who pulled a prediction out of his rear and then saw it miss badly.

 
I think the descriptive value of Football Outsiders' work is considerable. I think there are some flaws in their "secret formula" that tend to overvalue passing offenses with high completion percentages and undervalue "big play" offenses, but they provide genuine insight into what teams are playing well. I also think they tend to overvalue their own insights, almost as if they think their metrics should be prescriptive of what happens in the NFL - and if a lower-rated team somehow pulls off a bunch of big plays to upset a more highly-rated team, the game's outcome was in some way flawed. That's just my sense.

More broadly, I agree with you. Their rankings do a good job of exposing teams whose Win/Loss records are at significant variance with how well they are actually playing. But they have very limited value in picking outcomes of games between two closely matched teams. Which gets back to my original point, that Silver seemed very amateurish by touting their statistics so highly in making his picks in the 2012 playoffs. In just over a month, he went from being the guy who called all 50 states in November to just another talking head who pulled a prediction out of his rear and then saw it miss badly.
DVOA is sort of a hybrid between being retrodictive (explaining why the Seahawks won their last game) and predictive (explaining why the Seahawks are likely to win their next game). It probably has more retrodictive value overall. It takes certain steps toward trying to be predictive (by, for example, discounting fumble recoveries since they're mostly random), but ultimately doesn't go very far in that direction (for example, it fails to discount fumbles themselves to the extent that fumble rates are uncorrelated from week to week).

I don't think any serious sports bettor would rely all that heavily on DVOA to predict future outcomes of games. That's why I was interested in your initial claim that Silver had done exactly that. Silver is a smart guy; if he had actually done that, it would have caused me to reconsider my current view that doing so is stupid, and I might have learned something. But alas, it appears that he did no such thing, so ... false alarm.

 
Paul Krugman isn't exactly my go-to guy for political analysis, but I agree with his take on 538 completely.
Krugman and Silver are in a tiff of sorts.
"I haven't been surprised by Krugman's criticism because I've fired some shots at the New York Times editorial page, of which he's a member," Silver told TPM in an email on Monday, adding that Krugman "was full of praise for FiveThirtyEight while it was part of The New York Times."

The turning point may have come earlier this month, when Silver told New York Magazine that "op-ed columnists at the New York Times, WashingtonPost, and Wall Street Journal" are most like the hedgehog, a creature that the Greek poet Archilochus said knows "one big thing."

As Silver sees it, those hedgehogs run in contrast to FiveThirtyEight's scrappy crew of foxes, who know "many little things."

"Plenty of pundits have really high IQs, but they don’t have any discipline in how they look at the world, and so it leads to a lot of bull####, basically," Silver said in that interview.

When Silver was asked to identify an exception to such criticism, Krugman's name wasn't mentioned. The stats guru instead named Krugman's conservative colleague at the Times, Ross Douthat, as "someone who shows some originality."
I can't believe I'm standing up for Paul Krugman, but the suggestion that he has some kind of tribal loyalty to the NYT and that's what's driving his criticism is honestly pretty crazy. Krugman's passive-aggressive hatred of David Brooks, for example, is legendary. And Krugman is allegedly one of the toughest columnists for his editors to work with. (I've heard the same criticism from textbook reps, so I'm inclined to believe it). If there's on thing that Paul Krugman really does understand well, it's how to interpret economic with an underlying model in mind.

 
IvanKaramazov said:
Maurile Tremblay said:
IvanKaramazov said:
Paul Krugman isn't exactly my go-to guy for political analysis, but I agree with his take on 538 completely.
Krugman and Silver are in a tiff of sorts.
"I haven't been surprised by Krugman's criticism because I've fired some shots at the New York Times editorial page, of which he's a member," Silver told TPM in an email on Monday, adding that Krugman "was full of praise for FiveThirtyEight while it was part of The New York Times."

The turning point may have come earlier this month, when Silver told New York Magazine that "op-ed columnists at the New York Times, WashingtonPost, and Wall Street Journal" are most like the hedgehog, a creature that the Greek poet Archilochus said knows "one big thing."

As Silver sees it, those hedgehogs run in contrast to FiveThirtyEight's scrappy crew of foxes, who know "many little things."

"Plenty of pundits have really high IQs, but they don’t have any discipline in how they look at the world, and so it leads to a lot of bull####, basically," Silver said in that interview.

When Silver was asked to identify an exception to such criticism, Krugman's name wasn't mentioned. The stats guru instead named Krugman's conservative colleague at the Times, Ross Douthat, as "someone who shows some originality."
I can't believe I'm standing up for Paul Krugman, but the suggestion that he has some kind of tribal loyalty to the NYT and that's what's driving his criticism is honestly pretty crazy. Krugman's passive-aggressive hatred of David Brooks, for example, is legendary. And Krugman is allegedly one of the toughest columnists for his editors to work with. (I've heard the same criticism from textbook reps, so I'm inclined to believe it). If there's on thing that Paul Krugman really does understand well, it's how to interpret economic with an underlying model in mind.
This actually makes a whole hell of a lot sense. Isn't it the same argument that Hayek was making about intellectuals when they leave their field of expertise? That they really have no business being there, in those other categories, much like Krugman has taken his Nobel for economics and turned it into broader political advocacy, which he freely admits?

Or am I missing Silver's point?

 
The_Man said:
TobiasFunke said:
The_Man said:
Maurile Tremblay said:
I thought Silver devalued his brand big-time last year when he ventured into forecasting the NFL playoffs and just used Football Outsiders' DVOA as the basis for all his picks. For anyone who handicaps football on a semi-serious basis, it was an embarrassing case of amateur hour.
Do you have a link? I searched the domain for "DVOA" and got only one hit, an article not fitting that description.
http://blog.seattlepi.com/football/2013/01/10/statistician-nate-silver-predicts-seahawks-patriots-rematch-in-super-bowl/
Thanks, but that article (not written by Silver) seems to refute your previous statement. It says that FootballOutsiders' DVOA was picking Denver and San Francisco as the teams most likely to make the Super Bowl, while Silver was picking New England and Seattle.
Good point - so he was even worse than Football Outsiders, as at least they got San Francisco right (this was for the 2012 season).
You seem to be missing the point of the work of FootballOutsiders' DVOA metric (and by extension Silver citing to it). There's no "right" or "wrong." They're tools to determine the most likely outcome. Just because it doesn't happen doesn't mean it's flawed or bad. They'll be the first people to tell you that the teams they "pick" to make the Super Bowl are more likely than not to miss it.
I think the descriptive value of Football Outsiders' work is considerable. I think there are some flaws in their "secret formula" that tend to overvalue passing offenses with high completion percentages and undervalue "big play" offenses, but they provide genuine insight into what teams are playing well. I also think they tend to overvalue their own insights, almost as if they think their metrics should be prescriptive of what happens in the NFL - and if a lower-rated team somehow pulls off a bunch of big plays to upset a more highly-rated team, the game's outcome was in some way flawed. That's just my sense.

More broadly, I agree with you. Their rankings do a good job of exposing teams whose Win/Loss records are at significant variance with how well they are actually playing. But they have very limited value in picking outcomes of games between two closely matched teams. Which gets back to my original point, that Silver seemed very amateurish by touting their statistics so highly in making his picks in the 2012 playoffs. In just over a month, he went from being the guy who called all 50 states in November to just another talking head who pulled a prediction out of his rear and then saw it miss badly.
Whether it's flawed or not (and of course it is- no metric is perfect), I still don't see how he devalued his brand. Someone called him and asked for his predictions. He used what he considers the best tools available and made his picks. It's exactly what you'd expect him to do. Sports are unpredictable, especially sports with single elimination playoffs. Upsets happen. If that devalues his brand with certain people I'd say those people didn't really understand what he's doing in the first place. If you deal in probabilities eventually your "picks" are going to be wrong.

 
Thought this was pretty good...

Looking at a bunch of other posts, you can see that this is par for the course. And not one of the posts attempts a single quantitative prediction, which is what Silver has famously thrilled the world by doing in the past.

In sum, this so-called "data-driven" website is significantly less data-driven (and less sophisticated) than Business Insider or Bloomberg View or The Atlantic. It consists nearly entirely of hedgehoggy posts supporting simplistic theories with sparse data and zero statistical analysis, making no quantitative predictions whatsoever. It has no relationship whatsoever to the sophisticated analysis of rich data sets for which Nate Silver himself has become famous.

The problem with the new FiveThirtyEight is not one of data vs. theory. It is one of "data" the buzzword vs. data the actual thing. Nate Silver is a hero of mine, but this site is not living up to its billing at all.
 
From Carl Bialik:

The lower a team’s seed [in the March Madness brackets], the more likely [President Obama] is to overestimate its chances. The president has predicted a far smaller number of round-of-64 upsets than the tournaments have produced, and in the last seven tournaments he’s forecast just one men’s team with a seed below 9 to win more than one game.
Those two sentences contradict each other. Bialik is confusing projections about individual games for projections about groups of games. If each favorite has a 67% chance to win (which is an oversimplification, of course, but it demonstrates the principle involved), he should expect about ten or eleven upsets in the 32-game initial round. But in any particular game, he should expect zero upsets. (In any particular game, he should actually expect about 0.33 upsets, statistically, but that's not how brackets work. It's all or nothing -- each team gets either 0 wins or 1 win; no team is ever assigned 0.67 wins.)

The fact that Obama picked fewer upsets in individual games than one would rationally pick for the games as a group is not evidence that he's overestimating the chances of high seeds. Rather, the fact that he picked any upsets at all is evidence for the opposite proposition -- that Obama is underestimating the chances certain higher-seeded teams, at least relative to the selection committee's view.

 
isn't that just the old conundrum/confusion about the terms "high seed" or "low seed"? That is, when Bialik says "lower a team's seed", he's talking about a seed with a low number like a #1 or #2 seed.

 
isn't that just the old conundrum/confusion about the terms "high seed" or "low seed"? That is, when Bialik says "lower a team's seed", he's talking about a seed with a low number like a #1 or #2 seed.
No, Bialik is saying that Obama is picking too few upsets, and is therefore overrating teams that are seeded #1 or #2 (or anything better than #9).

It's the same mistake people make when they look at weekly FBG projections and say that we're picking too few RBs to run for more than 100 yards. In any given week, several RBs will rush for over 100 yards, but very few RBs are more-likely-than-not to do so.

Back to basketball, let's say there are 32 games, and in each game, the favorite is 67% likely to win and 33% likely to lose. If we are looking at the games as a group, we'll expect 10 or 11 upsets. If we are looking at each game in isolation, we'll expect zero upsets -- in any particular case, the favorite is likely to win.

Bialik is criticizing Obama's brackets as if they were group projections, but that's not how brackets work. When filling out brackets, you don't get to say "I expect 10 upsets" in the abstract. You have to name which particular upsets you expect before the games are played. That is to say, brackets are numerous projections about individual games -- not projections about characteristics of the group.

So the fact that Obama picks fewer upsets than are likely to occur -- just like when FBG projects fewer 100-yard RBs than are likely to occur -- is not evidence that he's overrating the top teams.

 
Last edited by a moderator:
ok, was just hung up on your use of the word "contradiction". didn't really see those statements as contradictory, but rather (based on your explanation) just clunky verbiage around a nuanced topic. I might even say you're being pedantic here.

 
ok, was just hung up on your use of the word "contradiction". didn't really see those statements as contradictory, but rather (based on your explanation) just clunky verbiage around a nuanced topic. I might even say you're being pedantic here.
:gauntlettossed:

 
Polling analyst fight!

I don’t like to call out other forecasters by name unless I have something positive to say about them — and we think most of the other models out there are pretty great. But one is in so much perceived disagreement with FiveThirtyEight’s that it requires some attention. That’s the model put together by Sam Wang, an associate professor of molecular biology at Princeton.

That model is wrong — not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls. This is because instead of estimating the uncertainty empirically — that is, by looking at how accurate polls or polling averages have been in the past — Wang makes several assumptions about how polls behave that don’t check out against the data.

There’s a rich record of those assumptions failing and resulting in highly overconfident forecasts. In 2010, for example, Wang’s model made Sharron Angle the favorite in Nevada against Harry Reid; it estimated she was 2 points ahead in the polls, but with a standard error of just 0.5 points. If we drew a graphic based on Wang’s forecast like the ones we drew above, it would have Angle winning the race 99.997 percent of the time, meaning that Reid’s victory was about a 30,000-to-1 long shot. To be clear, the FiveThirtyEight model had Angle favored also, but it provided for much more uncertainty. Reid’s win came as a 5-to-1 underdog in our model instead of a 30,000-to-1 underdog in Wang’s; those are very different forecasts.

There are a number of other examples like this. Wang projected a Republican gain of 51 seats in the House in 2010, but with a margin of error of just plus or minus two seats. His forecast implied that odds against Republicans picking up at least 63 seats (as they actually did) were trillions-and-trillions-to-1 against.
 
Last edited by a moderator:
Polling analyst fight!

I don’t like to call out other forecasters by name unless I have something positive to say about them — and we think most of the other models out there are pretty great. But one is in so much perceived disagreement with FiveThirtyEight’s that it requires some attention. That’s the model put together by Sam Wang, an associate professor of molecular biology at Princeton.That model is wrong — not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls. This is because instead of estimating the uncertainty empirically — that is, by looking at how accurate polls or polling averages have been in the past — Wang makes several assumptions about how polls behave that don’t check out against the data.There’s a rich record of those assumptions failing and resulting in highly overconfident forecasts. In 2010, for example, Wang’s model made Sharron Angle the favorite in Nevada against Harry Reid; it estimated she was 2 points ahead in the polls, but with a standard error of just 0.5 points. If we drew a graphic based on Wang’s forecast like the ones we drew above, it would have Angle winning the race 99.997 percent of the time, meaning that Reid’s victory was about a 30,000-to-1 long shot. To be clear, the FiveThirtyEight model had Angle favored also, but it provided for much more uncertainty. Reid’s win came as a 5-to-1 underdog in our model instead of a 30,000-to-1 underdog in Wang’s; those are very different forecasts.There are a number of other examples like this. Wang projected a Republican gain of 51 seats in the House in 2010, but with a margin of error of just plus or minus two seats. His forecast implied that odds against Republicans picking up at least 63 seats (as they actually did) were trillions-and-trillions-to-1 against.
I'll be interested to see Wang's response. Hard to argue with Silver's reasoning.

 
Polling analyst fight!

I don’t like to call out other forecasters by name unless I have something positive to say about them — and we think most of the other models out there are pretty great. But one is in so much perceived disagreement with FiveThirtyEight’s that it requires some attention. That’s the model put together by Sam Wang, an associate professor of molecular biology at Princeton.

That model is wrong — not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls. This is because instead of estimating the uncertainty empirically — that is, by looking at how accurate polls or polling averages have been in the past — Wang makes several assumptions about how polls behave that don’t check out against the data.

There’s a rich record of those assumptions failing and resulting in highly overconfident forecasts. In 2010, for example, Wang’s model made Sharron Angle the favorite in Nevada against Harry Reid; it estimated she was 2 points ahead in the polls, but with a standard error of just 0.5 points. If we drew a graphic based on Wang’s forecast like the ones we drew above, it would have Angle winning the race 99.997 percent of the time, meaning that Reid’s victory was about a 30,000-to-1 long shot. To be clear, the FiveThirtyEight model had Angle favored also, but it provided for much more uncertainty. Reid’s win came as a 5-to-1 underdog in our model instead of a 30,000-to-1 underdog in Wang’s; those are very different forecasts.

There are a number of other examples like this. Wang projected a Republican gain of 51 seats in the House in 2010, but with a margin of error of just plus or minus two seats. His forecast implied that odds against Republicans picking up at least 63 seats (as they actually did) were trillions-and-trillions-to-1 against.
Sounds like Silver is butthurt that Wang's model was better than his for 2012

 
Sounds like Silver is butthurt that Wang's model was better than his for 2012
You understand this stuff better than that I think.

I didn't post the part of Silver's explanation about 'model calibration' but that's what's relevant here. If you say something's 50/50 you should be wrong 50% of the time if you make a pick. Likewise, if you say something is effectively infinity to one of never happening it should never happen.

The question I'd like to ask Silver is how that idea jibes with him getting 99/100 states right in the last two Presidential elections. He should have missed more if his approach to modeling is right.

ETA: nope. I'm wrong. There's about a 13% chance he'd get all the states right if his predictions were completely independent of each other. But they aren't. When the polls go wrong it tends to happen all at once. So you'd expect to see his predictions be very right when nothing is hinky, but then go 'wrong' all at once when the polling is wrong across the board (as happens from time to time).

 
Last edited by a moderator:
Sounds like Silver is butthurt that Wang's model was better than his for 2012
You understand this stuff better than that I think.

I didn't post the part of Silver's explanation about 'model calibration' but that's what's relevant here. If you say something's 50/50 you should be wrong 50% of the time if you make a pick. Likewise, if you say something is effectively infinity to one of never happening it should never happen.

The question I'd like to ask Silver is how that idea jibes with him getting 99/100 states right in the last two Presidential elections. He should have missed more if his approach to modeling is right.
His margin of error/probability factor tightens up a lot once the race is nearly over. I'm not sure if that explains the whole amount, but it probably does explain a good bit of it.

 
Joseph said:
wdcrob said:
Slapdash said:
Sounds like Silver is butthurt that Wang's model was better than his for 2012
You understand this stuff better than that I think.

I didn't post the part of Silver's explanation about 'model calibration' but that's what's relevant here. If you say something's 50/50 you should be wrong 50% of the time if you make a pick. Likewise, if you say something is effectively infinity to one of never happening it should never happen.

The question I'd like to ask Silver is how that idea jibes with him getting 99/100 states right in the last two Presidential elections. He should have missed more if his approach to modeling is right.
His margin of error/probability factor tightens up a lot once the race is nearly over. I'm not sure if that explains the whole amount, but it probably does explain a good bit of it.
Nerd alert!

I went back and got the final %s for each of his calls in the 2012 race. So those already reflect his estimated error (which does shrink at the end).

I edited my post above, but the gist of it is that his final estimates aren't independent of each other. As long as there's nothing wrong with the polling he's got a great chance of beating the odds in each state. The issue is that when the polls are wrong they tend to be wrong in the same direction. So in cases when he misses he (and everyone else) will miss a bunch at once. Silver accounts for this possibility in his models, Wang doesn't.

So, in a "normal poll results" election Silver and Wang will get most or all states right. In a "not normal poll results" election Silver and Wang will both get more states wrong.

The only difference (at the Prez level) is that Silver is showing you an estimate of the possibility that the polls are wrong, while Wang isn't.

Where it comes into play is that things will very rarely happen that Silver predicts won't happen while things happen all the time that Wang says won't happen (i.e. + 63 house seats instead of +51 when Wang's margin of error is two seats -- six standard deviations). Silver's range of outcomes will be better -- especially over time.

 
Last edited by a moderator:
wdcrob said:
ETA: nope. I'm wrong. There's about a 13% chance he'd get all the states right if his predictions were completely independent of each other. But they aren't. When the polls go wrong it tends to happen all at once. So you'd expect to see his predictions be very right when nothing is hinky, but then go 'wrong' all at once when the polling is wrong across the board (as happens from time to time).
Even if they were independent, things that happen 13% of the time happen a lot. That's better odds than making an inside straight draw, which nobody says "shouldn't" happen.

 
wdcrob said:
ETA: nope. I'm wrong. There's about a 13% chance he'd get all the states right if his predictions were completely independent of each other. But they aren't. When the polls go wrong it tends to happen all at once. So you'd expect to see his predictions be very right when nothing is hinky, but then go 'wrong' all at once when the polling is wrong across the board (as happens from time to time).
Even if they were independent, things that happen 13% of the time happen a lot. That's better odds than making an inside straight draw, which nobody says "shouldn't" happen.
Absolutely -- but since Silver got 49/50 in 2008 as well I think you'd really have to question his models if each state call was independent. That's what I was thinking until I played with it a little bit.

 
wdcrob said:
ETA: nope. I'm wrong. There's about a 13% chance he'd get all the states right if his predictions were completely independent of each other. But they aren't. When the polls go wrong it tends to happen all at once. So you'd expect to see his predictions be very right when nothing is hinky, but then go 'wrong' all at once when the polling is wrong across the board (as happens from time to time).
Even if they were independent, things that happen 13% of the time happen a lot. That's better odds than making an inside straight draw, which nobody says "shouldn't" happen.
Absolutely -- but since Silver got 49/50 in 2008 as well I think you'd really have to question his models if each state call was independent. That's what I was thinking until I played with it a little bit.
Of the 50 states, though, how many were 99.99999% already determined? 49/50 and 50/50 sounds impressive, but if 40 of them were basically set and everyone was going to get those right, isn't it really all about who's best at nailing the remaining 10?

 
I would think it's much harder, as a general rule, to predict the outcomes of non- Presidential years, especially in close elections. The problem is nobody knows who is going to show up.

 
  • Sam Wang // Sep 17, 2014 at 11:29 am

    I agree, 2010 was not stellar – though I very clearly indicated a GOP takeover. Also, note that others were also off – here’sa roundup of the predictions. Finally, note that I used a different kind of polling data, which was of lower reliability than Senate/Presidential polls.

  • Sam Wang // Sep 17, 2014 at 11:00 am

    He’s assuming I didn’t learn anything from past elections. He might also be mixing up our snapshots and our predictions.
 
Correction – How Our Prediction *Really* Works September 17th, 2014, 11:10am by Sam WangI hear that the Princeton Election Consortium calculation has come under criticism for being statistically overconfident. I think there is confusion here, which requires a little explanation – and an appreciation for what I’ve learned since I started doing this in 2004.

The key difference is between the snapshot and the prediction. Our snapshots are precise because they give a picture of conditions today. Our November prediction builds in the possibility of change occurring in the coming seven weeks. That’s why the November prediction above (today at 70%) is less certain than the snapshot (today at 80%).

I explained this in 2012. As an example, when our current prediction method is applied to past Presidential races, they gave a cliffhanger in 2004, and clear Obama wins in 2008 and 2012. Polls suggest that this year, Senate control is also a cliffhanger.

I’m sure there are more points I have missed. Have at it in comments. Please be nice about everyone, including any rival sites. Nonsubstantive and rude comments will be heavily moderated.
 
Of the 50 states, though, how many were 99.99999% already determined? 49/50 and 50/50 sounds impressive, but if 40 of them were basically set and everyone was going to get those right, isn't it really all about who's best at nailing the remaining 10?
I just multiplied the chances of his prediction for each state for all fifty (i.e treated them like they're independent).

So the 100%s and 99.9% are accounted for. The chances he hit on all 50 in 2012 were only 13%. And the odds for 49 in 208 were probably something like ~20%. If they were independent, but they aren't.

 

Users who are viewing this thread

Back
Top