bobspruill
Breathe deeply
DISCLAIMERS:
This is not meant as criticism of FBG at all. I've used their projections for 4 years now with good success. The purpose here is to assess risks in depending upon FBG projections by determining where they are strong and where they are weak. This analysis only concerns 2005 projections, and as such the sample sizes are necessarily small. The comparisons made here are to the alternative method of using prior year's performance; this is not a competitive analysis of FBG to other sites' projections.
CONCLUSIONS:
(1) FBG, on balance, underestimates the consistency of the previous year's top 15 RB's.
(2) FBG, on balance, does a good job of assessing the value of new starters--those who are either changing teams or taking over a starting role--in the range 15-30. This includes rookie RB's.
(3) FBG projections are excellent at assessing RB rankings in the range 15-30 overall.
(4) Some combination (still to be investigated) of FBG projections and past year's performance is likely to provide a better predictor of RB ranking than either measure alone.
METHODS & DETAILED FINDINGS:
The focus here is on the rankings arrived at when stats are passed through the following scoring system:
1pt/5 yds rushing + 1pt/5 yds receiving + 1 PPR + 1pt/10 yds passing + 6 pts per TD passing or rushing or receiving - 3 pts per interception or fumble
The analysis was restricted to the top 40 RB's identified by using FBG stat projections with this scoring system. An initial comparison was made between the projected rank of these backs and their actual rank after week 16 of the 2005 regular season.
The correlation coefficient between projected rank and actual rank within this data set was 0.55. Conventionally, this is interpreted to mean that FBG rankings accounted for roughly 30% of the variability in RB rankings. The RMS difference between projected and actual rank was roughly 24. This may seem high, but realize that we're only talking about the top end of the rankings here, and so the error is definitely not unbiased. That is, there's a lot more room to miss on the low side with these projections than on the high side.
With that in mind, it seems remarkable that, of the 40 backs, only 24 had actual ranks lower than their projected ranks; the other 16 all wound up ranked higher. Of those 16, 11 were projected to fall in the range of 15-30.
It seems clear that underestimating the value of a back in this range is far less serious an error than overestimating it. Experienced FFL players know that it's the choice of a second or third RB that often determines your team's fortunes. This is an exceptionally useful place to be picking backs who perform better than expected.
The link below gives a graphical representation of differences between FBG projected ranks and actual ranks under the given scoring system. The FBG rankings are along the x-axis, and the red "perfect" line shows what the graph would look like if FBG had predicted the ranking exactly. Thus, overestimations of value occured where the blue graph is above the "perfect" line, underestimations where it is below "perfect."
FBG projected RB rank versus actual rank
The same methods were used to analyze prior year's performance as a ranking method. Interestingly, the correlation coefficient here was higher among the top 40: 0.58, which we interpret to mean that prior year's rank accounted for roughly 35% of the variability in this year's ranking. This suggests that some method of averaging or combining the FBG projections with the prior year's ranking may yield a still higher correlation, even if the two (as seems certain) are not independent.
This possibility seems even more likely when you examine where prior year's rank was good as a predictor, and where it was bad. The overall RMS difference in prior rank (among last year's top 40) versus this year's rank was 41--dramatically higher than FBG's projections. Prior year's rank turned out to be a better predictor than FBG in the range 1-15 but a great deal worse from 16 on down. Thus, using prior year's rank to temper FBG projections at the top end would, this year at least, have been a smart thing to do.
Despite its superior performance at the top, it's also worth noting that prior year's rank was far more likely to overestimate value than underestimate it. Among last year's top 40, 28 declined in rank, 3 (2 of the top 15) stayed in exactly the same place, and only 9 rose.
Here's the graphical representation of differences in last year's rank versus this year's, plotted on the same scale as that used in the graph above:
Prior year's RB rank versus actual rank
It might well be said that difference in rank is not the most important measure of how good a set of fantasy predictions really is. After all, the difference between RB1 and RB5 is typically much larger, in terms of point production, than the difference between RB31 and RB35. Thus, average measures of error in predicting rank obscure what you might call the "consequential difference" in ranks--a measure that depends upon point production.
In an effort to investigate these differences without utterly losing the ranking concept, the following method was used: actual point production was compared to the point production of the back who wound up at that back's projected rank. For example, under FBG projections, LT was the top-ranked back; he has accumulated 516 points so far. In actuality, Alexander is the top-ranked back (the back you thought you were getting when you drafted LT); he has accumulated 544 points so far. Thus, the consequential difference in the LT ranking is -28: you missed out on 28 points by thinking that the #1 back was LT, when in fact it was Alexander.
Finally, in an effort to make these numbers scale-neutral (which is not to say scoring-neutral, although it comes closer), the differences were taken as a percentage of a baseline production: that of the #40-ranked back (Antowain Smith: 147). That makes the consquential difference in the LT ranking -18%; for comparison's sake, the consequential difference in the ranking of Deuce McAllister this year was -260%.
Over the entire top 40, FBG and prior year's rank look pretty comparable this way. FBG projections yieded an RMS consequential difference of 85%, while prior year's rank yielded an RMS consequential difference of...85%. The interesting results arise when you segment the samples.
Over the top 10 RBs, the RMS consequential difference for FBG was 115%; for prior year's rank, it was 75%--about a third less.
If you capture the top 15, it's 105% for FBG and 78% for the prior year. This is still a pretty large difference, and the bias of the errors is in favor of prior year's rank: FBG overestimated in 11 of the 15 cases, while prior year's rank only overestimated in 9. In the case of the FBG overestimations, 7 fell completely outside the top 15; for prior year's ranking, that number was only 5. This seems fairly good evidence to support the conclusion that being in the top 15 last year was a better predictor of value than was appearing in the FBG top 15.
On the other hand, in the range 15-30, FBG was by far the better performer. The RMS consequential error for FBG projections was 75%, while for prior year it was 91%. This may not look so large until you realize that this is the range where scoring production tends to bunch up, so a 30% difference in predictive value here is more dramatic than it is over 1-15.
As was indicated above, however, it's the bias of these errors where FBG projections really shine. Of the 16 backs included in each sample, FBG overestimated in only 6 cases, whereas prior year's ranking overestimated in 11. In two illustrative cases (Nick Goings, Onterrio Smith), the overestimation was by more than 100%.
Below 30, as you can imagine, prior year's rankings are mostly hopeless. FBG's aren't great, but they're certainly better.
Here, for the sake of illustration, are graphs of consequential ranking error for FBG and prior year by projected rank:
Consequential rank difference: FBG
Consequential rank difference: prior year
Finally, I was interested in how FBG performed in two types of cases where prior year's rankings are likely to be bad: for rookies, and for backs who are newly taking over a starting job after having started none or only some of the team's games the previous year. Here are the FBG top-30 backs I'd classify this way:
Willis McGahee (7), Kevin Jones (9), Julius Jones (12), Steven Jackson (13), Lamont Jordan (19), Mike Anderson (21), JJ Arrington (22), Cadillac Williams (26), Ronnie Brown (30)
There were certainly other hard calls (Priest vs. LJ), and there were many more in the 30-40 range, but I wanted to look in this area, where some of the hardest and most significant projection decisions are made, and stay away from projection questions that involve predicting injury.
In terms of raw ranking difference, FBG overestimated in 4 of the 9 cases. There were three serious errors in terms of consequential value: K. Jones (-118%), Arrington (-95%), and Lamont Jordan (+105%). On the other hand, the only further error over 50% in either direction was McGahee (-63%).
What can best be learned from these facts is to temper FBG projections in these "hard" cases when the resulting ranking is better than 20th. Only 1 of the 4 overestimations occurred below that point, and all but 1 of the serious errors occurred above it. Jordan, of course, highlights the potential rewards in taking a flyer, but it looks as though the risks outweigh the benefits in the top 15 at least.
This is not meant as criticism of FBG at all. I've used their projections for 4 years now with good success. The purpose here is to assess risks in depending upon FBG projections by determining where they are strong and where they are weak. This analysis only concerns 2005 projections, and as such the sample sizes are necessarily small. The comparisons made here are to the alternative method of using prior year's performance; this is not a competitive analysis of FBG to other sites' projections.
CONCLUSIONS:
(1) FBG, on balance, underestimates the consistency of the previous year's top 15 RB's.
(2) FBG, on balance, does a good job of assessing the value of new starters--those who are either changing teams or taking over a starting role--in the range 15-30. This includes rookie RB's.
(3) FBG projections are excellent at assessing RB rankings in the range 15-30 overall.
(4) Some combination (still to be investigated) of FBG projections and past year's performance is likely to provide a better predictor of RB ranking than either measure alone.
METHODS & DETAILED FINDINGS:
The focus here is on the rankings arrived at when stats are passed through the following scoring system:
1pt/5 yds rushing + 1pt/5 yds receiving + 1 PPR + 1pt/10 yds passing + 6 pts per TD passing or rushing or receiving - 3 pts per interception or fumble
The analysis was restricted to the top 40 RB's identified by using FBG stat projections with this scoring system. An initial comparison was made between the projected rank of these backs and their actual rank after week 16 of the 2005 regular season.
The correlation coefficient between projected rank and actual rank within this data set was 0.55. Conventionally, this is interpreted to mean that FBG rankings accounted for roughly 30% of the variability in RB rankings. The RMS difference between projected and actual rank was roughly 24. This may seem high, but realize that we're only talking about the top end of the rankings here, and so the error is definitely not unbiased. That is, there's a lot more room to miss on the low side with these projections than on the high side.
With that in mind, it seems remarkable that, of the 40 backs, only 24 had actual ranks lower than their projected ranks; the other 16 all wound up ranked higher. Of those 16, 11 were projected to fall in the range of 15-30.
It seems clear that underestimating the value of a back in this range is far less serious an error than overestimating it. Experienced FFL players know that it's the choice of a second or third RB that often determines your team's fortunes. This is an exceptionally useful place to be picking backs who perform better than expected.
The link below gives a graphical representation of differences between FBG projected ranks and actual ranks under the given scoring system. The FBG rankings are along the x-axis, and the red "perfect" line shows what the graph would look like if FBG had predicted the ranking exactly. Thus, overestimations of value occured where the blue graph is above the "perfect" line, underestimations where it is below "perfect."
FBG projected RB rank versus actual rank
The same methods were used to analyze prior year's performance as a ranking method. Interestingly, the correlation coefficient here was higher among the top 40: 0.58, which we interpret to mean that prior year's rank accounted for roughly 35% of the variability in this year's ranking. This suggests that some method of averaging or combining the FBG projections with the prior year's ranking may yield a still higher correlation, even if the two (as seems certain) are not independent.
This possibility seems even more likely when you examine where prior year's rank was good as a predictor, and where it was bad. The overall RMS difference in prior rank (among last year's top 40) versus this year's rank was 41--dramatically higher than FBG's projections. Prior year's rank turned out to be a better predictor than FBG in the range 1-15 but a great deal worse from 16 on down. Thus, using prior year's rank to temper FBG projections at the top end would, this year at least, have been a smart thing to do.
Despite its superior performance at the top, it's also worth noting that prior year's rank was far more likely to overestimate value than underestimate it. Among last year's top 40, 28 declined in rank, 3 (2 of the top 15) stayed in exactly the same place, and only 9 rose.
Here's the graphical representation of differences in last year's rank versus this year's, plotted on the same scale as that used in the graph above:
Prior year's RB rank versus actual rank
It might well be said that difference in rank is not the most important measure of how good a set of fantasy predictions really is. After all, the difference between RB1 and RB5 is typically much larger, in terms of point production, than the difference between RB31 and RB35. Thus, average measures of error in predicting rank obscure what you might call the "consequential difference" in ranks--a measure that depends upon point production.
In an effort to investigate these differences without utterly losing the ranking concept, the following method was used: actual point production was compared to the point production of the back who wound up at that back's projected rank. For example, under FBG projections, LT was the top-ranked back; he has accumulated 516 points so far. In actuality, Alexander is the top-ranked back (the back you thought you were getting when you drafted LT); he has accumulated 544 points so far. Thus, the consequential difference in the LT ranking is -28: you missed out on 28 points by thinking that the #1 back was LT, when in fact it was Alexander.
Finally, in an effort to make these numbers scale-neutral (which is not to say scoring-neutral, although it comes closer), the differences were taken as a percentage of a baseline production: that of the #40-ranked back (Antowain Smith: 147). That makes the consquential difference in the LT ranking -18%; for comparison's sake, the consequential difference in the ranking of Deuce McAllister this year was -260%.
Over the entire top 40, FBG and prior year's rank look pretty comparable this way. FBG projections yieded an RMS consequential difference of 85%, while prior year's rank yielded an RMS consequential difference of...85%. The interesting results arise when you segment the samples.
Over the top 10 RBs, the RMS consequential difference for FBG was 115%; for prior year's rank, it was 75%--about a third less.
If you capture the top 15, it's 105% for FBG and 78% for the prior year. This is still a pretty large difference, and the bias of the errors is in favor of prior year's rank: FBG overestimated in 11 of the 15 cases, while prior year's rank only overestimated in 9. In the case of the FBG overestimations, 7 fell completely outside the top 15; for prior year's ranking, that number was only 5. This seems fairly good evidence to support the conclusion that being in the top 15 last year was a better predictor of value than was appearing in the FBG top 15.
On the other hand, in the range 15-30, FBG was by far the better performer. The RMS consequential error for FBG projections was 75%, while for prior year it was 91%. This may not look so large until you realize that this is the range where scoring production tends to bunch up, so a 30% difference in predictive value here is more dramatic than it is over 1-15.
As was indicated above, however, it's the bias of these errors where FBG projections really shine. Of the 16 backs included in each sample, FBG overestimated in only 6 cases, whereas prior year's ranking overestimated in 11. In two illustrative cases (Nick Goings, Onterrio Smith), the overestimation was by more than 100%.
Below 30, as you can imagine, prior year's rankings are mostly hopeless. FBG's aren't great, but they're certainly better.
Here, for the sake of illustration, are graphs of consequential ranking error for FBG and prior year by projected rank:
Consequential rank difference: FBG
Consequential rank difference: prior year
Finally, I was interested in how FBG performed in two types of cases where prior year's rankings are likely to be bad: for rookies, and for backs who are newly taking over a starting job after having started none or only some of the team's games the previous year. Here are the FBG top-30 backs I'd classify this way:
Willis McGahee (7), Kevin Jones (9), Julius Jones (12), Steven Jackson (13), Lamont Jordan (19), Mike Anderson (21), JJ Arrington (22), Cadillac Williams (26), Ronnie Brown (30)
There were certainly other hard calls (Priest vs. LJ), and there were many more in the 30-40 range, but I wanted to look in this area, where some of the hardest and most significant projection decisions are made, and stay away from projection questions that involve predicting injury.
In terms of raw ranking difference, FBG overestimated in 4 of the 9 cases. There were three serious errors in terms of consequential value: K. Jones (-118%), Arrington (-95%), and Lamont Jordan (+105%). On the other hand, the only further error over 50% in either direction was McGahee (-63%).
What can best be learned from these facts is to temper FBG projections in these "hard" cases when the resulting ranking is better than 20th. Only 1 of the 4 overestimations occurred below that point, and all but 1 of the serious errors occurred above it. Jordan, of course, highlights the potential rewards in taking a flyer, but it looks as though the risks outweigh the benefits in the top 15 at least.
f course, you might say that using that as a way of measuring the quality of projections actually circumvents the projections somewhat. What I was after in doing this analysis was a way of assessing the projections as they would be used during a fantasy draft. In order to use the projections in the way this average implies, you'd have to make your own additional projection in advance of the draft about how many weeks the player is likely to play. In an ideal world, this factor should be bound up in the projection itself.
I was in the middle on QBs and TEs and was in the top 3 in WRs. 
euce (FBG #4, 2004 #18, 2005 #51)
Interesting. It would be cool to see the actual numbers for each staff member.
