bobspruill
Breathe deeply
DISCLAIMERS:
This is not meant as criticism of FBG at all. I've used their projections for 4 years now with good success. The purpose here is to assess risks in depending upon FBG projections by determining where they are strong and where they are weak. This analysis only concerns 2005 projections, and as such the sample sizes are necessarily small. The comparisons made here are to the alternative method of using prior year's performance; this is not a competitive analysis of FBG to other sites' projections.
This is the third analysis of this sort I've posted. You'll find the one for RB's here and the one for WR's here. You might want to look at these threads for further explanation of some of the things mentioned here.
CONCLUSIONS:
(1) FBG was more successful than prior year's performance at identifying QB's likely to wind up in the top 30, especially for those ranked in the range 18-30; however, prior year's ranking was a slightly better predictor of relative rank within the top 18.
(2) As with the RB and WR rankings, FBG QB rankings represented less downside risk and more upside potential than prior year's rank.
(3) Prior year's rank and FBG projections were equally good predictors of whether a QB projected to finish in the top 10 actually did so--both identified 4 of them. Taken together, they identified 6.
(4) Prior year's performance may be useful as an aid to making small adjustments within the FBG projected top 30; however, in 2005 neither method was able to distinguish well among the top 15 QB's.
METHODS & DETAILED FINDINGS
For the purposes of this analysis, the top 30 ranked QB's were considered. FBG projections, last year's player stats, and this year's player stats through week 16 were passed through the following scoring system:
1pt/5 yds rushing + 1pt/5 yds receiving + 1 PPR + 1pt/10 yds passing + 6 pts per TD passing or rushing or receiving - 3 pts per interception or fumble
As with the previous analyses, two primary methods of assessing the predictive value of these projections were used. The first considered difference in ranks.
For those projected by FBG to fall in the top 30 QB's, the correlation between predicted rank and actual rank was 0.29. This is usually interpreted to mean that FBG ranking accounted for 9% of the variation in actual rank. The RMS difference in ranks over the top 30 for these projections was 13.6.
For those who fell in the top 30 QB's in the 2004 season, the correlation coefficient was 0.58. 2004 performance thus accounted for roughly 33% of variability in 2005 performance. The RMS difference in ranks over the top 30 for these projections was 15.8.
So how could the prior year's QB rankings correlate better with this year's while at the same time exhibiting a greater average error in the ultimate rank? The answer to that question is, believe it or not, AJ Feely.
The correlation coefficients primarily measure predictive value in relative rank--that is, among those predicted to be in the top 30, this coefficient tells you to what degree QB A being ranked above QB B indicates that QB A is likely to wind up outperforming QB B. AJ Feely, as it happens, was QB30 in 2004 and QB72 this year. Thus, prior year's rankings were quite accurate in predicting he would be below the others on the list, but the big difference in raw rank skews the RMS measure. Without him, the RMS difference in ranks is 13.9--virtually the same as the FBG rankings.
Here are graphical representations of the ranking errors for both the FBG rankings and prior year's performance. The red line indicates what the graph would look like if the ranking were perfect; cases where the blue line (actual) are above the red are those in which the player was overvalued; cases where the blue line is below are those in which the player was undervalued.
FBG projection versus actual rank
Prior year's rank versus actual
In an effort to understand how the two correlations could have been so different, the sample group was further divided into three segments--top 10, middle 10, bottom 10--to examine correlations within these subsamples. While the numbers thus obtained are probably not tremendously useful in themselves, this process can facilitate side-by-side comparisons of which regions of the rankings were responsible for the overall difference.
top 10: FBG -0.27, prior year -0.25
This reflects turnover within the top 10 QB's in 2005. Those near the bottom of the top 10 last year tended to perform better than those near the top, and of course FBG's rankings are not independent of prior year's performance.
middle 10: FBG 0.38, prior year 0.62
This reflects the degree to which relative rank in 2004 held up in 2005 for QB's below the top tier. FBG's rankings included several in the bottom of this segment who performed significantly better than projected.
bottom 10: FBG -0.30, prior year 0.57
The high correlation here for prior year is not much of a virtue, in fact. FBG rankings included several in the range 20-30 who performed well this season--but near the bottom of this segment, as it happened; prior year included such worthies as the aforementioned AJ Feely. In this case, the negative correlation can't really be held against the FBG rankings, as they were able to identify several QB's (E. Manning, Dilfer, Frerotte) who were barely on the prior year's radar. The negative correlation here is primarily a consequence of these three being ranked near the bottom of the top 30 and outperforming those immediately above.
In all, as relative rankings FBG was less successful than was prior year within the subsamples they put in the top 30, but FBG was better able to identify QB's who belonged there. FBG's top 30 included 25 members of the 2005 top 30 and all of the top 10, while the prior year's included only 22 members of the top 30 and 9 of the top 10. These differences may seem inconsequential until you realize that, in each case, the difference is 10% of the target group's size. It would be overstating matters to say that the difference is dramatic, but it is there. FBG's primary advantage over prior year's rank arose from QB18 or so on down. Above that, the two were equally successful.
These results should be kept in mind when considering the other major measure used here: consequential points difference. The meaning of this measure is explained in both of the other threads, so I'll simply repeat that this is a way of assessing the relative gain / loss in production associated with a ranking error.
For FBG rankings, the RMS consequential difference in ranks was 79.3%, whereas it was 78.4% for prior year--essentially the same. These numbers were also essentially the same over the top 10 projected QB's, over the top 15, and in the range 15-30.
To aid in a detailed discussion of what gave rise to these similarities, here are graphical representations of the consequential points differences for both FBG and prior year.
FBG: consequential difference in ranks
Prior year: consequential difference in ranks
If we leave aside the size and simply consider the direction of error in these cases, FBG seems superior to prior year. FBG underestimated value in 14 of the 30 cases, whereas prior year did so in only 9. Of course, not all the FBG underestimations are true indications of upside value. If you got Eli Manning at QB27, you got outstanding value; on the other hand, Jake Plummer at QB11 doesn't represent much in the way of value, since he was QB4 in 2004 and in many cases probably didn't last long enough for that value to be realized.
A quick look at the relative sizes of these errors is probably in order. As in the WR analysis, errors were considered to fall into 4 categories based on the RMS points differences: significantly undervalued, somewhat undervalued, somewhat overvalued, significantly overvalued. Under these breakdowns, FBG comes out ahead again. For FBG, 4 of the top 30 were significantly undervalued, 10 somewhat undervalued, 10 somewhat overvalued, and 6 signficantly overvalued. For prior year, the equivalent numbers were 1, 8, 13, and 8. Prior year's rank was a very tiny bit more likely to get near the actual ranking in 2005, but it was much more likely to overestimate value than FBG was. Over the top 17, the two methods of projection performed virtually identically under this measure.
So what we got in the 2005 QB projections was a group of players who were more likely to score near the top, but those projections were less good than prior year's performance at predicting the rankings within subsamples of that group. However, FBG projections in the top half, roughly, of the region considered were no better than prior year's performance, and perhaps slightly worse at assessing relative rank within this group. These facts do have implications for how the projections are used in a draft context.
The consensus among many FFL players is not to waste an early pick on a QB, since good values are available later and early-round QB's represent more risk than early-round RB's. 2005 was perhaps more of an object lesson in this truism than past seasons have been, but it seems likely that, whatever means you choose, QB1-QB10 provide for the most part roughly equal chances of landing a QB near the top. This year, in fact, QB5-QB15 would have been even better, but this seems likely to have been a feature of 2005 more than something that can be counted on going forward.
This is not meant as criticism of FBG at all. I've used their projections for 4 years now with good success. The purpose here is to assess risks in depending upon FBG projections by determining where they are strong and where they are weak. This analysis only concerns 2005 projections, and as such the sample sizes are necessarily small. The comparisons made here are to the alternative method of using prior year's performance; this is not a competitive analysis of FBG to other sites' projections.
This is the third analysis of this sort I've posted. You'll find the one for RB's here and the one for WR's here. You might want to look at these threads for further explanation of some of the things mentioned here.
CONCLUSIONS:
(1) FBG was more successful than prior year's performance at identifying QB's likely to wind up in the top 30, especially for those ranked in the range 18-30; however, prior year's ranking was a slightly better predictor of relative rank within the top 18.
(2) As with the RB and WR rankings, FBG QB rankings represented less downside risk and more upside potential than prior year's rank.
(3) Prior year's rank and FBG projections were equally good predictors of whether a QB projected to finish in the top 10 actually did so--both identified 4 of them. Taken together, they identified 6.
(4) Prior year's performance may be useful as an aid to making small adjustments within the FBG projected top 30; however, in 2005 neither method was able to distinguish well among the top 15 QB's.
METHODS & DETAILED FINDINGS
For the purposes of this analysis, the top 30 ranked QB's were considered. FBG projections, last year's player stats, and this year's player stats through week 16 were passed through the following scoring system:
1pt/5 yds rushing + 1pt/5 yds receiving + 1 PPR + 1pt/10 yds passing + 6 pts per TD passing or rushing or receiving - 3 pts per interception or fumble
As with the previous analyses, two primary methods of assessing the predictive value of these projections were used. The first considered difference in ranks.
For those projected by FBG to fall in the top 30 QB's, the correlation between predicted rank and actual rank was 0.29. This is usually interpreted to mean that FBG ranking accounted for 9% of the variation in actual rank. The RMS difference in ranks over the top 30 for these projections was 13.6.
For those who fell in the top 30 QB's in the 2004 season, the correlation coefficient was 0.58. 2004 performance thus accounted for roughly 33% of variability in 2005 performance. The RMS difference in ranks over the top 30 for these projections was 15.8.
So how could the prior year's QB rankings correlate better with this year's while at the same time exhibiting a greater average error in the ultimate rank? The answer to that question is, believe it or not, AJ Feely.
The correlation coefficients primarily measure predictive value in relative rank--that is, among those predicted to be in the top 30, this coefficient tells you to what degree QB A being ranked above QB B indicates that QB A is likely to wind up outperforming QB B. AJ Feely, as it happens, was QB30 in 2004 and QB72 this year. Thus, prior year's rankings were quite accurate in predicting he would be below the others on the list, but the big difference in raw rank skews the RMS measure. Without him, the RMS difference in ranks is 13.9--virtually the same as the FBG rankings.
Here are graphical representations of the ranking errors for both the FBG rankings and prior year's performance. The red line indicates what the graph would look like if the ranking were perfect; cases where the blue line (actual) are above the red are those in which the player was overvalued; cases where the blue line is below are those in which the player was undervalued.
FBG projection versus actual rank
Prior year's rank versus actual
In an effort to understand how the two correlations could have been so different, the sample group was further divided into three segments--top 10, middle 10, bottom 10--to examine correlations within these subsamples. While the numbers thus obtained are probably not tremendously useful in themselves, this process can facilitate side-by-side comparisons of which regions of the rankings were responsible for the overall difference.
top 10: FBG -0.27, prior year -0.25
This reflects turnover within the top 10 QB's in 2005. Those near the bottom of the top 10 last year tended to perform better than those near the top, and of course FBG's rankings are not independent of prior year's performance.
middle 10: FBG 0.38, prior year 0.62
This reflects the degree to which relative rank in 2004 held up in 2005 for QB's below the top tier. FBG's rankings included several in the bottom of this segment who performed significantly better than projected.
bottom 10: FBG -0.30, prior year 0.57
The high correlation here for prior year is not much of a virtue, in fact. FBG rankings included several in the range 20-30 who performed well this season--but near the bottom of this segment, as it happened; prior year included such worthies as the aforementioned AJ Feely. In this case, the negative correlation can't really be held against the FBG rankings, as they were able to identify several QB's (E. Manning, Dilfer, Frerotte) who were barely on the prior year's radar. The negative correlation here is primarily a consequence of these three being ranked near the bottom of the top 30 and outperforming those immediately above.
In all, as relative rankings FBG was less successful than was prior year within the subsamples they put in the top 30, but FBG was better able to identify QB's who belonged there. FBG's top 30 included 25 members of the 2005 top 30 and all of the top 10, while the prior year's included only 22 members of the top 30 and 9 of the top 10. These differences may seem inconsequential until you realize that, in each case, the difference is 10% of the target group's size. It would be overstating matters to say that the difference is dramatic, but it is there. FBG's primary advantage over prior year's rank arose from QB18 or so on down. Above that, the two were equally successful.
These results should be kept in mind when considering the other major measure used here: consequential points difference. The meaning of this measure is explained in both of the other threads, so I'll simply repeat that this is a way of assessing the relative gain / loss in production associated with a ranking error.
For FBG rankings, the RMS consequential difference in ranks was 79.3%, whereas it was 78.4% for prior year--essentially the same. These numbers were also essentially the same over the top 10 projected QB's, over the top 15, and in the range 15-30.
To aid in a detailed discussion of what gave rise to these similarities, here are graphical representations of the consequential points differences for both FBG and prior year.
FBG: consequential difference in ranks
Prior year: consequential difference in ranks
If we leave aside the size and simply consider the direction of error in these cases, FBG seems superior to prior year. FBG underestimated value in 14 of the 30 cases, whereas prior year did so in only 9. Of course, not all the FBG underestimations are true indications of upside value. If you got Eli Manning at QB27, you got outstanding value; on the other hand, Jake Plummer at QB11 doesn't represent much in the way of value, since he was QB4 in 2004 and in many cases probably didn't last long enough for that value to be realized.
A quick look at the relative sizes of these errors is probably in order. As in the WR analysis, errors were considered to fall into 4 categories based on the RMS points differences: significantly undervalued, somewhat undervalued, somewhat overvalued, significantly overvalued. Under these breakdowns, FBG comes out ahead again. For FBG, 4 of the top 30 were significantly undervalued, 10 somewhat undervalued, 10 somewhat overvalued, and 6 signficantly overvalued. For prior year, the equivalent numbers were 1, 8, 13, and 8. Prior year's rank was a very tiny bit more likely to get near the actual ranking in 2005, but it was much more likely to overestimate value than FBG was. Over the top 17, the two methods of projection performed virtually identically under this measure.
So what we got in the 2005 QB projections was a group of players who were more likely to score near the top, but those projections were less good than prior year's performance at predicting the rankings within subsamples of that group. However, FBG projections in the top half, roughly, of the region considered were no better than prior year's performance, and perhaps slightly worse at assessing relative rank within this group. These facts do have implications for how the projections are used in a draft context.
The consensus among many FFL players is not to waste an early pick on a QB, since good values are available later and early-round QB's represent more risk than early-round RB's. 2005 was perhaps more of an object lesson in this truism than past seasons have been, but it seems likely that, whatever means you choose, QB1-QB10 provide for the most part roughly equal chances of landing a QB near the top. This year, in fact, QB5-QB15 would have been even better, but this seems likely to have been a feature of 2005 more than something that can be counted on going forward.
Last edited by a moderator:
