The Rule of 26-27-60 (1 Viewer)

[JDub15]

There's a plausible explanation for why coaches would go with a sub-optimal strategy, though. Their continued employment could very well hinge on it. Just look at all the flak Belichick took when he made what was statistically the right play by going for it on 4th down against Indy. If that was Lovie Smith, he'd have been fired before the week was out. I can't think of a single plausible explanation as to why a scout would go with a sub-optimal strategy, though.

Why wouldn't the same effect occur for scouts? Their recommendations to the front office are very visible and if they go around touting unorthodox players over 6'4" QBs with a laser arm their mistakes are going to be much more noticeable than your average scout. Their decisions are still subject to variance in a similar way that going for it on 4th and 2 is- except that there will probably be several years- not several games or several plays- before their more correct decisions will be paying off.

I have to somewhat agree with this. I would use the same analogy as a stock broker - if you recommend for your client to buy Microsoft and Microsoft fails, hey you were recommending a consensus top stock. If you had recommended some esoteric software company that no one had heard of and that stock fails - well now you are an idiot and you get fired. I think the same thing happens in scouting to some extent - the consensus starts to creep in and scouts go with the "safe" player even if it's the wrong player. How the hell did Jason Smith of Baylor wind up being the number 2 overall pick in 2009 draft?

 

[Ray_T]

The 26 threshold for the Wonderlic score is explained, and the 60% completion percentage cutoff seems reasonable, since it is a round number and is sometimes used as a barometer for good completion percentage.But why is 27 starts the specific threshold for that element of the criteria? Unless that can be explained, it seems like it might be curve fitting to use that specific number. Is it the case that college teams nowadays tend to play an average of 13.5 games per year, meaning it is equivalent to 2 years of starting? But what about teams that don't make conference championship and/or bowl games? Wouldn't they have fewer than 27 games over 2 seasons? It seems like a number that will generally require college QBs to start games in 3 or more college seasons. I'm sure that is a great indicator of possible NFL success, since it's intuitive that being good enough to start in 3+ seasons is a reasonable indicator of talent/skill... and a larger sample size of 3+ seasons is always better than a smaller sample size, from an evaluation standpoint. But using an odd number like 27 seems fishy to me.

my thought is that if the team is not making bowl appearances that will factor in. If he were a better QB or was playing on a better team he'd have those bowl and/or playoff appearances. I'm sure that is part of what goes into the formula.

 

[JDub15]

i dont care about 26 and 27. 60 would concern me. if you cant hit wrs that are open by 5 yards 2/3 of the time then how are you going to hit them when theyre open by 2 feet?

Even that has to be taken in context. Jay Cutler couldn't hit his WRs 60% of the time... but I doubt he ever saw a receiver at Vanderbilt who was open by 5 yards. He's done just fine hitting his receivers at the NFL level, though.

I remember David Lewin (or whoever wrote the LCF piece for FO/ESPN Insider at the time) actually endorsed Cutler on this basis. Essentially he said in that situation you have to look at what the guy who came before him did with the same offense / relative level of talent and found that Cutler was vastly superior to his predecessor and thus his <60% accuracy in college wasn't indicative of his inability to play at an NFL level.That being said his poor body language and petulant-QB-ness certainly were there in college The FO guys used the exact same argument to bet against Matthew Stafford becoming a successful NFL QB - under 60% completion percentage and his predecessor (some 4th rounder who washed out of the NFL) actually had better stats in the same offensive system.

 

[Tau837]

We're going to have to agree to disagree, however, I do agree that the SI article didn't employ proper statistical analysis.

I don't really understand the disconnect you are having here. The first article on Drew Brees linked above talks about just Drew Brees. It doesn't attempt to make any kind of general statement/proposition to apply to all QBs, or to all QBs who meet a specific criteria. That's the difference. Why is this hard to understand?

You're completely missing the point and I'm not the one that's keeping this subject alive--I said we'll have to agree to disagree. SI put together a single sample of data that they arbitrarily determined, crunched it and offered a projection. Chase put together a single sample of data using criteria that he arbitrarily determined, crunched it and offered a projection. He's using more mechanics, but he's doing the same thing that SI did. He's arguing that SI should use one set of data, come up with a hypothesis and test that hypothesis on a second set of data. OK, fine, I'd like to see that too, but why is he criticizing SI's study as only "useful in retrospect" when he's also coming to a conclusion using only a single set of data. The point here is that if you use a single set of historical data to come to a conclusion, that conclusion is only applicable to that set of data whether you use more rigorous mathematical mechanics or whether you're coming up for a rule to grade QBs or a floor/likely performance for a QB. Again, I'm not criticizing FBGs, its the nature of football and the limited sample sizes we have to work with. FBGs handles it their way, Football Outsiders attempts to overcome that by DVOA/DYAR metrics that look at individual plays, etc. We all live with it.

You really don't see a difference with coming up with a "rule" that can be applied to all QBs and looking at relevant past history when considering a single player's outlook for this year? "The Rule of 26-27-60" is offered in the article as a predictive tool that can be applied to all future QB prospects. Chase's article about Brees has nothing to do with projecting other QBs.SI's article is suggesting a predictive rule, which requires intellectually honest validation to determine if it is valid or not. Chase isn't doing anything of the sort in his article. He's just talking about Brees's projection for this year. Furthermore, Chase provided a comprehensive sample of data to support his position, while the SI article conveniently left out the names of QBs that don't make their "rule" look so good, as pointed out earlier in this thread.

 

[TWP]

There's a plausible explanation for why coaches would go with a sub-optimal strategy, though. Their continued employment could very well hinge on it. Just look at all the flak Belichick took when he made what was statistically the right play by going for it on 4th down against Indy. If that was Lovie Smith, he'd have been fired before the week was out. I can't think of a single plausible explanation as to why a scout would go with a sub-optimal strategy, though.

Why wouldn't the same effect occur for scouts? Their recommendations to the front office are very visible and if they go around touting unorthodox players over 6'4" QBs with a laser arm their mistakes are going to be much more noticeable than your average scout. Their decisions are still subject to variance in a similar way that going for it on 4th and 2 is- except that there will probably be several years- not several games or several plays- before their more correct decisions will be paying off.

I have to somewhat agree with this. I would use the same analogy as a stock broker - if you recommend for your client to buy Microsoft and Microsoft fails, hey you were recommending a consensus top stock. If you had recommended some esoteric software company that no one had heard of and that stock fails - well now you are an idiot and you get fired. I think the same thing happens in scouting to some extent - the consensus starts to creep in and scouts go with the "safe" player even if it's the wrong player. How the hell did Jason Smith of Baylor wind up being the number 2 overall pick in 2009 draft?

Yup -- agree. I think the baseball comparison is a great one. In that case, the As did something different and IMMEDIATELY saw a drastic improvement in performance. They play a ton of games and everyone gets into a ton of clear statistical situations in every game. In football, you might not see your results improve for years. And you get one or two key injuries (which happen all the time) and you get no improvement at all, due purely to luck. Due to the number of games and availability of clean statistics, baseball is going to be a much more efficient system than football will be.

 

[Futz]

Matt Stafford only completed 58% of his college passes.

 

[JDub15]

I still prefer the LCF developed by Football Outsiders since their model adds the requirement that a QB be drafted in the first or second round. The SI study apparently seeks to replace the task of scouting whereas the LCF doesn't apply to QBs drafted in the 3rd round or later. IMO the LCF is more of a secondary filter beyond draft status that helps you determine if scouts had enough data to evaluate a QB or did a single team make a mistake in drafting a player off of a limited data set or measurables or whatnot. The LCF also doesn't have specific benchmarks, but is of a "more is better" nature.

Also, from what I understand, the LCF was developed by actual stats people with actual statistical backgrounds who did actual stat-like things such as backtesting for predictive power.With that said, I'm always, always leery of anything like that designed to predict inefficiencies in the NFL market. Not because I believe that inefficiencies don't exist- I'm sure there are dozens of them- but because I think the NFL is such a massive high-stakes business that as soon as an inefficiency is discovered, the market is going to correct for it pretty darn quickly. If Lewin really stumbled onto something when he discovered that NFL scouts were underrating completion percentage and number of starts, then I'm sure NFL scouting departments would have discussed it and corrected for it already, rendering the LCF useless going forward.

Great posting. I think FO has already come out and acknowledged this. LCF is completely dependent on the scouting departments separating out the flukey system QB's from legit NFL prospects, hence its requirement that the player must be drafted in the first two rounds. I.E., LCF was should be used to sort out the few QBs at the top of the board.If scouting departments are factoring LCF into their projections, then it invalidates the entire system.

 

[Tau837]

The 26 threshold for the Wonderlic score is explained, and the 60% completion percentage cutoff seems reasonable, since it is a round number and is sometimes used as a barometer for good completion percentage.But why is 27 starts the specific threshold for that element of the criteria? Unless that can be explained, it seems like it might be curve fitting to use that specific number. Is it the case that college teams nowadays tend to play an average of 13.5 games per year, meaning it is equivalent to 2 years of starting? But what about teams that don't make conference championship and/or bowl games? Wouldn't they have fewer than 27 games over 2 seasons? It seems like a number that will generally require college QBs to start games in 3 or more college seasons. I'm sure that is a great indicator of possible NFL success, since it's intuitive that being good enough to start in 3+ seasons is a reasonable indicator of talent/skill... and a larger sample size of 3+ seasons is always better than a smaller sample size, from an evaluation standpoint. But using an odd number like 27 seems fishy to me.

my thought is that if the team is not making bowl appearances that will factor in. If he were a better QB or was playing on a better team he'd have those bowl and/or playoff appearances. I'm sure that is part of what goes into the formula.

As a N.C. State fan, I can tell you that Russell Wilson has been the best QB in the ACC for the past 2 years running, yet State is only 11-14 over that time frame, with just one bowl appearance. Wilson was 4th in the NCAA in TD passes last year, but State's defense allowed an average of 35.3 points per game against opponents other than Murray State and Gardner-Webb. Wilson doesn't play defense. What more is he supposed to do to earn another bowl game? Furthermore, even if State did play another bowl game, that would have only been a max of 26 starts, so he would have also had to have gotten State into a conference championship game to qualify for this criteria without playing 3 seasons.Anyway, Wilson isn't a great NFL prospect due to his size, but he is just one example that illustrates what I was saying in the post you responded to.

 

[SSOG]

There's a plausible explanation for why coaches would go with a sub-optimal strategy, though. Their continued employment could very well hinge on it. Just look at all the flak Belichick took when he made what was statistically the right play by going for it on 4th down against Indy. If that was Lovie Smith, he'd have been fired before the week was out. I can't think of a single plausible explanation as to why a scout would go with a sub-optimal strategy, though.

Why wouldn't the same effect occur for scouts? Their recommendations to the front office are very visible and if they go around touting unorthodox players over 6'4" QBs with a laser arm their mistakes are going to be much more noticeable than your average scout. Their decisions are still subject to variance in a similar way that going for it on 4th and 2 is- except that there will probably be several years- not several games or several plays- before their more correct decisions will be paying off.

Like I said, the fact that all these studies keep finding that the draft is a relatively efficient marketplace suggests that that's not the case. And the scout isn't the only link in the chain, either. There's also the director of college scouting, the GM, and the head coach- the guys who take the scouts recommendations, filter them, and make a decision. If they know that the scouts overrate guys without experience or underrate guys with good completions percentages, what reason would they have for not factoring that into their decision-making process and compensating? Maybe with a top-10 pick the scrutiny is so intense that you'd rather go with the popular mistake rather than risk the public ridicule, but I doubt any GM has ever lost his job because the owner or the media thought he took the wrong 2nd round QB.

 

[Faust]

i dont care about 26 and 27. 60 would concern me. if you cant hit wrs that are open by 5 yards 2/3 of the time then how are you going to hit them when theyre open by 2 feet?

Even that has to be taken in context. Jay Cutler couldn't hit his WRs 60% of the time... but I doubt he ever saw a receiver at Vanderbilt who was open by 5 yards. He's done just fine hitting his receivers at the NFL level, though.

I remember David Lewin (or whoever wrote the LCF piece for FO/ESPN Insider at the time) actually endorsed Cutler on this basis. Essentially he said in that situation you have to look at what the guy who came before him did with the same offense / relative level of talent and found that Cutler was vastly superior to his predecessor and thus his <60% accuracy in college wasn't indicative of his inability to play at an NFL level.That being said his poor body language and petulant-QB-ness certainly were there in college The FO guys used the exact same argument to bet against Matthew Stafford becoming a successful NFL QB - under 60% completion percentage and his predecessor (some 4th rounder who washed out of the NFL) actually had better stats in the same offensive system.

Matt Stafford's predecessor was Joe Tereshinski III who was never drafted in the NFL. Tereshinski's predecessor was D. J. Shockley who was drafted in the 7th round - is that who you were thinking of?

 

[SSOG]

The 26 threshold for the Wonderlic score is explained, and the 60% completion percentage cutoff seems reasonable, since it is a round number and is sometimes used as a barometer for good completion percentage.But why is 27 starts the specific threshold for that element of the criteria? Unless that can be explained, it seems like it might be curve fitting to use that specific number. Is it the case that college teams nowadays tend to play an average of 13.5 games per year, meaning it is equivalent to 2 years of starting? But what about teams that don't make conference championship and/or bowl games? Wouldn't they have fewer than 27 games over 2 seasons? It seems like a number that will generally require college QBs to start games in 3 or more college seasons. I'm sure that is a great indicator of possible NFL success, since it's intuitive that being good enough to start in 3+ seasons is a reasonable indicator of talent/skill... and a larger sample size of 3+ seasons is always better than a smaller sample size, from an evaluation standpoint. But using an odd number like 27 seems fishy to me.

my thought is that if the team is not making bowl appearances that will factor in. If he were a better QB or was playing on a better team he'd have those bowl and/or playoff appearances. I'm sure that is part of what goes into the formula.

I think calling it a "formula" and speculating that there was some rational thought about what the breakpoints should be is giving it far too much credit. It's a classic case of curve-fitting. Lopez looked at the data and said "okay, where can I draw the line that leaves most of the good QBs on one side and most of the bad QBs on the other side". I'd be willing to be the real reason why the cutoff is 27 games is because there are either some stud QBs out there with 27 starts in college... or because there are some total busts out there with 26 starts in college.

 

[Maurile Tremblay]

Like I said, the fact that all these studies keep finding that the draft is a relatively efficient marketplace suggests that that's not the case.

What are "all these studies"? The only paper on the NFL draft I'm familiar with is Massey-Thaler, which concludes the opposite.Also, one glaring example of inefficiency that I think should be obvious to all — yet it persists — is the insane discount rate applied to future picks. In the NFL draft marketplace, a second rounder this year is worth roughly two second-rounders (or one first-rounder) next year. How can that possibly make sense?

 

[Chase Stuart]

Like I said, the fact that all these studies keep finding that the draft is a relatively efficient marketplace suggests that that's not the case.

What are "all these studies"? The only paper on the NFL draft I'm familiar with is Massey-Thaler, which concludes the opposite.Also, one glaring example of inefficiency that I think should be obvious to all — yet it persists — is the insane discount rate applied to future picks. In the NFL draft marketplace, a second rounder this year is worth roughly two second-rounders (or one first-rounder) next year. How can that possibly make sense?

I won't claim this to be the be-all, end-all of discussion, but I did a comprehensive study of this and totally agree with SSOG: http://www.pro-football-reference.com/blog/?p=527

 

[JDub15]

i dont care about 26 and 27. 60 would concern me. if you cant hit wrs that are open by 5 yards 2/3 of the time then how are you going to hit them when theyre open by 2 feet?

Even that has to be taken in context. Jay Cutler couldn't hit his WRs 60% of the time... but I doubt he ever saw a receiver at Vanderbilt who was open by 5 yards. He's done just fine hitting his receivers at the NFL level, though.

I remember David Lewin (or whoever wrote the LCF piece for FO/ESPN Insider at the time) actually endorsed Cutler on this basis. Essentially he said in that situation you have to look at what the guy who came before him did with the same offense / relative level of talent and found that Cutler was vastly superior to his predecessor and thus his <60% accuracy in college wasn't indicative of his inability to play at an NFL level.That being said his poor body language and petulant-QB-ness certainly were there in college The FO guys used the exact same argument to bet against Matthew Stafford becoming a successful NFL QB - under 60% completion percentage and his predecessor (some 4th rounder who washed out of the NFL) actually had better stats in the same offensive system.

Matt Stafford's predecessor was Joe Tereshinski III who was never drafted in the NFL. Tereshinski's predecessor was D. J. Shockley who was drafted in the 7th round - is that who you were thinking of?

This is from the FO article:"Stafford was directly preceded at Georgia by the recently retired David Greene; both spent their entire college careers under head coach Mark Richt in similar offensive systems. Stafford's college numbers are actually worse than Greene's, with the latter completing 59 percent of his passes and averaging 8.01 yards per attempt to Stafford's 7.83. If Stafford was really a star in the making, wouldn't he have put up better numbers, in the same system, than a guy who washed out of the NFL without taking a professional snap? If it was our $25 million guaranteed, the answer would need to be yes."

 

[Faust]

i dont care about 26 and 27. 60 would concern me. if you cant hit wrs that are open by 5 yards 2/3 of the time then how are you going to hit them when theyre open by 2 feet?

Even that has to be taken in context. Jay Cutler couldn't hit his WRs 60% of the time... but I doubt he ever saw a receiver at Vanderbilt who was open by 5 yards. He's done just fine hitting his receivers at the NFL level, though.

I remember David Lewin (or whoever wrote the LCF piece for FO/ESPN Insider at the time) actually endorsed Cutler on this basis. Essentially he said in that situation you have to look at what the guy who came before him did with the same offense / relative level of talent and found that Cutler was vastly superior to his predecessor and thus his <60% accuracy in college wasn't indicative of his inability to play at an NFL level.That being said his poor body language and petulant-QB-ness certainly were there in college The FO guys used the exact same argument to bet against Matthew Stafford becoming a successful NFL QB - under 60% completion percentage and his predecessor (some 4th rounder who washed out of the NFL) actually had better stats in the same offensive system.

Matt Stafford's predecessor was Joe Tereshinski III who was never drafted in the NFL. Tereshinski's predecessor was D. J. Shockley who was drafted in the 7th round - is that who you were thinking of?

This is from the FO article:"Stafford was directly preceded at Georgia by the recently retired David Greene; both spent their entire college careers under head coach Mark Richt in similar offensive systems. Stafford's college numbers are actually worse than Greene's, with the latter completing 59 percent of his passes and averaging 8.01 yards per attempt to Stafford's 7.83. If Stafford was really a star in the making, wouldn't he have put up better numbers, in the same system, than a guy who washed out of the NFL without taking a professional snap? If it was our $25 million guaranteed, the answer would need to be yes."

Greene was before D.J. Shockley, so they are not making an apples to apples comparison, as the talent around each QB would be different among other factors.

 

[Donnybrook]

This thread has to have some "what about?s", so I'll start:

Wonderlic

Matt Leinart 35

Brett Favre 22

Dan Marino 16

I looked up Marino's college stats. He had a 57.6 completion percentage. http://www.dolphinsinfo.com/marinop.htm#college Edit to add: Favre's college completion percentage was 53%. http://www.officialbrettfavre.com/bio/

So #1 and #2 QBs in NFL TDs don't fit the criteria.

 

[SSOG]

Like I said, the fact that all these studies keep finding that the draft is a relatively efficient marketplace suggests that that's not the case.

What are "all these studies"? The only paper on the NFL draft I'm familiar with is Massey-Thaler, which concludes the opposite.Also, one glaring example of inefficiency that I think should be obvious to all — yet it persists — is the insane discount rate applied to future picks. In the NFL draft marketplace, a second rounder this year is worth roughly two second-rounders (or one first-rounder) next year. How can that possibly make sense?

Chase's study was one. I've read several studies that have all concluded that the 1st guy taken at any position outperformed the 2nd guy taken at that position, who in turn outperformed the 3rd guy taken at that position, and so on down the line. There might be inefficiencies in how front offices VALUE their draft picks, but I think they're pretty efficient in the way that they EXECUTE those draft picks.Also, the discount rate applied to future picks is super easy to explain. That's another one that boils down to job security. Discounting future picks might not be the optimal long-run strategy for the franchise, but it can still be the optimal long-run strategy for the guy executing the picks on the franchise's behalf.

 

[az_prof]

I see a lot of people dumping on this study....and for good reasons.But I also see that the basic premises are essentially correct. More starts = more experiance = better prepared = more success. Higher wonderlic = better able to quickly assimilate data = better able to read defenses and adjust to complex schemes = more success. Higher conmpletion % = more accurate passer = more success.In other news, the sky is blue and grass is usually some shade of green....but there are exceptions to both.

My problem with this is the wonderlic part. Have you guys ever read some of the questions? And there is such a large range on the wonderlic among the NFL's greatest QBs. If the wonderlic could test football intelligence that would be different. I find it hard to take seriously anything that uses the wonderlic as a data point.

 

[AB in DC]

I still prefer the LCF developed by Football Outsiders since their model adds the requirement that a QB be drafted in the first or second round. The SI study apparently seeks to replace the task of scouting whereas the LCF doesn't apply to QBs drafted in the 3rd round or later. IMO the LCF is more of a secondary filter beyond draft status that helps you determine if scouts had enough data to evaluate a QB or did a single team make a mistake in drafting a player off of a limited data set or measurables or whatnot. The LCF also doesn't have specific benchmarks, but is of a "more is better" nature.

Also, from what I understand, the LCF was developed by actual stats people with actual statistical backgrounds who did actual stat-like things such as backtesting for predictive power.With that said, I'm always, always leery of anything like that designed to predict inefficiencies in the NFL market. Not because I believe that inefficiencies don't exist- I'm sure there are dozens of them- but because I think the NFL is such a massive high-stakes business that as soon as an inefficiency is discovered, the market is going to correct for it pretty darn quickly. If Lewin really stumbled onto something when he discovered that NFL scouts were underrating completion percentage and number of starts, then I'm sure NFL scouting departments would have discussed it and corrected for it already, rendering the LCF useless going forward.

Great posting. I think FO has already come out and acknowledged this. LCF is completely dependent on the scouting departments separating out the flukey system QB's from legit NFL prospects, hence its requirement that the player must be drafted in the first two rounds. I.E., LCF was should be used to sort out the few QBs at the top of the board.If scouting departments are factoring LCF into their projections, then it invalidates the entire system.

I think Lewin himself made this point when Kevin Kolb was drafted -- Kolb had LCF-favorable numbers, but he's a QB that might not have been drafted in the top two rounds before this kind of research became well-known.You could almost say that guys like Kolb will be the test of whether LCF is still relevant.

 

[Maurile Tremblay]

Like I said, the fact that all these studies keep finding that the draft is a relatively efficient marketplace suggests that that's not the case.

What are "all these studies"? The only paper on the NFL draft I'm familiar with is Massey-Thaler, which concludes the opposite.Also, one glaring example of inefficiency that I think should be obvious to all — yet it persists — is the insane discount rate applied to future picks. In the NFL draft marketplace, a second rounder this year is worth roughly two second-rounders (or one first-rounder) next year. How can that possibly make sense?

Chase's study was one. I've read several studies that have all concluded that the 1st guy taken at any position outperformed the 2nd guy taken at that position, who in turn outperformed the 3rd guy taken at that position, and so on down the line. There might be inefficiencies in how front offices VALUE their draft picks, but I think they're pretty efficient in the way that they EXECUTE those draft picks.

How likely is the 1st guy taken to outperform the 2nd guy taken at that position? I read somewhere that it's about 52% — which leaves a lot of room for plenty of inefficiencies of all sorts to exist in scouting.

Also, the discount rate applied to future picks is super easy to explain. That's another one that boils down to job security. Discounting future picks might not be the optimal long-run strategy for the franchise, but it can still be the optimal long-run strategy for the guy executing the picks on the franchise's behalf.

Why wouldn't scouts be subject to the same kinds of distorting pressures? "Nobody ever got fired for drafting guy with these combine numbers," or something like that.