What's new
Fantasy Football - Footballguys Forums

This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

New QB rating system set to debut? (1 Viewer)

Faust

MVP
How to identify NFL's best quarterbacks

By Mike Sando

The late Don Smith never claimed his passer-rating formula was perfect.

Quite the opposite, in fact.

"Some people call it a quarterback rating system, but that really is not what it is," Smith told me during a 2002 interview. "It’s simply a passing statistic."

I've actually defended Smith's rating system because the quarterbacks with the highest ratings -- Tom Brady, Philip Rivers and Aaron Rodgers led the way last season -- usually are the best quarterbacks. But there's so much more to quarterbacking than passing stats for touchdowns, interceptions, attempts, completions and yardage.

Game situations should count for something, and now they do.

With input from football people, including ESPN analyst Trent Dilfer, our statistical analysts have developed a 100-point ratings scale for quarterbacks taking into account advanced stats, game situations and relevant non-passing stats, including fumbles and sacks, to evaluate quarterbacks far more thoroughly. The methodology is complex -- one of the formula's key algorithms spans some 10,000 lines -- but the resulting "Total Quarterback Rating" (QBR for short) beats the old passer rating in every conceivable fashion. The ratings scale will debut this season.

I've been bugging the Stats & Information team for a sneak peak ever since learning former NBA statistical analyst Dean Oliver had joined our production analytics unit and was playing a prominent role in QBR development. Oliver, a Caltech grad with a Ph.D. in statistical applications, revolutionized how NBA teams use advanced statistics. Menlo College professor Ben Alamar, who has consulted with the San Francisco 49ers, is also part of the team.

Our stats team has been using game video to track stats relating to pressure, personnel, formation, game situation and more since 2008. The QBR stat represents a significant leap in harnessing those statistics for something more.

The old formula Smith created treated stats the same regardless of circumstance. A touchdown pass thrown against a prevent defense during a blowout defeat equals one thrown against pressure to win the game. A 5-yard completion on third-and-4 counts the same as a 5-yarder on third-and-15. A critical quarterback scramble, sack or fumble doesn't even factor.

"There is no way to statistically say how effective a guy is under fire," Smith lamented during our 2002 conversation. "None of that can be put into something like this."

Now it can, along with a whole lot more.

The QBR formula takes into account down, distance, field position, time remaining, rushing, passing sacks, fumbles, interceptions, how far each pass travels in the air, from where on the field the ball was thrown, yards after the catch, dropped balls, defensed balls, whether the quarterback was hit, whether he threw away the ball to avoid a sack, whether the pass was thrown accurately, etc. Each play carries "clutch weight" based on its importance to game outcome, as determined by analyzing those 60,000 plays since 2008. The stats adjust for quarterbacks facing an unusually high number of these situations.

"If it is a running clock late in the game, maybe you only get a few yards here or there, that is the right football play to make," Jeff Bennett, senior director of ESPN's production analytics team, said Sunday. "We spent a month learning about ratings to make sure quarterbacks couldn’t game the system, so they're not afraid to throw that deep pass at the end of the first half and risk an interception."

I've seen an outline for the rating system breaking down 2010 quarterbacks into six general categories, from top tier to poor. Precise rating numbers were not yet available. The quarterbacks under consideration broke down as follows:

Top tier: Brady, Peyton Manning, Matt Ryan, Michael Vick, Rodgers and Drew Brees.

Well above average: Josh Freeman, Eli Manning and Philip Rivers.

Above average: Ben Roethlisberger, Tony Romo, Joe Flacco, Matt Schaub, David Garrard and Kerry Collins.

Around average: Matt Cassel, Ryan Fitzpatrick, Mark Sanchez, Carson Palmer, Colt McCoy, Kyle Orton and Jon Kitna.

Below average: Shaun Hill, Jason Campbell, Jay Cutler, Matt Hasselbeck, Chad Henne, Donovan McNabb, Sam Bradford and Alex Smith.

Poor: Derek Anderon, Brett Favre and Jimmy Clausen.

ESPN plans to enlist several quarterbacks when introducing the stat during an hour-long "SportsCenter" special Aug. 5 at 8 p.m. ET. We'll be referencing the stat on the blogs and elsewhere. Bennett said he's allocating enough manpower to produce ratings on game days, a huge help for those of us analyzing player performances shortly after games.

"We want to reward a good football play," Bennett said.
 
Last edited by a moderator:
:popcorn:

Interested to get reactions from Chase and Doug if information is ultimately provided about the formula.

 
:popcorn:

Interested to get reactions from Chase and Doug if information is ultimately provided about the formula.
I agree, I can't wait to see what some of the deep stats guys on FBG will say about this. Peter King talked a little bit about it in today's MMQB:

MMQB Excerpt:

Stat (News) of the Week

ESPN will release this week a proposal for a new passer rating, called the Total QBR, or Total Quarterback Rating. It has been developed by several quarterbackmeisters at the network -- most notably Trent Dilfer -- and by some stat heads in the network's production analytics department. The point: Passer rating, developed in 1973 to measure a passer's efficiency, does that, but it doesn't necessarily measure what makes a quarterback great.

So the analysts at ESPN have taken every game played in the NFL since 2008 and measured the quarterback's contribution to the result on every play except handoffs. They say they've divined a system to rate quarterback performance in every game, and for full seasons, on a scale of 1 to 100 (no more 158.3 rating).

"This is a game-changer,'' Dilfer said. "Mark my words: This is the number scouts and coaches and the media will use to quantitatively discuss and judge the ability of quarterbacks going forward.''

If it sticks, of course. You know how the sporting public (and the larger American public) is with new ideas. But judge for yourself. ESPN will explain the proposed Total QBR in a special Friday night show at 8 Eastern with Dilfer and the Monday night crew -- Mike Tirico, Ron Jaworski and Jon Gruden.

Passer rating has probably declined in significance, with some of the smarter analysts all but ignoring it. But it still has juice. The problem, of course, is that in passer rating, so much is ignored. A series of short completions with big yards after the catch mean the same thing as a series of harder-to-complete long passes. The ESPN formula weighs out the yards after the catch, and weighs in things like lost fumbles in the pocket and sacks taken. And timing. A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should.

"We've included every play that a quarterback has direct control over,'' said Jeff Bennett, the senior director of production analytics at ESPN. "We think a rating system should evaluate all the quarterback's contributions in the context of the game.''

It's be interesting to see if the Total QBR gets traction and usurps passer rating. I think we're ready for a system that scores on a scale of 1 to 100, takes more factors into account than passer rating, and involves the measurement of so-called clutch play. Like OPS (on-base plus slugging) and WAR (wins above replacement value) in baseball, it's time for a more thoughtful number to judge how quarterbacks play. Is this the one? I don't know. But I like rethinking passer rating.
 
Am I the only one turned off by how hard Dilfer is pimping this?

I feel like I basically always disagree with this guy, and he's claiming it's going to change the scouting industry. :rolleyes:

 
I think it is likely that there are too many judgement calls being opaquely codified into a horrendously complex formula that will be cited (at least by Dilfer) as being more meaningful than it really is. Sort of like reading a Football Outsiders article, only worse.

 
Last edited by a moderator:
I think it is likely that there are too many judgement calls going being opaquely codified into a horrendously complex formula that will be cited (at least by Dilfer) as being more meaningful than it really is. Sort of like reading a Football Outsiders article, only worse.
Sounds about right...
 
I think it is likely that there are too many judgement calls going being opaquely codified into a horrendously complex formula that will be cited (at least by Dilfer) as being more meaningful than it really is. Sort of like reading a Football Outsiders article, only worse.
:goodposting:
 
I think it is likely that there are too many judgement calls going being opaquely codified into a horrendously complex formula that will be cited (at least by Dilfer) as being more meaningful than it really is. Sort of like reading a Football Outsiders article, only worse.
:goodposting:
LOL...Dilfer and ESPN are pimping this new system hard...it remains to be seen if it will gain any traction with the scouting community, analysts, or even the fans at large!
 
Hopefully this will consider type of offense, opponent, sacks, offensive line play, the running game and QB rushing ability. Passer rating is a bit embarrassing at this point.

 
Hopefully this will consider type of offense, opponent, sacks, offensive line play, the running game and QB rushing ability. Passer rating is a bit embarrassing at this point.
They touch on it in the linked article above, explaining some differences using examples from various QBs:Excerpt:
In preparing this item, I asked ESPN Stats & Information for examples of quarterbacks with similar passer ratings and disparate QBR numbers. We might then better illustrate how passer rating differs from QBR. Smith and Tennessee's Kerry Collins provided one such example. Their passer ratings were nearly identical, but Collins scored much higher in QBR.Collins' 56.0 rating in the new metric was six points higher than the average established by all quarterbacks across roughly 60,000 plays since 2008. Smith was well below average in QBR at 40.0, affirming what I suspect most of us intuitively know about these quarterbacks. In this example, the QBR seems quite helpful.Smith took sacks at a higher rate than Collins. They threw interceptions at a nearly identical rate, but the QBR formula considered Smith's interceptions far more costly to his team, according to Alok Pattani, one of two analytics specialists assigned to the QBR project. QBR also viewed Bradford and Hasselbeck more favorably than it viewed Smith.In another example, QBR rated Peyton Manning much higher than it rated Matt Schaub last season, even though Schaub finished with a slightly higher passer rating (92.0 to 91.9). Schaub took twice as many sacks while fumbling three times as frequent.Manning's QBR was 69.5, right around where we might expect a Pro Bowl-caliber quarterback to rate. Schaub's QBR was 57.8, above average but not as high as his passer rating ranked in relation to other quarterbacks.
It should be interesting when they roll this out on Friday and explain the evaluation criteria in further depth and detail.
 
It really seems like they are just merging the strength of the offensive line into the passer rating. That may be a better measure of the passing offense, but I'm not sure it's a better measure of the QB's ability. Seems like between the passer rating and some knowledgeable observation you can make a better judgement.

 
I think it is likely that there are too many judgement calls going being opaquely codified into a horrendously complex formula that will be cited (at least by Dilfer) as being more meaningful than it really is. Sort of like reading a Football Outsiders article, only worse.
:goodposting:
:goodposting: Fully expect this to be a trainwreck.

That said, I'm curious to see how ESPN will react when their numbers don't give with what they say. What will they say about Bradford having the 5th lowest QBR when they've been all ZOMG on him?

I'm also very curious to see how they stuck Matt Ryan in the top tier. Actually, I'm not. I suspect it will look similar to what Brian Burke has done: http://wp.advancednflstats.com/playerstats.php?pos=QB

Still, things stick out. I wonder how they got Sanchez in the same tier as Flacco. I'll certainly check this out, but with low expectations.

 
Does anyone care about the QB rating in the first place? If your good, your good, if your bad, your bad.

4 stats I care about are Wins, TDs, INTs, and Yards. What else matters?

 
Last edited by a moderator:
Two more thoughts:

1) In theory, this could be awesome. ESPN has all the resources you would need: access to perfect video of every play, limitless budget, and (the ability to hire) super smart statisticians. But once I hear Trent Dilfer's name, I can't help but :shrug: . Still, if enough smart people at ESPN got involved, this could be the best measure of a QB out there. But I'm not holding my breath.

2) The QBR will almost certainly be be much closer to the explanatory end of the explanatory/predictive spectrum. Say Team A is 11-0, having won every game by 3 points. Team B is 9-2, having won 9 games by 10 points and lost two games by 1 point. Both teams have faced the same schedule. "Winning percentage" is an explanatory stat; Team A is better than Team B because Team A has a 1.000 WP and Team B has a .818 WP. But winning percentage isn't very predictive. Team A has outscored its opponents by 33 points; Team B has outscored its opponents by 88 points. On a neutral field, knowing nothing else, Team B should be expected to beat Team A by 5 points. That's because, on average, Team B outscores its opponents by 5 more points than Team A does. "Scoring Margin" is a predictive stat; it won't explain what happened in the past all that well, but it does a very good job of predicting the future. Interceptions and touchdowns are explanatory stats more than they are predictive ones. Yards per attempt and sack rate, on the other hand, are excellent predictive stats.

Both stats have their usefulness. But I suspect the ESPN QBR will heavily rate things like "clutch plays" and "third down conversions" and turnovers. Those stats, IMO, are likely to be explanatory but not very predictive. Since they will be a key part of the formula, they will likely match up with things like team wins. But I'm not sure how much value as a predictive tool the ESPN QBR will be (i.e., saying that QB A has a 65 QBR so he will do better than QB B who has a 60 QBR won't make any sense, since the stat isn't designed to predict the future but rather explain the past).

 
I agree, I can't wait to see what some of the deep stats guys on FBG will say about this. Peter King talked a little bit about it in today's MMQB:

MMQB Excerpt:

Stat (News) of the Week

Passer rating has probably declined in significance, with some of the smarter analysts all but ignoring it. But it still has juice. The problem, of course, is that in passer rating, so much is ignored. A series of short completions with big yards after the catch mean the same thing as a series of harder-to-complete long passes. The ESPN formula weighs out the yards after the catch, and weighs in things like lost fumbles in the pocket and sacks taken. And timing. A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should.
The point of having "input from football people" is to make sure the statheads don't misinterpret how the game is played. So how does Trent Dilfer, Mr. WCO, stand behind a system that weighs out YAC? This is potential scary in two ways:1) YAC, of course, is attributable to QBs. Joe Montana made a living out of placing the ball in the right spot to enable his WRs to pick up YAC. If this system makes a 5-yard pass that's on the money that lets the WR go for 30 more yards equivalent to a 5-yard pass that's half a step behind the WR who catches it and gets tackled, that's really bad.

2) Reading out YAC is doubly troublesome because it will hurt QBs in WCO offenses just by virtue of throwing short passes. A screen pass, by definition, will be a bad pass if you read out YAC. If you want to just say "a screen pass is easy, any QB can throw it, so we're going to delete from our file all screen pases" that would be somewhat justifiable. But I doubt they'll do that. They may remove the value of the screen pass from the numerator, but they probably won't remove the attempt from the denominator. So if you throw 5% of your passes behind the LOS, they will judge your performance on 95% of your production based on 100% of your attempts. That's silly (again, I'm just speculating).

Re: point #2: A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should. If the goal is to create an excellent explanatory stat, that's fine. But ESPN will surely say this stat is highly predictive as well. That begs the question: are quarterbacks who throw 17 yard completions in a tie game with 2 mins left more likely in the next game to complete that pass than one who threw a completion in the 2nd quarter in the previous game? Probably not.

 
Prediction: This stat will strongly correlate with the old-style passer rating ( r > 0.5 ). In the end, they'll be measuring most of the same things.

 
I agree, I can't wait to see what some of the deep stats guys on FBG will say about this. Peter King talked a little bit about it in today's MMQB:

MMQB Excerpt:

Stat (News) of the Week

Passer rating has probably declined in significance, with some of the smarter analysts all but ignoring it. But it still has juice. The problem, of course, is that in passer rating, so much is ignored. A series of short completions with big yards after the catch mean the same thing as a series of harder-to-complete long passes. The ESPN formula weighs out the yards after the catch, and weighs in things like lost fumbles in the pocket and sacks taken. And timing. A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should.
The point of having "input from football people" is to make sure the statheads don't misinterpret how the game is played. So how does Trent Dilfer, Mr. WCO, stand behind a system that weighs out YAC? This is potential scary in two ways:1) YAC, of course, is attributable to QBs. Joe Montana made a living out of placing the ball in the right spot to enable his WRs to pick up YAC. If this system makes a 5-yard pass that's on the money that lets the WR go for 30 more yards equivalent to a 5-yard pass that's half a step behind the WR who catches it and gets tackled, that's really bad.

2) Reading out YAC is doubly troublesome because it will hurt QBs in WCO offenses just by virtue of throwing short passes. A screen pass, by definition, will be a bad pass if you read out YAC. If you want to just say "a screen pass is easy, any QB can throw it, so we're going to delete from our file all screen pases" that would be somewhat justifiable. But I doubt they'll do that. They may remove the value of the screen pass from the numerator, but they probably won't remove the attempt from the denominator. So if you throw 5% of your passes behind the LOS, they will judge your performance on 95% of your production based on 100% of your attempts. That's silly (again, I'm just speculating).

Re: point #2: A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should. If the goal is to create an excellent explanatory stat, that's fine. But ESPN will surely say this stat is highly predictive as well. That begs the question: are quarterbacks who throw 17 yard completions in a tie game with 2 mins left more likely in the next game to complete that pass than one who threw a completion in the 2nd quarter in the previous game? Probably not.
Chase,

Appreciate your popping into this thread and adding your viewpoint...like many, I am curious to see how their new system works and on the surface with the resources of ESPN this thing could be killer...or it could completely flop.

 
Yeah, I've got nothing against ESPN, but this rating system sounds like a trainwreck. It seems to heavily apply context in some areas while ignoring it in other areas (like sacks). I don't see a suggestion that sacks where receivers weren't open or the QB got less than the league average dropback time will be counted less than a QB simply holding the ball too long. Similarly, a Qb with a full complement of healthy receivers and linemen in the 4th quarter might nevertheless have an easier time of it than one dealing with a pathcwork offense early in the game. It's perfectly appropriate to say that this stuff comes out in the wash, but I imagine that a lot of the stuff the rating is taking into account would to.

 
The Hype continues:

QBR: No such thing as passer perfection

By Mike Sando

The official NFL stat sheet said Kurt Warner played a perfect game for the Arizona Cardinals against Miami back in 2008.

Total Quarterback Rating (QBR) knew better.

While Warner was brilliant that day, completing 19 of 24 passes for 361 yards and three touchdowns, his performance could have been statistically superior. He scored a 97.7 on the 100-point QBR scale.

Warner took two sacks for 12 yards in that game, affecting his QBR. The 75-yard touchdown pass he threw to Larry Fitzgerald traveled only 17 yards in the air. About half his passing yardage that day came after the catch.

QBR, set to debut during a "SportsCenter" special Friday night, keeps moving the carrot as quarterbacks chase perfection. Albert Larcada, ESPN Stats & Information analytics specialist for the QBR project, illustrated how this happens by analyzing the three performances since 2008 featuring 158.3 ratings.

Drew Brees posted a 158.3 rating against New England during the 2009 season. That effort translated to 98.6 by QBR standards. Brees took one sack for 4 yards. The 18-yard touchdown pass he threw to Pierre Thomas featured 25 yards after the catch. (Thomas caught it behind the line of scrimmage.)

No matter how well a quarterback plays from a statistical standpoint, he could have fared better.

Completing all 20 pass attempts, each for a 99-yard touchdown, would shatter records. But the performance wouldn't rate as high as one featuring 21 touchdown passes of that length. And so on.

That's why it's misleading to say a quarterback played a "perfect" game when his passer rating maxed out at 158.3 under the formula in place since 1973.

Tom Brady's 158.3 rating against Detroit last season translated to 94.6 in QBR. Brady's receivers made huge gains after the catch in that game, to a degree much greater than they would have done typically.

In theory, a perfectly executed short pass could free up a receiver for additional yards.

"It’s true a perfect pass could set up additional air yards for a receiver," Larcada explained, "but on average we found YAC to be mostly on the receiver, and pretty strongly so. Good receivers get more YAC per completion than bad receivers. Good quarterbacks don’t necessarily get more YAC per completion than bad ones."

More YAC means fewer air yards when all else is equal, hurting a player's QBR number.

"Even with a perfect pass, the receiver still must have the speed/moves/quickness to create the additional yardage," Larcada said.

QBR takes into account many more variables. It grades each play in relation to how it affects game outcomes, putting more weight on a killer interception than a meaningless one on a Hail Mary at the end of a half.

We'll continue the discussion Thursday.
 
QBR takes into account many more variables. It grades each play in relation to how it affects game outcomes, putting more weight on a killer interception than a meaningless one on a Hail Mary at the end of a half.We'll continue the discussion Thursday.
Again, I think it's important to make some distinctions here. A) Meaningless INT at end of first halfB) Bad throw turned INT on first play of gameC) Bad throw turned INT at crucial point in final minuteYes, it's bad that QB Rating (and ANY/A, a stat I like) treat all of these as identical. But as far as grading a QB's ability, I think B and C are much more similar than A and B. It sounds like ESPN will view A and B as much more similar, because C is going to be a killer. Again, I think it's fine if you're clear on what you're saying. But it will be all to easy to say QB X is better than QB Y because his ESPN QBR is better, when that's not really what the ESPN QBR is saying.
 
was kinda in to this until i read ...

Laranda explained..."Good quarterbacks don’t necessarily get more YAC per completion than bad ones."

More YAC means fewer air yards when all else is equal, hurting a player's QBR number.

"Even with a perfect pass, the receiver still must have the speed/moves/quickness to create the additional yardage," Larcada said.



i thought good QBs hit their WR in stride and bad ones get their WR hit after having to extend their catch radius

 
lol @ Matt Ryan > Phillip Rivers.
Only explanation is a severe weight is placed on high leverage plays; i.e., plays late in close games. Ryan was extremely successful in a very small sample size in those situations last year. Or, in other views, he was really clutch last year.
 
If ESPN is doing it, it is guaranteed to ####ty
This. The only reason I know about it is just now reading this thread. I don't watch ESPN (except for games) and I don't listen to ESPN and haven't watched Sportscenter in years.
 
Does anyone care about the QB rating in the first place? If your good, your good, if your bad, your bad.4 stats I care about are Wins, TDs, INTs, and Yards. What else matters?
Yards per attempt. Sustaining drives / getting first downs consecutively.
 
lol @ Matt Ryan > Phillip Rivers.
Is that really a silly notion?By traditional measures, I guess it would be.We don't know a ton about this stat yet, but Ryan over Rivers in 2010 certainly makes it more intriguing. ESPN claims the stat places significant value on situational performance and Ryan over Rivers in 2010 certainly would make it seem like they are backing up that claim.I don't see anyone claiming that Ryan is better than Rivers, but did Ryan have a better year in 2010 than Rivers? It's certainly possible. QB is the most important position in sports and the NFL is a QB league. One QB went 13-3 in a great division, including the defending SB Champs and a good young team, while another missed the playoffs in a mediocre to bad division. Both were outstanding statistically.This is the exact situation that this new stat is supposedly designed for.Certainly, there can be good debate about whether or not that's a good thing, but it's intriguing none the less. It's certain to have it's flaws, but it's nice to see the stat attempt to look past the traditional QB measures. Not to mention, ESPN will be criticized if this stat simply mirrors traditional QB measures (actually, it's been prematurely criticized in this very thread for that).
 
QBR takes into account many more variables. It grades each play in relation to how it affects game outcomes, putting more weight on a killer interception than a meaningless one on a Hail Mary at the end of a half.

We'll continue the discussion Thursday.
Again, I think it's important to make some distinctions here. A) Meaningless INT at end of first half

B) Bad throw turned INT on first play of game

C) Bad throw turned INT at crucial point in final minute

Yes, it's bad that QB Rating (and ANY/A, a stat I like) treat all of these as identical. But as far as grading a QB's ability, I think B and C are much more similar than A and B. It sounds like ESPN will view A and B as much more similar, because C is going to be a killer.

Again, I think it's fine if you're clear on what you're saying. But it will be all to easy to say QB X is better than QB Y because his ESPN QBR is better, when that's not really what the ESPN QBR is saying.
Why would you think that?Did I miss someone saying that desperation EOH hail mary INTs will be given the same treatment as general INTs (such as first play of the game)?

I was under the impression that they would treat A,B, and C differently, as they should be.

 
I agree, I can't wait to see what some of the deep stats guys on FBG will say about this. Peter King talked a little bit about it in today's MMQB:

MMQB Excerpt:

Stat (News) of the Week

Passer rating has probably declined in significance, with some of the smarter analysts all but ignoring it. But it still has juice. The problem, of course, is that in passer rating, so much is ignored. A series of short completions with big yards after the catch mean the same thing as a series of harder-to-complete long passes. The ESPN formula weighs out the yards after the catch, and weighs in things like lost fumbles in the pocket and sacks taken. And timing. A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should.
The point of having "input from football people" is to make sure the statheads don't misinterpret how the game is played. So how does Trent Dilfer, Mr. WCO, stand behind a system that weighs out YAC? This is potential scary in two ways:1) YAC, of course, is attributable to QBs. Joe Montana made a living out of placing the ball in the right spot to enable his WRs to pick up YAC. If this system makes a 5-yard pass that's on the money that lets the WR go for 30 more yards equivalent to a 5-yard pass that's half a step behind the WR who catches it and gets tackled, that's really bad.

2) Reading out YAC is doubly troublesome because it will hurt QBs in WCO offenses just by virtue of throwing short passes. A screen pass, by definition, will be a bad pass if you read out YAC. If you want to just say "a screen pass is easy, any QB can throw it, so we're going to delete from our file all screen pases" that would be somewhat justifiable. But I doubt they'll do that. They may remove the value of the screen pass from the numerator, but they probably won't remove the attempt from the denominator. So if you throw 5% of your passes behind the LOS, they will judge your performance on 95% of your production based on 100% of your attempts. That's silly (again, I'm just speculating).

Re: point #2: A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should. If the goal is to create an excellent explanatory stat, that's fine. But ESPN will surely say this stat is highly predictive as well. That begs the question: are quarterbacks who throw 17 yard completions in a tie game with 2 mins left more likely in the next game to complete that pass than one who threw a completion in the 2nd quarter in the previous game? Probably not.
Again, why would you even speculate that they would do something so stupid (as removing attempts from the numerator without also removing them from the denominator)?The YAC issue is a good one and clearly worth debate. I can see both sides of that one.

 
lol @ Matt Ryan > Phillip Rivers.
Is that really a silly notion?By traditional measures, I guess it would be.We don't know a ton about this stat yet, but Ryan over Rivers in 2010 certainly makes it more intriguing. ESPN claims the stat places significant value on situational performance and Ryan over Rivers in 2010 certainly would make it seem like they are backing up that claim.I don't see anyone claiming that Ryan is better than Rivers, but did Ryan have a better year in 2010 than Rivers? It's certainly possible. QB is the most important position in sports and the NFL is a QB league. One QB went 13-3 in a great division, including the defending SB Champs and a good young team, while another missed the playoffs in a mediocre to bad division. Both were outstanding statistically.This is the exact situation that this new stat is supposedly designed for.Certainly, there can be good debate about whether or not that's a good thing, but it's intriguing none the less. It's certain to have it's flaws, but it's nice to see the stat attempt to look past the traditional QB measures. Please don't compare Rivers situation to Ryans. By eek 3, SD was pulling wrs off the street and also lost Gates for about 7 games. Ryan got full seasons out of all of his guys. If you took Tony G and Roddy out of the equation, Matty Ice would most likely be Matty No Dice. Also...the divisions were not as unequal as you paint them to be. Sure N.O is a super bowl champ, but they had a bad year last year. Carolina was above and beyond the worst team in the league and was pretty much two guarenteed wins for them, no matter how bad you think the raiders or the broncos were, you still had to line up and play in order to win, unlike the panthers. Not to mention, ESPN will be criticized if this stat simply mirrors traditional QB measures (actually, it's been prematurely criticized in this very thread for that).
 
lol @ Matt Ryan > Phillip Rivers.
Is that really a silly notion?By traditional measures, I guess it would be.We don't know a ton about this stat yet, but Ryan over Rivers in 2010 certainly makes it more intriguing. ESPN claims the stat places significant value on situational performance and Ryan over Rivers in 2010 certainly would make it seem like they are backing up that claim.I don't see anyone claiming that Ryan is better than Rivers, but did Ryan have a better year in 2010 than Rivers? It's certainly possible. QB is the most important position in sports and the NFL is a QB league. One QB went 13-3 in a great division, including the defending SB Champs and a good young team, while another missed the playoffs in a mediocre to bad division. Both were outstanding statistically.This is the exact situation that this new stat is supposedly designed for.Certainly, there can be good debate about whether or not that's a good thing, but it's intriguing none the less. It's certain to have it's flaws, but it's nice to see the stat attempt to look past the traditional QB measures. Please don't compare Rivers situation to Ryans. By eek 3, SD was pulling wrs off the street and also lost Gates for about 7 games. Ryan got full seasons out of all of his guys. If you took Tony G and Roddy out of the equation, Matty Ice would most likely be Matty No Dice. Also...the divisions were not as unequal as you paint them to be. Sure N.O is a super bowl champ, but they had a bad year last year. Carolina was above and beyond the worst team in the league and was pretty much two guarenteed wins for them, no matter how bad you think the raiders or the broncos were, you still had to line up and play in order to win, unlike the panthers. Not to mention, ESPN will be criticized if this stat simply mirrors traditional QB measures (actually, it's been prematurely criticized in this very thread for that).
 
Does anyone care about the QB rating in the first place? If your good, your good, if your bad, your bad.4 stats I care about are Wins, TDs, INTs, and Yards. What else matters?
I'd de-emphasize TDs. They're typically way overweighted- a guy who throws for 70 yards on a drive and then sees his RB punch it in from the 1 yard line had a better drive than a guy who watched his coach call 12 straight runs and then tossed a wounded duck to the far corner off of play-action.Also, yards are good, but YPA is better. If you want to reward volume stats (which is reasonable- a guy with 4,000 yards and 7.8 ypa probably had a better season than a guy with 3,000 yards and 8.0 ypa), then you could do something like attempts*(ypa-x) where x is some baseline (league average, worst starter, or some arbitrarily selected number).Wins is a terrible "stat" to keep for QBs. If you could somehow track "wins above what an average QB would have done when placed on that team", then it'd be okay... but you can't. According to the wins "stat", Mark Sanchez is better than Philip Rivers... but does anyone really believe that if they switched teams, San Diego would win more and New York would win less?Really, the point is that all stats are meaningless without context. TDs are great... unless the QB is losing TDs through no fault of his own. Yards are great, but you face the compiler dilemma. Wins are impossible to attribute (even mostly attribute) to a single player. Another hugely important stat is 3rd down conversion percentage (one of the most powerful explanatory stats in football), but that one really needs to be adjusted for distance before it becomes predictive. If I had to pick a mainstream stat, I'd probably go with YPA or one of its derivatives. Hell, I'd even be fine with Passer Rating if they just stopped double-counting comp%.
 
QBR: Closer look at NFC West quarterbacks

By Mike Sando

Total Quarterback Rating (QBR) should affirm what we know most of the time while occasionally challenging what we think.

The newly developed metric would have no value if it tried to convince us Derek Anderson was better than Tom Brady (it does not do this). If it told us Chad Pennington was better in 2008 than Peyton Manning was in 2010 -- it does make that case -- we should at some point revisit those specific seasons for those specific quarterbacks. I've been most interested in learning what QBR reveals about performances relative to the passer-rating formula in use since 1973.

For example, Matt Hasselbeck finished with similar passer ratings in 2009 and 2010, but QBR favored his 2010 season by a wide margin. Something about Hasselbeck's 2009 season did not sit well with the QBR formula. Even his 2008 season, complete with a career-low 57.8 passer rating, held up better. Why was this?

Alok Pattani from our analytics team pointed to a couple factors:

[*]Hasselbeck made greater contributions as a runner in 2008 than in 2009.

[*]Hasselbeck fumbled only once in 259 plays during the 2008 season. Seattle recovered. He fumbled 18 times in 1,139 plays during the 2009 and 2010 seasons. Seattle lost eight of those 18 fumbles. That would have helped 2008 make up ground.

There were other factors, but these ones were prominent. Separately, Hasselbeck's production on third down improved steadily since 2008. His third-down completion percentage rose from 36.7 in 2008, 53.6 in 2009 and 60.9 last season. His sacks per pass play also steadily declined on third down.

I was also curious why Hasselbeck's QBR for 2010 doubled the 2008 QBR for then-San Francisco 49ers quarterback J.T. O'Sullivan, even though their passer ratings were similar. One key: O'Sullivan wasn't nearly as good on third-down pass plays across categories that included completion percentage, touchdowns, interceptions, sack percentage and conversion rate.

Understanding such disparities requires understanding QBR. Once QBR becomes established, considering the computations will not be relevant for those merely interested in the bottom line. We'll simply need to know that a season-long QBR around 50 would be near average, while a season-long QBR in the 65 range would reflect Pro Bowl-caliber play. Players will rarely reach an 80 QBR for a season, and it's impossible to reach 100 because a quarterback could, in theory, always complete one more pass for one more yard, etc.

ESPN plans to unveil details more fully during a "SportsCenter" special Friday night at 8 p.m. ET. A news release promoting the new metric offers the following details regarding QBR:

[*]Total QBR measures all of the significant contributions by a quarterback during the course of a game and accounts for precisely how much he impacts his team’s performance and chances of winning.

[*]Total QBR is based on all of a quarterback’s plays (rushing, passing, sacks, fumbles, interceptions, penalties, etc.), and it calculates the per-play net impact of the quarterback on the ability to score. Each play is weighted by the situation (i.e., down and distance, field position, time during the game) and its importance to the game’s outcome.

[*]Another variable, division of credit, assigns a percentage to how much credit a quarterback should get for a positive play -- or blame for a negative play. ESPN analyst Trent Dilfer helped come up with additional data to consider, including how far a pass travels in the air, where the ball was thrown on the field, yards after catch, whether the quarterback faced pressure, etc.

[*]QBR draws from 60,000 plays over the past three years to assign credit or blame for every play involving the quarterback. It measures quarterbacks on a 100-point scale, whereas the NFL's passer rating formula maxes out at 158.3.

According to those standards, Kurt Warner is the only NFC West quarterback since 2008 to post a Pro Bowl-caliber season.

And now, on to the chart showing passer rating and QBR figures for NFC West quarterbacks since 2008.
 
Guide to the Total Quarterback Rating: Explaining the methodology and statistics behind ESPN's new QB metric

By Dean Oliver

Dir. Analytics, ESPN Stats & Information

Early in a scoreless game, a quarterback throws a 20-yard pass just by the reaching arms of a defender and into the hands of his intended receiver, who holds on despite the distraction, then scampers the remaining 15 yards for a touchdown.

Another quarterback, down 30-10 with five minutes left in the fourth quarter, throws a 3-yard screen pass to a running back, who maneuvers another 32 yards through prevent defense to pick up a first down deep in opponent territory.

Both are called good plays, but labeling them as "good" isn't enough. Each play has a different level of contribution to winning, and each play illustrates a different level of quarterback contribution. What is the quarterback's contribution to winning in each situation? Coaches want to know this, players want to know this and fans want to know this.

The Total Quarterback Rating is a statistical measure that incorporates the contexts and details of those throws and what they mean for wins. It's built from the team level down to the quarterback, where we understand first what each play means to the team, then give credit to the quarterback for what happened on that play based on what he contributed.

At the team level, identifying what wins games is not revolutionary: scoring points and not allowing points. Back in the 1980s, "The Hidden Game of Football" did some pioneering work on that topic and how yardage relates to points. We went back and updated what that book did … then we went further. At the individual level, more detailed information about what quarterbacks do is really necessary. Brian Burke at AdvancedNFLStats.com has done very good work in advancing that effort, and FootballOutsiders.com has done some of this by charting data, but over the past three years ESPN has charted football games in immense detail. By putting all these ideas together and incorporating division of credit, we have built a metric of quarterback value, the Total Quarterback Rating, either Total QBR or QBR for short.

What follows is a summary of what goes into QBR. It took several thousand lines of code to implement, but we'll keep this shorter.

Win Probability and Expected Points

The goal behind any player rating should be determining how much a player contributes to a win. We went back through ten years of NFL play-by-play data to look at game situation (down, distance, yard line, clock time, timeouts, home field, field surface and score), along with the ultimate outcome of the game, to develop a win probability function.

This function treats every win the same, regardless of whether it was 45-3 or 24-23, though there is clearly a difference between such games. The first game represents total domination, whereas the other represents two fairly evenly matched teams. Because win probability treats every win the same, it misses some of what goes into the win, specifically many of the points that represent domination or the points that lead up to a last-second victory. So, although QBR uses win probability to assess how "clutch" a situation is, it uses expected points as the basis of evaluating quarterbacks. It has more of the details, and understands the difference between wins, but still strongly relates to wins in general.

The concept of expected points was discussed as early as the mid-1980s with Pete Palmer and "The Hidden Game of Football," where they talk about "point potential." Their idea was that, as you move closer to the opponents' end zone, you are actually gaining points. Brian Burke took it further to note that third-and-10 from midfield, for instance, has fewer expected points than first-and-10 from midfield. In other words, down and distance also matter in terms of points. We took this even further to look at clock time, home field, timeouts and field surface to generate the expected points for any team given its situation in a drive. One particular situation to note is that, at the end of the half, a team is less likely to score any points than at most times of the game, just because the half is going to expire.

It's useful to mention here that expected points are expected net points. It's possible that a team has expected points less than 0. This simply implies that the other team is generally more likely to score. This usually happens when a team is backed up deep in its own side of the field, especially if it is third or fourth down.

What then happens is an evaluation of expected points added. How does a team go from 1.1 expected points to 2.1? However they do it, that is 1.0 expected points to be distributed to the offensive players on the field. But how they do it is what determines how credit is given to a quarterback.

Dividing Credit

Division of credit is the next step. Dividing credit among teammates is one of the most difficult but important aspects of sports. Teammates rely upon each other and, as the cliché goes, a team may not be the sum of its parts. By dividing credit, we are forcing the parts to sum up to the team, understanding the limitations, but knowing that it is the best way statistically for the rating.

On a pass play, for instance, there are a few basic components:

• The pass protection

• The throw

• The catch

• The run after the catch

In the first segment, the blockers and the quarterback have responsibility for keeping the play alive, and the receivers have to get open for a QB to avoid a sack or throw the ball away. On the throw itself, a quarterback has to throw an accurate ball to the intended receiver. Certain receivers may run better or worse routes, so the ability of a QB to be on target also relates somewhat to the receivers. For the catch, it may be a very easy one where the QB laid it in right in stride and no defenders were there to distract the receiver. Or it could be that the QB threaded a needle and defenders absolutely hammered the receiver as he caught the ball, making it difficult to hold on. So even the catch is about both the receiver and the QB. Finally, the run after the catch depends on whether a QB hit the receiver in stride beyond the defense, and the ability of a receiver to be elusive. Whatever credit we give to the blockers, receivers and quarterback in these situations is designed to sum to the team expected points added.

The ESPN video tracking has been useful in helping to separate credit in plays like these. We track over-throws, under-throws, dropped passes, defended passes and yards after the catch. The big part was taking this information and analyzing how much of it was related to the QB, the receivers and the blockers. Not surprisingly, pass protection is related mostly to the QB and the offensive line, but yards after the catch is more about what the receiver does. Statistical analysis was able to show this and we divided credit based on those things.

As a relevant side note, statistical analysis showed that what we call a dropped pass was not all a receiver's fault either. A receiver may drop a ball because he wanted to run before catching it, because the defense distracted him, because it was a little bit behind him or because he was about to get hit by a defender. If the defender was there a half second before, the defender would have knocked the ball free and it would have been called a "defended pass", not a "dropped pass." There are shades of gray even on a dropped pass and analysis showed that. Drops are less a QB's fault than defended passes or underthrows, but the QB does share some blame.

On most other plays, quarterbacks receive some portion of credit for the result of the play, including defensive pass interference, intentional grounding, scrambles, sacks, fumbles, fumble recoveries (Carson Palmer once recovered a teammate's fumble that saved the game for the Bengals) and throwaways. On plays where the QB just hands off to a running back, we didn't assign any credit to the QB. Our NFL experts did suggest that some QBs are very good at interpreting defenses pre-snap and identifying better holes for their backs.

However, they also told us that it would be nearly impossible to incorporate. Because they suggested this, we built in the ability to give credit for QBs when they just handed off, but we couldn't find the right analysis to do it in 2011.

Clutch Index

The final major step is to look at how "clutch" the situation was when creating expected points. A normal play has a clutch index of 1.0. For instance, first-and-goal from the 10-yard line in a tie game at the start of the second quarter has a clutch index of almost exactly 1.0. A more clutch situation, one late in the game where the game is close -- the same situation as above but midway through the fourth quarter, for example -- has a clutch index of about 2.0. Maximum clutch indices are about 3.0 and minimum indices are about 0.3.

These clutch index values came from an analysis of how different situations affect a game's win probability on average. One way to think of it is in terms of pressure. A clutch play is defined before the play by how close the game appears to be. Down four points with three seconds to go and facing third-and-goal from the 3-yard line -- that is a high-pressure and high-clutch index situation because the play can realistically raise the odds of winning to almost 100 percent or bring them down from around 40 percent to almost 0 percent. The same situation from midfield isn't as high pressure because it's very unlikely that the team will pull out the victory. Sure, a Hail Mary can pull the game out, but if it doesn't work, the team didn't fail on that play so much as they failed before then. On third-and-goal from the 3-yard-line, failure means that people will be talking about that final play and what went wrong.

The clutch indices are multiplied by the quarterback's expected points on plays where they had a significant contribution, then divided by the sum of the clutch indices and multiplied by 100 to get a clutch-valued expected points added per 100 plays.

A Rating from 0 to 100

The final step is transforming the clutch-valued expected points rate to a number from 0 to 100. This is just a mathematical formula with no significance other than to make it easier to communicate. A value of 90 and above sounds good whether you're talking about a season, a game or just third-and-long situations; a value of four or 14 doesn't sound very good; a value of 50 is average and that is what QBR generates for an average performance.

That being said, the top values in a season tend to be around 75 and above, whereas the top values in a game are in the upper 90s. Aaron Rodgers may have gone 31 of 36 for 366 yards, with three passing TDs, another TD running, 19 first down conversions and eight conversions on third or fourth down in one game -- for a single game Total QBR of 97.2 -- but he can't keep that up all year long. Pro Bowl-level performance for a season usually means a QBR of at least 65 or 70. We don't expect to see a season with a QBR in the 90s.

Defensive Adjustment

With this rating, we have intentionally not adjusted for opponents. This doesn't mean that we won't adjust for opponents as we use it, but that we want QBR to be flexible for many purposes, and keeping opponents' strength out gives us that flexibility. As it stands, QBR can be broken down for all sorts of situations -- red zone, third-and-long, throwing to a certain receiver, in bad weather, against different defensive formations. We didn't want to muddy it up with opponent adjustments that aren't as useful for those situations. How to implement a defensive adjustment for third-and-long may also be different than one for the whole season. Beyond this, a defensive adjustment is often not a constant factor. A defense that looks good in Week 4 may actually not be so good after a few more weeks. Because it isn't a constant thing, it makes sense to leave that for analysis, rather than constant incorporation into QBR.

There will be analyses that we do on ESPN that will suggest the use of an opponent adjustment, but we will do that when needed, not up front.

Concluding Thoughts

What underlies QBR is an understanding of how football works and a lot of detailed situational data. What it yields are results that should reflect that. It illustrates that converting on third-and-long is important to a quarterback. It shows that a pass that is in the air for 40 yards is more reflective of a quarterback than a pass that is in the air for five yards, where the receiver has 35 yards of run after the catch. These premises should sound reasonable to football fans. They come out of a lot of statistical analysis, but they are also consistent with what coaches and players understand.

As we neared the end of the development of QBR, we talked to Ron Jaworski and Greg Cosell at NFL Films about its evolution. Cosell said at one point, "Football is not complex, but it is very detailed." I realized then that QBR is like that. It is very detailed, accounting for a lot of different situations, but it is not particularly complex. It really does try to see the game the way we have gotten used to seeing it in its elegant simplicity. We hope you, the fan, appreciate it as well.

Dean Oliver is one of the pioneers in sports analytics. Author of "Basketball on Paper", the standard for doing analytics in basketball, Oliver applied his work to personnel and coaching matters for five successful years in the front office of the Denver Nuggets. Oliver is now the Director of Analytics for ESPN. He joined the company in 2011 to build the analytics group, which works across a number of sports.
 

Users who are viewing this thread

Back
Top