Retrospective Analysis of the Weekly IDP Projections (1 Viewer)

[Islander]

I have completed the first phase of my offseason analysis of the IDP projections that come out every week from John Norton. I mentioned in this thread that I would try to analyze retrospectively some numbers to find out if there were biases in the projections or not. Sometimes during the season I felt like there were biases, but it was mostly based on random observations during the season rather than extensive quantitative analysis. So I tried to prove or disprove my beliefs about the flaws (or lackthereof) of the weekly projections.

However, before I go to the results, let me mention of few things.

Like I mentioned in the thread referred to above, my intent is not to bash the projections. It’s the best source of IDP projections I know of, and I use them. If I was doing my own projections, I don’t pretend they would be better. However, in the spirit of continuous improvement, I think it’s fair to point out the problems and I hope it’s not received as an all-out complaint, but rather as a suggestion for improvement. If the projections improve, everybody here will benefit.

I also do not think any single person can come up with the perfect set of projections every week. There are so many players, so many stat categories to project, you have to consider matchups, etc. But in a perfect world, we would have one or several people doing weekly projections and FBGs on this board would point out where projections are clearly too optimistic or pessimistic, and the person who made the projections would tweak them based on feedback received to end up with a better set of projections come Sunday. In 2007 I would love to see weekly projections from more than one person (on offense as well). In addition, maybe we should set up a weekly thread where people can post “the projected number tackles for Vilma this week is way too high / too low because XYZ”. Then the projections would get refreshed if the projections-maker (Norton or others) agrees with the feedback.

Also, let me describe what I think “perfect” projections are. A perfect projection is the number that would turn out to be exact on average if the game was played 100 times. So if Vilma plays the Dolphins at home, let’s say Norton projects 7 tackles. To me 7 is a perfect projection is Vilma would get 700 tackles in the 100 games the Jets would play the Dolphins at home. Of course we cannot know how many tackles Vilma would get in those 100 games because they won’t play those games. But you have to keep this in mind in order to make perfect projections. If Norton projects 12 tackles for Vilma in that game, and Vilma happens to get 12 tackles in that game, I do not think it was a perfect projection. I think it was luck that it turned out Vilma got 12 (from reading previous comments especially in the shark pool, many people disagree with this statement). I view player stats as probability distributions. If in those hypothetical 100 games, let's assume Vilma would get the following number of tackles

0: 2 times

1: 3 times

2: 4 times

3: 5 times

4: 6 times

5: 9 times

6: 12 times

7: 18 times

8: 12 times

9: 9 times

10: 6 times

11: 5 times

12: 4 times

13: 3 times

14: 2 times

As you can see, it’s a probability distribution with a mean of 7.0. If that was the case, I would project 7.0 tackles. If the outcome is 12, that does not mean my projection of 7.0 was wrong. The outcome was different from the projection simply because it’s a random process. In addition, if the projection was 12 and the outcome is 12, that does not mean the projection was correct (in fact, a projection of 12 was way too high in this particular example, because if Vilma played that game 100 times, he would get 70 tackles, not 120 tackles, and the projection of 12 would be too high 91% of the time). Think of projecting the outcome of a die roll where people could bet over/under on your projection. The perfect projection is 3.5. If the outcome is 6, it does not mean your projection of 3.5 was wrong.

Several people use the weekly projections in different ways. Some use them simply as a cheatsheet so they don’t care about the numbers, they only care about the rankings, so Norton needs to consider how the rankings will come out based on the projections. Others convert the projected stats into their scoring system to come up with a new cheatsheet. If that’s all you do, then whether all projections are 10% overstated or 10% understated for all players, you don’t care. But if you use the projections for other purposes, that may impact what you do. So again, in the perfect world, all projections would be unbiased for all players, and regardless what you use projections for, you will have something very good to work with. However, I fully understand that some people don’t place high value about all players being over/underprojected 10% across the board – they might only care about projections of certain players relative to one another (is player X projected more fantasy points than player Y?).

Another argument I frequently read in the IDP forum but also in the shark pool: “well there is only a 0.5 pt difference between the projection of LB10 and LB20 or between WR10 and WR20, so it does not mean much”. Let me express my views on this. I totally disagree. Half a point is big. If you bet on sports, you already know that the casino has a small advantage (the juice, or the vig) and to be able to beat the casino, you need to exploit lines that may be off by as little as 0.5 pt. The difference between a 54% success rate and a 51% success rate is enormous. In the Vilma example above, if the projection was 6.5 tackles instead of 7.0 tackles, the projection would be too high 41% of the time and it would be too low 69% of the time. That is far from being a good projection. In my view, anything more than 0.3 or 0.4 difference in projected fantasy points is statistically relevant and I disagree with the argument that it’s not much. I have read from some posters before that they thought a difference in projections of 2 pts is not much. In my eyes, it is huge. If you were a casino offering a line that is 2 pts off (for over/under fantasy points of a certain player), you would go out of business in a heartbeat if you were taking bets from sharks that know their stuff.

I have not done all the number crunching I want to do yet, but I have some preliminary results that I want to throw out there as a basis of discussion before I move on. I think the FBG crowd that hangs out here in the fantasy offseason is sharper than what we see between Sept-Dec so hopefully this discussion will be of good quality.

Here is what I did.

I took the projections for all DLs, LBs, DBs for each week in 2006. I looked at the actual stats for each IDP of each game in 2006. For each player and each week, I compared the projection to the actual result. I compiled numbers by position, by week, by stat category, etc.

The first thing I wanted to find out is whether there is a bias in the overall projections (not the breakdown by player). Many people don’t care about this, as I mentioned above, but I do, so I calculated season totals.

*** Note: I am missing week #5 stats because the link on the FBG site is incorrectly pointing to the week #6 stats. I will fix my numbers once I have the week #5 stats, but until then, my season totals are for all weeks except week #5. Can someone on the FBG staff fix the link to the week #5 stats and year-to-date stats after 5 weeks? ***

Here are the results:

Position Stat Actual Projected A-P Average WeeksTooHigh WeeksTooLow

DL Tkl 5417 5679 -262 -16.4 12 4

DL Asst 1989 1713 276 17.3 4 11

DL Sack 716 917 -201 -12.6 14 1

DL INT 23 2.5 20.5 1.3 2 12

DL FF 174 178.5 -4.5 -0.3 9 6

DL FR 133 125.9 7.1 0.4 8 7

DL PD 384 198 186 11.6 1 14

DL Pts 9776 10103.3 -327.3 -20.5 10 6

LB Tkl 7037 8550 -1513 -94.6 16 0

LB Asst 2683 3157 -474 -29.6 14 2

LB Sack 306.5 274.5 32 2.0 5 11

LB INT 92 63.3 28.7 1.8 4 12

LB FF 137 138 -1 -0.1 8 7

LB FR 91 96 -5 -0.3 7 8

LB PD 469 468 1 0.1 8 7

LB Pts 11638 12141.2 -503.2 -31.4 12 3

DB Tkl 8796 9266 -470 -29.4 14 2

DB Asst 2101 1682 419 26.2 1 15

DB Sack 90.5 24 66.5 4.2 1 15

DB INT 369 420.9 -51.9 -3.2 13 3

DB FF 160 62.5 97.5 6.1 1 15

DB FR 105 35.1 69.9 4.4 1 15

DB PD 1580 1756 -176 -11.0 13 3

DB Pts 14857.5 13813.8 1043.7 65.2 3 13

Total Tkl 21250 23495 -2245 -140.3 16 0

Total Asst 6773 6552 221 13.8 6 10

Total Sack 1113 1215.5 -102.5 -6.4 12 4

Total INT 484 486.7 -2.7 -0.2 9 7

Total FF 471 379 92 5.8 1 15

Total FR 329 257 72 4.5 2 14

Total PD 2433 2422 11 0.7 7 9

Total Pts 36271.5 36058.3 213.2 13.3 8 8

In the table above, you can see that Norton consistently overprojected tackles for LBs. He projected 8550 tackles all year (except week #5) and there were only 7037 tackles by LBs (excluding week #5). So he was off by 1513 tackles, for an average of 94.6 tackles too high per week. He was too high all 16 weeks (next to last column of table), and he was not too low in any week. For some categories, the sum of the last 2 columns is not 16 (for example, PDs for DLs 1+14=15) because the projection was equal to the actual result in a particular week.

Those numbers pretty much match what I expected before compiling the results. Norton projected too many sacks for DLs, too many INTs and PDs for DBs, not many tackles for LBs. Those are the bread & butter of each of the three positions. In addition, Norton projects too few INTs and PDs for DLs, too few sacks and INTs for LBs, and too few sacks and FFs / FRs for DBs. Those are the stats that are generally not the bread and butter of each position. I have not done this analysis with 2005 stats, but I am pretty sure we would find similar results.

Note that for the total fantasy points for all positions total (last row of table), Norton did pretty well. In 8 weeks he was too high, and in 8 weeks he was too low. He was off by only 13.3 pts per week on average for all NFL players (only 0.6% off). This is pretty close to zero and one could say there was virtually no bias. But the breakdown by position / stat category could be improved.

When looking at aggregate stats only, the biases should be fairly easy to fix. Total tackles for all LBs are pretty stable week to week. Even for the not so stable categories, like INTs for DLs, if you know there are about 1.5 INT by a DL each week (in 2006 there were 23 INTs by DLs in 16 games), then I think the weekly projections should have about 1.5 INTs total for all DLs. Norton only projected 2.5 INTs for DLs all year – clearly too low.

Hopefully I did not screw up on my spreadsheets and calculations. But if I spot or if anybody spots a problem, please let me know and I will re-post the numbers.

That’s it for now, later on I will post results for other analyses I plan on doing. Among other things, I will try to show that I believe the top players at each position have overstated projections. In other words, not only do LBs have too many projected tackles in total, but it is even more the case for the top LBs in the NFL. For all positions, I believe the projected fantasy points of the top players are too high on average, even though for all players of the NFL in total, Norton’s projections were pretty much in line. Finally, I will try to show that if there is a player who is not that good but Norton put him very high on the projections for that week (probably due to a favorable matchup), then on average those players were projected too many fantasy points.

For the moment I would appreciate comments on what I have so far.

 

[stevegamer]

Think of projecting the outcome of a die roll where people could bet over/under on your projection. The perfect projection is 3.5. If the outcome is 6, it does not mean your projection of 3.5 was wrong. Several people use the weekly projections in different ways. Some use them simply as a cheatsheet so they don’t care about the numbers, they only care about the rankings, so Norton needs to consider how the rankings will come out based on the projections. Others convert the projected stats into their scoring system to come up with a new cheatsheet. If that’s all you do, then whether all projections are 10% overstated or 10% understated for all players, you don’t care. But if you use the projections for other purposes, that may impact what you do. So again, in the perfect world, all projections would be unbiased for all players, and regardless what you use projections for, you will have something very good to work with. However, I fully understand that some people don’t place high value about all players being over/underprojected 10% across the board – they might only care about projections of certain players relative to one another (is player X projected more fantasy points than player Y?). Another argument I frequently read in the IDP forum but also in the shark pool: “well there is only a 0.5 pt difference between the projection of LB10 and LB20 or between WR10 and WR20, so it does not mean much”. Let me express my views on this. I totally disagree. Half a point is big. If you bet on sports, you already know that the casino has a small advantage (the juice, or the vig) and to be able to beat the casino, you need to exploit lines that may be off by as little as 0.5 pt. The difference between a 54% success rate and a 51% success rate is enormous. In the Vilma example above, if the projection was 6.5 tackles instead of 7.0 tackles, the projection would be too high 41% of the time and it would be too low 69% of the time. That is far from being a good projection. In my view, anything more than 0.3 or 0.4 difference in projected fantasy points is statistically relevant and I disagree with the argument that it’s not much. I have read from some posters before that they thought a difference in projections of 2 pts is not much. In my eyes, it is huge. If you were a casino offering a line that is 2 pts off (for over/under fantasy points of a certain player), you would go out of business in a heartbeat if you were taking bets from sharks that know their stuff.

A lot to digest, but I want to point out a few things about your assumptions:1. 3.5 is a perfect projection for an average roll of a standard d6, but it is a worthless projection for any one roll. The question becomes are people looking for projections to match what actually occurs, or the average of what occurs. I think it's very hard to determine what an average result would be for a non-repeating event.2. I think lots of folks ignore the half-to-two point variances, because one game is close to impossible to predict accurately within much closer tolerances than that.3. The casino analogy is somewhat flawed. A casino could go out of business being accurate if they don't get money on both sides. Even if they nailed your "perfect" projection point where it was a 50-50 over/under, what matters it that they project what the bettors think, not actual performance.4. I do think the consistent overprojection of certain statistics is very relevant - and in fact matters much more than the points scored variance. This overprojection will show up as significant noise int heprojections, which will skew the points scored projections.

 

[Biabreakable]

Nice post and very interesting.

I think this can be very usefull. I have approached offensive projections this way, wholisticly. Never looked at the defensive side in this way. Looking at total numbers of tackles and breaking them down by position.

When I choose IDP starters it is more based off gut matchup and how the player has been performing recently with only a little cosideration to how the player has performed in past years or even 4 weeks ago.

So coming from my neophite IDP perspective I can see how this can sharpen the tools we have available and make more informed conclushions.

I will add specific questions/comments later as I need to re-read the post.

In regards to your belief that a perfect projection should be based off of the probobility average in 100 games.

An NFL season is only 16 games long. So your perfect projection would be based off of 6.25 seasons of a player not missing a game. Injuries, players role in the defense and match ups are all going to change over that period of time. So while I understand the scientific method your employing here I am not really confident in it being a applicable control group to the question at hand. Which is what is a likely expectation/projection for player X this week. To me it would make more sense to break these numbers down by 16 and how players in similar situations have performed over a 16 game season as a basis for projections. I do a think that looking at historical trends in seasons would be usefull to increase the sample size. But it seems more logical to me to break things down this way.

When you mention people using the projections for other purposes than creating cheat sheets to help them decide who to start each week I wonder what other purposes people might use them for?

For making trade evaluations perhaps. What else?

I am somewhat suprised to see that defensive backs had more total tackles than linebackers did in 2006 which makes me wonder if this has always been the case but I just wasn't aware of it? Or if the total number of tackles for defensive backs has been increasing over the years compared to Lbers?

I realise that more defensive backs are on the field than linebackers are in 4-3 defenses and nickle/dime packages so this may have a lot to do with the total number. And teams pass the ball more than they run so this may have a effect on this as well but in any case interesting to me because I did not realise this before.

In regards to the Dlinemen and making interceptions. For the most part this is one of the flukiest plays ever made in the NFL. How can one predict this? One thing I might look at is Dlinemen getting passes defensed. The ones who are good at getting thier hands up and knocking balls down should have a better chance to tip a pass and intercept it. But a lot of the time that a play like this happens it is totaly on the Qb. The Dlinemen suddenly realises he has the ball. RRRUUUUNNN FOREST

 

[John Norton]

Very interesting stuff.

Before this thread gets too long I wanted to pass along a little background info. Some customers look at the projections for what they are, simply projections. However they are also used to drive the lineup and draft dominator tools that have become a staple of the FBG package. Many customers simply want to see where their guys "rank" while others will look harder at the specific numbers. The dilemma I/we face is that it's very difficult to make everyone happy with one set of projections but doing multiple sets of projections each week or having several writers do them simply isn't practical for us at this point.

There are several considerations that go into the numbers each week. I have a nifty tool that gives me the averages for each team broken down by position. For example this is the Cardinals '07 per game averages.

NAME POS YEAR WEEK TM TKL AST SCK INT FF FR PD

PER-GAME AVG db . . . 23.1 3.1 0.3 0.7 0.6 0.3 3.4

PER-GAME AVG dl . . . 10.6 3.4 1.4 0.1 0.4 0.6 0.9

PER-GAME AVG lb . . . 16.3 2.9 0.6 0.3 0.3 0.3 1.0

I also chart the tackle production by position for teams playing against each opponent. like this,

ARI MLB WLB SLB

1-SF 8-0 6-0 7-0-1

2-Sea 6-6 6-0-1 8-0-1

3-Stl 4-2 2-0 3-0

4-Atl 2-2 7-3 2-0

5-KC 2-3 5-1 1-0-1

6-Chi 11-8 13-2 3-1

7-Oak 5-0 4-1 0-0

8-GB 2-3-1 7-2 1-0

Bye

10-Dal 7-2 3-1 2-0-1 5-0

11-Det 6-2* 4-0 4-1

12-Min 0-1 1-2 2-0

13-Stl 6-0 1-1 3-0

14-Sea 4-0 3-1 3-5

15-Den 3-1 4-2 2-1

I then consider the averages and the history, factor in the variables of the match up such as injuries to linemen, backup quarterbacks or whatever might apply that could influence the numbers that week. I first project the overall team numbers. Say I am working on the Cardinals. They average 50 solo tackles per game. If they have an opponent with a strong offense/running game I would expect them to go over the average by a few that week and may project 55 tackles. Obviously if they play a horrible offensive team or one that passes more than they run, the tackle numbers would project below average. Once I have established my expectations for the team I set about distributing those numbers among the guys who I expect to see action. I use an excel file that has all the significant contributors on the defensive roster listed. I plug in the numbers then tweak them until I get a balance that I like.

There are a multitude of ways to analyze the numbers. I could spend 40 hours a week doing so and still be no better off simply because there is a human element involved and the numbers are far from absolute. Using the Vilma example. If 7 is the so called "perfect number" for him I could project him at 7 each week. According to this theory that would be very good for the season ending analysis but in reality the season is a roller coaster ride and I am not trying to project his average, I am trying to predict what he will do against a specific opponent in a specific game with a ton of variables specific to only that game.

Another factor here using Vilma as an example; Heading into the season I expected Vilma to be near the 100 tackle mark. As such I over projected him early in the season. Once it became painfully evident that he was not going to be that productive I began to back down his projected numbers but because of the high expectations early on, I still had him over-projected for the season. Likewise there are players who go the other way. Gerald Hayes comes to mind as a guy who I didn't project with big numbers prior to or early in the season, but he came on strong. I'll be the first to admit that predetermined expectations can get in the way, particularly early in the season, and there are undoubtedly players that I never did adjust quite far enough either way.

I find it both interesting and very surprising that I was so far off on the tackle numbers for linebackers. That's something I will take a serious look at. As for over projecting the good players, that comes down to the probability factor and the difficulty of trying to make rankings out of projections. How often are you going to project Zach Thomas for fewer than 5 tackles? It happened 4 times this season and he was over projected all 4 times. I can't drop his projected numbers for the next game to get the overall numbers back in line. I have to project the next game for what I expect from that game. In short I'm not so much concerned with the season totals as they don't really apply to what I am trying to do. My goal is to be as accurate as possible from week to week.

It was mentioned that I only projected 2.5 interceptions for defensive linemen this year while there were actually 23 picks by DL on the season. That comes down to the probability factor as well. It's highly unlikely that a lineman is going to make an interception. With a few exceptions (IE Jason Taylor and/or some of the zone blits schemes) linemen are not going to be covering receivers and the vast majority of the picks they do get are going to be off batted passes. How can I project or predict those 23 pure luck picks among 256 games?

Biabreakable mentioned his surprise that D-Backs make more tackles overall. At first glance you may not expect that but in reality there are many more of them that see game action than the other positions. When you look at an average box score the bottom is normally littered with special teams guys and dime backs who are 1-0. All those add up. This is very likely a contributing factor to my over-projection of linebackers as well. I don't have 6-7 defensive backs on my spread sheet in most cases and probably have a tendency to give many of those extra tackles to the linebackers.

One of the issues I struggle with while doing projections each week is finding a balance between projecting to create a logical ranking for those who look only at the ranking, or projecting the actual box score for those who concentrate more on the raw numbers. It can be a major catch 22 for me. One thing we did differently this year was to break down big plays into decimal points so that I could project a guy with a portion (say .2) of an interceptions. No one catches half and interception so this is obviously geared toward projecting to create a ranking. Personally I would prefer to project actual box scores but giving one guy a full interception would seriously skew the rankings so I am trying to find a happy medium. There are times that I have a feeling a guy will be huge, if I project him big and he lives up to his billing it's expected and I don't hear a word about it, but if he doesn't, I have mud on my face and hear a lot of chin music from you guys about the top ranked linebacker for the week going 5-2 <_<

As I have said before we are always looking to improve the product. Thanks for the feedback

 

[Biabreakable]

I was just looking at some historical numbers to see how some of these things matched up compared to other seasons.

Dline interceptions in 2005 were 13 total(9 players with only 1) FF were 206(72 players with only 1) PD were 391(58 players with only 1) Sacks were 726.5( 40 had only 1 10 had .5)

Dline interceptions in 2004 were 28 total(only one player had more than 1) FF were 206(71 players with only 1) PD were 415 (61 players with only 1) Sacks were 745.5 (50 had only one 11 had .5)

I point out the player who only had one time out of 16 games of getting awarded with a play in each category because I don't think this is high enough to really be able to expect that player to repeat the play in a projection. They only have a 6.25% chance of accomplishing this over a season. A player with 2 has a 13% chance so that would be the minimum I could see myself possibly giving the player a chance to do it in a game.

The numbers besides interceptions for Dline do have some consistency from year to year that I can see projecting after taking out the 1's and .5s though.

If anyone wants to add to these breakdowns I would be interested to see the numbers for 2006/2003 and for other positions but I got tired of counting on NFL.com

My mothod for projecting using 2-3 year averages for players who are actualy producing somthing significant in the scoring categories may be more subjective than a wholistic projection. But it saves me time and keeps my focus limited to players I might actualy roster/start. The weakness of this is me missing players who may be on the verge of becoming starting caliber performers such as Trent Cole did this year.

Trent Cole is an example of a player who did decently as a rookie 38 solo tackles 5 sacks 2 PD 1 FF but still was able to fly under my radar.