"An Indepth Look at Accuracy of Weekly Projections" (1 Viewer)

[David Dodds]

A lot of people have posted a very simple question. Are FBG projections any good? I have always contended that we have the best processes and these processes result in the best projections. Almost any site can come up with decent preseason projections / rankings by averaging mulitiple sets of data, comparing against ADP, etc. But the weekly stuff is tough because you have such a short window to get it right. I know what we do to come up with numbers and it's exhausting. We simulate the games. We turn these simulations into player projections that track to league wide levels for TDs, yards, etc. Others have done studies and we generally rate very high in these studies, but almost without fail these comparisons tackle the task in weird ways that we are usually left scratching our head at. And the fact that we change our projections within minutes of the game starting, we have always been a bit suspect that others were using the right data.

Here is our general plan. I say general, because I want this board to kick it around before we settle on a way to score this, etc.

The study will go from week 6 through week 16 of this fantasy season. We will only use sites that have projections (and we will settle on which sites will be tracked before the study starts). All projections will be converted to fantasy points using FBG scoring.

Based on actual fantasy points, we will record the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs. Those are the scorecard players. Each site will have it's projections for said players converted to fantasy points and compared.

For example: P. Manning is a top 20 QB in wek 8 and throws for 260 yards, 2 passing TDs, 1 interception and has 2 rushing yards. FBG scoring calculates this as 265/20 +4*2 - 1 + 2/10 = 20.2 fantasy points... If site A projected 290 yards and 2 TDs and site B had it 265 yards, 1.8 TDs, 0.6 ints, and 3 yards. Then the comparison would look like this:

Site A translates to 22.5 FP and Site B translates to 265/20 + 1.8*4 -.6*1 + 3/10 = 13.25 + 7.2 -.6 +.3 = 20.15

Site A = 1 - ( |20.2 - 22.5| / 20.2 ) = 88.6% accuracy for P. Manning

Site B = 1 - ( |20.2 - 20.15| / 20.2 ) = 99.8% accuracy for P. Manning

The plan is to then add up for each position, an overall, etc for each site by week.

I know this isn't an exact way to meaure things. I am not sure it makes sense to add all 20 QBs and assign equal weights. I am not sure 20 QBs is the right number to assess, etc. I am posting this thread because I want it all out in the open on how best to do this.

We will publish the full plan PRIOR to beginning this study. This will include documenting the times we are extracting info, etc. We will likely need some help from people to certify our processes and methods as we embark on this.

We will publish the results REGARDLESS of how we finish in this. We contend we have the right processes and these processes will lead to the best data. If we find that to be false, then we hope the results will help point to areas where we can improve.

Andy Hicks from our staff will be in charge of this study. Let us know your thoughts, sites you would like assessed, etc and we will see what we can do here. We are looking to have the full plan in place by Sep 30th.

 

[gianmarco]

I'm glad this is being approached, but this is my only "beef" with what's proposed and what I would prefer seeing myself.

Ranking Peyton in the top 20 doesn't take much to do. Getting extra points because you predicted a slightly better day vs. another site doesn't mean your projections are necessarily better.

To me, it's the "against the grain" rankings that matter most. Guys that you are considerably higher than or lower than the consensus. Those are the calls that can make or break a fantasy week. For example, most wouldn't have advocated starting Matt Forte this week over Addai. However, if someone had actually done this and given a sound reason why, that means a lot more to me than a site that ranks Peyton #3 vs. #6. Likewise, if a site warned us that Palmer might be in for a tough day and to pursue other options when every other site has him ranked anywhere in their top 5-10, then those are the things I'd like to know.

Anybody can visit a multitude of sites and average out the info to get an idea of who to start or how to rank a player. And oftentimes, these average rankings work fine. It's the small things like knowing to start Deshaun Jackson or Dante Rosario with confidence for x and y reasons that would make me trust one site over another. If I see a site making statements like that and they have a track record of being right more often than others, I would pay much more attention to them on those items that actually matter more than a simple average ranking. Likewise, if a site continually makes "against the grain" projections that are usually wrong without consequence, it would be nice to know that they are simply putting out those things to be different and that doesn't help me much at all.

When bye weeks and injuries kick into full gear, it's these types of sit/start decisions that may not be obvious that can make or break a season.

 

[Moe Green]

 

[tomarken]

This is a fantastic idea David, really looking forward to it. For the record, before the study or anything else, I want you to know that even if FBG's comes in dead last, as long as I am playing fantasy football I will be a Footballguys subscriber. What you guys offer is so much better than the competition it's really not close. Anyway, this will be an interesting exercise, I'm sure I'll have some ideas in the next few weeks but for starters:

I am not sure it makes sense to add all 20 QBs and assign equal weights.

I think it may be beneficial to assign larger weights to lower ranked players. For example, most people are not typically facing a decision between starting the #3 QB or the #5 QB. Even if your top 10 QB's are not all in the correct order, most people are going to start their studs. (For example, I don't care whether you have Romo ranked higher than Peyton, or Peyton ranked higher than Romo - 99% of people don't have to decide between them and will be starting whichever of the two they are lucky enough to own anyway.)On the other hand, I often find myself referring to your projections when trying to decide between starting, say, the #14 QB and the #18 QB. Nailing these projections is much more useful for most fantasy footballers, I would imagine. I know it is for me. So these types of projections should weigh more heavily on your "accuracy score" IMO.

 

[supersecretid]

I VERY strongly think you need to use logarithmic rankings.

If you project Manning at 15, the difference between him actually scoring 16 and actually scoring 17 is bigger than the difference between him actually scoring 50 and actually scoring 51.

In the long run it's likely to balance out across sites, but I think this would be a better methodology. With all the stat heads you guys have around I'm sure you could implement this pretty easily. I would get a hold of a couple of stat geniuses, both in-house and out and ask for any other tips on making this study as meaningful as possible.

 

[Dinsy Ejotuz]

IMO as long as you're clear about your methodology and make the raw before and after data available to people, you're covered no matter what. Either people like your method, or they can figure out how to do it in a way that makes them happy.

 

[beavers]

.

 

[jonmhend]

I look at Fantasy Index and Fantasy Guru. I'd like them included.

Haven't thought about the process much, but I'd consider using a mechanism to toss outliers so they don't overly skew the results. Maybe toss the 2 worst QB, 4 RB, 5WR, and 2 TE predictions for each source.

 

[JKL]

While your projections should inform your list order rankings, and if you have better weekly projections you will do well, I would prefer a rank order evaluation as opposed to just looking at absolute value differences between a player's projection and actual results. This is because of the difference between drafting, and making a Yes/No choice of starting a player among more limited options. Let's say Site A, who is a bit wacky, projects Jay Cutler to throw for 500 yards and 5 TD's, while most people have more realistic projections and have him ranked about 6th in a weekly rank. If he throws for 350 and 3 TD's and is the highest scorer at QB, the more conservative projections were closer, but the outrageous projection correctly predicted Cutler to be the best play of the week.

Here is my thought. Let's say you have 10 different sites you are monitoring as part of the project. Find the top 12 QB's for each. There will be alot of consensus and slight shuffling, but there will be some differences. Any QB who appears somewhere in a top 12 list is then included--let's say there were 16 distinct QB's who appear in someone's top 12. These 16 QB's are then assessed, Won-Loss style, head to head at the end of the week's games. Your QB1 goes head to head against the other 15 (if he was actually #1, you would get a 15-0 for putting him there), your QB2 then goes head to head against the remaining 14, QB3 then H2H against the remaining 13, etc. Let's say you have Matt Cassel in your top 12 this week while most do not. If he actually puts up a top 10 score, then you are going to, relatively speaking, have a higher winning percentage than those that had him 16 out of 16. But if no one puts him top 12, then he doesn't count against the starters if he has a good performance.

I don't care if someone correctly predicts Kyle Orton over Damon Huard. I do care if you are going to tell me Matt Schaub is a fantasy start or not on a given week relative to some other site. This method would seem to do pare it down to those players that someone-anyone- considers a fantasy starter. You could do the same at other positions, just use higher numbers, like top 24 running backs or top 30 wide receivers.

 

[gianmarco]

While your projections should inform your list order rankings, and if you have better weekly projections you will do well, I would prefer a rank order evaluation as opposed to just looking at absolute value differences between a player's projection and actual results. This is because of the difference between drafting, and making a Yes/No choice of starting a player among more limited options. Let's say Site A, who is a bit wacky, projects Jay Cutler to throw for 500 yards and 5 TD's, while most people have more realistic projections and have him ranked about 6th in a weekly rank. If he throws for 350 and 3 TD's and is the highest scorer at QB, the more conservative projections were closer, but the outrageous projection correctly predicted Cutler to be the best play of the week.Here is my thought. Let's say you have 10 different sites you are monitoring as part of the project. Find the top 12 QB's for each. There will be alot of consensus and slight shuffling, but there will be some differences. Any QB who appears somewhere in a top 12 list is then included--let's say there were 16 distinct QB's who appear in someone's top 12. These 16 QB's are then assessed, Won-Loss style, head to head at the end of the week's games. Your QB1 goes head to head against the other 15 (if he was actually #1, you would get a 15-0 for putting him there), your QB2 then goes head to head against the remaining 14, QB3 then H2H against the remaining 13, etc. Let's say you have Matt Cassel in your top 12 this week while most do not. If he actually puts up a top 10 score, then you are going to, relatively speaking, have a higher winning percentage than those that had him 16 out of 16. But if no one puts him top 12, then he doesn't count against the starters if he has a good performance.I don't care if someone correctly predicts Kyle Orton over Damon Huard. I do care if you are going to tell me Matt Schaub is a fantasy start or not on a given week relative to some other site. This method would seem to do pare it down to those players that someone-anyone- considers a fantasy starter. You could do the same at other positions, just use higher numbers, like top 24 running backs or top 30 wide receivers.

and pretty much what I was trying to say as well.

 

[supersecretid]

While your projections should inform your list order rankings, and if you have better weekly projections you will do well, I would prefer a rank order evaluation as opposed to just looking at absolute value differences between a player's projection and actual results. This is because of the difference between drafting, and making a Yes/No choice of starting a player among more limited options. Let's say Site A, who is a bit wacky, projects Jay Cutler to throw for 500 yards and 5 TD's, while most people have more realistic projections and have him ranked about 6th in a weekly rank. If he throws for 350 and 3 TD's and is the highest scorer at QB, the more conservative projections were closer, but the outrageous projection correctly predicted Cutler to be the best play of the week.Here is my thought. Let's say you have 10 different sites you are monitoring as part of the project. Find the top 12 QB's for each. There will be alot of consensus and slight shuffling, but there will be some differences. Any QB who appears somewhere in a top 12 list is then included--let's say there were 16 distinct QB's who appear in someone's top 12. These 16 QB's are then assessed, Won-Loss style, head to head at the end of the week's games. Your QB1 goes head to head against the other 15 (if he was actually #1, you would get a 15-0 for putting him there), your QB2 then goes head to head against the remaining 14, QB3 then H2H against the remaining 13, etc. Let's say you have Matt Cassel in your top 12 this week while most do not. If he actually puts up a top 10 score, then you are going to, relatively speaking, have a higher winning percentage than those that had him 16 out of 16. But if no one puts him top 12, then he doesn't count against the starters if he has a good performance.I don't care if someone correctly predicts Kyle Orton over Damon Huard. I do care if you are going to tell me Matt Schaub is a fantasy start or not on a given week relative to some other site. This method would seem to do pare it down to those players that someone-anyone- considers a fantasy starter. You could do the same at other positions, just use higher numbers, like top 24 running backs or top 30 wide receivers.

and pretty much what I was trying to say as well.

The counter to the argument of rank being most impt, and lower ranks being more impt is that not all leagues are alike.Many people play in salary cap games or even stock market-based games. Also many people will base a trade on what improves their chances for the given week. Also you just never know who will be on the wire depending on the strength of a league and how quirky the set-up may be. These all get at the same thing: for many ppl (not all, not even most) every ranking is important, and actual numbers are impt too.

 

[Mimo]

IMO some guys need to tone down the expectation of the site. This site does well with projections, analysis etc. I was reading in this trend that some of us want to see the small things like analysis on why we should start the Dante Rosarios, Deshan Jackson or Hank Basketts of the world with confidence in week 1. You have to be kidding me if anyone thought to even play some of these guys week one. I know we already know it but this hobby we have requires a whole lot of luck too.

 

[puckalicious]

I'm assuming you'll toss out evaluations on players that get injured during the game? Hopefully nobody gets penalized for predicting Brady to get 25 points in week 1 when he actually got 3.

Although this may need to be a bigger discussion. At what point do you say the injury skewed the rankings vs not? Certainly a season ending knee injury in the 1st quarter should be viewed differently than a guy that sits in the 4th quarter due to a sore hammy.

 

[AcerFC]

This should be interesting

 

[Iwannabeacowboybaby!]

What you're embarking on is a huge project and could be too big if you let it.

I think you should focus the study on:

Top 12 QB's

Top 24 RB's

Top 36 WR's

Top 12 TE

I say that because that's your basic starter for standard leagues.

Maybe you get a bonus point for any player that you nail on the money.

For example, if FBG predict MJD to be RB 11 and he actually finishes RB 11, you get a bonus for not only predicting him in the top 24 which I'm sure every site will do, but you had him where he should be.

It may not be rocket science to pick Tony Romo in the top 12 QB's, but it's still solid work if you pick him QB no. 3 and he actually finishes there. Equally impressive if you pick Jake Delhomme to finish No. 12 and he finishes 12, awesome, you deserve a bonus point.

You could even set up a negative detractor for certain things if you wish. For example, you could say any top 5 pick that doesn't finish in the top 12 QB's or 24 RB's or 36 WR's gets a negative point.

If you picked Peyton Manning to finish 4th and he finishes 16th for the week, then you deserve a slap on the wrist, because guess what, you were off.

 

[The Dude]

I think it's a lot if work with no payoff. So you rank well - great - but is it significant? Or you rank poorly - is that significant? There's a huge amount of randomness - one week you can easily be tops and the next week bottom. It's only usefule if there is something you can learn to make your projections better - and I don't think you will find it - not that yours cannot be improved, there's just no magic ingredient that makes it all work.

I think it would make better sense to evaluate your rankings. Doesn't matter if Romo is ranked 3rd and projected 300 yards passing and 2 TD - it just matters that he finishes 3rd - whether it's with 350 yards passing or 195.

 

[tomarken]

It absolutely matters what the projections are, far more than the actual numerical rankings (which are determined by the projections anyway). What about the millions of leagues with flex positions? If I can start Larry Fitz or Frank Gore as my flex player, what good does it do me to know that Fitz is WR3 and Gore is RB5?

I think this is an important enough point to be settled early on in this process: I propose that we agree to work off of projections, and not numerical rankings. We should agree to this before we get any farther into discussing methods of evaluating the various sites.

 

[Gatorman]

david,

I think it works, and it is going to be hard to look at it in terms of ranking rather than projections, so I would stick to projections. I would also add that you may want to add a few raw categories to help us understand the "shape factor" if you will of what you are looking at (this will help with those who are looking at rankings rather than projections)...

Namely:

If manning has 22 or 18 and his projections were 20, it looks like the same error, but it is not. One was an overestimation and one was under. I believe that a value should be there for over and under the baseline, as well as one where you are way off (non injury way off).

Figure it should look like this:

Fantasy points Points projected per site acuuracy (+ or -) and than flag the outliers.

figure if you are off my more than 25% (one way or another), that is an outlier (you can also try t calculate a better outlier baseline)

IMHO

Gator

 

[Iwannabeacowboybaby!]

After thinking on it more, I too think it should be done on projections and not rankings because of the Flex position.

Some of the hardest decisions a fantasy owner has to make is deciding upon what player to use in that 1 flex spot.

Do you use this WR, this WR or that RB for that 1 position. Those types of decisions win and lose games every week.

 

[David Dodds]

I know we will definitely use projections and not rankings. The second you use rankings, every site has an excuse. We score our rankings differently, etc. And the point about the flex is right on. Most people have to make those decisions every week.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

 

[tomarken]

I know we will definitely use projections and not rankings. The second you use rankings, every site has an excuse. We score our rankings differently, etc. And the point about the flex is right on. Most people have to make those decisions every week.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

While I agree that this is important and should be done by every site, isn't including it in the analysis also a little bit of self-serving scale-tipping? You already know FBG's does this!

 

[ComfortablyNumb]

I think you are placing way too much emphasis on quantitative vs qualitative weekly projections given the week to week variability and uncertainty of NFL stats. One week is just too small a sample to predict stats with any accuracy.

I'd rather see a simple average of the staff members independent weekly rankings, like you do in preseason.

 

[T Bell]

David, how many points are awarded to players who are not within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs in your rankings, but who end up there, or who are within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs but who don't end up placing there when it's all said and done?

 

[tomarken]

David, how many points are awarded to players who are not within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs in your rankings, but who end up there, or who are within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs but who don't end up placing there when it's all said and done?

The quote below makes it sound like these would not be factored in.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

 

[Bri]

Been wanting this for a long time

How will ya deal with injuries?

I don't think there's much to predicting Derrick Mason(for example) will get 4 catches for 60 yards but, the week he gets 9 catches for 100+, that's what I want to know most. Is there a way to put more weight toward getting the highest(and lowest I suppose) weeks right?

 

[T Bell]

David, how many points are awarded to players who are not within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs in your rankings, but who end up there, or who are within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs but who don't end up placing there when it's all said and done?

The quote below makes it sound like these would not be factored in.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

Isn't that a major omission? It means that this study turns a blind eye to the ability of sites to predict sleepers in a given week, right?

 

[Aaron Rudnicki]

David, how many points are awarded to players who are not within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs in your rankings, but who end up there, or who are within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs but who don't end up placing there when it's all said and done?

The quote below makes it sound like these would not be factored in.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

Isn't that a major omission? It means that this study turns a blind eye to the ability of sites to predict sleepers in a given week, right?

as long as the a sleeper winds up among the highest-scoring 20 QBs, 35 RBs, 50 WRs, and 15 TEs in a given week, he will be counted. if a player gets injured and doesn't wind up in that group, he will not be counted.

 

[tomarken]

David, how many points are awarded to players who are not within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs in your rankings, but who end up there, or who are within the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs but who don't end up placing there when it's all said and done?

The quote below makes it sound like these would not be factored in.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

Isn't that a major omission? It means that this study turns a blind eye to the ability of sites to predict sleepers in a given week, right?

No - if a sleeper is predicted to finish in the top 20 (or whatever) and actually finishes in the top 20, they will be counted. This doesn't fit into the categories you described.

 

[Bri]

In regards to 50 WRs and 35 RBs. When you get past the especially good bunch and get into the backup RBs and maybe guys like Amani Toomer or Darrell Jackson or Ladell Betts. Is it really an accomplishment to correctly predict 25 yards rushing, 2 catches for 10 yards for Ladell? Or 2 for 35 for Amani and Darrell? In your math formula wouldn't these players result in some of the best (highest %) and easiest projections?

If so, maybe you're going a smidge too deep

 

[Aaron Rudnicki]

Based on actual fantasy points, we will record the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs. Those are the scorecard players. Each site will have it's projections for said players converted to fantasy points and compared.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

 

[elshagon]

I agree in projections being used over rankings. rankings don't mean much unless you apply your leagues scoring system to them. One thing I've seen suggested in the past is to perhaps place a note on players that move up/down the most in early projections compared to later in the week. It's nice to see the thinking involved in the ranking change, whether it be because of injury, weather, etc. The study is cool because it will let us compare FBG's to other sites. I know just here on FBG's there's certain staff I trust with their RB rankings, but not so much for QB's. I have a feel for some positions more so than others, it will be good to see if FBG's are better at predicting certain player positions over others/

 

[Bri]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

 

[tomarken]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

Yes.

 

[yonder]

If the measure of accuracy is solely based on the projections and rank is ignored, I think the dataset needs to be larger.

The true value of these projections come in the mid-tier players (in my opinion) as the studs are going to be in the lineups no matter what the projections are.

 

[GoBears84]

I know this isn't an exact way to meaure things. I am not sure it makes sense to add all 20 QBs and assign equal weights. I am not sure 20 QBs is the right number to assess, etc. I am posting this thread because I want it all out in the open on how best to do this.

David,I applaud the effort. It's not an easy thing to do. You may remember that last year I took a stab at the accuracy of the 2006 Projections. It was time consuming but informative.

Having been through a similar effort I think I can add some insight.

First, consider taking the squared residual rather than simple difference. It will also allow you to manipulate the data easier and make more valid comparison.s

Second, if you haven't already - talk to a statistician. I was amazed at the input I received from statistics professors who frequent the board.

Finally, while all the sharks are here in the pool, not all of them are mathematicians and there are a number of really smart FBG subscribers who aren't always in the Shark Pool. I might suggest putting this out as part of one of the weekly emails to see if you can get input from a real expert.

And I agree with another poster -- I expect that FBG's can't be 100% correct 100% of the time. But the insight, the data and the tools are top notch and allow me to make informed decisions.

Best regards.

 

[tomarken]

Ooh, how's this for a crazy idea...

By week six or whenever this is starting, we'll have PPG averages for everyone, yes?

Then instead of using the top-20 or whatever, why don't you compare projections for the 20 players whose performance that week deviated the most from their season average?

Say, for example, that through six weeks Amani Toomer is averaging 3.5 fantasy points per game. In week 7 he scores 14 fantasy points. He would be one of the players you would use.

Assume also that Peyton Manning is averaging 20 PPG through six weeks, and in week 7 he scores 11 points. He might be another one of the players you would use.

Etc.

This would allow you to compare the abilities of the various sites to correctly identify players who will significantly under- and over-perform their expected production.

 

[Driver]

A lot of people have posted a very simple question. Are FBG projections any good? ....

The study will go from week 6 through week 16 of this fantasy season. We will only use sites that have projections (and we will settle on which sites will be tracked before the study starts). All projections will be converted to fantasy points using FBG scoring.

Based on actual fantasy points, we will record the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs. Those are the scorecard players. Each site will have it's projections for said players converted to fantasy points and compared.

For example: P. Manning is a top 20 QB in wek 8 and throws for 260 yards, 2 passing TDs, 1 interception and has 2 rushing yards. FBG scoring calculates this as 265/20 +4*2 - 1 + 2/10 = 20.2 fantasy points... If site A projected 290 yards and 2 TDs and site B had it 265 yards, 1.8 TDs, 0.6 ints, and 3 yards. Then the comparison would look like this:

Site A translates to 22.5 FP and Site B translates to 265/20 + 1.8*4 -.6*1 + 3/10 = 13.25 + 7.2 -.6 +.3 = 20.15

Site A = 1 - ( |20.2 - 22.5| / 20.2 ) = 88.6% accuracy for P. Manning

Site B = 1 - ( |20.2 - 20.15| / 20.2 ) = 99.8% accuracy for P. Manning

The plan is to then add up for each position, an overall, etc for each site by week.

I know this isn't an exact way to meaure things. I am not sure it makes sense to add all 20 QBs and assign equal weights. I am not sure 20 QBs is the right number to assess, etc. I am posting this thread because I want it all out in the open on how best to do this.

I know we will definitely use projections and not rankings. The second you use rankings, every site has an excuse. We score our rankings differently, etc. And the point about the flex is right on. Most people have to make those decisions every week.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

Excellent idea! Good luck with it. A couple comments:1. Please include fantasyguru.com (John Hansen)

2. Regarding selecting the top-20 QBs based on actual fpts scored each week, I think you might be introducing some bias in determining which site has the "most accurate" projections. For example, what you are proposing is similar to evaluating the performance of 10 portfolios (containing 25 stocks each) recommended by an investment professional at the beginning of the time period. Then, looking at the performance of all stocks during a specified time period, and picking the top-20 (best-performing) stocks -- and measuring each portfolio's performance based on how many of the best-performing stocks they contained.

I haven't stated the problem too clearly. Basically the problem is that you haven't included the "poor or average performers." I think a better approach would be to include (in the group of QBs to be evaluated, for example): (1) the top-20 highest-scoring QBs each week, and (2) any QB ranked in the top-15 (projections) of any site being evaluated. I think this would eliminate some of the problematic bias, and provide a more stable group of players, at each position, for evaluation purposes.

Including only the top-20 scoring QBs each week would probably benefit FBG, compared to other sites. But the major problem is that you don't want to exclude Peyton Manning because he scored outside of the top-20 one week, and Drew Brees because he scored outside the top-20 the next week. Manning (in the first week) and Brees (in the second week) would have actual points much less than predicted points for virtually every site. But you don't want to exclude these data points -- IMO they are as valid as Matt Ryan scoring much higher than predicted (and being included in the group of the top-20 scoring QBs). You want to measure both positive and negative differentials between "predicted" and "actual." Maybe focus on average "absolute % difference" to measure the average difference between predicted and actual (either positive or negative). And I think you want to divide by predicted points (22.5 rather than 20.2) in the example above for Manning, Site A.

I think this is an extremely important project. But you want to do it right (and I may be off base in some of my suggestions). Good luck.

 

[a_troll00]

In general, I don't put a whole lot of weight in weekly projections. I just tend to look at matchups instead. Then again, you can play a great player against an ideal matchup... and he could still lay an egg. That's football for you.

 

[davearm]

I know we will definitely use projections and not rankings. The second you use rankings, every site has an excuse. We score our rankings differently, etc. And the point about the flex is right on. Most people have to make those decisions every week.Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

Dodds you're right on with projections vs. rankings. The point of the exercise is to see who is best at projecting stats (yards, TDs, etc), because whoever does that the best will also have the best rankings.And using FBG scoring is fine; any standard scoring system will work.As you allude to, the primary challenge is how best to draw the sample. From what you've described, your sample will change week to week. Seems to me you could do just as good a job by settling on a core group of players that encompasses both regular starters and fringe guys by simply tracking the top 20 QBs, top 30 RBs, top 40 WRs, and top 20 TEs by final ADP, and using that same sample of 110 guys throughout the year. Or to put it another way, I don't see the problem with assessing the accuracy of the predictions for the same, consensus best players every week.I'm also onboard with a side study to see how a particular site's cumulative yards and TDs projections track with the leaguewide actuals. Looking at some site's weekly projections, you're left feeling like every game is going to end 38-34.As far as timing goes, that seems like a nonissue. Just collect the data after the games have been played. I can't imagine sites are fudging their numbers ex post.Anyway, I'll be very interested to see the findings here. Great idea.

 

[Aaron Rudnicki]

after reading Driver's post, I wonder if this type of analysis would be biased towards websites that tend to overproject for players. In other words, if Website A tends to be more generous with ALL of their projections each week, then they are more likely to have higher projections for the players who come out of nowhere to post big games. This would likely give them a big edge here compared to a potentially small hit they'd take from overprojecting the top players. Meanwhile, a comparable website that may be more realistic in their projections for all players could wind up looking worse in comparison by being more realistic on the top players and coming in lower on the sleeper guys.

 

[Pictus Cat]

I would think every site that does projections would jump on this.

Nothing to lose and FBG guys to attract.

I think an analysis of what projections would have been most usefull would be nice too.

For example, x's projections would have lead you to start x over y, which would have been a right call.

 

[T Bell]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

Yes.

No, it's both. You have to compare the top 20 as predicted with the actual top 20. That's why I'm asking how you deal with players in one set but not in the other.

 

[Aaron Rudnicki]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

Yes.

No, it's both. You have to compare the top 20 as predicted with the actual top 20. That's why I'm asking how you deal with players in one set but not in the other.

the plan is to just look at the top-20 QBs in ACTUAL points scored and compare how accurate each site was in predicting their production. The plan is not to compare QBs who were predicted to finish in the top-20 each week.

 

[davearm]

A lot of people have posted a very simple question. Are FBG projections any good? ....

The study will go from week 6 through week 16 of this fantasy season. We will only use sites that have projections (and we will settle on which sites will be tracked before the study starts). All projections will be converted to fantasy points using FBG scoring.

Based on actual fantasy points, we will record the top 20 QBs, top 35 RBs, top 50 WRs and top 15 TEs. Those are the scorecard players. Each site will have it's projections for said players converted to fantasy points and compared.

For example: P. Manning is a top 20 QB in wek 8 and throws for 260 yards, 2 passing TDs, 1 interception and has 2 rushing yards. FBG scoring calculates this as 265/20 +4*2 - 1 + 2/10 = 20.2 fantasy points... If site A projected 290 yards and 2 TDs and site B had it 265 yards, 1.8 TDs, 0.6 ints, and 3 yards. Then the comparison would look like this:

Site A translates to 22.5 FP and Site B translates to 265/20 + 1.8*4 -.6*1 + 3/10 = 13.25 + 7.2 -.6 +.3 = 20.15

Site A = 1 - ( |20.2 - 22.5| / 20.2 ) = 88.6% accuracy for P. Manning

Site B = 1 - ( |20.2 - 20.15| / 20.2 ) = 99.8% accuracy for P. Manning

The plan is to then add up for each position, an overall, etc for each site by week.

I know this isn't an exact way to meaure things. I am not sure it makes sense to add all 20 QBs and assign equal weights. I am not sure 20 QBs is the right number to assess, etc. I am posting this thread because I want it all out in the open on how best to do this.

I know we will definitely use projections and not rankings. The second you use rankings, every site has an excuse. We score our rankings differently, etc. And the point about the flex is right on. Most people have to make those decisions every week.

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week). I also think it's important to total a site's projections and see how many yards, TDs, etc they were predicting. Even if they missed for a week, I would want to know that the site framed their numbers in reality (and didn't overstate TDs by 15%).

Excellent idea! Good luck with it. A couple comments:1. Please include fantasyguru.com (John Hansen)

2. Regarding selecting the top-20 QBs based on actual fpts scored each week, I think you might be introducing some bias in determining which site has the "most accurate" projections. For example, what you are proposing is similar to evaluating the performance of 10 portfolios (containing 25 stocks each) recommended by an investment professional at the beginning of the time period. Then, looking at the performance of all stocks during a specified time period, and picking the top-20 (best-performing) stocks -- and measuring each portfolio's performance based on how many of the best-performing stocks they contained.

I haven't stated the problem too clearly. Basically the problem is that you haven't included the "poor or average performers." I think a better approach would be to include (in the group of QBs to be evaluated, for example): (1) the top-20 highest-scoring QBs each week, and (2) any QB ranked in the top-15 (projections) of any site being evaluated. I think this would eliminate some of the problematic bias, and provide a more stable group of players, at each position, for evaluation purposes.

Including only the top-20 scoring QBs each week would probably benefit FBG, compared to other sites. But the major problem is that you don't want to exclude Peyton Manning because he scored outside of the top-20 one week, and Drew Brees because he scored outside the top-20 the next week. Manning (in the first week) and Brees (in the second week) would have actual points much less than predicted points for virtually every site. But you don't want to exclude these data points -- IMO they are as valid as Matt Ryan scoring much higher than predicted (and being included in the group of the top-20 scoring QBs). You want to measure both positive and negative differentials between "predicted" and "actual." Maybe focus on average "absolute % difference" to measure the average difference between predicted and actual (either positive or negative). And I think you want to divide by predicted points (22.5 rather than 20.2) in the example above for Manning, Site A.

I think this is an extremely important project. But you want to do it right (and I may be off base in some of my suggestions). Good luck.

This is well-said. Taking the actual top 20 performing QBs for a given week introduces a clear bias by excluding the subset of guys that underperform (perhaps dramatically underperform) their projection that week.

That's definitely a measure you'd want to include in the study.

 

[davearm]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

Yes.

No, it's both. You have to compare the top 20 as predicted with the actual top 20. That's why I'm asking how you deal with players in one set but not in the other.

the plan is to just look at the top-20 QBs in ACTUAL points scored and compare how accurate each site was in predicting their production. The plan is not to compare QBs who were predicted to finish in the top-20 each week.

The latter is a better approach, for the reasons outlined above.

 

[T Bell]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

Yes.

No, it's both. You have to compare the top 20 as predicted with the actual top 20. That's why I'm asking how you deal with players in one set but not in the other.

the plan is to just look at the top-20 QBs in ACTUAL points scored and compare how accurate each site was in predicting their production. The plan is not to compare QBs who were predicted to finish in the top-20 each week.

Ok, got it. It's purely a projections comparison, not a rankings comparison.

 

[NCPanthersFan]

It is out of the box thinking like this that ALWAYS keeps me coming back for year after year. Whether or not you did this - the community you have created at FBG (both subscriber and non-subscriber) is the real magic. Draftsharks is one I would like to see.

Thanks for the product guys.

 

[davearm]

Some others might have misunderstood what I said when I said we will use the Top 20 QBs, etc. That list will be determined by actual fantasy performance. So if Kyle Orton busts out a huge game, his game counts. I think this gives us a good sampling of a lot of different players (instead of looking at who gets ranked as a top 20 QB which would likely be the same players almost every week).

bump for this key piece as it doesn't seem like everyone is understanding it.

so it's top 20 AFTER the games not before?

Yes.

No, it's both. You have to compare the top 20 as predicted with the actual top 20. That's why I'm asking how you deal with players in one set but not in the other.

the plan is to just look at the top-20 QBs in ACTUAL points scored and compare how accurate each site was in predicting their production. The plan is not to compare QBs who were predicted to finish in the top-20 each week.

Ok, got it. It's purely a projections comparison, not a rankings comparison.

In essence it's really both, but it's the projections you want to focus on, because the rankings are muddied by applying different scoring rules.

 

[Das Boot]

 

[GregR_2]

What you're embarking on is a huge project and could be too big if you let it.I think you should focus the study on:Top 12 QB'sTop 24 RB'sTop 36 WR'sTop 12 TEI say that because that's your basic starter for standard leagues. Maybe you get a bonus point for any player that you nail on the money.For example, if FBG predict MJD to be RB 11 and he actually finishes RB 11, you get a bonus for not only predicting him in the top 24 which I'm sure every site will do, but you had him where he should be.It may not be rocket science to pick Tony Romo in the top 12 QB's, but it's still solid work if you pick him QB no. 3 and he actually finishes there. Equally impressive if you pick Jake Delhomme to finish No. 12 and he finishes 12, awesome, you deserve a bonus point.You could even set up a negative detractor for certain things if you wish. For example, you could say any top 5 pick that doesn't finish in the top 12 QB's or 24 RB's or 36 WR's gets a negative point.If you picked Peyton Manning to finish 4th and he finishes 16th for the week, then you deserve a slap on the wrist, because guess what, you were off.

I disagree very much. That may be how many players actually get started. But if you are looking to someone else (FBGs) to give you input, you need input not just on the players you decided in the end to start, but on the players who you were considering starting. And that is a lot more than the number of players who actually started. Often nearly twice as much. David's numbers are a lot better as that is a more realistic sized group of players whose rankings people would actually care about.