Predictive validity of preseason SOS rankings? (1 Viewer)

[cobalt_27]

Anyone have links to any analyses that establishes that these rankings have any utility?  If so, are certain positions more durable/reliable than others?

 

[Ignoratio Elenchi]

http://www.footballperspective.com/does-pre-season-strength-of-schedule-matter/

 

[cobalt_27]

http://www.footballperspective.com/does-pre-season-strength-of-schedule-matter/

Thanks.  This covers predictive validity of team record, and it's exceptionally weak to the point of being pretty meaningless.

Based on these data, I'd expect player adjustments/predictions based on SOS are equally meaningless.  But, I'd be curious to see studies that have looked at that.  Amazing we get inundated with SOS articles and virtually nothing I've seen has established that it's at all meaningful.

 

[tangfoot]

My best guess is that SOS is essentially meaningless until 4-6 games have been played in the current season.  At that point you should have enough data to start determining a trend.

 

[cobalt_27]

My best guess is that SOS is essentially meaningless until 4-6 games have been played in the current season.  At that point you should have enough data to start determining a trend.

Perhaps.  But, I'm skeptical.  And surprised no available data seems to be run on this, given how widely accepted SOS is a conceptual framework for decision-making in FF circles.

 

[DawnBTVS]

My best guess is that SOS is essentially meaningless until 4-6 games have been played in the current season.  At that point you should have enough data to start determining a trend.

Even this can be flawed with players coming back from suspensions or injuries early in the year by Weeks 5-8 too. Through Week 6 of 2017, the Patriots looked disastrous against the pass and like obvious must-start opposing QBs but then Week 7 hit and they completely transformed only allowing Pittsburgh to top 245 yards through the end of the season.

The 49ers were capable of being gashed on the ground then Weeks 12-16 they allowed 90, 62, 90, 90, and 92 yards rushing. Even the LA Chargers went through a 5 of 7 week stretch where they were pretty dominant against the run (allowing under 90 yards in 4 of those weeks) despite looking pretty poor overall for the season.

I've come to rely less on SOS as the years have gone by and instead focused on potential game plans or injuries in trying to ascertain how games may go.

 

[Biabreakable]

Anyone have links to any analyses that establishes that these rankings have any utility?  If so, are certain positions more durable/reliable than others?

Great question, I am curious as well.

I would think some value could be gained by being aware of the strength of defenses on a teams schedule and there are some folks who do projections based on each game for the whole season before the season has begun.

I even tried to make projections like this once I learned about it, but I had less confidence in the projections trying to do it this way. Once I started to get to game 6 I figured a lot of things could have changed by then.

I know teams change a lot every year and even game to game. Even teams with stability in personnel and coaching staffs can have some huge swings from year to year. Some times this is because of injuries, but not always. Some times teams just have a bad or good game for whatever reason.

There are times when teams actually perform at a very similar level for two consecutive seasons or longer, but most teams can't. Most teams defenses will perform at a different level from one season to the next and obviously from week to week as well, depending on their opponent which is always changing.

I agree with the idea of looking at the first 4 weeks of a season and start considering that data for starting decisions and trades at that time, which is similar to SOS analysis to help inform your decisions, but it is still only a small factor on my decisions.

I wonder what folks doing predictions for daily fantasy think of SOS? Are match ups a huge factor in those decisions? 

I guess I would like to hear the thoughts of people who find SOS useful.

 

[Alex P Keaton]

Anyone have links to any analyses that establishes that these rankings have any utility?  If so, are certain positions more durable/reliable than others?

I shared my own analysis of this years ago on FBG.  It's been a long time, but my recollection was:

- prior year "Fantasy Points allowed" by a team's defense in year N are not highly predictive for year N+1

- there is major regression to the mean in "Fantasy Points allowed" from year N to year N+1 for an individual team's defense

- that said, the best defenses typically remain above average and the worst defenses typically remain below average.....from year N to N+1.....just to a lesser degree

 

[Chaka]

Ian Allen at fantasy index did a piece on this a few years back. I have the article somewhere, not getting it now, but the bottom line is that it is virtually useless except at the extremes. I think it's came to the conclusion that if your opponents had a combined record of 137-119 you should downvalue a little. And if their record is 140-116 then it's even worse.  On the flip side a record of 119-137 or 116-140 demonstrates a benefit to players. Everuthing else is negligible. He backed it up pretty well as I recall. 

 

[cobalt_27]

I would think some value could be gained by being aware of the strength of defenses on a teams schedule and there are some folks who do projections based on each game for the whole season before the season has begun.

Exactly the sort of thing that needs to be addressed.  Some value could be gained by being aware of strength of defense if--and only if--our models for strength of defense are valid.  If they are not valid, then there is no value in applying erroneous SOS data into an equation (or even a heuristic) to enhance or diminish a player's value/ranking.  My takeaway from the limited analyses that have been conducted on SOS or SOD (defense) is that they are rubbish, which then suggests we should not use SOS to guide our rankings.

That's not to say that player values don't go up/down depending on quality of the opponent.  I'm pretty sure they do.  I just don't have confidence in our ability to accurately predict the strength of schedule during the preseason.  Perhaps if we isolated only the two tails at the 10th percentile (i.e., whom we believe are REALLY GOOD or REALLY BAD), maybe there's better correlation there between what we predict and what actually happens...and then we could make adjustments for a limited number of players based on those data.  But, I just haven't seen that hypothesis fleshed out in any good analyses yet.  

 

[cobalt_27]

Ian Allen at fantasy index did a piece on this a few years back. I have the article somewhere, not getting it now, but the bottom line is that it is virtually useless except at the extremes. I think it's came to the conclusion that if your opponents had a combined record of 137-119 you should downvalue a little. And if their record is 140-116 then it's even worse.  On the flip side a record of 119-137 or 116-140 demonstrates a benefit to players. Everuthing else is negligible. He backed it up pretty well as I recall. 

I wrote the post above this before seeing yours.  This is in line with what I'm thinking and gives me a little hope we might salvage something from preseason SOS rankings.  I'll take a look at Allen's analysis if I can find it later today.  But, this is promising.

 

[Long Ball Larry]

Exactly the sort of thing that needs to be addressed.  Some value could be gained by being aware of strength of defense if--and only if--our models for strength of defense are valid.  If they are not valid, then there is no value in applying erroneous SOS data into an equation (or even a heuristic) to enhance or diminish a player's value/ranking.  My takeaway from the limited analyses that have been conducted on SOS or SOD (defense) is that they are rubbish, which then suggests we should not use SOS to guide our rankings.

That's not to say that player values don't go up/down depending on quality of the opponent.  I'm pretty sure they do.  I just don't have confidence in our ability to accurately predict the strength of schedule during the preseason.  Perhaps if we isolated only the two tails at the 10th percentile (i.e., whom we believe are REALLY GOOD or REALLY BAD), maybe there's better correlation there between what we predict and what actually happens...and then we could make adjustments for a limited number of players based on those data.  But, I just haven't seen that hypothesis fleshed out in any good analyses yet.  

I have been curious for a while about how much the opposing defense really affects the offensive players.  I had tried to start playing around with a while ago (scatterplotting fantasy points vs. defense rating, or something like that, by position and wasn't seeing a clear line), but that was very rudimentary.  I would like to know how much of a factor the defense really is vs. individual talent, offense factors, weather, etc.

 

[zftcg]

I've noticed that ESPN's Bill Barnwell, who uses quantitative analysis extensively in his writing, almost always discusses SOS in terms of either Vegas projected win totals or Pythagorean expectation rather than previous year's record. I'd be curious to see if those are any more accurate. (Also found this 2007 article that says Pythagorean expectation is a better predictor of next season win totals than previous record)

Ultimately, though, as fantasy players we're not interested in the accuracy of predictions in the abstract ("Are the Pats facing a tough schedule this year?") but in terms of specific decisions ("Should I bump Stafford up/down based on his SOS?") IMO the more specifically you try to apply it, the less predictive ability you're going to see. In that sense I'm with @Cobalt 27 in having such low confidence in its predictive power that I'm reluctant to assign any weight to it in my analysis. The one example where I might consider it at the margin is in a case like Dez Bryant last year, where we knew heading into the season that he would be facing a murderer's row of elite corners who typically follow around WR1s. Even in that case, though, if I had followed that logic I would have been right, but not necessarily for that reason; I think most would attribute Dez's 2017 season far more to a drop-off in his talent than to his CB match-ups. (As it happens, I ignored all of it and drafted him in the second round)  

 

[Leroy Hoard]

Along the lines of zftcg's post above, there was an article that adjusts for point differential and that supposedly makes last year's records of opponents more accurate. I believe Vegas projections are based more on this model.

 

[zftcg]

Along the lines of zftcg's post above, there was an article that adjusts for point differential and that supposedly makes last year's records of opponents more accurate. I believe Vegas projections are based more on this model.

Yes, "Pythagorean expectation" basically just means point differential. Let's say two teams both score exactly as many points as they give up. The Rams get a few breaks and go 10-6, the Cards are unlucky and go 6-10. From a PE perspective, they should have both gone 8-8, and there is no difference between the two. So if New England is going to play the Rams next season and Green Bay is playing Arizona, both NE and GB have the same SOS based on those opponents.

I'll admit I have no idea how Vegas calculates over/unders for team wins. I had always assumed it was just a wisdom-of-crowds kind of thing, where if they lean too far in the wrong direction the sharps will hammer them back to an equilibrium. But maybe they use PE to make the initial calculations? I'd be curious to know how it works.

 

[Leroy Hoard]

Yes, "Pythagorean expectation" basically just means point differential. Let's say two teams both score exactly as many points as they give up. The Rams get a few breaks and go 10-6, the Cards are unlucky and go 6-10. From a PE perspective, they should have both gone 8-8, and there is no difference between the two. So if New England is going to play the Rams next season and Green Bay is playing Arizona, both NE and GB have the same SOS based on those opponents.

I'll admit I have no idea how Vegas calculate's over/unders for team wins. I had always assumed it was just a wisdom-of-crowds kind of thing, where if they lean too far in the wrong direction the sharps will hammer them back to an equilibrium. But maybe they use PE to make the initial calculations? I'd be curious to know how it works.

I think you pretty much have it with that last part. Start out with what they really think and then adjust to projected and current betting so as to lessen their risk while still maxing profit.

 

[Leroy Hoard]

I think you pretty much have it with that last part. Start out with what they really think and then adjust to projected and current betting so as to lessen their risk while still maxing profit.

Actual numbers per this website: https://datashoptalk.com/nfl-predicting-2018-wins/

2018 NFL Win Predictions

New England Patriots 10.47

Minnesota Vikings 9.96

New Orleans Saints 9.75

Baltimore Ravens 9.66

Jacksonville Jaguars 9.54

Philadelphia Eagles 9.53

Los Angeles Rams 9.43

Pittsburgh Steelers 9.34

Carolina Panthers 9.13

Atlanta Falcons 9.09

Dallas Cowboys 9.02

Tennessee Titans 8.95

Kansas City Chiefs 8.81

Detroit Lions 8.68

Cincinnati Bengals 8.59

Seattle Seahawks 8.58

Los Angeles Chargers 8.46

Washington Redskins 8.33

Green Bay Packers 8.15

Arizona Cardinals 7.92

Houston Texans 7.89

San Francisco 49ers 7.58

Buffalo Bills 7.45

Denver Broncos 7.22

Oakland Raiders 7.16

Cleveland Browns 7.05

Tampa Bay Buccaneers 6.98

New York Jets 6.96

Miami Dolphins 6.91

Chicago Bears 6.67

Indianapolis Colts 6.35

New York Giants 6.25

 

[zftcg]

Actual numbers per this website: https://datashoptalk.com/nfl-predicting-2018-wins/

Interesting. I'd never seen them all laid out like that before. Is it normal to have such a narrow distribution? Obviously it's highly unlikely that all 32 teams end up with between 6 and 10 wins.

 

[Leroy Hoard]

Interesting. I'd never seen them all laid out like that before. Is it normal to have such a narrow distribution? Obviously it's highly unlikely that all 32 teams end up with between 6 and 10 wins.

I just think the regression to the mean sometimes outweighs the far ranges that previous win totals predict. Cleveland won't go 0-16 again for instance.

 

[Dismattle]

Believe the biggest precursor to SOS success when ranking players will happen to involve common sense...  (Not For Long) It is any given Sunday!

Let's say Team A finishes a poor Season and somehow ends up drafting an RB in the next yr's draft (Giants?) while Team B finds some success during the Season and also drafts an RB (Seahawks?) leaving Team C who has a great Season while relying on the potential of a 2nd yr RB (Vikings)  We then could discuss such things as Division games which is Teams played twice possibly taking into consideration Playoff matchups  We could try an use some logic about how the NFL uses the prior yr's finish in matching up games for the next Season  This info could be quite helpful in determining success Or it may very well appear to cancel out some perceived notion 

Divisionally its somewhat hard to really see much separation amongst expected competition but possibly a slight edge to the Vikings which one would expect to be canceled out by the scheduling of games  However, none of this info has much bearing when considering the scope of SOS  Fact is that Barkley has one of the roughest schedules while Cook may have the easiest according to normal SOS trends

Now it's time to go back to the beginning!  Its near impossible to try an rank Team A with a very realistic ranking  Giants, in this case, have drafted what some to believe a generational talent which means expected yards cannot possibly be factored accordingly  (3 down back is quite capable of many yds from scrimmage even if the run is shut down)  Once again Team B is coming off a successful Season so any additions should be quite beneficial in potential  All of which leaves us with Team C who should have much more potential for regression as opposed to improvement 

Here's a piece from a Guru concerning SOS w/ a catchy title: 

These factors play major roles in winning games and thus, a team’s end of season record.  But they are not descriptive of a team’s strength  http://www.sharpfootballanalysis.com/blog/2018/ignore-virtually-all-offseason-nfl-strength-of-schedule-information

Now is when I wish I could try an explain the value of Penny in relation to SOS or any precursor really  A guy drafted by a good Team with a good Coach with a somewhat healthy track record for evaluating talent who quite possibly not only ignored some rankings but could have overlooked other Team needs.  However, since I'm unable to convey such logic by any means I'm going to end my post with yet another version of SOS in some respects  One which happens to rank Seattle near the bottom at #30 using some pretty sound logic (Vegas betting odds)

This article will focus on 2018 SOS using 2018 Vegas projected win totals. My method starts by taking three of the largest, most reputable Vegas sportsbooks (Westgate, South Point, CG Technology) to build a model creating a consensus line which factors in juice.  http://www.rotoworld.com/articles/nfl/79906/298/superior-strength-of-schedule

TBH I really do question a lot of the validity to even what is the latest variant of SOS  I mean I'm not really sure I've been all that much fond of Eli for any length of time  OBJ's success has almost exceeded my expectations  I truly value the potential of Engram to be somewhere in the neighborhood of 800 yds w/ ten TD's  In so far as ADP I've witnessed proof of a Guru taking Barkley at fifth overall... (shout out to @FantasyTaz)  I dunno I mean Eli isn't getting any younger  OBJ is I believe talking about not attending TC (contract related)  Giants have the latest addition to their arsenal which doesn't equate success as much as usage which could mean less to eat for the TE too

When its all said and done do I personally still enjoy the bringing up of SOS info for Fantasy discussion?  I clicked your link didn't I

 

[Ilov80s]

I don’t pay attention to SoS except for the start of the season and the extremes (best 3 or so Ds and 3 or so worst).

 

[toonarmy]

My co-workers Yahoo league has been going since 2000, and I have records of top 10 performance since 2004. The defense rankings are based on whatever Yahoo was using at the time, but currently it's

Sack 1

Interception 2

Fumble Recovery 1 (think the default is actually 2)

Touchdown 6

Safety 2

Block Kick 2

Points Allowed 0 points 10

Points Allowed 1-6 points 7

Points Allowed 7-13 points 4

Points Allowed 14-20 points 1

Points Allowed 21-27 points 0

Points Allowed 28-34 points -1

Points Allowed 35+ points -4

Extra Point Returned 2

The table below shows the number of top 3 defenses that returned to the top 10 the following year, plus the same for top 5 and top 10 of the previous year.

Year Top3 Top5 Top10

2004 2       4        5

2005 2       2        4

2006 2       4        7

2007 2       2        6

2008 1       2        4

2009 2       3        4

2010 2       3        4

2011 1       3        4

2012 1       2        4

2013 1       1        4

2014 0       1        4

2015 1       2        3

2016 2       4        5

2017 1       1        3

What the above suggests is that the predictive value of the previous year for the top 10 is about 40%, though for top 3 performers it was pretty good, though that has not been the case since 2011 (except for 2016).

Hope that helps.

 

[cobalt_27]

The above is a good start but needs a little more statistical rigor to answer the question about predicting even the top defenses.  I think it approaches the question and, broadly seems supportive of the idea we might take some stock in how we gauge top-tier defenses.  

But, in the end, even if there is statistical significance, is it “clinically meaningful?”  In other words, is it even helpful if the pool of predictable defenses is limited to just 3 or—at best—5?

 

[Tanner9919]

Anyone have links to any analyses that establishes that these rankings have any utility?  If so, are certain positions more durable/reliable than others?

fantasyindex.com does a run on this topic every year, and basically implies that they're useless/meaningless for the most part..SOS for playoff weeks, that's something that might be important, but as a whole, SOS really doesn't matter..

if you're talking about easiest SOS For a RB< there might be something there.or easiest SOS for a QB..