On Processes and Outcomes (1 Viewer)

[Adam Harstad]

Hey guys, my first article for the site has gone live (link available here), and I wanted to get some discussion going on it. For those who are familiar with my posting history, there won't be anything earthshaking in there- I've posted before on the idea of separating our processes from their outcomes and evaluating both independently- but I thought it would be good to get the entire thought process spelled out a little bit more formally and get it gather it all in one place for future reference.

So, anyone have any particular thoughts on the idea of evaluating processes independent of outcomes? Some real-world examples from their own experiences? Does anyone have some examples of good processes or bad processes? I'm also interested in suggestions or feedback on the piece itself; I want to write about what you guys want to read about, so feel free to bring the criticism (constructive or otherwise). TIA.

 

[ConnSKINS26]

Your old piece on this from your old rankings site is still one of my favorite "dynasty basics" articles to recommend to people.

 

[Grace Under Pressure]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

 

[Adam Harstad]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I have a couple of processes I've adopted in my dynasty league that have paid dividends for me over the years. For one, when I've studied past trends, I've discovered that injured players were almost always rated too low. The discount applied to injured players was not commensurate with the risk that they were actually demonstrating. I've integrated that more into my process now- I've lowered my personal injury discounts, and I make a point of always gauging interest whenever another owner's player gets injured. That's a combo good process/bad process (underrating injured players = bad process, countering for that instinct = good process). Another process that has been useful is that I've changed the way I've ranked rookies. I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

I used to heavily discount players who were character risks. Roger Goodell came into the league and talked big about cleaning it up, and I bought into it hook, line, and sinker. I thought these character risks were walking timebombs, ready to detonate into year-long suspensions at any moment. When I went back and actually evaluated outcomes, however, this proved to be a pretty bad process. It turned out that Goodell overreached and was quickly smacked down. I think it would have been easy to continue adding a "knucklehead factor" to my rankings, but they were more accurate without it, so it had to go.

I used to be a big fan of wide receivers who were also standout punt returners. This was mostly just based on hunches and anecdotal data. After a couple of years, I examined the evidence again, and found no real evidence that players who were standout returners were more likely to develop into top WRs, even if there were plenty of anecdotes supporting the theory. That was a bad process, so I ditched it.

Really, evaluating processes doesn't require fancy regressions or rigorous data-driven analysis. Simply questioning from time to time what you believe, why you believe it, and whether that belief is warranted is usually enough. Humans are a mess of biases, and a lot of random beliefs and preferences will creep into the machinery over time. Even a crude examination of those beliefs is usually enough to separate the good from the bad.

As a special aside, I don't think "trusting your gut" is necessarily bad process. I do think that as we gain experience and acquire expertise in fantasy, we start to unconsciously notice a lot of patterns. Sometimes gut feelings are just patterns that our unconscious mind is warning us about that our conscious mind has not yet seen. If someone has a sufficient level of expertise, I think gut feelings are probably pretty useful. For the majority of fantasy football owners, though, I agree that if your gut is telling you anything, it's probably to take it easy on the hot sauce.

 

[ducktales]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

 

[Drop]

Here is an old thread that I still think about quite a bit as I try to evaluate my processes: http://forums.footballguys.com/forum/index.php?showtopic=595123&hl=

I'm trying to refine my process for identifying the conditions for a super elite fantasy TE to emerge. The struggle comes from sorting through mixed results and trying to determine what is solid and what is random. Developing a good process is tricky in that the feedback you immediately get won't always be a positive result and it will make you wonder if you're dong it right. I'm always in favor of refinement in light of new information, drawing the foundation for such from parallels that I can identify whether obvious or subtle. I don't get a complete picture of the puzzle, but having clarity on certain specific things allows me to avoid situations that are clouded by hype.

The thread linked is goes back a couple of seasons, so note the date on the posts as you read them.

 

[msommer]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

Troy Williamson was a prime example of this. His ability to drop passes was pretty much unequaled, the Vikings could with better success have picked Roddy White, Vincent Jackson or even Reggie Brown at 7.

When the Bills took J.P. Losman 22nd overall and passed on Matt Schaub it didn't make J.P. the more successful fantasy football qb

I am sure that there are many counter examples to this (and then we could discuss Tom Brady, Alfred Morris, Arian Foster etc ad nauseam). IMHO reading everything and then putting your own thoughts on top of that have served me better. And yes, I keep in mind that virtually nothing negative (about playing ability) will be written about rookies between the draft and TC - if there is, there goes the alarm bells again.

One process I've adopted is not to pick late round QBs, well knowing that would have made me miss Tom Brady. The roster space then used for shoring up other positions with players on which we have more data points or a closer path to starting. Most late round QBs if they are lucky become #2 on their team year two or three. Few end up starting, even on other teams, Tom Brady and to some extent Matt Cassell and A.J Feeley are notable exceptions to this although fewer make it to fantasy relevance. When/if they make it they can be had on the wire or through trades.

 

[spreagle]

I agree injuries and character risks land a lot of player on the fantasy "do not draft" lists and make for good value draft targets. Here is another example where situation landed a good player on the do not draft list. No one wanted to draft Gronkowski in our keeper league one year after his rookie season because, as one owner said, "I wouldn't know who to start, Gronkowski or Hernandez". There seemed to be a perception at the time that each NFL team only had one startable TE and even though gronk/hernandez were top ten TEs, both dropped ridiculously late in our draft (I picked up hernandez on waivers actually). In retrospect I think most fantasy owners are lazy and like to set it and forget it, and two good TEs on one team was something my league owners didn't want to deal with week to week. That "only one stud TE per team" perception doesn't seem to apply to other positions like WR (guys like Austin may be undervalued due to Dez) but I think it may apply to some NFL teams with two decent RBs, with the better-receiving 3rd down RB being undervalued by fantasy drafters.

 

[Coeur de Lion]

Not to speak for SSOG, but he's not saying to blindly follow NFL draft order in a fantasy draft. Rather break guys in tiers based on draft position (ie "late 1st / early 2nd round RBs" or whatever works for you), then adjust for opportunity and situation within those tiers. I'd love to hear what type of "homework" a fantasy owner is doing that can identify either overdrafted or underdrafted players better than the NFL franchises that invest millions of dollars and thousands of man-hours into the process.

That said, I think that a guy falling down the board is much stronger evidence than a guy taken way earlier than the pre-draft consensus. A guy taken really high can be the result of one franchise making a mistake and reaching (ie Al Davis with DHB). When a guy that was projected to be drafted high falls way down the board (ie Jonathan Dwyer) it means that every NFL franchise is saying "no thanks" multiple times -- it seems pretty unlikely that 32 teams are all missing the boat simultaneously on a single player.

IMO it's definitely bad process to believe that a fantasy geek can look at a few YouTube highlights and see things that NFL teams miss.

 

[Adam Harstad]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

It's like the stock market. I can look at the stock market right now and say definitively that some stocks are wildly underpriced and some stocks are wildly overpriced. I just can't say which ones are which. There's a group of people (hedge fund managers) who make millions of dollars a year because they think they *CAN* distinguish which is which. The truth, though... is that they can't. They really can't. Long-term analysis says that as many as 80% of hedge funds fail to outperform the market, and that there are no reliable indicators ahead of time which funds are going to belong to the successful 20%. The fact that most existing hedge funds boast impressive resumes is because of survivorship bias- the ones that don't outperform go out of business, and everything that's left has a great track record... but as they say, past results don't guarantee future performance.I'm not saying that you should just blindly follow draft order. Fantasy players place different weights and values on what's important. If an NFL team gets a guy who winds up being the 14th best QB in the league, that NFL team made a great pick. If a fantasy owner gets a guy who winds up being the 14th best QB, he's bye-week filler at best. NFL teams want tight ends who can block and catch. Fantasy teams couldn't care less about that first part. If an NFL team takes a kicker in the 1st round, I still wouldn't touch him in my rookie draft, regardless of draft position- kicker is just such a low-value position. I might prefer 3rd round RBs over 2nd round WRs just because of the immediacy of production (fantasy teams have shorter rosters and no practice squad) and the relative value of the position (owing to positional scarcity). Also, fantasy owners have to consider situation, while NFL teams do not.Just because you use draft position as a starting point doesn't mean you don't tweak and adjust to reflect the differences between weights and values between fantasy football and reality football. I'm not advocating just opening the wikipedia page for the 2013 draft and going straight down the list. I am advocating relying heavily on team scouting evaluations, because they're very, very good at it. The NFL is a very efficient marketplace.I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

 

[Drop]

I don't understand what you mean by "The NFL is a very efficient marketplace."

 

[Adam Harstad]

I don't understand what you mean by "The NFL is a very efficient marketplace."

Efficient-market hypothesis"In finance, the efficient-market hypothesis (EMH), or the Joint Hypothesis Problem, asserts that financial markets are "informationally efficient". In consequence of this, one cannot consistently achieve returns in excess of average market returns on a risk-adjusted basis, given the information available at the time the investment is made."Edit: An efficient market is one which has accounted for and accurately priced all available information. The only way to consistently beat an efficient market is to have information that was not available (e.g. through insider trading). NFL scouting departments accumulate huge volumes of information on the prospects (far more information than any fantasy football owners has access to), and they're good at weighing and synthesizing that information to come up with a player value that relatively accurately reflects the risk and potential reward. The theory is that you can't say, for example, "this RB is small, so I'm going to downgrade him" because NFL teams will have already considered his size, and that will already be reflected in his draft position.It's possible at times that the market is biased and systematically underrates assets of a certain class. Once upon a time, the market was biased against black quarterbacks. Some might argue that today's market is biased against short quarterbacks, and Russell Wilson proves it. I could just as easily argue that the fact that Wilson (and Drew Brees) are so unique in the league suggests that maybe the NFL is right to be wary of shorter quarterbacks. Either way, even if the market exhibits some sort of bias, there is a huge financial incentive to recognize that bias and correct for it. In the late '90s, the Steelers recognized that the market was biased against 'tweener LB/DEs, so they drafted a bunch late and reaped huge rewards. The market quickly saw what was happening and corrected, and now you regularly see 'tweeners drafted in the 1st round. In baseball, the Oakland A's discovered that there was a market bias causes on-base percentage to be improperly priced. They exploited that oversight and managed to post the best record in baseball over a 3-4 year span. Once the market found out what they were doing, though, it adapted and started pricing OBP accurately, and the A's advantage disappeared overnight.That's an efficient market in a nutshell: a market where, given the information we have available to us, we can not readily identify who was overpriced and who was underpriced.

 

[Drop]

What does the efficiency lead to though? League profit? Team championships? I'm unclear how you gauge the 32 various motivations.

 

[Adam Harstad]

What does the efficiency lead to though? League profit? Team championships? I'm unclear how you gauge the 32 various motivations.

The efficiency doesn't lead to anything, it's the result of something. It's not a cause, it's an effect. Profit motive, (or "desire to win" or whatever- the nature of the motivation is less important than the intensity of the motivation), means teams have a huge financial interest in getting it right, which leads to them doing lots of research and hiring lots of really smart people to maximize their chances of getting it right, which leads to an efficient market.

 

[Adam Harstad]

I'm glad there's interest in the whole "efficient market hypothesis" and how it impacts rookie rankings, because that was the next piece I was planning on writing. In the meantime, if anyone wants some more reading on the subject, the always-excellent Chase Stuart has run the numbers and found that the correlation between how well a team drafts in year N and how well it drafts in year N+1 is just 0.07 (essentially no correlation at all). This is just another point supporting the idea that the draft is efficient. Even the best talent evaluators don't consistently outperform the market. Even the worst teams don't consistently underperform the market. Six years ago, A.J. Smith and Bill Polian were genius GMs. Today, they're unemployed. Six years ago, the 49ers were a joke. Now they're the most talented roster, from top-to-bottom, in the entire NFL. Six year ago, the Bengals were a punchline who refused to pay for a scouting department. Since then, they've landed on Andre Smith, Geno Atkins, Andy Dalton, A.J. Green, Carlos Dunlap, and a host of other super-young, super-talented players. Random chance dictates that you'll have some pretty spectacular failure clusters (Matt Millen and the Oakland Raiders) and some pretty spectacular success clusters (the 49ers and the Seahawks), but that those are going to be randomly distributed, and that the past performance is not going to hold much value when predicting future performance.

 

[Instinctive]

 

[cheese]

Another way of looking at this is that it's basically blackjack theory. If you split 10's against a 6, end up with 2 20's and win twice as much, did you play it better or worse?

The issue I see with it is that in fantasy football, it isn't as cut and dry which processes are good and which are bad. I fear that the presence of confirmation bias would lead almost all of us to use this as a reason to write off any failures as "bad luck" despite our good process. In many cases with FF, the only clues we will ever get about whether our process was good or bad is our outcomes. The last thing most people need is another excuse to ignore those clues and keep doing what they're doing. In fact, I would say that is already the default mentality for most players. Personally, I think you'd be closer to "correct" by examining each failure and coming up with a small set of lessons learned.

I don't disagree with you at all regarding your overall thoughts on process vs outcome. I just think in FF it points players farther down the path they're already on... which is to say more denial and bias regardless of outcome and less innovation and evolving as a player.

 

[Drop]

I'm glad there's interest in the whole "efficient market hypothesis" and how it impacts rookie rankings, because that was the next piece I was planning on writing. In the meantime, if anyone wants some more reading on the subject, the always-excellent Chase Stuart has run the numbers and found that the correlation between how well a team drafts in year N and how well it drafts in year N+1 is just 0.07 (essentially no correlation at all). This is just another point supporting the idea that the draft is efficient. Even the best talent evaluators don't consistently outperform the market. Even the worst teams don't consistently underperform the market. Six years ago, A.J. Smith and Bill Polian were genius GMs. Today, they're unemployed. Six years ago, the 49ers were a joke. Now they're the most talented roster, from top-to-bottom, in the entire NFL. Six year ago, the Bengals were a punchline who refused to pay for a scouting department. Since then, they've landed on Andre Smith, Geno Atkins, Andy Dalton, A.J. Green, Carlos Dunlap, and a host of other super-young, super-talented players. Random chance dictates that you'll have some pretty spectacular failure clusters (Matt Millen and the Oakland Raiders) and some pretty spectacular success clusters (the 49ers and the Seahawks), but that those are going to be randomly distributed, and that the past performance is not going to hold much value when predicting future performance.

He takes data from 2000-2007 for all teams, but I don't see how it's valid given that the staffs drafting varied and even changed in terms of the team that a draft mind was working for. What does it prove that would stand up to mathematical peer review?

 

[Adam Harstad]

Another way of looking at this is that it's basically blackjack theory. If you split 10's against a 6, end up with 2 20's and win twice as much, did you play it better or worse? The issue I see with it is that in fantasy football, it isn't as cut and dry which processes are good and which are bad. I fear that the presence of confirmation bias would lead almost all of us to use this as a reason to write off any failures as "bad luck" despite our good process. In many cases with FF, the only clues we will ever get about whether our process was good or bad is our outcomes. The last thing most people need is another excuse to ignore those clues and keep doing what they're doing. In fact, I would say that is already the default mentality for most players. Personally, I think you'd be closer to "correct" by examining each failure and coming up with a small set of lessons learned. I don't disagree with you at all regarding your overall thoughts on process vs outcome. I just think in FF it points players farther down the path they're already on... which is to say more denial and bias regardless of outcome and less innovation and evolving as a player.

Sure, confirmation bias is the most insidious danger in pretty much any endeavor. That doesn't mean it can't be overcome, you just have to take certain steps. Adopting a more scientific mindset is a great start- instead of coming up with theories and then trying to prove them, come up with theories and try to DISPROVE them. Deliberately seeking out disconfirming evidence can help short circuit the natural confirmation bias routines. Seeking outside opinions is another good strategy- whoever you ask is going to have their own biases, but at least their biases will be different from yours. During the season, keep a notebook and record all your thought processes behind every move every week (write down who you added on waivers, who you dropped, who you considered, who you rejected, and why, write down your reasoning behind trades, etc). Writing it down in real time helps guard against hindsight bias. Then make friends with someone who is not in any of your leagues, and after the season, show him your notes. Better yet, after the season, review all your moves and try to guess what you were thinking at the time, then compare that to what you wrote in your notebook and see how accurate your guesses are (usually, not very).Cognitive bias will always be hanging over you and holding you back- it's simply too integrated into our mental processes to ever truly shake free. Our minds are really terribly designed for handling things like fantasy football. We've just got way too much faulty code. I don't think anyone should let that sad reality hold them back from trying to improve, though. Never let the perfect be the enemy of the good.This is another subject I hope to do some writing about- our biases, how we can identify them, and what steps we can take to try to short circuit them (or, even better, to take advantage of them in our opponents).

 

[Drop]

"come up with theories and try to DISPROVE them"

 

[Instinctive]

Adam, what did you do for a living before this? I'm a Finance major (among other things) and apply the EMH to pretty much everything, but a lot of what you tend to put forth seems to come more from a statistics background. Just curious.

 

[Adam Harstad]

I'm glad there's interest in the whole "efficient market hypothesis" and how it impacts rookie rankings, because that was the next piece I was planning on writing. In the meantime, if anyone wants some more reading on the subject, the always-excellent Chase Stuart has run the numbers and found that the correlation between how well a team drafts in year N and how well it drafts in year N+1 is just 0.07 (essentially no correlation at all). This is just another point supporting the idea that the draft is efficient. Even the best talent evaluators don't consistently outperform the market. Even the worst teams don't consistently underperform the market. Six years ago, A.J. Smith and Bill Polian were genius GMs. Today, they're unemployed. Six years ago, the 49ers were a joke. Now they're the most talented roster, from top-to-bottom, in the entire NFL. Six year ago, the Bengals were a punchline who refused to pay for a scouting department. Since then, they've landed on Andre Smith, Geno Atkins, Andy Dalton, A.J. Green, Carlos Dunlap, and a host of other super-young, super-talented players. Random chance dictates that you'll have some pretty spectacular failure clusters (Matt Millen and the Oakland Raiders) and some pretty spectacular success clusters (the 49ers and the Seahawks), but that those are going to be randomly distributed, and that the past performance is not going to hold much value when predicting future performance.

He takes data from 2000-2007 for all teams, but I don't see how it's valid given that the staffs drafting varied and even changed in terms of the team that a draft mind was working for. What does it prove that would stand up to mathematical peer review?

He mentioned 2000-2007 because he was referencing a previous article where he developed the methodology for grading drafts, but in the next paragraph he said that he had taken that methodology and applied it to all drafts since 1970 for that analysis. So we're looking at about a 40-year sample.I agree that staffs have changed a lot over that period. I would argue that this is just another manifestation of the efficiency of the marketplace. I think some people really, truly do suck at evaluating talent (Matt Millen, Al Davis in his later years). The point is that the guys who suck get replaced. Other guys really are ahead of the curve, but then everyone else figures out what they're up to and catches up. If a team has a deficit in talent evaluation, they will fix that deficit. If a team has an advantage in talent evaluation, the rest of the league will close the gap. In the end, the teams that drafted well in the past are no more likely to continue drafting well in the future, while the teams that drafted poorly in the past are no more likely to continue drafting poorly. The massive stakes leave teams highly incentivized to close any gaps, and the market remains efficient.As for how the idea would stand up to peer review... I'd guess probably pretty well. Cade Massey presented findings at the MIT Sports Analytics conference which largely reached the same conclusions. Teams just don't outperform the market average in talent evaluation over an extended timeline.

 

[Adam Harstad]

Adam, what did you do for a living before this? I'm a Finance major (among other things) and apply the EMH to pretty much everything, but a lot of what you tend to put forth seems to come more from a statistics background. Just curious.

I studied music in college, and worked most recently in health care. I actually have very little background in either statistics or finance, and what I have is mostly self-taught, so if I wind up totally messing up the theory or application and looking like a fool, I'd appreciate it if you'd let me know rather than letting me persist in my bad processes.

 

[Instinctive]

Adam, what did you do for a living before this? I'm a Finance major (among other things) and apply the EMH to pretty much everything, but a lot of what you tend to put forth seems to come more from a statistics background. Just curious.

I studied music in college, and worked most recently in health care. I actually have very little background in either statistics or finance, and what I have is mostly self-taught, so if I wind up totally messing up the theory or application and looking like a fool, I'd appreciate it if you'd let me know rather than letting me persist in my bad processes.

Haha. Our faculty focuses mostly on value investing by finding market inefficiencies, but you seem to have the efficient market hypothesis down pretty well. I haven't done any grad-level work on it yet but it comes up in most classes. I'd let you know, but it seems highly unlikely that situation will come up. Thanks for the answer man.

One thing to think about with the EMH, though, is that it is built on the theory of rational expectations - which has two major things to look out for: The optimal forecast is not going to be accurate because (1) People may not be physically able to appropriately integrate all available information, even though ti is assumed that they have it, and (2) the market can be unaware of relevant information.

Additionally, any news that is both new and unexpected will offset the market for a short period of time - THAT is what I am trying to focus on in my FF trading and such. Whenever news comes out, right away, is when I am most active in my trade offers. You quickly have to assess 2 things: is the market overreacting to the news, or is it under reacting? We only have that opportunity for a short time before market efficiency will settle on an appropriate price.

As far as inefficiencies go, I'm on board with the market undervaluing injured guys - we just don't seem to get the power of modern medicine yet - but we should also take a look at suspensions. I pretty much constantly try to buy suspended players (current example: Josh Gordon), which tends to work far better in redraft than in dynasty (many dynasty people seem to get that 2 games in 6 years is no big deal, but redraft players tend to see 2/15 (I don't play week 17) as a HUGE deal when it really isn't.

 

[MAC_32]

Re rookie rankings…I have had a lot of success utilizing my evaluations to maximize the talent acquired in rookie drafts, and over the first year or two of their career.

I expect the NFL to have a better hit rate than the average dynasty fantasy football owner, but that includes a laundry list of owners that don’t know much outside of the top prospects, top college teams, and in some cases their favorite college team until January. They’re playing from the curve, those are the types of owners that should use draft order as their baseline.

I think that is a bad idea given the information base I work with. I start following players very early in their college career. No one is going to bat 1.000, and I miss my share of prized prospects, but the important thing is my miss rate on the players I actually pick is very low. I consistently avoid the early round landmines and usually find under valued stocks in the later rounds.

I wish I could translate my process into writing, but I really can’t. Or at least don’t know how to. There are too many independent variables and they’re constantly changing. How I evaluate Justin Hunter is going to be different than Cordarelle Patterson and Da’Rock Rogers…and until last year they were all on the same team playing the same position. In the end, it’s a balancing act – performance, measurables, character, round selected, organization, surrounding talent, expected role, among other things. Pigeon holing it into a specific process is not going to yield optimal results. People like a ratings number…92…84…88…it makes it easier. But this is not an easy process and I don’t think applying ratings numbers is an accurate way to evaluate these players when creating rookie/young player boards. Bundling them into tiers works then fine tune your target selection to get the players you want, at value. That’s still not all of it though.

Again, I have no idea how to explain it, it just works for me.

 

[Adam Harstad]

Adam, what did you do for a living before this? I'm a Finance major (among other things) and apply the EMH to pretty much everything, but a lot of what you tend to put forth seems to come more from a statistics background. Just curious.

I studied music in college, and worked most recently in health care. I actually have very little background in either statistics or finance, and what I have is mostly self-taught, so if I wind up totally messing up the theory or application and looking like a fool, I'd appreciate it if you'd let me know rather than letting me persist in my bad processes.

Haha. Our faculty focuses mostly on value investing by finding market inefficiencies, but you seem to have the efficient market hypothesis down pretty well. I haven't done any grad-level work on it yet but it comes up in most classes. I'd let you know, but it seems highly unlikely that situation will come up. Thanks for the answer man. One thing to think about with the EMH, though, is that it is built on the theory of rational expectations - which has two major things to look out for: The optimal forecast is not going to be accurate because (1) People may not be physically able to appropriately integrate all available information, even though ti is assumed that they have it, and (2) the market can be unaware of relevant information. Additionally, any news that is both new and unexpected will offset the market for a short period of time - THAT is what I am trying to focus on in my FF trading and such. Whenever news comes out, right away, is when I am most active in my trade offers. You quickly have to assess 2 things: is the market overreacting to the news, or is it under reacting? We only have that opportunity for a short time before market efficiency will settle on an appropriate price. As far as inefficiencies go, I'm on board with the market undervaluing injured guys - we just don't seem to get the power of modern medicine yet - but we should also take a look at suspensions. I pretty much constantly try to buy suspended players (current example: Josh Gordon), which tends to work far better in redraft than in dynasty (many dynasty people seem to get that 2 games in 6 years is no big deal, but redraft players tend to see 2/15 (I don't play week 17) as a HUGE deal when it really isn't.

It should be noted that my belief in the EMH begins and ends at NFL scouting departments and the NFL draft. I don't think the fantasy market is anywhere close to efficient, and I don't think there are anywhere near enough incentives (or, perhaps more importantly, barriers to entry) to get it there. The really high-stakes stuff is probably pretty close, but casual leagues? No way. And one of the reasons I'm so drawn to dynasty is that not only is there no efficient market, there's hardly anything that passes for a market at all. Player valuations fluctuate so wildly from team to team, from league to league, and from format to format that there's a massive number of inefficiencies to exploit. That's why my rookie rankings hew so closely to draft position (under the assumption that the draft is an efficient market), while my veteran rankings deviate so wildly from ADP (under the assumption that the market for dynasty veterans is very inefficient).

 

[Coeur de Lion]

Additionally, any news that is both new and unexpected will offset the market for a short period of time - THAT is what I am trying to focus on in my FF trading and such. Whenever news comes out, right away, is when I am most active in my trade offers. You quickly have to assess 2 things: is the market overreacting to the news, or is it under reacting? We only have that opportunity for a short time before market efficiency will settle on an appropriate price. As far as inefficiencies go, I'm on board with the market undervaluing injured guys - we just don't seem to get the power of modern medicine yet - but we should also take a look at suspensions. I pretty much constantly try to buy suspended players (current example: Josh Gordon), which tends to work far better in redraft than in dynasty (many dynasty people seem to get that 2 games in 6 years is no big deal, but redraft players tend to see 2/15 (I don't play week 17) as a HUGE deal when it really isn't.

Totally agree here. Any injury or police blotter action generally prompts me to immediately send out a flurry of offers. It can work really well.One thing to keep in mind regarding the off-field stuff is the massive difference between substance abuse vs other types of behavioral nonsense. If you get arrested multiple times for non-substance abuse issues, the punishment is discretionary. The penalties for substance abuse issues are mandatory and generally much harsher than anything the NFL lays down for even pretty severe non-drug offenses.IMO it's a pretty safe bet to ignore wife beating, bar fights, gun issues, and the like, as absurd as that sounds. Even the Pacman Joneses, Brandon Marshalls, and Kenny Britts of the NFL are mostly looking at a few games total for arrest records as long as my arm. An ongoing issue with drug or alcohol addiction is far more likely to wind up in multiple and/or lengthy suspensions for the player involved.

 

[Drop]

As for how the idea would stand up to peer review... I'd guess probably pretty well. Cade Massey presented findings at the MIT Sports Analytics conference which largely reached the same conclusions. Teams just don't outperform the market average in talent evaluation over an extended timeline.

I watched the talk and he used a similar method of considering teams and not staffs. What does the Bengals 1995 staff have to do with the Bengals 2011 staff? Is the old Browns staff carried over to the Ravens in the data? Teams are irrelevant, it's all about the people who evaluate the competitive landscape and make the pick, and that is further complicated by player development, competition, injuries, opportunity, and luck. There are many questions I have about the logic of the methodology used. The guys on the 1995 Bengals staff may be picking for another team now and competing against the Bengals 2011 staff. I don't think the study is thorough enough, but it's something to think about and probably just means I have to learn a lot more stuff. Seems like the process is bad, regardless of the outcome. Thanks for all the responses, it's appreciated.

 

[ducktales]

IMO it's definitely bad process to believe that a fantasy geek can look at a few YouTube highlights and see things that NFL teams miss.

People discover talent on youtube all the time, what an absurd statement.

I don't think the average fantasy player will be able to detect when a NFL franchise reached for a player or when a skilled player dropped, but this guy isn't an average fantasy player, he is a staff member on the biggest fantasy site in the world.

Lastly,

“Whether you think you can, or you think you can't--you're right.” - Henry Ford

 

[ducktales]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

I think you are being unfair to Waldman here, before you said scouts were smarter than you and thats why you piggyback on them --like anyone could-- now you're saying you could do what waldman and other scouts do if you put in the time and you don't because what waldman does isn't productive? The time could be put into more productive things? Wow, what the subconscious mind makes the conscious mind believe in order to protect the delicate ego is amazing.

 

[Coeur de Lion]

IMO it's definitely bad process to believe that a fantasy geek can look at a few YouTube highlights and see things that NFL teams miss.

People discover talent on youtube all the time, what an absurd statement.

I don't think the average fantasy player will be able to detect when a NFL franchise reached for a player or when a skilled player dropped, but this guy isn't an average fantasy player, he is a staff member on the biggest fantasy site in the world.

Lastly,

Whether you think you can, or you think you can't--you're right. - Henry Ford

We'll have to agree to disagree. I love FBGs as much as anyone, but you're putting them on a pretty big pedastal if you think any of them are going to out-scout the NFL. Given the resources involved IMO it's simply not possible. And I'm fairly certain that most of the staff here would agree with me.

 

[Instinctive]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

I think you are being unfair to Waldman here, before you said scouts were smarter than you and thats why you piggyback on them --like anyone could-- now you're saying you could do what waldman and other scouts do if you put in the time and you don't because what waldman does isn't productive? The time could be put into more productive things? Wow, what the subconscious mind makes the conscious mind believe in order to protect the delicate ego is amazing.

I don't think the point is that what Waldman does is unproductive, I think the point is that if Harstad also did it, it would be unproductive for him. If you gave me 1000 hours, I bet I could reach a fairly similar level of scouting as well. Do you realize just how much time that is? It is over 40 days straight, all day every day, of work. It's a 40 hour per week job for 6 months. I feel like if I did anything with a professional time commitment for a full 6 months I'd get pretty damn good at it, you know?

 

[Coeur de Lion]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

I think you are being unfair to Waldman here, before you said scouts were smarter than you and thats why you piggyback on them --like anyone could-- now you're saying you could do what waldman and other scouts do if you put in the time and you don't because what waldman does isn't productive? The time could be put into more productive things? Wow, what the subconscious mind makes the conscious mind believe in order to protect the delicate ego is amazing.

I don't think the point is that what Waldman does is unproductive, I think the point is that if Harstad also did it, it would be unproductive for him. If you gave me 1000 hours, I bet I could reach a fairly similar level of scouting as well. Do you realize just how much time that is? It is over 40 days straight, all day every day, of work. It's a 40 hour per week job for 6 months. I feel like if I did anything with a professional time commitment for a full 6 months I'd get pretty damn good at it, you know?

And then realize that any NFL team has a dozen or more people whose full time job is to invest that time 12 months a year, backed by years of experience and millions of dollars.Guys like Waldman, EBF, etc who watch a ton of college ball and break down film are certainly much better scouting than the vast majority of fantasy owners, myself included. They're certainly not better than an NFL personnel department though. That's no insult, it's just a reflection of the resources involved. Give Waldman a full time staff of scouts, a few million bucks to fly around watching live games and practices, access to players / coaches / doctors / etc and then it's a level playing field. Barring that it's absurd to even make the comparison.

 

[ducktales]

this is crazy to say you need a full staff and millions of dollars to do quality scouting.

http://www.fannation.com/truth_and_rumors/view/171150-want-attention-from-nfl-scouts-try-youtube

it takes 0 dollars to watch youtube videos

If you guys are going to become cheerleaders for Mr Adam -- Adam, can't speak for yourself?--, at least bring up valid points.

Adam himself said it wasn't a matter of time, it was a matter of intelligence; he said scouts are smarter than him and its silly to try do anything but follow them. Then when he realized how useless that made him look he changed it and said he could do it but it would be a waste of time because he could spend it on more productive things.

Whats more productive than helping subscribers get the best player available in drafts? I guess writing non football articles?

 

[ducktales]

I also want to take a moment and thank Mr Waldman, I have great respect for you because you are useful, you're just not another person repeating information we can all easily attain and that my friend is true value.

 

[MAC_32]

Great topic. I'd like to read more about some examples of good and bad processes by league type, i.e. auction league, dynasty league, etc. Perhaps some common identifying characteristics of good processes, like data driven analysis and use of metrics vs. superficial/fanboy from gut decisions, for example.

I used to rank rookies by just reading a lot of people's opinions and basically gauging hype. I've made a conscious shift, in recent years, to indexing rookie values a lot more closely to draft position. The reasoning is that NFL scouting departments are a lot smarter than I am, so it's silly not to cheat from their homework when I've got the chance.

anyone can make a rookie ranking based on draft order, so why would anyone bother reading your rookie rankings? NFL scouts aren't perfect, they may be smarter than you overall, but if you do your own homework combined with their homework you could avoid drafting a terrible player that got drafted too early --happens often enough-- and draft the better player that fell--once again, happens enough-- or you could just shrug your shoulders and call anyone that goes against NFL scouts silly because you aren't capable of doing it yourself --this is an unconscious defense mechanism to make you feel better about yourself--.

I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

I think you are being unfair to Waldman here, before you said scouts were smarter than you and thats why you piggyback on them --like anyone could-- now you're saying you could do what waldman and other scouts do if you put in the time and you don't because what waldman does isn't productive? The time could be put into more productive things? Wow, what the subconscious mind makes the conscious mind believe in order to protect the delicate ego is amazing.

I don't think the point is that what Waldman does is unproductive, I think the point is that if Harstad also did it, it would be unproductive for him. If you gave me 1000 hours, I bet I could reach a fairly similar level of scouting as well. Do you realize just how much time that is? It is over 40 days straight, all day every day, of work. It's a 40 hour per week job for 6 months. I feel like if I did anything with a professional time commitment for a full 6 months I'd get pretty damn good at it, you know?

And then realize that any NFL team has a dozen or more people whose full time job is to invest that time 12 months a year, backed by years of experience and millions of dollars.Guys like Waldman, EBF, etc who watch a ton of college ball and break down film are certainly much better scouting than the vast majority of fantasy owners, myself included. They're certainly not better than an NFL personnel department though. That's no insult, it's just a reflection of the resources involved. Give Waldman a full time staff of scouts, a few million bucks to fly around watching live games and practices, access to players / coaches / doctors / etc and then it's a level playing field. Barring that it's absurd to even make the comparison.

They are not better than the best personnel departments, but I think putting faith in most NFL personnel departments is a mistake. It's all about analysis and decision making. There are lots of personnel departments that are lousy at that, despite it being their full time jobs. Just like in any other business.

When a team like San Francisco, Green Bay, Giants, etc. picks a player I wasn't high on I take notice. Others, not so much.

 

[Adam Harstad]

As for how the idea would stand up to peer review... I'd guess probably pretty well. Cade Massey presented findings at the MIT Sports Analytics conference which largely reached the same conclusions. Teams just don't outperform the market average in talent evaluation over an extended timeline.

I watched the talk and he used a similar method of considering teams and not staffs. What does the Bengals 1995 staff have to do with the Bengals 2011 staff? Is the old Browns staff carried over to the Ravens in the data? Teams are irrelevant, it's all about the people who evaluate the competitive landscape and make the pick, and that is further complicated by player development, competition, injuries, opportunity, and luck. There are many questions I have about the logic of the methodology used. The guys on the 1995 Bengals staff may be picking for another team now and competing against the Bengals 2011 staff. I don't think the study is thorough enough, but it's something to think about and probably just means I have to learn a lot more stuff. Seems like the process is bad, regardless of the outcome. Thanks for all the responses, it's appreciated.

Well, it's not comparing the 1995 Bengals to the 2011 Bengals, it's only comparing Year N to Year N+1 (so the 2010 Bengals to the 2011 Bengals). It's like the "hot hand" studies in basketball- you don't need to compare each shot to all other shots that player has taken, you just need to compare it to the shot immediately preceding. If there's some sort of sequential dependency- if the "hot hand" exists, and a player is more likely to make his next shot if he's made several in a row- then you'd expect the data to be "streakier" than pure randomness would dictate (i.e. instead of a random distribution of shots, you'd expect a tendency for a lot of longer-than-expected hot streaks and cold streaks). The streakier the data, the higher the correlation would be between each shot and the shot immediately preceding it. Same concept here- if some teams were consistently better at scouting and some were consistently worse, you'd expect to see a lot of "hot streaks" (runs of a lot of good drafts in a row) and "cold streaks" (runs of a lot of bad drafts in a row). If that were the case, the correlation coefficient would be higher. If the data was perfectly random, the correlation coefficient would be zero. In this case, the correlation coefficient is very close to zero, which suggests that a team's draft performance in year N holds practically no predictive power for their draft performance in year N+1.

This isn't to say that everyone is equally good drafters. Surely some people are atrocious drafters. This just says that those atrocious drafters get replaced quickly, and the team regresses back to average. Over the long run, you wouldn't expect any particular team to outperform or underperform the league average in drafting.

 

[Matt Waldman]

My take on the effectiveness of football scouting. http://www.footballoutsiders.com/futures/2013/futures-front-office-overhaul

There's little that's very good about the NFL's process. Some just happen to do a better job with a bad process than others right now.

And thanks, Duck Tales - appreciate you seeing what I try to do.

And yes, I beleive directed effort + persistence yields positive results. It's a simple but elegant statement. Most focus on the simplicity until they try to do it. Then they appreciate its elegance.

 

[Coeur de Lion]

this is crazy to say you need a full staff and millions of dollars to do quality scouting. http://www.fannation.com/truth_and_rumors/view/171150-want-attention-from-nfl-scouts-try-youtube it takes 0 dollars to watch youtube videos If you guys are going to become cheerleaders for Mr Adam -- Adam, can't speak for yourself?--, at least bring up valid points. Adam himself said it wasn't a matter of time, it was a matter of intelligence; he said scouts are smarter than him and its silly to try do anything but follow them. Then when he realized how useless that made him look he changed it and said he could do it but it would be a waste of time because he could spend it on more productive things. Whats more productive than helping subscribers get the best player available in drafts? I guess writing non football articles?

I'm guessing more people would be willing to engage you if you came off as less of a bitter condescending argumentative DB, but whatever works for you.And the NFL must be the dumbest bunch of ultra-successful rich people in history, considering they waste millions of dollars on information they could get for free on YouTube. No one's saying there's no value in personally scouting guys from online highlight clips. Just that no part timer is outscouting the NFL from in front of a TV or computer screen.

 

[ducktales]

this is crazy to say you need a full staff and millions of dollars to do quality scouting. http://www.fannation.com/truth_and_rumors/view/171150-want-attention-from-nfl-scouts-try-youtube it takes 0 dollars to watch youtube videos If you guys are going to become cheerleaders for Mr Adam -- Adam, can't speak for yourself?--, at least bring up valid points. Adam himself said it wasn't a matter of time, it was a matter of intelligence; he said scouts are smarter than him and its silly to try do anything but follow them. Then when he realized how useless that made him look he changed it and said he could do it but it would be a waste of time because he could spend it on more productive things. Whats more productive than helping subscribers get the best player available in drafts? I guess writing non football articles?

I'm guessing more people would be willing to engage you if you came off as less of a bitter condescending argumentative DB, but whatever works for you.

I'm guessing you're a sensitive butterfly and i'm sorry sensitive butterfly if my strong opinions offend you.

 

[Adam Harstad]

IMO it's definitely bad process to believe that a fantasy geek can look at a few YouTube highlights and see things that NFL teams miss.

People discover talent on youtube all the time, what an absurd statement.

I don't think the average fantasy player will be able to detect when a NFL franchise reached for a player or when a skilled player dropped, but this guy isn't an average fantasy player, he is a staff member on the biggest fantasy site in the world.

Lastly,

“Whether you think you can, or you think you can't--you're right.” - Henry Ford

Footballguys is a pretty diverse crowd. We've got all types around here. If all of us were armchair scouts, they'd have to change the name to "Armchairscoutingguys.com". I'm not a college scout, and I make no claims to the contrary. I just don't watch enough college ball to think I can bring anything to the table in that regard. Surely if I devoted 1,000 hours to scouting, I could get pretty good at it... but I don't think it would be an efficient use of time. I can think of a lot of other things I could do with that 1,000 hours that would produce bigger dividends in my fantasy football ability. This is not a knock on Waldman, who is awesome at what he does and an indispensable resource. That's just not my thing. I feel I can create a big enough value add elsewhere to justify my presence, but as always, reasonable minds are welcome to disagree.

Also, it's not a question of whether people notice talent on youtube, it's a question of whether people notice talent on youtube enough to beat the market. Everyone has their success stories. I've got a few of my own- I was high on Percy Harvin and low on Chad Jackson after seeing them play at Florida. Everyone has their failure stories, too- I thought Ben Troupe was the next big thing at TE, and I thought Matt Stafford was going to flame out. The question is whether the successes outweigh the failures. I mean, if you just throw darts at the board and label every player you hit as "overrated", you'll be right half the time. You'll be wrong half the time, too. A lot of people think they can consistently beat the market. I think that belief is usually wrong, a manifestation of hubris brought on by selective memory and confirmation bias. I made the stock market analogy a while back, and I think it's apt- 100% of hedge fund managers believe they can beat the market average. These guys are well educated professionals making millions of dollars a year, working insane hours and with unlimited resources at their disposal. And yet, the overwhelming majority of them fail, and good luck guessing ahead of time who the successful ones will be. You would almost always be better off ignoring the actively managed mutual fund accounts entirely and just investing in a low-fee index fund. In this example, that's what relying on NFL scouting departments to do the leg work qualifies as- a low-fee (i.e. low-time-investment) index fund that will, over the long runs, produce similar returns to the actively managed mutual funds.

 

[Adam Harstad]

I think you are being unfair to Waldman here, before you said scouts were smarter than you and thats why you piggyback on them --like anyone could-- now you're saying you could do what waldman and other scouts do if you put in the time and you don't because what waldman does isn't productive? The time could be put into more productive things? Wow, what the subconscious mind makes the conscious mind believe in order to protect the delicate ego is amazing.

If I'm being unfair to Waldman, then I offer him my deepest apologies. I don't question the value of what he does. He makes scouting available to the mainstream. I think the NFL has a lot more resources, and I think the average team has a dozen guys who are just as good as Matt Waldman at what he does. This is not an insult to Waldman- quite the opposite, I think he does NFL-caliber scouting work, which is the highest praise I can think of for a scout. The NFL is very stingy, and they don't share the results of that hard work with the public. Matt Waldman is far less stingy, and he's happy to share the results of his hard work with the public. If I had a choice between paying $20 to get access to an NFL team's scouting notes and paying $20 to get access to Waldman's scouting notes, I'd take the NFL team's every single time. I don't know of any NFL teams that are sharing, though, which makes Waldman the absolute best around for a guy like me.

Still, as good as Waldman is, he is not better than an entire NFL scouting department. No one man could be. And while the NFL scouting departments aren't willing to share all of their detailed scouting notes with me, once a year they have a special party where they all get together and tell the whole world how they value every single rookie in the league. That party is called the "NFL draft", and for me, it's appointment viewing. When those massive NFL scouting departments start shouting to the world which players they think are the best, I listen, because each scouting department is the equivalent of a dozen Matt Waldmans, and then some, because those scouting departments have access to medical records and incident reports which remain sadly beyond Waldman's reach, and those scouting departments have entire medical staffs (sort of like a cadre of Jene Bramels) who will poke and prod and draw blood and accurately gauge a prospect's fitness and medical status, which Waldman simply doesn't have the ability to do. Those scouting departments have psychologists to gauge a player's mental state. They have access to college disciplinary records, even the ones that didn't get reported to the public. They are hulking behemoths who spend untold amounts of resources making sure that their information is as accurate and as thorough as it can possibly be. Most importantly, they are very, very, very motivated to get it right. If Matt Waldman makes a bad call, then we all gather together and commiserate and recognize that stuff happens and move on. If an NFL scout makes a bad call, they risk losing their job. If team makes a lot of bad calls, they're losing millions of dollars in revenue every year. The professionals are much more incentivized to get every call right, every time. I'm not saying that Waldman doesn't similarly strive to be right every time, I'm just saying the consequences of being wrong are far less catastrophic for Matt Waldman than they are for an NFL team.

Waldman is great. Bramel is great. These guys are brilliant, and giants in their field. NFL scouting departments are better. That's why I weigh the opinions of NFL front offices so much more heavily when I come up with my rankings.

 

[squistion]

I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

Like working on being more concise in your writing?

 

[Instinctive]

this is crazy to say you need a full staff and millions of dollars to do quality scouting.

http://www.fannation.com/truth_and_rumors/view/171150-want-attention-from-nfl-scouts-try-youtube

it takes 0 dollars to watch youtube videos

If you guys are going to become cheerleaders for Mr Adam -- Adam, can't speak for yourself?--, at least bring up valid points.

Adam himself said it wasn't a matter of time, it was a matter of intelligence; he said scouts are smarter than him and its silly to try do anything but follow them. Then when he realized how useless that made him look he changed it and said he could do it but it would be a waste of time because he could spend it on more productive things.

Whats more productive than helping subscribers get the best player available in drafts? I guess writing non football articles?

You realize highlight youtube videos are not the end all be all of scouting right?

Have you ever scouted or interacted with scouts? Do you have any idea all the other effort that goes into this?

I've built the recruiting clips that OU uses to scout for who we'll recruit, and the amount of work that I alone put into scouting JUST WIDE RECEIVERS is huge. And I don't get to make a recommendation or influence the process at all! I just watch and build the video montages for the real scouts and recruiters.

Now think about it at an NFL level. Every major college in the country. 100 or more players on every team. Quite a few are prospects at almost every D1, BCS school.

You think that all NFL scouts do is look at the youtube highlight videos? You may need a bit of a reality check, my friend. You're missing all the fundamentals that scouts look for, all the little things, the off ball plays, the habits, the techniques, the character interviews, the work ethic, the family medical history...

I'll put you, with 10 hours a day to look at youtube videos only - no other resources - against any single NFL scout. And I'd bet my entire IRA on the scout to outperform you as a talent evaluator more than 80% of the time. Hands down. I'd even give you nice odds as the underdog on top of that.

No offense, man, but you should really re-think just what you believe it takes to scout players, because the suggestion that youtube highlight vides and time is all it takes is absolutely ludicrous.

 

[Adam Harstad]

I love my man Waldman, and he's awesome at what he does. How much time do you think he spends scouting college players, though? 500 hours a year? 1,000? If I can crib off of the NFL scouts' homework and come up with a set of rookie rankings that is 95% as good in 1% of the time, that's a clear win for me. That's a huge "good process". I can use that 499-999 hours that I saved and put them to more productive purposes. I know that the NFL is wrong about some of these players, and some of these players will dramatically outperform or underperform their draft position. I'm just, in general, skeptical of anybody's ability to identify ahead of time which players will fall into which groups.

Like working on being more concise in your writing?

That would be a start.

I'm sorry that this was such a long letter, but I didn't have time to write you a short one.

-Blaise Pascal

 

[ducktales]

IMO it's definitely bad process to believe that a fantasy geek can look at a few YouTube highlights and see things that NFL teams miss.

People discover talent on youtube all the time, what an absurd statement.

I don't think the average fantasy player will be able to detect when a NFL franchise reached for a player or when a skilled player dropped, but this guy isn't an average fantasy player, he is a staff member on the biggest fantasy site in the world.

Lastly,

“Whether you think you can, or you think you can't--you're right.” - Henry Ford

Footballguys is a pretty diverse crowd. We've got all types around here. If all of us were armchair scouts, they'd have to change the name to "Armchairscoutingguys.com". I'm not a college scout, and I make no claims to the contrary. I just don't watch enough college ball to think I can bring anything to the table in that regard. Surely if I devoted 1,000 hours to scouting, I could get pretty good at it... but I don't think it would be an efficient use of time. I can think of a lot of other things I could do with that 1,000 hours that would produce bigger dividends in my fantasy football ability. This is not a knock on Waldman, who is awesome at what he does and an indispensable resource. That's just not my thing. I feel I can create a big enough value add elsewhere to justify my presence, but as always, reasonable minds are welcome to disagree.

Also, it's not a question of whether people notice talent on youtube, it's a question of whether people notice talent on youtube enough to beat the market. Everyone has their success stories. I've got a few of my own- I was high on Percy Harvin and low on Chad Jackson after seeing them play at Florida. Everyone has their failure stories, too- I thought Ben Troupe was the next big thing at TE, and I thought Matt Stafford was going to flame out. The question is whether the successes outweigh the failures. I mean, if you just throw darts at the board and label every player you hit as "overrated", you'll be right half the time. You'll be wrong half the time, too. A lot of people think they can consistently beat the market. I think that belief is usually wrong, a manifestation of hubris brought on by selective memory and confirmation bias. I made the stock market analogy a while back, and I think it's apt- 100% of hedge fund managers believe they can beat the market average. These guys are well educated professionals making millions of dollars a year, working insane hours and with unlimited resources at their disposal. And yet, the overwhelming majority of them fail, and good luck guessing ahead of time who the successful ones will be. You would almost always be better off ignoring the actively managed mutual fund accounts entirely and just investing in a low-fee index fund. In this example, that's what relying on NFL scouting departments to do the leg work qualifies as- a low-fee (i.e. low-time-investment) index fund that will, over the long runs, produce similar returns to the actively managed mutual funds.

Matt Waldman might not be a nfl scout but calling him an armchair scout is derogatory

Being able to evaluate talent isn't your thing?

you are comparing scouting --at a non nfl level-- to throwing darts at a draft board and arbitrarily saying bust or underrated? Lol?

 

[Instinctive]

My take on the effectiveness of football scouting. http://www.footballoutsiders.com/futures/2013/futures-front-office-overhaul

There's little that's very good about the NFL's process. Some just happen to do a better job with a bad process than others right now.

And thanks, Duck Tales - appreciate you seeing what I try to do.

And yes, I beleive directed effort + persistence yields positive results. It's a simple but elegant statement. Most focus on the simplicity until they try to do it. Then they appreciate its elegance.

I want to step outside the debate for a second and be clear here: You are awesome at what you do, and the amount of experience you have and time you put into it is incredible. I want to make sure that I don't come off as belittling that, because it is probably better than I could ever be at it, even if I did enter our hypothetical and work at it like a full time job. I could be very good, but there's no way I could reach your level without the years of experience and dedication you have for the art.

So thank you for doing what you do - I certainly don't have the time, nor the skill, to replicate it. Or to make it my job and even try.

 

[ducktales]

this is crazy to say you need a full staff and millions of dollars to do quality scouting.

http://www.fannation.com/truth_and_rumors/view/171150-want-attention-from-nfl-scouts-try-youtube

it takes 0 dollars to watch youtube videos

If you guys are going to become cheerleaders for Mr Adam -- Adam, can't speak for yourself?--, at least bring up valid points.

Adam himself said it wasn't a matter of time, it was a matter of intelligence; he said scouts are smarter than him and its silly to try do anything but follow them. Then when he realized how useless that made him look he changed it and said he could do it but it would be a waste of time because he could spend it on more productive things.

Whats more productive than helping subscribers get the best player available in drafts? I guess writing non football articles?

You realize highlight youtube videos are not the end all be all of scouting right?

Have you ever scouted or interacted with scouts? Do you have any idea all the other effort that goes into this?

I've built the recruiting clips that OU uses to scout for who we'll recruit, and the amount of work that I alone put into scouting JUST WIDE RECEIVERS is huge. And I don't get to make a recommendation or influence the process at all! I just watch and build the video montages for the real scouts and recruiters.

Now think about it at an NFL level. Every major college in the country. 100 or more players on every team. Quite a few are prospects at almost every D1, BCS school.

You think that all NFL scouts do is look at the youtube highlight videos? You may need a bit of a reality check, my friend. You're missing all the fundamentals that scouts look for, all the little things, the off ball plays, the habits, the techniques, the character interviews, the work ethic, the family medical history...

I'll put you, with 10 hours a day to look at youtube videos only - no other resources - against any single NFL scout. And I'd bet my entire IRA on the scout to outperform you as a talent evaluator more than 80% of the time. Hands down. I'd even give you nice odds as the underdog on top of that.

No offense, man, but you should really re-think just what you believe it takes to scout players, because the suggestion that youtube highlight vides and time is all it takes is absolutely ludicrous.

You may need to step outside the stone age you are living in and realize the internet is a vast source of information --it isn't like the old days where scouts HAD to travel around the country--, be it a players youtube video --to see their skill in action-- or their twitter account --to see if there is character issues potentially-- etc

 

[Instinctive]

this is crazy to say you need a full staff and millions of dollars to do quality scouting.

http://www.fannation.com/truth_and_rumors/view/171150-want-attention-from-nfl-scouts-try-youtube

it takes 0 dollars to watch youtube videos

If you guys are going to become cheerleaders for Mr Adam -- Adam, can't speak for yourself?--, at least bring up valid points.

Adam himself said it wasn't a matter of time, it was a matter of intelligence; he said scouts are smarter than him and its silly to try do anything but follow them. Then when he realized how useless that made him look he changed it and said he could do it but it would be a waste of time because he could spend it on more productive things.

Whats more productive than helping subscribers get the best player available in drafts? I guess writing non football articles?

You realize highlight youtube videos are not the end all be all of scouting right?

Have you ever scouted or interacted with scouts? Do you have any idea all the other effort that goes into this?

I've built the recruiting clips that OU uses to scout for who we'll recruit, and the amount of work that I alone put into scouting JUST WIDE RECEIVERS is huge. And I don't get to make a recommendation or influence the process at all! I just watch and build the video montages for the real scouts and recruiters.

Now think about it at an NFL level. Every major college in the country. 100 or more players on every team. Quite a few are prospects at almost every D1, BCS school.

You think that all NFL scouts do is look at the youtube highlight videos? You may need a bit of a reality check, my friend. You're missing all the fundamentals that scouts look for, all the little things, the off ball plays, the habits, the techniques, the character interviews, the work ethic, the family medical history...

I'll put you, with 10 hours a day to look at youtube videos only - no other resources - against any single NFL scout. And I'd bet my entire IRA on the scout to outperform you as a talent evaluator more than 80% of the time. Hands down. I'd even give you nice odds as the underdog on top of that.

No offense, man, but you should really re-think just what you believe it takes to scout players, because the suggestion that youtube highlight vides and time is all it takes is absolutely ludicrous.

You may need to step outside the stone age you are living in and realize the internet is a vast source of information --it isn't like the old days where scouts HAD to travel around the country--, be it a players youtube video --to see their skill in action-- or their twitter account --to see if there is character issues potentially-- etc

My main statement stands, regardless of how amazing the Internet is (and I agree, the info you can find is fantastic), there are a lot of things that aren't on it. Like all of the off-ball, non-highlight plays. All the practices a player goes through to see how he carries himself. Determining work ethic, family situation, who a guy spends his time with, etc...

And if you're going to invest millions of dollars in someone, and you think "I'll just check his twitter" is the best route, feel free. But you're missing a lot of things about the way the world works. I've got a twitter and a facebook, and they are pretty much representative only of my good side. A large portion of us college students are recognizing the value of manipulating social media to benefit you in employment searches.

I'm a 21 year old (who has lurked on this site since 07, and posted avidly since 09 (for example). I think I understand the value of the Internet...). I've grown up in this Internet and media age. Let's not pretend like I'm in the "stone age," but maybe recognize instead that while the Internet is an INCREDIBLE tool, it is not the end all be all. Like I said, even at the amateur level (admittedly, the very top tier of the amateur level) here at OU, every single step of scouting for recruiting takes hours of work and tons of information NOT available on the Internet. Do you think we're stupid? One of the best football colleges in the country, and we wouldn't just pull these things off the Internet if we could? Come on, man. Use your head.

 

[Coeur de Lion]

My take on the effectiveness of football scouting. http://www.footballoutsiders.com/futures/2013/futures-front-office-overhaul There's little that's very good about the NFL's process. Some just happen to do a better job with a bad process than others right now. And thanks, Duck Tales - appreciate you seeing what I try to do. And yes, I beleive directed effort + persistence yields positive results. It's a simple but elegant statement. Most focus on the simplicity until they try to do it. Then they appreciate its elegance.

I want to step outside the debate for a second and be clear here: You are awesome at what you do, and the amount of experience you have and time you put into it is incredible. I want to make sure that I don't come off as belittling that, because it is probably better than I could ever be at it, even if I did enter our hypothetical and work at it like a full time job. I could be very good, but there's no way I could reach your level without the years of experience and dedication you have for the art. So thank you for doing what you do - I certainly don't have the time, nor the skill, to replicate it. Or to make it my job and even try.

Hope you understand that I'm in no way bashing you Matt. I'm a big fan. The fact that so many of us pay for your RSP in this age of readily avaiable free information speaks volumes.