Why do rankings use averages for consensus? (1 Viewer)

[videoguy505]

3 User(s) are reading this topic (0 Guests and 0 Anonymous Users)

3 Members: Dave Baker, Doug Drinen, thunderclap

Tease.

 

[freshly_shorn]

"my stats professor says you're all wrong" is not a great argument.

How about the stats books and stats rules? Are those a good argument? The problem is y'all have tried to reframe my argument in a few different ways. In the immortal words of George Herbert Walker Bush, read my lips:It is statistically meaningless to average a group of rankings.Almost all of you have agreed with that technicality. None of you have acknowledged my later sentiment about 'assigning' meaning to it.Guys, I'll be the first to admit that I've pointed out a mere technicality. No one seems to acknowledge my repeated sentiments that since you guys have assigned meaning to the average, and the audience is willing to accept your meaning, then it really is OK.Oh- and I'm also posting the Prof's remarks because Chase Stuart was interested, not just as part of my argument. Just another expert saying averaging rankings is statistically meaningless.

Theory and application are two different animals. Sure it's incorrect to average ordinal data, but in this case it's useful. If you want a measure of variability among the rankings, you could take take the standard deviation of rankings (though CV would probably be more useful). Kurtosis, why not? Is it theoretically correct? No. But it provides useful information which is all that really matters, in application.I understand you're just pointing out a technicality having been freshly imbued with statistical knowledge but in the end no FF data is statistically significant. It all comes down to the user's judgement.

At least you understand the point. And (as no one has yet acknowledged), I agree that it can still be useful to a point, as long as one takes it with a grain of salt, as the old saying goes.

 

[freshly_shorn]

I'm fresh out of a graduate statistics class

I probably should have stopped here.I'll take the word of people who have actual experience in this sort of analysis over someone who just learned about it in grad school.

Put it another way, academia is where you learn the rules. Real life is where you learn when to break the rules.

The only question is whether the mean is more useful than another measure. If so, use the mean. If not, use the alternative. And thanks to MT and Jayman for finally getting to this question. I'm not sure which number is more useful, but I think these guys are finally getting somewhere.

Thanks, Confucius, I'll keep that in mind.

 

[NattyDread]

As a graduate student in the social sciences, this has been a very interesting thread. Thanks for freshly_shorn for inititating the discussion. From my perspective, the general conclusions seem to be:

1. Taking the mean of ordinal data has no statistical meaning

2. Taking the mean of ordinal data may have functional meaning

For the purposes of the rankings, both the mean and median would have value for me in trying to assess central tendency and the effect of outliers in the rankings. I for one would like to see the median added to the rankings page alongside the average.

 

[joey]

As a graduate student in the social sciences, this has been a very interesting thread. Thanks for freshly_shorn for inititating the discussion. From my perspective, the general conclusions seem to be:1. Taking the mean of ordinal data has no statistical meaning2. Taking the mean of ordinal data may have functional meaningFor the purposes of the rankings, both the mean and median would have value for me in trying to assess central tendency and the effect of outliers in the rankings. I for one would like to see the median added to the rankings page alongside the average.

After all the stats nerd dust has settled, I'd like to add thatI too would like to see a median column added to the rankings.Sounds like it could be useful.Yes. I said "useful". Not "correct" or "better" or whatever. Just plain old usefuland that's all I'm looking for

 

[freshly_shorn]

I think it would also be useful to see 'MDP'- median draft position. Since median does deal with outliers differently from mean, I think you get closer to where a given player is 'most likely' to go. But, you'd also have to know the min/max (which most mock draft examinations provide).

I am also surprised one of the admins didn't point out that right above the rankings there is a blurb about how rankings = points, which are then averaged. This implies, then, that participants know that their ranking is going to be numerically interpreted. As I mentioned way back in some distant post where I quote from a stats book, you "can" treat such ordinal data as interval under those conditions, with caution.

Hopefully FBG will consider adding median to their rankings... it'd be cool to see it as part of ADP, too.

In all seriousness, thanks to the real stats guys and admins for taking this seriously. I can see why this is called the shark pool.

 

[Majorum]

This whole discussion is amusing me, so i decided to finally register and want to add my view to this topic...

What we are looking for here is the FBG staff consensus. Assuming each staff member is dead set on their respective ranking, if you lock them all in a room and ask them to produce one ranking, it seems to me that you simply ask everyone who their #1 pick is, and if you have a majority, you move on. If not, just drop the player(s) with the least amount of vote, and revote on the remaining players until you get majority. This procedure assures that every staff member is taken into account at every step (which the median does not), while also disregarding the numerical difference between the rankings (which the mean does). Since the rankings of each staff member is out there, it's a simple matter of coding the algorithm, which I did...

Consensus|Mean|Diff|Player1 |1 |0 |RB LaDainian Tomlinson, SD2 |2 |0 |RB Steven Jackson, STL3 |3 |0 |RB Larry Johnson, KC4 |5 |-1 |RB Shaun Alexander, SEA5 |4 |1 |RB Frank Gore, SF6 |6 |0 |RB Brian Westbrook, PHI7 |7 |0 |RB Willie Parker, PIT8 |10|-2 |RB Clinton Portis, WAS9 |8 |1 |RB Joseph Addai, IND10|9 |1 |RB Rudi Johnson, CIN11|12|-1 |QB Peyton Manning, IND12|14|-2 |RB Laurence Maroney, NE13|11|2 |RB Reggie Bush, NO14|15|-1 |RB Travis Henry, DEN15|16|-1 |WR Steve Smith, CAR16|13|3 |WR Chad Johnson, CIN17|19|-2 |RB Willis McGahee, BAL18|17|1 |RB Maurice Jones-Drew, JAX19|18|1 |WR Larry Fitzgerald, ARI20|20|0 |WR Torry Holt, STL21|24|-3 |WR Reggie Wayne, IND22|22|0 |TE Antonio Gates, SD23|27|-4 |WR Marvin Harrison, IND24|23|1 |RB Edgerrin James, ARI25|21|4 |RB Ronnie Brown, MIA26|25|1 |WR Terrell Owens, DAL27|29|-2 |RB Cedric Benson, CHI28|32|-4 |RB Marion Barber III, DAL29|28|1 |WR Roy Williams, DET30|26|4 |QB Carson Palmer, CIN31|34|-3 |RB Cadillac Williams, TB32|30|2 |QB Tom Brady, NE33|36|-3 |RB Thomas Jones, NYJ34|35|-1 |WR Randy Moss, NE35|33|2 |QB Drew Brees, NO36|31|5 |WR Anquan Boldin, ARI37|45|-8 |RB Adrian Peterson, MIN38|42|-4 |WR T.J. Houshmandzadeh, CIN39|40|-1 |WR Javon Walker, DEN40|38|2 |WR Andre Johnson, HOU41|41|0 |WR Donald Driver, GB42|37|5 |WR Marques Colston, NO43|39|4 |WR Lee Evans, BUF44|44|0 |RB Marshawn Lynch, BUF45|46|-1 |QB Donovan McNabb, PHI46|43|3 |RB Deuce McAllister, NO47|48|-1 |RB Jamal Lewis, CLE48|47|1 |RB Ahman Green, HOU49|49|0 |QB Marc Bulger, STL50|51|-1 |TE Tony Gonzalez, KC51|54|-3 |TE Todd Heap, BAL52|50|2 |WR Plaxico Burress, NYG53|52|1 |QB Michael Vick, ATL54|56|-2 |RB Brandon Jacobs, NYG55|53|2 |RB Warrick Dunn, ATL56|66|-10|WR Braylon Edwards, CLE57|62|-5 |RB LaMont Jordan, OAK58|55|3 |RB DeAngelo Williams, CAR59|65|-6 |TE Jeremy Shockey, NYG60|74|-14|TE Kellen Winslow Jr, CLE61|60|1 |WR Laveranues Coles, NYJ62|61|1 |WR Chris Chambers, MIA63|64|-1 |WR Santana Moss, WAS64|67|-3 |WR Hines Ward, PIT65|79|-14|RB Tatum Bell, DET66|71|-5 |WR Calvin Johnson, DET67|57|10 |WR Darrell Jackson, SF68|68|0 |RB Fred Taylor, JAX69|63|6 |TE Alge Crumpler, ATL70|72|-2 |WR Joey Galloway, TB71|58|13 |QB Jon Kitna, DET72|70|2 |RB Vernand Morency, GB73|59|14 |RB LenDale White, TEN74|69|5 |RB Julius Jones, DAL75|77|-2 |RB Kevin Jones, DET76|73|3 |WR Deion Branch, SEA77|84|-7 |TE Vernon Davis, SF78|83|-5 |RB Chester Taylor, MIN79|81|-2 |WR Reggie Brown, PHI80|80|0 |TE Jason Witten, DAL81|78|3 |TE Chris Cooley, WAS82|82|0 |TE Ben Watson, NE83|76|7 |RB Jerious Norwood, ATL84|75|9 |QB Philip Rivers, SDI did not come up with a definitive tie-breaker procedure (which is not required if the number of rankings is prime ) so who is selected in a tie is sort of random (more precisely, its decided by the order in which the program process the data). If I remember correctly, Portis is an example of a player jumping a few spots in a tie-breaker. Other funny stuff include Steve Smith jumping ahead of Chad Johnson, which happens because at that point in the draft, 5 staffer would select McGahhe, 5 would select Smith and 4 would pick Chad (off the top of my head), so once you drop Chad, the extra votes push Smith ahead, while the majority prefers Johnson over Smith in a vacuum.In the end, the correct way to get a consensus comes down to how you want to define 'most likely to go next'. I think the procedure mentioned above is as close as you can get.

Majorum

 

[freshly_shorn]

This whole discussion is amusing me, so i decided to finally register and want to add my view to this topic...What we are looking for here is the FBG staff consensus. Assuming each staff member is dead set on their respective ranking, if you lock them all in a room and ask them to produce one ranking, it seems to me that you simply ask everyone who their #1 pick is, and if you have a majority, you move on. If not, just drop the player(s) with the least amount of vote, and revote on the remaining players until you get majority. This procedure assures that every staff member is taken into account at every step (which the median does not), while also disregarding the numerical difference between the rankings (which the mean does). Since the rankings of each staff member is out there, it's a simple matter of coding the algorithm, which I did...

Consensus|Mean|Diff|Player1 |1 |0  |RB LaDainian Tomlinson, SD2 |2 |0  |RB Steven Jackson, STL3 |3 |0  |RB Larry Johnson, KC4 |5 |-1 |RB Shaun Alexander, SEA5 |4 |1  |RB Frank Gore, SF6 |6 |0  |RB Brian Westbrook, PHI7 |7 |0  |RB Willie Parker, PIT8 |10|-2 |RB Clinton Portis, WAS9 |8 |1  |RB Joseph Addai, IND10|9 |1  |RB Rudi Johnson, CIN11|12|-1 |QB Peyton Manning, IND12|14|-2 |RB Laurence Maroney, NE13|11|2  |RB Reggie Bush, NO14|15|-1 |RB Travis Henry, DEN15|16|-1 |WR Steve Smith, CAR16|13|3  |WR Chad Johnson, CIN17|19|-2 |RB Willis McGahee, BAL18|17|1  |RB Maurice Jones-Drew, JAX19|18|1  |WR Larry Fitzgerald, ARI20|20|0  |WR Torry Holt, STL21|24|-3 |WR Reggie Wayne, IND22|22|0  |TE Antonio Gates, SD23|27|-4 |WR Marvin Harrison, IND24|23|1  |RB Edgerrin James, ARI25|21|4  |RB Ronnie Brown, MIA26|25|1  |WR Terrell Owens, DAL27|29|-2 |RB Cedric Benson, CHI28|32|-4 |RB Marion Barber III, DAL29|28|1  |WR Roy Williams, DET30|26|4  |QB Carson Palmer, CIN31|34|-3 |RB Cadillac Williams, TB32|30|2  |QB Tom Brady, NE33|36|-3 |RB Thomas Jones, NYJ34|35|-1 |WR Randy Moss, NE35|33|2  |QB Drew Brees, NO36|31|5  |WR Anquan Boldin, ARI37|45|-8 |RB Adrian Peterson, MIN38|42|-4 |WR T.J. Houshmandzadeh, CIN39|40|-1 |WR Javon Walker, DEN40|38|2  |WR Andre Johnson, HOU41|41|0  |WR Donald Driver, GB42|37|5  |WR Marques Colston, NO43|39|4  |WR Lee Evans, BUF44|44|0  |RB Marshawn Lynch, BUF45|46|-1 |QB Donovan McNabb, PHI46|43|3  |RB Deuce McAllister, NO47|48|-1 |RB Jamal Lewis, CLE48|47|1  |RB Ahman Green, HOU49|49|0  |QB Marc Bulger, STL50|51|-1 |TE Tony Gonzalez, KC51|54|-3 |TE Todd Heap, BAL52|50|2  |WR Plaxico Burress, NYG53|52|1  |QB Michael Vick, ATL54|56|-2 |RB Brandon Jacobs, NYG55|53|2  |RB Warrick Dunn, ATL56|66|-10|WR Braylon Edwards, CLE57|62|-5 |RB LaMont Jordan, OAK58|55|3  |RB DeAngelo Williams, CAR59|65|-6 |TE Jeremy Shockey, NYG60|74|-14|TE Kellen Winslow Jr, CLE61|60|1  |WR Laveranues Coles, NYJ62|61|1  |WR Chris Chambers, MIA63|64|-1 |WR Santana Moss, WAS64|67|-3 |WR Hines Ward, PIT65|79|-14|RB Tatum Bell, DET66|71|-5 |WR Calvin Johnson, DET67|57|10 |WR Darrell Jackson, SF68|68|0  |RB Fred Taylor, JAX69|63|6  |TE Alge Crumpler, ATL70|72|-2 |WR Joey Galloway, TB71|58|13 |QB Jon Kitna, DET72|70|2  |RB Vernand Morency, GB73|59|14 |RB LenDale White, TEN74|69|5  |RB Julius Jones, DAL75|77|-2 |RB Kevin Jones, DET76|73|3  |WR Deion Branch, SEA77|84|-7 |TE Vernon Davis, SF78|83|-5 |RB Chester Taylor, MIN79|81|-2 |WR Reggie Brown, PHI80|80|0  |TE Jason Witten, DAL81|78|3  |TE Chris Cooley, WAS82|82|0  |TE Ben Watson, NE83|76|7  |RB Jerious Norwood, ATL84|75|9  |QB Philip Rivers, SD

I did not come up with a definitive tie-breaker procedure (which is not required if the number of rankings is prime ) so who is selected in a tie is sort of random (more precisely, its decided by the order in which the program process the data). If I remember correctly, Portis is an example of a player jumping a few spots in a tie-breaker. Other funny stuff include Steve Smith jumping ahead of Chad Johnson, which happens because at that point in the draft, 5 staffer would select McGahhe, 5 would select Smith and 4 would pick Chad (off the top of my head), so once you drop Chad, the extra votes push Smith ahead, while the majority prefers Johnson over Smith in a vacuum.In the end, the correct way to get a consensus comes down to how you want to define 'most likely to go next'. I think the procedure mentioned above is as close as you can get.Majorum

Interesting, and

 

[Macho Camacho]

as has been said a few times, it seems like the biggest issue in this thread is semantics. FBG offers this method of rankings because it's much quicker and much less painless than actually taking the time to do projections for hundreds of different players. thus, they're able to give to give sets of rankings to appease the diehards among us who eat this stuff up, generate discussion and traffic to the web site, and to serve people with abnormally early drafts.

if you're convinced the data is "meaningless," then ignore it. but it certainly serves many purposes and can be useful to many people.

 

[Giggity]

as has been said a few times, it seems like the biggest issue in this thread is semantics. FBG offers this method of rankings because it's much quicker and much less painless than actually taking the time to do projections for hundreds of different players. thus, they're able to give to give sets of rankings to appease the diehards among us who eat this stuff up, generate discussion and traffic to the web site, and to serve people with abnormally early drafts.if you're convinced the data is "meaningless," then ignore it. but it certainly serves many purposes and can be useful to many people.

This is actually one of the better threads this offseason. Looking at how players are ranked, averaged, projected, is really at the heart of FBGs. I don't think anyone is saying use average to rank players is useless, I believe it's merely a discussion on possible improvements, and how those improvements differ.

 

[CalBear]

As we get even further afield, stat geeks may be interested to look at Arrow's Impossibility Theorem. Kenneth Arrow, from a junior university in the South Bay, proved that it is impossible for a voting system based on ranked preferences to consistently and accurately reflect the rational desires of the voters in the system.

The Impossibility Theorem is too stat geeky for most people to grasp, but the short version is that, in the realm of ranked-choice voting, the Impossibility Theorem is similar to Goedel's Incompleteness Theorem and Heisenberg's Uncertainty Principle; a mathematical proof that a set of criteria which intuitively might seem to be easily satisfiable are, in fact, mutually exclusive.

The upshot is, there is no "correct" way to turn a list of ranked choices from n people (where n>2) into a list of ranked choices which meets the reasonable criteria put forth by Arrow. Something will not be satisfied. All you can do is decide which failure characteristics you want your system to have.

 

[Chase Stuart]

As we get even further afield, stat geeks may be interested to look at Arrow's Impossibility Theorem. Kenneth Arrow, from a junior university in the South Bay, proved that it is impossible for a voting system based on ranked preferences to consistently and accurately reflect the rational desires of the voters in the system.

The Impossibility Theorem is too stat geeky for most people to grasp, but the short version is that, in the realm of ranked-choice voting, the Impossibility Theorem is similar to Goedel's Incompleteness Theorem and Heisenberg's Uncertainty Principle; a mathematical proof that a set of criteria which intuitively might seem to be easily satisfiable are, in fact, mutually exclusive.

The upshot is, there is no "correct" way to turn a list of ranked choices from n people (where n>2) into a list of ranked choices which meets the reasonable criteria put forth by Arrow. Something will not be satisfied. All you can do is decide which failure characteristics you want your system to have.

Not to get too stat geeky, but the reasonableness of each criterion imposed by Arrow is questionable. Moreover, it's not that it's impossible for a voting system to work, it's just that it will fail a nonzero number of times. That's a pretty significant difference (unless your consistent and accurately language was meant to imply an infinite number of trials).

 

[CalBear]

Not to get too stat geeky, but the reasonableness of each criterion imposed by Arrow is questionable. Moreover, it's not that it's impossible for a voting system to work, it's just that it will fail a nonzero number of times. That's a pretty significant difference (unless your consistent and accurately language was meant to imply an infinite number of trials).

To bring this back to the original discussion, I think it's reasonable to expect that fairly realistic cases could be constructed to demonstrate failures of either mean or median as a ranked-preference voting choice for fantasy rankings. Arrow's paradox certainly doesn't imply that you should throw up your hands and not worry about your voting system; there are certainly voting systems which will meet desirable criteria more often than others, and it's therefore reasonable to be having this conversation about whether mean, median, or ranked-choice elimination would be the most effective way to rank players based on a set of FBG staff rankings. I'm not going to do the math, but I'm pretty sure the ranked-choice elimination method described above fails Arrow's "independence of irrelevant alternatives" test; however, that doesn't mean that it's not an improvement over mean or median, in terms of producing desirable results.

 

[freshly_shorn]

Not to get too stat geeky, but the reasonableness of each criterion imposed by Arrow is questionable. Moreover, it's not that it's impossible for a voting system to work, it's just that it will fail a nonzero number of times. That's a pretty significant difference (unless your consistent and accurately language was meant to imply an infinite number of trials).

To bring this back to the original discussion, I think it's reasonable to expect that fairly realistic cases could be constructed to demonstrate failures of either mean or median as a ranked-preference voting choice for fantasy rankings. Arrow's paradox certainly doesn't imply that you should throw up your hands and not worry about your voting system; there are certainly voting systems which will meet desirable criteria more often than others, and it's therefore reasonable to be having this conversation about whether mean, median, or ranked-choice elimination would be the most effective way to rank players based on a set of FBG staff rankings. I'm not going to do the math, but I'm pretty sure the ranked-choice elimination method described above fails Arrow's "independence of irrelevant alternatives" test; however, that doesn't mean that it's not an improvement over mean or median, in terms of producing desirable results.

That is the larger question which should be culled from this debate: what is the best way to represent a quick ranking consensus? It appears that without experimentation, you would provide at least both the mean and median, though the ranked-choice-elimination method also seems interesting. At any rate, many have said there is no perfect way to do this, but there are a few different ways that appear informative. What FBG ultimately does with this, who knows, but they do provide the information in such a format that we can compute our own results.