In science and statistics, if the results of randomized trial do not follow a normal distribution then something is probably causing that. It could be many things, including the design of the study, poor data collection, confounding variables that were unaccounted for, etc. But once you eliminate the obvious, all that's left is data.
Now, nothing Joel did approaches a rigorous study of the matter. What he did do was point out something you will never hear on ESPN. And yes, he probably did it on an attempt to disparage a rival (it is TV after all), but that doesn't make him wrong.
In this case, human bias is the likely culprit. Computers generally do not make mistakes like that unless told to do so. We have 2 unrelated computer systems that use different methods/algorithms for determining rankings. Both system vary from the human rankings with essentially the same error. I can tell you that my algorithm has no bias for or against any one team or conference. I am willing to assume that that is true of any computer ranking system that is not attempting to mimic or predict the CFP rankings.
All humans are biased. Even when we're aware of being biased, we cannot stop it. There is a study out there that shows this very explicitly. It essentially says that even when you can do the math (and not everyone remembers or understands normalization), you will set aside the results you arrive at if those results oppose your world view.
Anyway, I though it might be instructional to compare my rankings to the CFP and do a very simple bias check. What we have is the CFP top 25, conference, Ray Ranking, and the difference between CFP and RR. If the number is positive the CFP has the team ranked higher, and if the number is negative they have the team ranked lower:
The average difference is 0.8 places, but I've explained in previous posts how that really doesn't tell the full story. The standard deviation is 4.14 places, so there's a good bit of variance considering we're talking about only 25 spaces to work with. But far more interesting, and finally getting to the point here, is this:
The CFP has ranked the SEC schools 2 full places higher than my computer algorithm, on average. They have ranked the Big 12 an average of 1 place higher while ranking the B1G an average of 1.17 places and the Pac 12 an average of 2.33 places lower. Again, small sample size. Next season I'll run this each CFP week and see if there's a trend.
Let's take a look at the AP:
Average difference is 1.0, and SD is 5.2 places, so a little more variation in the rankings overall, but the real issue is this:
-->
The SEC is ranked nearly 4 full places higher than the computer on average. The Big 10 and Pac 12 are ranked an average of 2.83 and 3.0 places lower than the computer. One conference ranked systematically higher and two ranked systematically lower by human voters. Because they just know.
The real tragedy is the indoctrination we have all been through for the past however many years. We have been trained to believe that the AP (and now the CFP) know more because hey, they watch the games and they've been there forever and they just know. And we have ESPN and Fox with their vested interests in selling ad time telling us how right or wrong the other one is. And they want us to believe that being partners with the conference TV channels doesn't affect their "reporting."
Well, now we watch the games, too. And we have Bill James, who showed the world that there is a better way to evaluate talent, an objective way. And we know that humans are biased. All humans. This is why we need that paradigm shift away from voters and committees towards computer ranking systems that treat all teams equally. And let the chips fall where they may!
No comments:
Post a Comment