I think there's an assumption that if a guy played football at Harvard, he must be smart. And sure, he can't be a dummy. But it doesn't necessarily mean that he's unusually smart.
The way Ivy League athletics work is that coaches are allowed to designate a number of applicants where they can essentially put their thumb on the scale. Again, these kids can't be total academic basket cases. But they definitely don't meet the standards of the general population of applicants. I actually went to an Ivy (with a good football program), and there were definitely guys on the team who were not exactly lighting it up in the classroom.
Think of it this way. If I told you there were two random students, one of whom went to Harvard and the other who went to the University of Tennessee, it would be reasonable to assume that the Harvard kid was smarter. But if I told you those two students were Ryan Fitzpatrick and Peyton Manning, that assumption becomes less reasonable. Because the reason Peyton went to UT was not that he wasn't smart enough. It's that he was too good at football. If he had wanted to go to Harvard, they would have found a way to admit him. And based on what I've seen of him, I'd say there's a pretty good chance that he's smarter than Fitz.
ETA: I just looked up, and apparently Fitz got 1580 on his SATs while Peyton got a 1030. So while my general point about Ivy League athletes still holds, I may have gotten it wrong in this specific case. Though I still think Peyton is plenty smart.