There is definitely a bit of an old boys club at the top, especially when it comes to the boards and executives of most companies. That being said, a great CEO is worth every penny, regardless of what they make, while a bad CEO is a horrible misallocation of capital. That is not unlike most high profile jobs.
I think there are at least three factors contributing to the runaway salaries of Fortune 500 CEOs.
(1) The most commonly given reason, and perhaps the least convincing of the three: it's an old boys club. The executives at one company are board members of another. They vote on each other's salaries, and even if there isn't an explicit
quid pro quo in place, they all must realize that high pay for executives in general is in their own best interest.
(2)
Companies try to avoid paying their executives a below-average amount because they don't want to signal to the world (i.e., to institutional investors) that they lack faith in their leadership. Ever since well-meaning regulations took effect that required public companies to disclose their compensation to executives, the companies that paid a below-average amount have sought to raise their executive pay up to the average. But this just creates a new, higher average, and the system works as a ratchet, constantly pushing the average upwards.
(3) Companies are bigger now than they have historically been, what with global markets and international trade deals and whatnot. This increases the value of a very good CEO over his peers, just as adding 1.5 PPR for tight ends increases the value of Rob Gronkowski over Vernon Davis. When a company is doing 50 billion dollars in annual sales, the marginal value of a CEO who oversees 5.3% annual growth over a replacement-value CEO who would have engineered only 5.1% growth may be about a hundred million dollars. The pay of CEOs has risen sharply over the past few decades because they are simply worth more now. (At least the good ones are, and companies are optimistic that their own executives will be good ones.)