There are a couple of different common ways for keeper compensation. Each has different impacts, and there is no "right" answer but it comes down to what you prefer.
One method is Every team keeps X players, and they essentially don't cost draft picks. If such a league keeps 2 players, then essentially the draft starts in the third round. You can still call it the 1st, but the point is, every team has an identical number of picks, it's as if everyone had to spend their first 2 picks on their first 2 players. And every team should keep 2 players since not keeping a player doesn't give you any extra pick to use. Compared to other methods, this method helps the most the teams with the best studs. If someone has two top 5 players and another team's best two players are the equivalent of 5th round players, then you could equate it to the latter team having to use his first 2 picks on 5th round players while the better team got to use his first two picks on two elite players. So bad teams may be forced to use what would have been early picks on players not worth that slot, compared to redraft. There tend to be very few studs available for teams with the early picks when the draft does start. Further, teams with multiple elite players are likely to come out even further ahead as they can trade them to teams who don't have as good of keepers, and who don't pick at the top of the beginning round of the draft where they could take that player with their own pick. Unless the league institutes some additional rule to limit how many times a player can be kept, the player could be kept throughout his career.
Another method is some form of Keep them at a round they were drafted or kept the year before, possibly minus some adjustment. Free agents either can't be kept, or are given some set round number. The "adjustment" might be, keep them at one round earlier than where they were drafted, and you can't keep someone with a round less than 1. This method results in keepers being primarily break out players or those who were undervalued, since they present the best value. It means that many top picks will still be available in the draft, and top picks eventually recycle. However, a downside is it can created the biggest disparity between the cost of keeping the player and his value. Depending on league parameters and how early in preseason drafts happened, Arian Foster, Clinton Portis, and Priest Holmes are examples of late picks who ended up being consistently elite players. In the previous system, Foster would essentially cost a 1st or 2nd in a keep 2 league. In a league he is kept at Previous Round minus 1, he might have been kept in the 6th round and 5th rounds the last 2 years, which obviously makes him much more valuable than in the previous setup. Some teams might not even have a player worth the draft pick they'd need to give up. Good teams aren't as likely to be able to trade keeper worthy players for draft picks as the last setup, and probably get less (later round picks). If leagues do lower the round at which the player must be kept, then players cannot be kept throughout a long career, but some could still be keeper worthy through their effective career.
The final common method is You can keep X players, and doing so costs your earliest X picks. If not kept, you use those picks normally. In this system, if you can keep up to 4 players, they would cost your 1st, 2nd, 3rd, and 4th round picks and your first selection would be in the 5th round. If another team only kept 2, they would cost their 1st and 2nd round pick, and then the team would draft available players with their 3rd and 4th round picks. A team whose roster has few or no players better than those thrown back could keep 0 and select 4 players who weren't kept before the team who kept 4 would draft. This system tends to be somewhere in the middle of the other two in most aspects. Players tend to be kept for picks closer to their actual value than the previous 2 systems. Only players worthy of the first X rounds will get kept (assuming your owners have a modicum of sense), as otherwise they could throw that player back and there would have to be a better available who another team had to cut. In addition, such a team would get an earlier crack at rookies which can make up for not having keeper worthy players. Good teams are helped... if you keep 2 you could still keep Foster and Rice with 1st and 2nd round picks... but bad teams have a way of using a pick before the round in which everyone again has picks. Bad teams may choose to not keep anyone just in order to get first shot at high profile rookies like an Adrian Peterson, Trent Richards, Calvin Johnson, Andrew Luck or RG3. Actually even good teams may throw all their players back to do this if they really think someone will be special. Without some additional method of limiting how many times a player can be kept, a player could be kept throughout their career, so long as they continue to be worth an early selection. While teams may trade for keeper worthy players not being kept, the price paid should be less than the first system since the team acquiring the player can still get the player by just not keeping players, and then is only competing against other teams who didn't keep the same number of players. Such trades are probably for more than trades in the 2nd system, since such trades in the second system are more likely for middle round players being kept with a late pick, not for earlier round quality players, so that would limit what is given up to late picks.
Also worth mentioning, when my own leagues both went keeper many years ago, we debated whether we should put a limit on how many years in a row a player could be kept. I went back for about 5 years and posted who the top 10 fantasy QBs, RBs, and WRs were each year. RBs there is an extreme amount of turnover. It's a rare RB who stays in the top 5 for a long time, and generally half of the top 10 RBs turn over each year. Even amongst QBs, the amount of turnover is more than you might expect. I'd suggest any leagues considering this do similar and look and see just how much turnover there is and thus how much players will get thrown back into the draftable pool naturally before deciding if such a rule is needed. We did not adopt such a rule, and I really can't think of a time in the last 10 years now I've wished we had it. Even the top players like an Edge, Faulk, Priest, Tomlinson, Peterson, etc, get traded after a few years as teams are looking to get max value for them before they start dropping off. That, or they drop off in performance, or get injured. I personally don't think such rules are needed, and that's my opinion after watching leagues that keep 4 players, where it would be more pronounced if being able to keep a player unlimited was going to be a problem. I can say honestly I've always had amongst the best set of keepers year after year, but that's only happened because I'm actively acquiring new players. Most years at least 2 of my 4 keepers change, and while I've had stud RBs pretty consistently during that time, their identity has continually changed.
Ok, so now having attempted an objective discussion of the three main ways of doing keepers... I'll add my personal preference. I much prefer the third system. I think giving teams the option of first choice from non-drafted players is a great way of giving smart owners a chance to turn around a bad team. (While I suppose bad owners may blindly keep players who aren't worth it and get their just desserts.) I don't like forcing teams to use picks on players who aren't worth it, which the first system does. And I don't like the second system's outcome of some teams having significant players at fantastic values while others not only don't have anything comparable, but get no additional advantage to offset it. As an example, in the latter system I'd have had Arian Foster keepable with 9th and 8th round picks these last 2 years. That's just way too big an advantage considering I'd still have my 1st, 2nd, etc round picks to use on players same as teams who don't have an obscenely cheap Foster.
I also like the third system for the ability to cut bait on a bad team when you see some rookies coming out you like. Despite having had good keeper options, I threw my roster back into the undrafted pool so I could use my earliest picks on Adrian Peterson his rookie year. I like this particular element of strategy that the third system adds, which there really is nothing comparable in the other two.
They all can work, and some aspects of the other systems may appeal to a group more. I've done all three and think the actual pluses and minuses make the third the better system by a fair margin, but any of them will work for a league.