Six of one, half dozen of the other.
Our younger daughter has a May birthday, is small for her age, was very shy and we considered holding her back, but ultimately did not. Now, she just finished 3rd grade, but will be entering an accelerated cluster for 4th graders next year, where they will start the 5th grade curriculum.
On the flip side, my older daughter just finished 5th grade, and one of her classmates was a "redshirt" (or however you want to describe it). The classmate was a bully and behavioral problem early on in 1-3rd grades, but in the last couple of years has matured, and was one of 3 kids with straight A's in 5th grade.
I think it worked out in each case.
I do think the "danger" in holding kids back in the hopes that they are "smarter" is that the learning curve at a young age is massive, but that it flattens out as kids get older. So you may find yourself with a kid who is reading before his/her classmates, but by the time kids are learning substantive material, your kid will be about where he/she is going to be in life.