It has to be case by case. I was one of the youngest in my class, turning five a few days after the first day of kindergarten, and I was smallish until I was a sophomore in HS. It might have been a small social hinderance in grade school, but wasn't an issue academically or in sports. (I wasn't THAT much smaller).
My son was born a couple weeks AFTER the cutoff date, and they made me wait a year for kindergarten. The next year, he was so far ahead of all the other kids academically that they promoted him within 2 months to the first grade. Like me, he's on the smallish side for his grade (now 7th) but is doing very well academically. Socially he's holding his own, but is certainly not "popular" (I think I'm kind of glad of that).
My son was born a couple weeks AFTER the cutoff date, and they made me wait a year for kindergarten. The next year, he was so far ahead of all the other kids academically that they promoted him within 2 months to the first grade. Like me, he's on the smallish side for his grade (now 7th) but is doing very well academically. Socially he's holding his own, but is certainly not "popular" (I think I'm kind of glad of that).