I was a PD for about 3 years combined and worked for a PD all through law school.  Furthermore, I handle a large chunk of the caseload my current firm gets with our public defense contracts (some jurisdictions, usually the smaller ones, won't have a formal public defender's office but will instead contract the work out to private defense attorneys) so I still do plenty of "public defense".  So, in total, I have near seven years of working public defense. I'll try to answer as many as I can.
First, concerning strictly the work, I love public defense work.  It offers the most rewarding parts of practice for me (negotiations, trials, and client contact), is never boring, and doesn't include the negative aspect of paying clients expecting the world from you and you wanting to deliver it solely so you can get paid.  I have some crazy stories and when I get together with my civil friends I usually have the most interesting stories. There are certainly times where it's a thankless job (i.e. you did your best to get a guy who did some awful things a good plea deal and you have a victim cursing at you,  a monster for a client with unreasonable expectations who treats you like dirt, and a public who doesn't even think you're a "real lawyer" yet helped carry out some injustice). Nonetheless, that stuff really didn't bother me a ton.
Second, regarding the hours worked, I imagine this is strictly jurisdiction specific.  In my experience, I never really had one of those nightmarish caseloads you may see in the movies.  Frankly, unless I had a big trial I had to prep for, I rarely worked more than 40 hours/week.  Hell, when I worked for a major city's public defense offense the office kept the caseload so manageable and the prosecutor's had no authority to deviate off a plea matrix I maybe did 25 hours of actual work a week.  I work far more now in private practice than I ever did as a PD (despite the common notion that private attorneys aren't overworked like PDs).
Third, regarding pay, again that's jurisdiction specific.  In the two states I've worked (MN in school, AZ for work) the pay was pretty uniformly scaled across the state.  Since it's public record, I can tell you in AZ a starting PD makes in the mid-50s.  For me, as a single dude fresh out of law school, the 50k or so I made coming out of school and living in a rural area was more than enough to pay the bills and throw money at girls and Vegas on the weekends.  Obviously, in a city where the cost of living is higher, or if one has a family and hefty student loans, I'm sure it's more difficult.  In some of the other cities, however, I don't think it's quite as good.  I have had buddies go work in the Boston and LA PDs and they barely made enough to cover their student loans.  Nonetheless, an experienced attorney who takes a job as the jurisdiction's "Public Defender" could make six figures and I know the federal guys are making near that, too.
Finally, as far whether it's a common "starting job" I'd say it's a very common (and absolutely worthwhile) starting position for those that want to do criminal defense, but it's not necessarily a common "starting job" across the board.  I would imagine as well that since the starting salaries do tend to be a bit lower, there are probably a higher rate of public defenders from lower-tiered schools, but that's maybe a loose correlation.  Heck, in my experience when I started I came in with a Harvard, a Georgetown, a Cal, and a University of Chicago grad.  As Krista indicated, for some public defense is what these people view as their missions in life and it truly isn't about the money for them.
That was way too long, but hopefully that's helpful.