My family set them off every year for as long as I can remember. We lived in rural VA on 4+ acres, then later on a 40 acre farm. We started with what we could buy illegally in rural Virginia, and slowly got bigger until we were going to PA and stocking up every summer. I've been the one lighting them since I was probably 8 years old or so. My Dad showed me what to do, and made it very clear that I light them and run. If I light them, and they don't go off, I would let him deal with them. Never had an injury shy of some minor burns from the burning fuses. Was one of my favorite parts of childhood honestly. I enjoyed our home show more than the big ones at the park.
There are a lot of things that are dangerous because people are stupid. Not saying fireworks aren't dangerous, just that most of the folks who get seriously injured are largely to blame more than the fireworks. If you give an idiot enough time, he'll find a way to hurt himself with most anything.
I also think it's a city/country mentality difference. My wife is from a more urban area, and the 1st time I said I wanted to get fireworks to set off here in NJ, she assumed I just meant sparklers. When I came home with some rockets, mortar launchers, etc. she about had a heart attack and was convinced I would burn the house down and blow my hand off. She just went into Pittsburgh and saw a great show. We didn't have that option, so we made our own.