I respect anyone who makes thoughtful, conscious choices about their eating habits no matter what conclusions they settle on.
I think factory farming is an ethical (and environmental) catastrophe. I strongly agree with vegans about avoiding animal products that come from factory farms.
I part ways with vegans elsewhere pretty strongly. I kind of view veganism as vaguely comparable to fundamentalist Christianity. At the heart of Christianity, arguably, is an absolutely wonderful ideal of loving your neighbor and being kind to everyone. But the fundamentalists get lost, in my view, in rigid adherence to ethically arbitrary rules at the margin (such as opposing gay marriage, or, in some variations, refusing to come in contact with women during their menstrual cycles or whatever). Similarly, at the heart of veganism is a genuine concern for animal welfare that I'm fully on board with. But the idea that eating honey is worse than eating almonds, or that eating oysters is worse than eating wheat, is ethically arbitrary and, IMO, wrong. (Almond farming is harder on bees than honey production is. Wheat production kills a lot more sentient beings than oyster aquaculture does -- including cute ones like mice, not just scary ones like snakes and potato bugs.)
The conclusions I've settled on myself are that hunted wild game (including wild-caught fish if caught responsibly -- no nets that entangle dolphins, etc.), along with meat, milk, and eggs from pasture-based farms are more than just acceptable. I view them as net positives from an ethical standpoint (at least under certain conditions). Products from conventional factory farms, on the other hand, I'd avoid 100% if I were more strong willed. I generally try, but I make plenty of exceptions when I'm at restaurants. I should try to make fewer exceptions.
I do think that it doesn't have to be all or nothing. Reducing your consumption of conventional meat by 10% is better than nothing. Baby steps are fine.
Probably the best thing a person can do as a single dietary change is to cut out chicken. Substitute any other meat instead if you're going to eat meat. As Will MacAskill
explains: "If you crunch the numbers on amount of harm done per meal, or per calorie consumed, then by far the strongest argument is to cut out chicken, then (non-free range) eggs, then pork. The argument for cutting out beef, and especially the argument for cutting out milk, is much, much weaker. Chickens suffer the most of all the animals, they're in the worst conditions, and you kill more chickens in the typical American diet than you do beef cows or dairy cows, simply because those animals are so much larger." (I'm not persuaded that chickens suffer more than pigs, but I think the other points are correct.)