I think overall a big problem is that all the focus is now on how the person feels vs. the intent of the statement or action. I bump into somebody by accident, that's not a big deal because, well, it was not intentional. I post something that offends somebody, that's a big deal because even if I might have been ignorant about what I posted, if they feel attacked, now it's a micro-agression or I am a ____ist, whatever.
To answer the OP, yes you can be insensitive or ignorant and not be a ____ist. It's just that today's call out culture really doesn't help in us taking a breath and thinking about the situation a bit as everybody hops on and calls for the person's job.