Let me preface this with a few things.
I'm drunk. I like Northam, and I'm a liberal. But I don't really care if he has to take the fall to prove a point. To demonstrate that this is not cool is important enough that a rich dude losing his political career is a an acceptable casualty, imo.
But 1984 was a very long time ago. Far closer to segregation being the norm in the South than it is to today.
I went to college in the late 90's in NC. The things that seemed acceptable then, thankfully, are far different than what we consider acceptable now. In my fraternity, as of the year 2000, we still had something called the "Sharecropper's Ball". For that party, as of the early 90's, they still had people in black face pretend to pick cotton during the party. Most everybody in the fraternity had a confederate flag in their room (though the "nuanced" guys had the real flag, not the battle flag). The initiation robe closely resembled the KKK robe/hood. This was a regular mainstream fraternity.
Now, I hated this fraternity, and only lasted a couple years. But it wasn't for any of the above. I wouldn't have considered these guys to be racist back then. They were just eastern NC ########s (from a lot of money, which is the main reason I didn't like them. I was a poor kid paying my own way through school, and we never saw eye-to-eye). And they were born in a time where this sort of stuff was acceptable. I didn't like these dudes for a lot of reasons, but at the time, they were just normal dudes (and quite popular I might add).
Again, I have no idea if Northam will survive this politically. Or should. Like I said, I don't care. Just adding my own anecdotal experience from a southern school in the 90's. And Northam is at least 20 years older than I am.