The conservative position is that because civil rights have advanced so much, the Voting Rights Act is no longer necessary.Frankly I don't see how anyone can think otherwise. Republicans have steadfastly refused to renew the 1965 Voting Rights Act, and their voters have supported that refusal.
Hard to imagine more concrete proof of "wanting to go back to the 1950s" than literally rolling back one of the laws passed shortly thereafter that moved the country forward.
Mind you, I strongly disagree with this position. But it’s a far cry from wanting to return to segregation. Not the same thing at all.