The US has a god-awful history with literacy tests, and I don't trust the government to devise a test that would accurately weed out folks who shouldn't be voting.
That said, a very large proportion of voters are incompetent. Not in the sense that they have any mental disabilities like autism or Down syndrome, but more in the sense that they just aren't intellectually sophisticated enough to rightly exercise coercive power over the rest of us. (Quick disclosure -- my wife works with high-functioning autistic and Down's folks. That isn't what I'm talking about here). What I'm talking about are folks who don't know who the current VP is, or who can't identify the Speaker of the House. People who can't name a single Supreme Court Justice. These are people are so completely disconnected from the government as it actually exists that I don't trust their judgement and don't want them voting.
On a higher level, people who have no working understanding of basic government and basic economics worry me. People who fly coach should not be voting for who gets to fly the plane.
The best argument I have run across for this viewpoint is
Against Democracy by Jason Brennan. I don't find this book completely persuasive because it requires that you buy in to a series of philosophical parlays, but it gets the overall point across.