Critics absolutely loathed the Velvet Underground. They might have set a record for negative reviews and garnered a reception that completely demolished the aspirations of the band itself. They disintegrated, slowly but surely, with members leaving after realizing they wouldn't live the rock n' roll dream.
It's only the modern-day hagiography the band receives from critics that would lead anyone to believe that anyone listened to them because they were critical darlings. Even in the nineties, I was under no critical spell when I first heard the Andy Warhol album. It just sounded so great, so of-the-moment back then. Wasn't until later I knew how much they were revered. That came from other artists, actually. Artists that were influenced by and incorporated the use of the Velvet Underground's "innovations" into their music were what caused the historical tide to turn towards acceptance and lionization of the Velvet Underground. They were a musician's band before they were critical darlings.