No. No, it's not. Fantasy football isn't a scoring competition - it's a series of weekly head-to-head matchups. It makes perfect sense to adjust your lineups based on your opponent and the situation you find yourself in. If you don't do this, you are leaving wins on the table.
The OP is asking a very good question, but your thinking is backwards. If he's got Gronk and Gordon it should make you more apt to start Brady, not less, because those guys can't blow up for him without benefiting you as well. The only time I'd veer from this thinking is if I were a huge underdog, in which case a big day from NE probably doesn't get you any closer to your opponent and I'd probably fade the Pats and hope for the best.
Same thinking goes for starting or sitting a DST vs. your opponent's opposing QB. If you're a big favorite it often doesn't make sense, because he'll need a big QB day to beat you anyway, which will hurt your DST score and make it easier for him. But if you're a big underdog you should increase variance, which means you should always stack the DST against his QB since his loss doubles your potential gain.
There are many other examples, but it's much more nuanced than "always start your best players".