This is a casual work environment (shorts / polos), we have had (after hours) happy hours and drinks on the back patio, but a line needs to be drawn during work hours IMO.
Am I jerk, or should I not worry about it?
Not sure you are a jerk, but if your company is encouraging drinks on the patio, you have some issues.
Drinking after work isn't nearly the same as drinking during the day. I recognize it may be position specific, but, for example, my boss, other associates, and I will oftentimes have beers in the office at the end of the day or week. But if I had one at lunch with work still needing to be done, I'd expect to be immediately fired for it.