Columbia has an iconic campus, a real stadium, and is in Manhattan so that's a good option if you want to see a college game in person. Rutgers is much closer than West Point, and they might actually be able to win against Liberty. It's a great atmosphere, although admittedly I'm biased. However, it's not West Point. West Point campus is really cool. The game is different too with all of the cadets going to every home game. There's a museum there too. Rutgers is fun, but it's fun like going to any other Big10 game. West Point is different and worth the experience.
I didn't realize the ferry was such a long ride, but I would probably still do it. I would probably take it there for the experience and if you want to leave early, or not deal with a 3 hour trip back, you can take a taxi or Uber to the train station and take the train back to NYC, although that would still probably take 2 hours or so in total. The train is a good way to travel around here. Just check the schedule from the station you would go through so you can time it right. I think the day overall is worth figuring out good travel plans back and forth. I know it's a long way if you're spending 6 hours travel time, but at night the view of NYC from the river is amazing. Either way, there are travel options for you around here for a day trip like that, and if I were to recommend any type of day trip from NYC, this one would be at the top of my list.