@The Z Machinenails it as usual.
Baltimore's "downtown" is steadily shifting east (away from Camden Yards) so the area isn't nearly as fun as it was in 90s or early aughts. If you were coming with your wife, I would recommend a hotel in Harbor East (closer to Little Italy and Fells Point) to be close to better food and more things to do, but then you would have a mile-plus walk to the ballpark, With a couple of kids, booking something right in the Inner Harbor seems a lot easier. Unfortunately, it's mostly chain restaurants down there, but I can highly recommend a great local Detroit-style pizza place (Underground Pizza Company) over at Power Plant.
For real local flavor, Faidley's at Lexington Market has the best crabcake in Bmore that's not made at my house. It's always a point of argument when Baltimoreans recommend tourists walk over to Lexington Market (it's a quick walk from the stadium) but as Z said, if you're not city-dumb, it's perfectly safe on Saturday at lunch. It was part of the drill when I drove up to O's games in high school or summer break to hit the Market first and bring a bunch of food in for the game and still take visitors there on the regular.
If you want to head the other direction, Cross Street Market in Fed Hill is less than a mile walk south from the stadium and has some really good local food spots as well. A recent renovation has made it a lot more bougey (as the kids would say) compared to its blue-collar routes, but that certainly fits what the neighborhood has become. And even then, it's only bougey for Baltimore...
