Why on earth would you go to Iceland with two little kids?
My wife and I went to Iceland last May for 11 days with a 3 and 5 year old. They loved it, as did we
We only spent a day on either end in Reykjavik, so I don't have tons to offer for options in the city. We rented two separate airbnb's. Place is full of cool architecture, so we found a super cool house up Laugavartan, and then another super cool house down in Selfoss.
Our days were spent like this. Woke up - had a delicious breakfast and looked at some maps of touristy places we wanted to check out (waterfalls, volcanos, glaciers, lava fields, etc.). We would pack some snacks and head out. Get to the destination and hike around for a few hours and eat some snacks. Go eat a semi-delicious lunch at some place nearby (I didn't love the food - and it was stupidly expensive), then drive around and find some hot spring to relax in. Kids loved the hot springs. Most days were spent exactly like this.
To me, it defiinitely is a great family vacation for a family that likes to explore. How old are the kids?
I would for sure check out Blue Lagoon. It was crowded, and like most things expensive, but it was awesome. We ate dinner at the restaurant on site which was very good.
If you rent a car, make sure to get a AWD vehicle. About half the roads that go to places AWD is required.
Any other questions feel free to PM me or post them here.