For what it's worth, I always ride public transit overseas - it's a key part of my cultural experience! You are truly in with the locals when you are riding the bus or the subway. You see parts of the community no tour will take you, for better or worse. I would always advocate spending time riding transit when travelling.
That being said, if you are bringing kids - it's a long trip, and exhausting for a day trip, so drive on. If you want to see Buchart Gardens, you may find you do not have enough time to do everything with public transit, so either plan to spend one night, or drive on, or take a taxi to speed things up (but there goes your cost savings!)
It depends what you are planning to do once you get to Victoria. If you are happy with a few hours around the Inner Harbour and making your time on the bus part of the experience, then go for it!