"Hopping over" to Vancouver Island is sort of like hopping over to Ireland for the day if you have 4 days to spend in England.
It can certainly be done, but most people severely underestimate the size of Vancouver Island (as it's an island in the same way that Ireland is an island) and they also underestimate the logistics of getting to/from the ferry terminal (as the terminals are a fair distance outside of Vancouver and are also a 35 minute drive outside of Victoria), the length of the ferry rides (90+ minutes) and so on.
If people do a day trip to Vancouver Island, they usually go to the city of Victoria at Vancouver Island's southern tip, as the city's compact, walkable, and there are enough options to do a round trip (whether that be a 12 hour bus tour, a round trip float plane trip, or simply a bus/ferry combo).
Typically I suggest 3 days minimum in Vancouver because, again, most people understimate how much there is to see and do, and how much time they'll take to see it. For example, simply a visit to Vancouver's Stanley Park can eat up a whole afternoon. Whale watching out of Vancouver is another activity that eats up most of your day as you're out on the water for 3-5 hours. And there are 2 days gone. For these reasons, I'd either do a day trip to Whistler or a day trip to Victoria, but don't try to do both if you only have 4 days in Vancouver. 5 or more days? Certainly. But with 4 days, I'd plan on one day trip.
Edited: 10 June 2012, 21:11