I posted our tentative itinerary and got some great feedback, which I am using to tweak things.
Originally we planned an overnight in Milford Sound. The only reason we thought we would do this is because we want to do at least one of the longer walks along the Te Anau-Milford Sound Road ... right now, Key Summit is the plan. Although we are early risers, we do like to take our time while traveling from point to point, and stop and explore frequently. That means that I'm not sure we would be able to make MS in time for the last cruise boat of the day. As much as I've read about the scenery along the road, I'd hate to rush it. Hence the overnight.
I guess the question is ... how miserable would we be on the trails along the Te Anau-Milford Road sandfly-wise. If we'd be fighting sandflies the entire time, then I guess we would skip the hikes (except for a couple of short walks) and easily make it to MS for the last cruise and then make it back in time to overnight in Te Anau instead.