If you don't mind spending money on taxis (they're rather expensive in this area), take a taxi from Portofino to Rapallo station which is four miles further away fro Portofino than Santa Margherita's. The 07.37 train woould get you to Florence (Firenze S. M. Novella) in 2h 23, and coming back the 18.28 will get you back to Rapallo in 2h 52 giving you an 8.5 hour day in Florence.
See http://www.trenitalia.com/tcom-en for train times to June 9 - times will be similar or the same on the later dates.
Personally, I'd choose to go somewhere nearer than Florence. And if you are not going to use taxis, the journey times really will be far too long.