The botanical gardens are beautiful to walk around and only about a 10 minute walk from the centre of the town. It's quite reasonable to get in and for a small extra fee you can feed the giant tortoises which is a great experience although you can see them for free. There are lots of interesting plants to see, including the coco de mer tree.
I'd definitely recommend going if you have a spare few hours. However don't forget insect repellent as there are quite a lot of mosquitos around.