Do not be fooled by the name. Inverary Castle is not a castle...at least in the fortress sense. It is a big, old manor house. Those from Toronto, think of Casa Loma but not nearly as big, nor as nice.
We lost a lot of valuable time on our holiday in expectations that were unfulfilled with this place. It is a very nice manor home (of which only a portion can be visited) in a pretty setting. If that is what you are looking for it is very worthwhile.
If you are looking for an thinking of a traditional castle - don't bother. No battlements, no moats, no fortress walls. One armaments room where I got the feeling that the weapons were always used as decorations rather than as instruments of war.