Yes, it's worth it. Cable Car rides are $6 each ride and regular fare $2 (regular fare does include a transfer). Doesn't take too much for the Passport to save some bucks.
Plus there are multiple intangible benefits: not fumbling for your fare, quicker boarding, no worry about transfers expiring, hopping on/off on a whim (especially cable card), and more.
Also, even if you don't ride enough to actually get your money's worth, the "extra" cost will only be a few dollars. Just a drop in the bucket of the total cost of your trip and in my mind worth paying for the convenience & other benefits.
Lastly, depending on what you're planning to do & how long you're in SF, you might want to get a Citypass which bundles a 7-day MUNI Passport with about a half dozen attractions (at a discounted price).
That's my $.02,
hlo