If you're OK with carrying your bags up one flight of stairs you'll be fine with the subway. Get out the back end at 96th Street and you'll be right there.
Be aware that, unique to Penn Station, the 1 is on a separate platform from the 2,3. Since you're going to 96th Street, which is an express stop, go to the 2,3 platform. If you prefer, there's a short elevator available up to the platform. That platform will have both uptown and downtown 2,3 trains. Make sure you get on an uptown one. For your trip the 2 and 3 cover an identical route (though they separate further north,) so just take whichever comes first.