My first trip to Colorado from the East Coast. We are not interested in skiing but primarily looking for the best mountain town atmosphere.
Welcome to the forum. Knowing time of year is important.
Aspen and Breckenridge are both towns that have a heritage that goes back before becoming a ski resort town. Each were 19th century mining centers and both preserve that history in the look and feel of the town. Aspen is more remote than Breckenridge, which is less than two hours from Denver. Aspen is more upscale than Breckenridge. Both have an excellent mountain town atmosphere.
Vail, on the other hand, is a ski resort town built in the 1960s as a ski area. While it doesn't have the heritage of Aspen and Breckenridge, it has a wonderful atmosphere. Look to stay in Vail Village for the best experience.
If you're looking to pay a visit in the winter, all three of them will carry a heavy cost based on skiing. Summertime is more moderate.
The OP said they are not skiing so they won't be Après-Skiing. This poster seems to be attempting some sort of record with posts (often two at a time for no reason) rather than actually offering suggestions.
As far as the richness of the social scene, it would be hard to top Aspen.