Baku's Old Town (or Icheri Sheher as it is called in Azeri language) is the core place if you are into visiting historical places. Take a walk from the legendary Maiden's Tower which is at the edge of the inner city to Shirvanshah's Palace through narrow streets passing by old buildings (most of them are hotels or offices though).