One of the museums that you could visit in the centre of the city is known as "The Culture House" - but it is really not set up for foreign visitors.  There are several floors to visit, but most of the descriptions are in Icelandic only.

If you want to see real practical history of this island, laid out in a meaningful way - you can go to two other places.  The first is within Reykjavik, just a bit further out of the centre.  The National Museum of Iceland has recently been completely renewed and updated.  

For a view of Iceland life over the last few hundred years you could do a lot worse than take a trip out along the southern coast road.  After almost 150km you get to Skogar where there is a well known broad waterfall (Skogafoss) - one that is the last of a series up the river. You can hike or drive to the others with varying degrees of difficulty.  However, in the same small village is the Skogar Folk Museum. This is a collection of items that was started in 1945 by one man.  There is now a fascinating museum of several rooms, as well as a series of relocated buildings.  These include a Turf Farm from 1830, a Schoolhouse originally built in 1901, and a Church with parts that are over 300 years old.  If you get a tour from the guy who started it (don't ask how old he is) he will keep you interested for a long time.  He frequently plays the organ in the church, and the piano in the schoolhouse.  A visit to this location is included in some day trips that go along the south coast road.