The Margaret Hoyt Smith House, an 1870 Victorian at 30 Highland Avenue in Rowayton, was demolished at the end of June 2006 to make way for twelve upscale homes. The house was the residence of Margaret Hoyt Smith, the first woman architect in Norwalk, designer of many significant structures throughout the state. Architecturally, this fine old house represented the classic Victorian style that is fast disappearing all over the city.

The developer, while not keen on demolition of the well-built and elegant home, was not able to incorporate the existing Victorian-era home with the more modern architecture of the new homes planned for the site. The only option was to move the house. The Norwalk Preservation Trust and the Rowayton Civic Association worked to find a site and a way to move the large 3500 square foot structure. This proved unfeasible. Even though a site and a willing owner were found within a reasonable distance, the large house could not be moved down Highland Avenue without the loss of the mature trees that lined the street. Thus, in spite of the best efforts of the community, Norwalk lost another piece of its heritage.