The Lee Library is the only remaining Carnegie library building in the Berkshires. Not only does the building have great beauty, but it has great historical significance.

The present building is on the site of Peter Wilcox's homestead. The Wilcox one-story, one-room log house was the location of Lee's first town meeting held in December 1777. Peter Wilcox is known for his part in Shay's rebellion.

The original part of the present building was built in 1907. The total cost of the building, including the lot and furnishings, was $35,500. Andrew Carnegie donated $12,000, the town appropiated $18,300, and the remainder was donated by concerned citizens. Lee Marble Works quarried and cut the marble used in the construction of the building. The original section of the building is Corinthian in style, with interior woodwork of polished birch.

The library was expanded in 1977. Marble quarried in Lee was again used in the construction. The new wing tripled the size of the library to a total of approximately 7,500 square feet. The addition includes the Betty Dennis Children's Room, stacks, a reading room, and the J. Peter Scolforo Gallery.