Town leaders agree to tear down Oak Street residence