Gayfers affordable housing project ‘in peril’ after concerns over the project continue