Last month, a High Court judge upheld the government's decision to grant permission for the 43-storey residential tower and public swimming pool (Planning, 25 September, p4).
But he gave English Heritage and Westminster leave to appeal. They had argued that the tower would cause irreparable damage to the setting of some of the capital's heritage landmarks including Somerset House and St James's Park.
However, in a statement this week, the bodies said: "Despite the leave to appeal, we have decided not to pursue the issue further. Reaching this decision has been difficult and we continue to believe that the development will permanently harm, not enhance, London's precious historic environment."
The Coin Street Community Builders scheme was backed by former communities secretary Hazel Blears despite having no affordable homes provision and a recommendation of refusal from inspector Philip Wilson.
The controversial development had also won support from the London Borough of Lambeth and local MP Kate Hoey.