Westminster City Council and conservation body English Heritage claim the Elizabeth House development between Waterloo Station and York Road would harm the setting of the Palace of Westminster across the river, and did not want the final decision to be left to the London Borough of Lambeth, which has already resolved to grant permission.
The scheme would include offices, homes and other uses in two buildings, one that is part-29 and part-14 storeys tall, and one which is 11 storeys.
But planning minister Nick Boles took the decision on behalf of secretary of state for communities and local government Eric Pickles that the application did not need to be called in.
Today, Mr Justice Andrew Collins, rejected the claim by Westminster and English Heritage that this was "irrational" in the light of a warning from international heritage body UNESCO that, if the plans proceed, it may put the Westminster World Heritage Site (WWHS), which includes the Houses of parliament and Westminster Abbey, onto its "danger list".
The judge said that the secretary of state had a "very wide" discretion on call in decisions and that, on balance, the claim must fail.
He said: "There is a view which could reasonably be taken that despite UNESCO's and the claimants' concerns the impact is not such as would damage the WWHS or other listed buildings and conservation areas.
"The defendant is, in exercising his judgment, entitled to regard the real risk, even the probability, of UNESCO placing the WWHS on the danger list as acceptable.
"It may be a surprising view to take but I am not persuaded that it quite reaches the level of irrationality."
In a statement, Elizabeth House said: "We are delighted with this outcome which clears the way for a planning permission.
"What is most important now is that the Elizabeth House regeneration is delivered for Waterloo, Lambeth and London.
"For too long Waterloo has lacked the investment and modern office stock to attract business to the area and these plans will create 8,700 much needed new jobs and transform the public realm around Waterloo Station used by tens of millions of people each year."