David and Ann Usher have been fighting to save their home at Small Fen Farm, Brandon, since 2012.
But Judge Nathalie Lieven QC today finally opened the way for council workmen to tear it down.
She rejected claims the couple had not been given a fair hearing and that the demolition would violate their human rights.
London's High Court heard that Forest Heath District Council issued an enforcement notice, requiring the house's demolition, in November 2012.
Since then, the couple have twice pleaded with planning inspectors that they should be allowed to keep their home.
But their arguments that they had merely "renovated" an existing building and that demolition was "excessive" were rejected.
The house was "harmful to the character and appearance" of the surrounding area, one inspector said.
Also rejected were claims that demolition would breach the couple's human rights and those of Usher's sick mother, Ellen.
On January 23 this year, the council told the Ushers it would take "direct action" and send in its own workmen to demolish the house.
The council said demolition would start at 8.30am on 30 January and warned the Ushers to remove all their possessions in advance.
This was, however, delayed after the couple launched their High Court challenge.
Their barrister, Ned Westaway, said they had not been given a fair opportunity to argue their case before the council.
They had been given only a week's notice of the "disproportionate and unreasonable" demolition decision, he argued.
The council, he added, should have considered every available alternative to sending in a demolition crew.
Dismissing their complaints, however, Judge Lieven said the couple had been given "every opportunity" to persuade the council to stay its hand.
They had employed their own planning consultant and had been told "in the clearest possible terms" of the council's intention to take direct action.
There was also nothing "irrational or disproportionate" in the council's refusal to extend the January 30 deadline.
Mr and Mrs Usher had "chosen" not to demolish their home themselves "despite being given very clear extended deadlines" by the council.
The council had considered the alternatives, but in the end was entitled to instruct its own workmen to do the job.
R on the application of Usher & Anr v Forest Heath District Council. Case Number: CO/433/2017