The plans for the new Gaelic Athletics Association (GAA) stadium were approved by environment minister Mark H Durkan in December last year and local residents then launched a legal challenge.
Mr Justice Horner said the decision making process had been "fundamentally flawed" but did not quash the decision.
He said there were failures in the environmental impact assessment, with an unrealistic reliance on the figure of 32,600 capacity as a baseline for the project. This meant the effects of large crowds on the ground and the surrounding roads had not been properly assessed, the judge said.
Both sides will now make further submissions before Wednesday when the judge will make a final decision.
Durkan responded that before then he "will consider what the judge has said very carefully". He added: "Clearly I am disappointed, both for my department and for the GAA."
Casement Park Project Board chairman Tom Daly said the group was "deeply disappointed". He said the proposed redevelopment "would have provided the opportunity of a world class provincial stadium" with economic and social benefits "including financial investment, new jobs, apprenticeships and community projects".
