Midlothian Council unanimously endorsed the view of planners that the scheme is contrary to the structure and local plans for the area. It also ruled that the potential benefits do not outweigh the development's negative visual and landscape impact.
Scottish Coal said it would study the council's decision before considering whether to lodge an appeal. The company wants to extract two million tonnes of coal over five years to supply the Cockenzie power station.
Objectors included local people and East Lothian Council, whose area borders the application site.