The current duke wanted to remove the interior walls and build 39 flats.
The proposed scheme also involved building six houses in the adjoining Buchanan Wood and felling several trees covered by preservation orders.
Loch Lomond and the Trossachs National Park planning board turned down the application after hearing objections and serious concerns that had been lodged by local people and organisations.
However, the rejection has left the door open for a revised scheme to be submitted. Board chairman Gillie Thompson said: "We would love to see Buchanan Castle either stabilised or retained by more sympathetic development."