R&G Homes was willing to accept a planning condition specifying that it could only sell the 14 flats to people who agreed to educate their children privately. The company came up with the idea after planners warned that the scheme was unlikely to be approved unless a way was found to reduce the burden on the already oversubscribed Linlithgow Academy.
However, West Lothian Council has decided that the novel scheme would be unworkable and approved the town centre project without conditions.
Deputy leader Willie Dunn said: "If these conditions were in place they would be challengeable and we could end up in court years down the line."
He added: "We have a policy that there is to be no growth in Linlithgow and the only way to deal with these situations is to judge each case on its merits."
R&G managing director Gifford Bruce said the state school ban, which would have been the first of its kind in Scotland, was only proposed as a last resort.