Kit Malthouse was speaking yesterday at a Conservative Party Conference fringe event organised by think-tank Planning Futures.
He also called for local authorities' local plans to outline their expectations for planning conditions. And he said planning inspectors shoudl back neighbourhood plans.
Malthouse said councils should be firmer in their viability negotiations with developers, following changes in the National Planning Policy Framework that aim to put authorities in a stronger position to defend their affordable housing policies.
This includes new planning guidance, which states that the price an applicant pays for land cannot be used to justify failing to meet local policies.
"If there's a change in the market, councils have to be firm, grasp their convictions with both hands and say 'Well, sorry boys, you are going to lose money on this one, you paid too much for the land, because the section 106 has to be paid.'
"Finding ways for us to enforce that and to make sure the development takes place as well, is something I'm giving some thought to", he said.
Earlier at the conference, Malthouse called for councils to be 'much tougher' with developers on build-out speed.
He also said that authorities should be more transparent in publishing a list of infrastructure items that are funded by section 106 planning gain agreements.
He said: "There's quite a lot we can do around transparency, making it clear to the public about what section 106 [payments are] going towards. Make it public, get it out there."
Such a move would give the public more confidence that money generated by development was going back into the community, he said.
The minister further called for local authorities' local plans to outline their expectations for planning conditions.
He said: "It would be great, wouldn't it, to get all that stuff done up front. Not just about contributions but also design and the general thrust of conditions. Then we give much more certainty.
"At the moment, because you don't get that in the local plans, most developers are going in [to negotiations over planning conditions] slightly in the dark.
"It becomes a prolonged iterative process, backwards and forwards, taking chunks out of each other. It usually results in a great long list of conditions."
Elsewhere, Malthouse said that he wants to "make sure that the decisions of the Planning Inspectorate reinforce neighbourhood planning".
He added: "We can't have a situation where neighbourhood plans are constantly overridden because then people won't bother."