Yesterday, a BBC investigation revealed that councils in England are holding £1.5 billion of unspent Section 106 money.
The BBC said that its investigation had also revealed that over the past five years £9.8 million of Section 106 cash has been returned to developers, often because it has not been spent within a set time period.
In a statement issued yesterday afternoon, Boles said that local people "would be surprised to hear that councils are hoarding millions of pounds that developers have given to the community to provide amenities or infrastructure as part of their planning permission".
He added: "Councils made these agreements with developers for the benefit of the community and should not be pocketing the cash. In many cases they could also risk losing the money and be forced to pay it back if unspent within a set time frame."
"Such levies add to the cost of building homes and business expansion. If there is no pressing planning need for them, they should not be charged in the first place."
Umbrella body the Local Government Association (LGA) has said it is "utterly wrong" to say councils are "sitting on" unspent money.
Mike Jones, chairman of the LGA’s Environment and Housing Board, said pulling together funding for local infrastructure schemes was a "complex process" which could can take time. "It is not surprising that there is some money that has not yet been spent", he said.