Robert Jenrick, the secretary of state for housing, communities and local government was speaking at a Conservative Party Conference fringe event yesterday afternoon organised by the free market Institute of Economic Affairs think tank.
On Monday, Jenrick announced that the government would push ahead with proposals to ease planning rules to allow the upwards extensions of properties and the demolition of commercial buildings to make way for new homes.
He also revealed that the government would allow councils to raise planning application fees in return for higher quality services, and will also consider how pre-commencement conditions can be cut further.
Speaking yesterday, Jenrick said the government wanted to make further changes to the planning system and was "looking much more broadly" than Monday's measures.
The first area of change would be focused on making it easier for "consumers", he said.
The aim was to allow "people in their everyday lives trying to make changes to their homes".
"How can we make that system as simple and cost-effective as possible?" he asked "Building upwards is part of that but there's more that we can do there."
As an example, he said the government wants local authorities "to meet the deadlines for planning decisions".
He said: "We are going to help provide them with extra money for that, but [in return there would be] potential to have your fees rebated automatically if the council aren't able to meet that deadline. So, a faster and more consumer-friendly service."
Secondly, the government wants to improve the planning system for small- and medium-sized builders "who are absolutely essential for the future of the industry".
Jenrick said ministers "have further ideas" for how to help such developers "get the good-quality service they deserve".
Finally, he said the government wants to focus on making it easier for "big developers who play an essential role in unlocking large parcels of land and getting major development done".
No further details were revealed about changes in the latter two areas.
Jenrick went on to say that we currently have a "very complex and contradictory planning system" that "needs bringing into the 21st century".
"But it needs to be done in a sensitive way that brings communities along in the process."
On town centres, he said that ministers "want to see whether there are further planning reforms we can do to help people assemble land, to regenerate land and to get housing into town centres".
Asked whether the government needs to look at relaxing rules on building on the green belt, Jenrick said it was not something that ministers were considering.
He said: "We have chosen to go for building on brownfield sites, for greater liberalisations like building upwards, densification in urban areas.
"I hear the arguments. Some parts of the green belt are not particularly beautiful but it's not our priority to take on that issue."