The truth is very different. We are merely against planning as it is currently constituted. At present, the planning industry follows a template created in 1947 as part of the push for a command economy. Born out of an era of "government knows best", the idea was for land to be rationed by means of a local plan that limits where buildings of each type can go. The beliefs behind this were distrust of people and private property.
The legacy of this is clear: misgivings about new development and a fear of change; only 103,000 homes built in 2010 despite house prices tripling between 1995 and 2010; the world's most expensive cities for office, retail and industrial space, throttling our ability to grow and compete; the rapid destruction of front gardens for parking and other areas of urban greenery to preserve "green belts" used for intensive agriculture.
I work near the beautiful Pimlico grid in central London. It was largely the creation of one planner, Thomas Cubitt. He did it without council plans and endless planning guidance, obeying only basic fire regulations. He used the notion of beauty that we all have within us to create one of the most heartwarming places I know.
Pimlico is beautiful because it is truly holistic: the ownership of the land and design were largely vested in a single owner and creator. As a result, the impact of each house and street on its neighbours mattered. This is, in fact, the key. Our opponents argue that we want a planning free-for-all. This is untrue. You must have a balance with these things, but the system should be about local people having a direct say on what planners allow, in effect recreating the single ownership of an area. Those who own neighbouring land are the best final arbiters of a scheme.
The argument that people don't know what is best has had 64 years since the Town and Country Planning Act 1947 to convince. It has not succeeded. Our most popular developments usually pre-date the act, while those who oppose new development grow stronger with each passing decade, even as our housing and development needs become more acute.
Ultimately, our view of planning is about our wider view of society. There is an opinion that we can't think for ourselves or be ourselves without descending into an ugly way of life from which only government control saves us. Pimlico refutes that view.
Planners would still exist in a reformed system. Those who want to control other people would not. But those who want to help create stunning places that will be admired for centuries would still have a place. Those who oppose this vision must set out an alternative - not pretend that the last few years have been a success.
Alex Morton is senior research fellow at Policy Exchange, and was the author of the think-tank's 2011 report that recommended that planning permission should no longer be required for residential conversions of offices.