The Local Government Association (LGA), the umbrella group representing councils, surveyed members about the impact of permitted development rights introduced since 2013.
According to the survey, nine out of ten councils are concerned about the quality and design of developments created under permitted development rules, and six out of ten raised concerns about safety.
Around two-thirds of councils said they believe affordable housing and infrastructure contributions have fallen, while a similar proportion believe demands on local services have increased as a result of permitted development rights allowing the conversion of buildings to residential use being extended.
More than four in five councils responding to the survey said that the costs of administering the prior approval process for permitted development are considerably higher than the fee set by central government.
The survey was sent to all 322 single-tier local authorities and shire districts in England in membership of the LGA. The response rate was 32 per cent.
One of the most significant changes to permitted development rights in recent years involved allowing office to residential conversions without full planning permission.
Introduced in May 2013, the policy was initially intended as a temporary measure but was made permanent three years later. Under the policy, developers are not required to provide affordable housing or other infrastructure contributions.
Analysis of recent government figures by the LGA suggests that the lack of developer contributions under permitted development rights has led to the loss of 10,500 affordable homes across England in the last three years.
The association warned that councils are being forced to deal with increased demands on public services created by new development without any contributions from developers towards affordable housing or infrastructure such as roads, schools and health services.
The government recently proposed a significant extension of permitted development rights that would allow the addition of new storeys to residential buildings, changes between various high street uses and the demolition of commercial buildings to make way for homes.
Martin Tett, housing spokesman at the LGA, said: "Permitted development rules are taking away the ability of local communities to shape the area they live in or to ensure homes are built to high standards with the necessary infrastructure in place, and have resulted in the potential loss of thousands of desperately needed affordable homes.
"Extending permitted development rules risks exacerbating these problems. Councils, which are answerable to their residents, must be given back their ability to oversee all local developments to ensure they are good quality and help build prosperous places."
A Ministry of Housing, Communities and Local Government spokesperson responded: "We are making the housing market work by reforming planning rules so we can build more properties that suit a range of needs.
"Permitted development rules provide flexibility, reduce bureaucracy and make the most effective use of existing buildings."
In April, a report by the Royal Institution of Chartered Surveyors (RICS) concluded that office-to-residential permitted development rules have allowed the development of "extremely poor-quality housing".
Last week, the Town and Country Planning Association said it is to launch a new campaign against the "scandal" of permitted development rights.