Esther McVey was speaking yesterday at a Conservative Party Conference fringe event organised by the Centre for Policy Studies (CPS) think tank.
During the event, journalist Liam Halligan criticised the standard of housing produced under PD rights, saying many were "poky office blocks that have been converted".
He said: "I visited some really terrible places in south London and Harlow that are absolutely outrgaeous.
"Under permitted development rights, the usual space requirements have been waived. We are developing the slums of tomorrow today. They're developing-world conditions."
In response, McVey said: "As we were looking to build more homes we did have permitted development, some of which has worked.
"But where it's not working, we will be changing the rules to ensure that they are habitable, because there are certain conditions that people must live in.
"Those [rules] will be altered going forward."
She later added: "What we are doing is making sure that standards are better."
Earlier this year, the previous housing secretary James Brokenshire announced a review of the impact of residential PD rights on housing quality, following growing concerns about the policy.
Meanwhile, one of the architects of the residential PD policy that was first introduced for office conversions in 2013, spoke in its defence at the same fringe event.
Alex Morton, now head of policy at the CPS, was a housing adviser in the Number 10 policy unit at the time.
He said: "Permitted development was something I was very enthusiastic about. I still think on balance it's a good thing.
"But you do get a kickback from people who say there are one or two examples of bad things that are happening.
"I hope when we go back to [review] permitted development, we don't reverse it.
"People say the flats are too small. This idea that somehow because of permitted development there's a housing crisis, it's nonsense.
"People are being over-crowded because there's not enough housing space. Creating more housing space is not a bad thing."