According to its analysis of data from the government and the Home Builders Federation (HBF), the backlog of unbuilt homes has grown by a further 100,000 in the last year alone.
Shelter says this shows the government's new planning reforms will not boost housebuilding by themselves, as planning permission is not the primary stumbling block to getting homes built.
The charity said it analysed housing supply and approvals by financial year, using the latest English housing pipeline figures from the HBF and Glenigan and new build completions figures published by the Ministry of Housing, Communities and Local Government (MHCLG).
It assumed "that developers take on average two years after full planning permission is granted to construct the units approved by local authorities" and consequently compared units approved between 2011/12 - 2016/17 with completions from 2013/14 – 2018/19.
It found that 382,225 homes "remain unbuilt", with the backlog of having increased by 101,223 units, when compared with the same position last year.
Shelter's chief executive Polly Neate said: "The chronic shortage of decent, genuinely affordable homes in this country is one that must be fixed. But the government's planning reforms fundamentally misdiagnose the problem.
"The idea that the planning system is stopping homes being built is a myth. Across the country hundreds of thousands of 'phantom homes' sit on sites with planning permission fully approved."
She said the government "should spend the cash its set aside for housing that much faster and start building social homes now".
Campaign to Protect Rural England (CPRE) campaigns and policy director Tom Fyans said: "The backlog of building in the midst of a housing crisis is one of this generation's untold scandals.
"The government urgently needs to give councils more powers to set stringent, enforceable deadlines for homes to be built, and to require more affordable homes that meet local needs, instead of disempowering councils and handing more power to developers."
But HBF planning director Andrew Whitaker called Shelter's figures "misleading" as they "include homes on sites with construction work ongoing, many already completed and being lived in, as well as ones with an initial planning permission but, frustratingly, still awaiting the final sign off builders need to start construction".
He added: "The time it now takes to process permissions to the point where builders can get onto site and start work continues to increase, causing frustration and delay to housing supply. Political indecision at a local level and a lack of capacity in local authority planning departments means applications are taking months or years to process.
"This is frustrating builders large and small across the country and threatens to undermine government and industry attempts to deliver more homes."
Have you advised on, or secured, a recent major permission? We want to know about big housing, retail and leisure, and commercial and industrial schemes approved between 1 June - 31 August 2020 for our next Biggest Permissions report. To qualify, schemes should be above the following thresholds: 500 homes; 5,000 square metres of retail or leisure space; or 20,000 square metres of commercial or industrial space. To get involved, or for more details, please email eleanor.kahn@haymarket.