Brian Berry, chief executive of the Federation of Master Builders (FMB), which represents small- and medium-sized building firms, also said that the body's members would support increased planning application fees as long as the money was "ring-fenced for planning departments".
Berry delivered a speech about the impact of Brexit on housing and planning at this morning's conference, which is organised by Planning.
He said delivery of new homes would be "compromised in the short and medium term" post-Brexit if the UK progresses the policy advocated by the government’s Migration Advisory Committee.
Referring to the committee’s suggestion that the government could consider a migration policy whereby skilled workers earning more than £30,000 are granted work permits, Berry pointed out that more than half of construction workers earn less than that.
"We would argue that they are skilled people, so this policy needs a rethink," he said.
Berry said the biggest issue facing the construction sector was the availability of skills.
He said that, according to a latest survey of FMB members, 54 per cent said they were having problems recruiting bricklayers and 56 per cent could not find carpenters and joiners.
"There is huge demand building up, but we haven’t got the people to do the work, and that’s why EU workers have been very important," he said.
"Our members rate EU workers – 94 per cent say they are good or very good – so there is a clear need to have EU workers in the building industry."
"Thirty-three per cent of out members say ending freedom of movement will be a major constraint on house building post-general election.
"We need the government to develop a new immigration policy that supports the skills needed for this country’s infrastructure."
Berry also identified the planning system as one of the key issues facing small builders.
"Planning continues to be an issue, and the cost and delay in the planning process in particular," Berry said.
"There are real concerns about the under-resourcing of planning departments, and it is so bad that our members are very supportive of increased planning fees provided money is ring-fenced for planning departments."
Only three per cent of FMB members said they saw a positive impact arising from the last increase in planning application fees, Berry said.
A range of solutions such as the pooling and sharing of staff, and the improved retention of planners, were required, he added.