Judges were unanimous in choosing a lattice-like building by Japanese architectural practice Kengo Kuma & Associates for the site at Craig Harbour.
"We have subjected each of the six proposals to exhaustive scrutiny," said judging panel member Mike Galloway, director of development at Dundee City Council.
"The Kengo Kuma design gives us something which is bold and ambitious but buildable and practical. It is a beautiful stone building which elegantly meets the requirements laid out in the brief. This is a harmonious and integrated response to this site which enhances the overall waterfront project."
The £47 million development is being overseen by Design Dundee, a partnership between the museum, the University of Dundee University, Abertay University, Dundee City Council and Scottish Enterprise.
Chairwoman Lesley Knox said the design "demonstrates a clear understanding of the city, offers a new experience of the river and will be as exciting internally as externally". She added: "This has the potential to be one of Europe's most exciting buildings."
The Scottish Government is expected to announce shortly how much it will contribute to the scheme, on which work could start in 2012. A feasibility study estimated that the museum would attract hundreds of thousands of visitors a year but called for radical improvements to Dundee's transport infrastructure.