Propaganda is largely responsible for this idea. Model Aryans get media exposure, the uglies don't. Look at the Nazi leadership, Hitler, Himmler and Goering were some of the ugliest leaders in history. Himmler looks like a melted thumb, he used to cover his own face out of embarrassment during photographs.

"Himmler appeared to be “a fairly short, ill-looking man in shabby civilian clothing,” and his unimposing presentation was nothing new. He had always been embarrassed by his weak chin and sought to cover it with his hand when being photographed, and his weak eyes forced him to wear spectacles (or a pince-nez) throughout his life."