It seems genetic science will disagree with you.

Loss of heterozygosity over time can be estimated with the following equation.

Ht = Ho (1 - 1 ) t

2Ne

heterozygosity at time t is a function of initial heterozygosity at time zero the amount of heterozygosity lost will be very small if Ne is large heterozygosity is lost with the passage of time ("to the power of t") why "2Ne"? because each individual receives genes from two parents To retain 90% of initial genetic variation after 100 years, Ne must equal 500, and N must equal 1,000 to 2,000 individuals. This way, Genetic drift caused by random mutations will outwiegh any damaging effects caused by inbreeding.

If you do not have enough individuals, your population gets too genetically similar over time, leading to risk with infectious diseases, and other maladies usually brought on by inbreeding.

Captive populations of large vertebrates often descend from a handful of individuals: for the Siberian tiger, 25; for the Mongolian wild horse, 13; an African gazelle began with one male and three females. Such populations face a high risk of inbreeding, which means mating with a close relative -- parent-offspring, siblings, cousins etc. Most species rarely inbreed in nature, because of larger population sizes and various behavioral factors, hence this consequence of rarity and captivity is unusual. Comparison of inbred vs. non-inbred captive mammal populations from zoos shows that reproduction and survival are reduced by inbreeding. This effect, termed inbreeding depression occurs because inbreeding results in increased homozygosity, which in turn increases the likelihood that deleterious recessive genes will be expressed. Normally, harmful, recessive alleles are so rare that they almost never occur in the homozygous state. Inbreeding increases the probability increases that two alleles present at a location are identical due to common descent. If a son mates with his mother, not too unlikely in a small population, the offspring might be "aa" for some trait. Furthermore, the offspring may well receive both those "a's" from the female who is simultaneously the mother and grandmother. In small populations, all individuals quickly become related, and as a consequence are more homozygous than non-inbred individuals and have lower levels of genetic diversity. Although drift and inbreeding both result in increased homozygosity, the mechanism and the consequences are different. Drift increases homozygosity by random loss of alleles, and harms the population’s long-term capacity to adapt. Rare, harmful, recessive genes are most likely to be lost, because they already are so rare. Inbreeding increases homozygosity by descent (individuals receive the identical gene through each parent), and facilitates the appearance of harmful, recessive alleles in the homozygous state. Individuals are less healthy as a consequence, and the effect is shorter-term. By a similar equation, it can be calculated that an Ne of 50 is large enough to keep the in-breeding coefficient at 1% per generation. Animal husbandry studies show this inbreeding rate to be as high as can be maintained in artificial selection. You might think it unlikely that our captive population will contain even one "Aa" individual. True enough, for that gene location. But there are 104 - 106 gene locations, and many lethal and deleterious genes are found in normal healthy populations, such as the NRE 220 class.

Edit: Sauce http://www.snre.umich.edu/~dallan/nre220/outline12.htm and disclaimer. I am not a genetic expert, you probably know more then me, but everything I've read so far shows me that a small initial population under the Ne=500 number leads to gene pool damage over time. As well as susceptibility to harmful recessive genetic traits. would you care to explain to me how the population of Northern Elephant seals looks at the moment, with regards to genetic deformities or possible large scale epidemic problems?