Actually, it's just that simple. As there are very few genes mixing with inbreeding, it gets in the way of evolution, thusly evolution finds ways to "fish" it out.
Also, inbreeding through several generations will create a weaker subspecies, that, in time, will just die out. Tests show that humans are most attracted to the "average" face, both female and male. This is most likely an attempt (and a good one at that) to remove wayward genes from the human genome. Evolution wants to be surefire, not quick.
The best way to explain most human feelings or urges, one needs to look towards evolution and social-anthropological behaviour, which is most obvious in pre-historic ages.
God, i sounded like a technocrat.