No, that's a perfectly reasonable question. I mean it's an obvious evidence that brings the whole Bible into doubt.
Even furthermore since all of the human race's genepool got bottlenecked after the Noachian flood.
Well, from a strictly biblical standpoint, Genesis 9 tells us...
22 Ham, the father of Canaan, saw his father's nakedness and told his two brothers outside.
23 But Shem and Japheth took a garment and laid it across their shoulders; then they walked in backward and covered their father's nakedness. Their faces were turned the other way so that they would not see their father's nakedness.
24 When Noah awoke from his wine and found out what his youngest son had done to him,
25 he said,"Cursed be Canaan! The lowest of slaves will he be to his brothers."
26 He also said, "Blessed be the LORD, the God of Shem! May Canaan be the slave of Shem.
27 May God extend the territory of Japheth; may Japheth live in the tents of Shem, and may Canaan be his slave."[2]
So you see, Ham saw his father naked, so he received the curse of Ham.
Now a non-biblical Jewish text, the Talmud says
Babylonian Talmud, Sanhedrin 108b: "Three copulated in the ark, and they were all punished—the dog, the raven, and Ham. The dog was doomed to be tied, the raven expectorates, and Ham was smitten in his skin"
So, smitten can mean "beaten" What happens to white skin when it's beaten? It turns dark.
So, Judeo-Christinaically speaking, Ham did some bad stuff, this cursed him, his skin turned dark and dark people are meant to be slaves.
Isn't religion wonderful?