David Goldenberg explains that the persistence that Ham was black originated from saying his name meant black.

David Goldenberg

David Goldenberg, The Curse of Ham: Race and Slavery in Early Judaism, Christianity, and Islam (Princeton: Princeton University Press, 2003), 142-143

Princeton University Press
David Goldenberg, Edward Blyden, Thomas Peterson, Josiah Priest
Reading Public

Of course, the biblical text does not describe anyone as Black. Nonetheless everyone assumed that Ham was Black and that he was somehow affected by the curse of slavery. It didn't matter whether one one supported the institution of Black slavery or not, or whether one was Black or not; everyone in the ninteenth-century America seemed to believe in the truth of Ham's blackness. As Edward Blyden, a Black scholar, clergyman, and statesman, wrote in 1869: "It is not to be doubted that from the earliest ages the black complexion of some of the descendants of Noah was known. Ham, it would seem, was of a complexion darker than that of his brothers." This was a "fact" accepted by almost all, including both abolitionists and a large number of Black clergy . . . In a study of the mythic world of the antebellum South vis-a-vis Blacks, Thomas Peterson showed that notion of Blacks "the children of Ham" was a well-entrenched belief . . .

But why should that be the case? Even putting aside for the moment the difficulty that according to the Bible, it was Canaan, not Ham, who was cursed, why was Ham identified with black Africa? Why the persistent centuries-long identification of Ham with the Black? The answer has to do with the name Ham itself, which was understood to be related to the Hebrew word for "black" or "brown," and thus indicated associations with the black African.

When Josiah Priest, whose works were very popular, wrote in 1843 that Ham was born a Negro, his proof rested on the meaning of Ham as "black." One writer later summarized Priest's position this way: "If Ham's name meant black and his descendants were black, these two circumstances will go far to prove that proposition in hand: viz., that Ham was a Negro.:"

Citations in Mormonr Qnas
Copyright © B. H. Roberts Foundation
The B. H. Roberts Foundation is not owned by, operated by, or affiliated with the Church of Jesus Christ of Latter-day Saints.