David O. McKay says that the Book of Abraham is the only reason for the priesthood ban.
Gregory A. Prince and William Robert Wright, David O. McKay and the Rise of Modern Mormonism (Salt Lake City, UT: University of Utah Press, 2005), pp. 80–81
We have the Pearl of Great Price, a translation by Joseph Smith of an early account of the creation of the world. In that is a passage which refers to a son of Egyptus, who claimed the right of Priesthood by descent. But descendants could not have it because they were descendants of colored people. That Pharaoh was a righteous man but was not given the priesthood because of his descent from Egyptus, who had colored blood in her veins. That is the only reason. It is founded upon that.