Brigham says only God can take the priesthood restriction off.
"Speach by Gov. Young in Joint Session of the Legislature, Feb 5th 1852 giving his views on slavery," Historian's Office reports of speeches, 1845-1885; Brigham Young, February 5, 1852; Church History Library, accessed January 21, 2022
Now then in the Kingdom of God on the earth, a man who has the African blood in him cannot hold one jot nor tittle of priesthood; why? because they are the <true> eternal principles the Lord Almighty has ordained, and who can help it, men cannot, the angels cannot, and all the powers of earth and hell cannot take it off, but thus saith the Eternal I am, what I am, I take it off at my pleasure, and not one partical of power can that posterity of Cain have, until the time comes the says he will have it taken away. That time will come when they will have the privilege of all we have the privilege <of> and more.