Brigham teaches that the personal character of Joseph (or any other leader) is not important in comparison to the truthfulness of their doctrinal teachings.
Brigham Young, "The Gospel Like a Net Cast Into the Sea—Good and Bad in the Church—Embrace Principles in Your Faith, not Men—Confess Only to Those Against Whom You Have Sinned—Economize the Gifts of God, Etc.," November 9, 1856, Journal of Discourses, 26 vols. (Liverpool: S.W. Richards, 1857), 4:77-78
We can pick out Elders in Israel right here who can beat the world at gambling, who can handle the cards, cut and shuffle them with the smartest rogue on the face of God's foot-stool. I can produce Elders here who can shave their smartest shavers, and take their money from them. We can beat the world at any game.
We can beat them, because we have men here that live in the light of the Lord, that have the Holy Priesthood, and hold the keys of the kingdom of God. But you may go through all the sectarian world, and you cannot find a man capable of opening the door of the kingdom of God to admit others in. We can do that. We can pray the best, preach the best, and sing the best. We are the best looking and finest set of people on the face of the earth, and they may begin any game they please, and we are on hand, and can beat them at anything they have a mind to begin. They may make sharp their two-edged swords, and I will turn out the Elders of Israel with greased feathers, and whip them to death. We are not to be beat. We expect to be a stumbling block to the whole world, and a rock of offence to them.
I never preached to the world but what the cry was, "That damned old Joe Smith has done thus and so." I would tell the people that they did not know him, and I did, and that I knew him to be a good man; and that when they spoke against him, they spoke against as good a man as ever lived.
I recollect a conversation I had with a priest who was an old friend of ours, before I was personally acquainted with the Prophet Joseph. I clipped every argument he advanced, until at last he came out and began to rail against "Joe Smith," saying, "that he was a mean man, a liar, money-digger, gambler, and a whore-master;" and he charged him with everything bad, that he could find language to utter. I said, hold on, brother Gillmore, here is the doctrine, here is the Bible, the Book of Mormon, and the revelations that have come through Joseph Smith the Prophet. I have never seen him, and do not know his private character. The doctrine he teaches is all I know about the matter, bring anything against that if you can. As to anything else I donor care. If he acts like a devil, he has brought forth a doctrine that will save us, if we will abide it. He may get drunk every day of his life, sleep with his neighbor's wife every night, run horses and gamble, I do not care anything about that, for I never embrace any man in my faith. But the doctrine he has produced will save you and me, and the whole world; and if you can find fault with that, find it. He said, "I have done."
It is the fashion in the world to embrace men in their faith, or a fine meeting house, or a genteel congregation, thinking, "O, what perfect order, and how pretty they look; how straight they walk to meeting, and how long their faces are during the services; how pretty that deacon looks under the pulpit; the people are so pretty, the meeting house is so nice, that we want to join such pretty people." Such feelings will take a people to hell. Embrace a doctrine that will purge sin and iniquity from your hearts, and sanctify you before God, and you are right, no matter how others act.