George Q. Cannon teaches that both Joseph and Brigham were fallible men and had failings and weaknesses.
George Q. Cannon, "The Latter-Day Saints Aspire to Celestial Glory," August 12, 1883, Journal of Discourses, 26 vols. (Liverpool: John Henry Smith, 1884), 24:274
Now, was not Joseph Smith a mortal man? Yes. A fallible man? Yes. Had he not weaknesses? Yes, he acknowledged them himself, and did not fail to put the revelations on record in this book [the Book of Doctrine and Covenants] wherein God reproved him. His weaknesses were not concealed from the people. He was willing that people should know that he was mortal, and had failings. And so with Brigham Young. Was not he a mortal man, a man who had weaknesses? He was not a God. He was not an immortal being. He was not infallible. No, he was fallible.