The 19th Wife: “You of all people…”

In the chapter where Ann Eliza goes to Beehive House to ask Brigham Young for a new stove, he balks at first but then gives in, telling her, “I always thought you of all people would understand what it meant to be my wife.” What does he mean by this? Is his statement justified at all?