And before long, [Thaddeus Stevens] obtained the services of a housekeeper, Lydia Hamilton Smith, a mulatto widow of great respectability with two children, who remained with him and took care of him for the rest of his life. Thad’s relations with Mrs. Smith have given rise to never ending speculation. Widely thought to be his mistress, both at the time and afterward, Mrs. Smith was treated with great respect by Stevens’s family and by himself. He always addressed her as “Madam,” gave her his seat in public conveyances, and included her in social intercourse with his friends. As William M. Hall remembered, she was regarded as a “virtuous and respectable woman.” “Certainly,” he commented, “whatever may have been the state of affairs at the inception of the relationship it existed for years at Washington in such form as to leave no discredit connected with it in the minds of Mr. Stevens’s acquaintances and friends and of the general public.”…Stevens himself never confirmed or denied it. In a largely noncommittal letter to W. B. Mellius in September 1867, in which he stressed the fact that he seldom responded to the many attacks upon him, he also stated that the libels concerning his domestic history were “totally without foundation.”
Hans L. Trefousse, Thaddeus Stevens: Nineteenth-Century Egalitarian (Chapel Hill: The University of North Carolina Press, 1997), 69-70.