Both his parents were free persons of color. Following private schooling in Wisconsin and Ohio, Adams graduated from Oberlin College. After a brief teaching stint in Louisville, in 1870 he followed his uncle, Joseph C. Corbin, to work in Arkansas in the Reconstruction. By 1874 he had risen from schoolteacher to assistant superintendent of public instruction.

The first two years of the war are fairly blank in the Whitman biography, but he surfaces again in the fall of 1862. His younger brother, George Washington Whitman, an officer in the Fifty-first Regiment of New York Volunteers, was reported in the New York papers to have been seriously wounded in the battle of Fredericksburg in December 1862. Whitman was dispatched to Washington, D.C., by anxious family members in Brooklyn to search for his brother in the more than forty wartime hospitals. Failing to find George there, he went to the battle site to find his brother only slightly wounded.