After Ulysses S. Grant resigned his commission in 1854, the family unsuccessfully cast about for a financial foothold. Julia Grant's prosperous family had always looked down on "dirt farmers," but, to their and her dismay, she found herself living a life of grinding poverty on a miserable, unproductive farm called "Hardscrabble" outside St. Louis. As she noted in her memoirs, the coming of the Civil War saved both her and her husband from desperate unhappiness.
No one understood Ulysses S. Grant better or knew how much he loved the army than his wife. She believed in him intensely and never wavered from her belief that he was destined for greatness. During the Civil War, Julia Grant spent more time in the field than any other general's wife. It was rumored that General Grant's aides summoned her to camp whenever he was drinking too much, but his letters tell a different story. Whenever he felt it safe enough, he would write anxious letters asking her to join him, which she eagerly did. An aide recalled evenings when she and the general would sit at headquarters in the field holding hands, looking shy and ruffled if anyone caught them. Julia Grant was nearly captured by the Confederates when they overran Holly Springs, Mississippi, in 1862, and she was with General Grant during the Vicksburg campaign in the spring of 1863.