Aubrey, the town, was officially founded 1867, when Civil War veteran Lemuel Noah Edwards (1838–1910) built the second frame house there. Edwards eventually gave each of his 10 children a lot on which to build a home. The Edwards family was instrumental in several civil developments. Dancing was not allowed, but the townspeople often gathered in the Edwards home for singing and listening music performed on an organ that Edwards had imported.
Eventually, Aubrey became known for the peanut farms that surrounded the town. By 2009 horse ranches surrounded Aubrey. Around that time, houses were constructed in Aubrey, replacing the grounds of the old peanut farms.