Dr. John Herndon Graves (1807-1889) and his family settled in this area in 1858. They acquired several large tracts of land over time, and Dr. Graves' daughter and her husband deeded the land around this site for Lilac Baptist Church in 1880. It was intended for use as a cemetery as well. According to oral history, another area resident subsequently deeded a strip of land across the south end of the cemetery. The earliest marked grave on this site is that of Calvin McCormick (d. 1870). Dr. John H. Graves' marker indicates that he was the earliest-born of those interred here. Other burials of interest include four Civil War veterans, one Spanish-American War veteran, nine veterans of World War I and a number of veterans of other wars and conflicts. Lilac Cemetery continues as a chronicle of Milam County history. Texas Historical Commission marker erected 2000