Savoy is located at 33°36′5″N, 96°21′50″W and is at the intersection of U.S. Highway 56 and Farm Road 1752, on the Missouri Pacific line ten miles west of Bonham in extreme west central Fannin County. A small part of Savoy is also in extreme eastern Grayson County.
Savoy was established about 1863 by Colonel William Savoy, a pioneer settler and landowner who had arrived in the area in the late 1850s. The settlement grew slowly until after the Civil War, when hundreds of settlers began arriving in the area. In 1873 a post office opened.
The Savoy Male and Female College was established in 1876 and was one of the first coeducational academic institutions in North Texas. The school provided educational opportunities regardless of ability to pay tuition for young men and women in the area and Native American students from reservations in Oklahoma. It also offered primary and preparatory classes for county school children.