Tong College was founded by Isabel Pembrugge at the same as the present church, it was built on the south side of the Church and had various ancillary buildings. These included a hospice for 13 elderly and infirm people. The Royal Licence of Henry IV, establishing the College, was granted in 1410. It granted approval, to Walter Swan, Clerk, and William Mosse, Clerk, to acquire the Church, the patronage, and to build the new Church and College. Swan was the incumbent of Pembridge in Herefordshire. The College was to have five chaplains and a Warden. The first Warden was installed in 1411.

The College was fought over during the Civil War. In April 1644, Col. Tiller, in a letter to Prince Rupert, states that he had driven the rebels out of the Church, and then out of the College and the Castle. He considered the Castle was worth keeping, but suggested that the College should be demolished. In the excavations of the College there was a level that indicated that it had been burned down around this time.

The college was partly excavated by Alan Wharton in the 1980s and he confirmed the ground plan that had been recored from crop marks in the 1920s.