Built in the middle of the 12th century by Gauthier I of Villebéon, grand chamberlain of the kings Louis VII and Philippe Auguste, Le Château de Nemours became from the 15th century owned by the Armagnac family. In 1673, Louis XIV gave the duchy of Nemours to his brother Philip of Orleans.

The castle became a museum in 1903 and was awarded the "Musée de France" label following the museum law of January 4, 2002.