The area along the Delaware River had long been settled by the Lenape, an Algonquian-speaking indigenous tribe that lived in the mid-Atlantic coastal areas at the time of European colonization. The English also called them the Delaware, after the river they named for colonial leader Thomas West, 3rd Baron De La Warr, the Delaware. Milford was founded in 1796 by Judge John Biddis, one of Pennsylvania's first four circuit judges.

He named the settlement after his ancestral home in Wales.Milford has a large number of buildings of historical significance, many constructed in the nineteenth century and early twentieth centuries. Some are listed on the National Register of Historic Places, while numerous others are included in the Milford Historic District.

Of the 655 buildings in the district, 400 of them have been deemed to be historically significant. The district is characterized by a variety of Late Victorian architecture. Grey Towers National Historic Site, the ancestral home of Gifford Pinchot, noted conservationist, two-time Governor of Pennsylvania, and first head of the U.S.

Forest Service, is located in Milford. It was designed by architect Richard Morris Hunt has been designated a National Historic Site. From 1904 to 1926, Grey Towers was the site of summer field study sessions for the Master's program of the Yale School of Forestry, together with the Forester's Hall, a commercial building that was adapted and expanded for this purpose.

Jervis Gordon Grist Mill Historic District, Hotel Fauchere and Annex, Metz Ice Plant, and Pike County Courthouse are also listed on the National Register of Historic Places. Nearby is Arisbe, the home of Charles S. Peirce, a prominent logician, philosopher and scientist in the late 19th century, and another NRHP property.The Pike County Historical Society Museum in Milford includes in its collection the "Lincoln Flag", which was draped on President Abraham Lincoln's booth at Ford's Theatre the night he was assassinated.

The flag was bundled up and placed under the President's head, and still bears his blood. It was kept by stage manager Thomas Gourlay. He passed it down to his daughter Jeannie, an actress who had appeared in the play, Our American Cousin, at the theatre that night. She later moved to Milford and the flag was donated to the museum after her death.The Milford Writer's Workshop, an annual science-fiction writer's event, was founded in 1956, and ran until it moved to the United Kingdom in 1972, where it is still running.In September 2007, Arthur Frommer's Budget Travel named Milford second on its list of "Ten Coolest Small Towns" in Pennsylvania.