Hartley once had a castle that stood guard over Kirkby Stephen. Originating with the Harcla family, Michael is known in the 13th century and then Andreas a younger son. See the legend of Andreas de Harcla. There are also two fiction books centred on this character by Adrian Rogan, The Northern Warrior and 1314 A succession of Musgrave Knights improved the castle until finally abandoning it and moving to Eden Hall. Unfortunately, there is nothing left to see of this castle which is now a farm.
Leading up towards the quarry and Hartley Fell there is a pretty village along the beck which flows down to Hartley Mill next to the River Eden just before Low Mill and Hartley Road into Kirkby Stephen.
There are several footpaths to and from Hartley including the Northern Viaduct Trust walk along the disused Stainmore Railway to Stenkrith Park. A leaflet is available at the Visitor Centre.