The name of the village of Annalong (Ford of the Ships) may well date back to the days of the Vikings when their long-ships found shelter in the 'wee hole' at the mouth of Annalong River where the harbour was later built. Until the gravel bar was breached at Kilkeel in the 1870s, Annalong was the principal harbour along the Mourne coast. Prior to 1800 granite kerbstones, locally known as 'kribben' were being exported from Annalong to the west coast ports of Britain in converted fishing boats. As the granite industry developed purpose-built schooners were used and by 1900 a fleet of over thirty such vessels operated from the harbour. The demand for granite has changed and to satisfy the demand large blocks of different coloured granites are now imported from all over the world to be cut and polished in the village.

Up until the second world war Annalong had a considerable fishing industry but the harbour was too small to accommodate the larger vessels and they moved to Kilkeel though the inshore fishermen continued to operate from Annalong. The harbour is now used mainly for pleasure craft.