And even worse is that the people that did know him will only think that he left yet again. They'll only speak of his title and his name like a ghost. The Hero of Time.
Ocarina of Time is a full blown tragedy. Where even the happy ending that everybody got to experience will never be felt by Link. Everyone that Link grew to knew wouldn't know him. His brothers in the Gorons, his acceptance in the Gerudos, his cherishment of Zoras. All will fade.