Hiyori forgot.
Yato made sure of that, he did everything he could to erase himself from her life. But something always remained. Scents really can bring up even very old and hidden memories, and the body remembers better than the mind. He couldn't erase any of that.