A pure hearted person takes a long time to recognise harm because they measure other people's intentions by the goodness of their own heart.
And narcissists know how to exploit that virtue.
They use your empathy to excuse their behavior, your patience to push your limits, and your willingness to see the good in them to keep you questioning what you already know.
Your pure heart was never the problem. Their exploitation of it was.