Honestly, Peter is partly to blame.
He spent four years convincing himself that staying away from MJ was the noble thing to do, when he could’ve simply approached her and given her the choice. Instead, he decided for both of them that their relationship was over.
Then Brand New Day proves how quickly that wall could’ve come down. It took less than three days around MJ and Ned for that old chemistry and connection to start resurfacing.
Peter didn’t lose MJ because they stopped caring about each other. He lost four years because he was too afraid to find out if they could start again.