you actually can speak up at anytime but you have decided that in order to maintain connection with this person you need to abandon yourself. it's clearly making you unhappy yet you stay anyway.
The best revenge is letting people live with themselves. I used to be petty, and pettiness feels so good. But nothing feels better than being unbothered and sexy and regulated and healed while a mentally ill whackjob ruins their own life because of who they are as a person.