Pastor, you should know that no matter how hard you try, no matter WHAT you do... there will always be those who just do not like you.
Just keep acting with love & integrity. Let the problem & the sin be theirs not yours.
As much as it depends on you, be at peace with all men.