You don't have to do what people want you to do. You have to do what’s within your capacity for the sake of Allah Alone. You’ll be surprised that you can actually do much more when your focus is Allah. So keep going, do your best and ignore the rest.