Honey, you've been so worried about others. Put yourself first. It's not your responsibility to change them or make them feel better. If they wanted to, they would.
Do things that make your present happy while you are preparing for your future. It doesn't matter what people will say as long as you know how your life works. Just do it! It's like taking or leaving the dream you wanted!