Coming back to say that first and foremost, there’s NO reason for you to cut out any food group from your diet, especially if it’s not detrimental to your health.
Secondly, the solution to your problem is not to deprive yourself of food, but to practice portion control.
Thanks
Lastly, because I’m very upset and also generous. If anyone at any point, who claims to be a health professional tells you to “cut out” anything from your diet, without giving you a valid (evidence -based) reason or explanation, please run fast and run far.