If you would feel entitled to turn down eating something because you were allergic to it, then you should feel entitled to turn down food because it wouldn’t help you reach your weight loss goals. Losing weight and being happy and healthier are just as legitimate reasons to say no as an allergy!