It’s not about the calories, it’s about the habit. Every time you give in to a craving and eat something unplanned (whether the food has 50 calories or 500 calories), you reinforce the habit of giving in, making it more likely you’ll give in the next time, and the time after that.