Yeah, I’d consider tracking your calories for a week or two. Maybe first start by taking some time off the bike, get some really good recovery, and then get back into training + tracking. Just as an experiment to make sure you’re not in a chronic (big) deficit. If you find you’re not in a deficit… I’d say it’s too much training.
Just be careful to not get too caught up in it. I know people that start tracking calories, and then it turns into a bad habit.
50% is a recovery ride, that should be easy. Like, just the weight of your legs falling on the pedals easy. Not having to think about it.