Trust how you feel. If it really is a recovery ride then it does exactly what it says - it helps you to recover as well or better than doing nothing at all i.e. you’ll feel better at the end of that ride than you did at the start, and you’ll feel better for your scheduled workout the next day than you would do after a day of complete rest.
If you’re doing a recovery ride, you start out feeling crap, and after a decent warm up (~15 minutes) you’re still feeling crap then that’s typically a good sign that you need a day off and I would shut it down there. If you’re doing a recovery ride and you feel fine at the time but you feel tired or heavy legged for your scheduled workout the next day then that’s typically a sign that you pushed too hard on the recovery ride and need to go easier.
I tend to ride (or do other aerobic exercise) every day unless I’m sick, injured or life gets in the way. Getting the blood flowing and the muscles moving seems to help me with recovery and feel better the next day. Good for mental health, sleep and habit-forming as well - the routine of doing some kind of exercise every day is one I formed 25+ years ago and has served me well through some times when sport and training has had to take a back seat.