That’s the nature of HRV - it isn’t an exact science, though the marketing people would have you believe otherwise.
It is still a powerful tool though once you;ve learned what is “normal” for you. For me it definitely does flag up when I’m overdone on the bike (low HRV, Low Resting HR), struggling with non-bike stress or about to get sick (very low HRV/High Resting HR). Day to day when things are normal it bounces around a bit but without hitting any extremes.