This happens to me all the time, but only once I start to exceed my average workout length. When I was doing mostly hour-long workouts with a 75-90 minute one thrown in and a longer ride on the weekend, sure enough after ~75 min in Z2 I would start to see HR drift up. Now most of my rides are 2-2.5 hrs, and on longer weekend rides I’ll start to see that HR drift up after 2.5-3 hrs in Z2.
I try not to worry about it and just go by watts; once you’re outside your regular comfort zone just trust the training and advice and it will work out.