Heart rate not dropping after hills and efforts

I’m 63 and do very a very similar volume and type of riding that you do. I’ve found that if I do that type of mileage at Z2 levels I’m fine but if I put too many bouts of intensity into them I gradually get more and more fatigued until I do a ride and I can tell from my HR on the first 20mins or so that i’m not right. I just dial it back and then take it really easy for a few days. By the way we’d kill for 20 degrees in the summer here in Lancashire let alone in the winter. Hope it goes well with the Doc.