What time of the day were you doing these rides? If you were feeling too-cold at the start but then warmed up & felt comfortable… I wonder if that would explain HR drift.
Also, check your cadence profile over the whole ride. If your self-selected cadence drops over the course of a ride, then it will encourage a slight drop in heart rate.
Lower cadences cause:
- Lower muscle pump
- Increased pedalling force for the same power output, ensuring greater recruitment of type-2 fibres → less oxygen is required for the same power output, therefore less cardiac activity is required.
This thread has a more in-depth explanation: