I confess to having this problem as well - although in my defense, my local terrain is ‘lumpy’ which doesn’t make it easy to ride at low intensity without going excruciatingly slow at times.
I’m planning on riding recovery weeks indoors throughout the summer. Sod’s law states that those weeks will undoubtedly be the best riding weather…