Oh, hello fellow ex-gymnast 
Honestly so much drift for an endurance ride is a lot. BUT - this is one try and you definitely should not draw conclusions from one ride. Being in your shoes, I would start to keep an eye on it.
PRO tip - register to intervals.icu - after upload, you will see the cardiac drift calculated in an intelligent way - it takes the warm-up and cool down out and takes a comparable time slot (you can influence it) - this is so much more easy!
At the other hand, if you already know that your base is weak, this drift it totaly possible. And honestly, don´t get misunderstood - traditional “base” plan on TR carries this name, but the focus is on the Sweetspot with aim to push your ftp higher. But you do not train that long steady endurance tempo, so you don´t get that much better in it… in the words of @ambermalika you are what you train for 
To build a very good stable base, you simply have to spend hours in the low zone and your body - rather quickly - adapts. In my case, I sacrified a few years ago something like 8 weeks in the spring and I was just ticking miles with a friends with one goal. At least 150km and brething by the nose. After 2 000km like this, I went to the lactate test and it proved a magnificent progress in that sense - I could actualy ride endlessely in my endurance zone without any cardio drift and super stable blood lactate. At the other hand, as soon as it ramped up, my lactate sky rocketed and I failed quickly the ramp test - predictably
That year, it was my plan - I build a super base like that and than did hard rides in the hills and it got me nicely through the Oetztaler Radmarathon (my first race)
Today, I wouldn´t be so strict, I don´t have any ultra endurance in sight, but in your case, I think this would be good to actualy add some really easy rides and focus on that side as a plus to the SS work
Besides, think very well about the total work load - if I read well your numbers, 165W is Z2 for you. So this is actualy not “that” easy - but that depends if you can handle the added stress. Longer Z1 rides are really a safe bet and a super way how to train easily your cardio system without risk of overload, if you have harder SS/VO2Max trainings other days…
Disclaimer - I am not a coach, nor a physio, just talking from my experiences and knowledge