I wouldn’t be discouraged necessarily…
I was there for 7 days last August and rode the same route, but minus Alpe d’huez, on day 2 of the trip. This was after an 80 mile day on day 1 which went over the Glandon and back over the Croix de Fer. Day 3 was a 50 mile loop taking in the Izoard, followed by a rest day and then a couple more days of riding.
I was around 66-67kg and 270ish FTP (so significantly less fit than you!) and had a 34-30 easiest gear.
Admittedly, my legs were quite sore at the start of each day! But I’d have thought with your level of fitness and weight, you should be able to make the most of your time and enjoy yourself, just keep it steady and sensible!