At your wight and height you have a lot of room for a lot of FTP growth. Because of this you can do a lot of short duration VO2 efforts to jump start the process of growing your fitness. I know this is the OPPOSITE of what some might suggest. But…
You mentioned above about doing 160 mile rides. Although that sounds impressive around the office, Those rides are not usually done the appropriate way to gain any type of fitness adaptation.
To really make a ride that long count toward your actual fitness the 17mph average speed should not even be looked at. You would want to try and stay in your real zone 2 to zone 3 power or dare i SAY IT ( HEART RATE ZONES ) for as much of that ride as possible to gain anything. Riding at 100 watts for 1 minute then spiking to 300 watts for 20 seconds over a hill then back down to 50 watts to recover before you level back out at 100 watts over and over again on that long 160 mile ride does nothing for your fitness and is actually just the easiest method to finish a ride that long.
Your other outdoor rides, spirited group rides where the power gets heavy over the rollers and hills, these are the rides that will get you stronger ( MOSTLY WHEN ) you have a lot of fitness growth left to uncover. You cant keep up because when you are doing your looonger slooower 160 mile rides your other buddies are doing harder 30 to 40 mile rides.
My personal advise, Do rides just like those group rides when you are solo. And they only have to be an hr long right now. Go out and hammer for an hr. How do you do that? Find out your tempo HEART RATE and or power and ride at that level for as much of the ride as you can AND on all of the hills and rollers you must POWER up them to the best of your ability. Spike the power on the hills but then dont let up after cresting the hill continue riding at the elevated ( TEMPO ) HR. You will most likely have to let up some to recover but the better you get at knowing your TEMPO and riding at it while still being able to hammer the rollers and hills the more you will gain from this entry level but effective style of training.
Pick a 15 mile loop around your home with some rollers and hills and ride it 3 days a week as hard as your can. Do that for 3 months, 3 days a week. Say Monday, Wednesday and Saturday. Don’t even worry about volume or longer rides yet. Use the rest of the week for recovery, maybe some easy recovery rides if you feel you can get it in. It will take you some time to feel what power you can put out over the hills but honestly don’t even look at power or HR on your rides for the next 3 month until AFTER your ride to analyze. Just be HONEST with yourself and admit if you think you didn’t go hard on the hills or if you let up too much on your tempo.
This doesnt mean go ALL OUT on the hills but it does mean getting out of the saddle on the rollers and pushing a bit harder up them until you crest over. There will be trial and error. Start small and short 15 mile rides. TEMPO on the flats PUSH IT on the hills and rollers. And at some point you might not feel up for pushing it hard on that day, dont do it, ride that day easy. but still get out there and ride the loop easy, and ride at least 3 days a week during this time. Short, Hard Tempo, Effective
This isn’t the best way to train forever but its a good way to jump start your motor.
Long rides are great but you wont get that strength to keep up with your buds by doing them without something else.
After 3 months, see where you are. Then you could potentially start to work on some structured ( SWEET SPOT BASE ) and bring it indoors for more structured training.