How important is specificity?

I am planning on buying a new bike and am torn between a Propel with SRAM Force and a TCR with ultegra di2. As I cracked the frame on my old TCR.

I came to the conclusion that I would get the propel (as I always dreamed of getting an aero bike) and instead of buying a second cassette and new trainer body I would just use my gravel bike (giant TCX) on the trainer which has its pros and cons.

I don’t wear out my expensive SRAM cassette, chain, and chainrings
There’s no constant putting on and taking off of the back wheel to put it on the trainer.
I won’t be dripping sweat all over my fancy new bike.

After taking an FTP test on the trainer (with the TCX) my FTP dropped significantly (after an additional full block of training, from 308 —> 265 and then re doing the test the next day and getting 295. While I have been training this whole time consistently and hitting power PRs outside. I attribute a 10-20 watt difference to the difference in position/metal frame/ any other excuse so that I don’t have to say I’m weaker.
The bike fit is slightly different changing the muscles used on the trainer

Which is making me second guess my decision to get the propel, and think I would be better off just buying a TCR with Shimano so I can use it on the trainer.

My question to you guys is how important is it to ride the bike you’re going to be racing on, on the trainer