I can see myself very well in your situation: from time to time my w/kg seems to be the only thing that matters – all the time, everywhere I go.
My solution for the problem is that I (try to) accept the fact that holding your FTP is difficult. Like you I train hard through base, build and specialty; but after that, I let it go. After peaking in late May with FTP of 275w, I went down to 265w in a couple of weeks. After that, I started doing long endurance work outside to enjoy summer. That bumped my FTP back to 275w for my next build phase. During the build from mid July to mid August I actually lost some watts (most likely due to trying to lose some weight simultaneously). However, it brought me back to proper fitness for some late summer races. After the racing was over, I have tried to maintain my fitness with 450-500 TSS weeks but, once again, my FTP has come down to 260w. Soon, after two weeks of off time, I’ll start to build my fitness again from SSB I.
To put it simply, I have good fitness for a month or so, and outside that I have fitness high enough that I can still enjoy everything I want to do on my bike. If you really look at it, having one’s FTP in 260w instead of 275w isn’t such a huge difference if you’re just riding easy and sprinting for the town signs. Letting my FTP go a little has truly allowed me to change my riding radically: during late summer/early fall I have had only one 1h proper training session, one 2h endurance session, and two long (2,5-3,5h) rides outside. Compare that to the stuff you have in SSB2 or General Build mid vol – quite a lot easier.
Also, one point to add about losing fitness: it may come back surprisingly fast. As mentioned above, after my break in early summer, I had my FTP down to 265w but got it back to my peak 275w with just 4 weeks of mostly endurance work (doing traditional base 2 mid vol outside with added volume). That is why I can be a little less frustrated about losing fitness: I know that soon it’s going to come back again when I change from maintenance to building again.
When it comes to not hitting your target of 4w/kg, I may have had a similar feeling. I did hit the 275w but during the two separate build sets I had (before my first peak in spring and during the summer), I never got above that number. It makes you mad to realise that you are stuck even if you are working as hard as you can. However, I try to approach it like this: I don’t know how much benefits I’m going to get from my training and that’s why every cycle is an experiment. If the results from the experiment are not what I was looking for, I have to analyse the process, come up with improvements and do the next experiment. As in many other places in life, not every experiment is a success – but there’s a huge amount of information out there to come up with improvements in your training “process”. To give you an example, my next experiment is to do the base, build and specialty with 15% added TSS. This is because: 1) I think I can handle it and 2) generally more training tends to give your more benefits.