So, a quick update:
In the end, I opted to upgrade my power meter to a an spider based one (rotor in-spider) and I couldn’t be happier with the results. Not only the described behaviour is gone, but now is much easier to maintain the required power. Here are the plots of two workouts:
On top you have the workout with Rotor inspider and on the bottom with the 4iiii left only source. The difference in power stability is huge!