I wouldn’t worry too much about it. One of the goals of sweet spot work is learning to push the pedals hard for long periods of time. Taking little rests is a bit like riding outside. If you can work up to 60 or 90 minutes of total sweet spot work, do it and you’ll get strong 
and those short rests won’t really matter.
3 Likes