I don’t know that really short breaks (a few minutes) have a big detrimental effect on fitness, but when you get in the habit, and then plan to stop ‘less’ during a race, I feel it’s certainly harder.
For example, if you do a couple 5-6 hr rides/month, and you are regularly used to stopping 3x for 5-10 minutes, vs stopping 2x for 2 minutes, the latter will better prepare you for stopping less during races.
Last year I really focused on very minimal stopping during my 5-6 hr weekend rides. I would carry all my hydration and fuel and would maybe stop twice for 2 minutes to pee. It was hot, and I would sweat a lot, so that seemed to help getting by peeing less.
Fast forward to now, and it’s much cooler where I’ve moved to and TONS of climbing. On a 5 hour ride I probably pee 4x and honestly, something about the terrain just makes me feel like I have to stop more. I also occasionally have to stop to strip layers, plus go under/over closed gates usually 2-3x a ride.
As a comparison, last year at Unbound, 5 hrs 38 minutes, I had one stop at the mid-way aid station for 2-3 minutes as I stopped at my wife’s car, peed in a jug while eating a snickers as she swapped bottles and refilled my hydration bladder.
I have a 6-6.5 hr race on May 31st, and plan to stop at 3 aid stations that will probably be 3-5 minutes a piece to refuel and pee. I don’t think my body is as conditioned as it was last year to ride longer without the stops, which really sucks, but is what it is.. I think it’s largely due to getting in the habit of stopping more during long rides (during indoor rides I never stop on rides 2 hrs or less).