I think there’s a ton of selection bias happening.
Not that the impression is incorrectly created by a few biased samples, but that generally folks who are ex-rowers have been greatly selected for size and absolute vo2max.
There are no hills on the water and the increase in boat drag due to weight doesn’t scale like total power, so bigger / more watts is always better. To make the boats, you’re always selecting the max power applied to the boat (technique matters a lot) for ~6 minutes. So, if you don’t compare favorably to other 6’3"+ folks in absolute vo2max, relatively quickly you’re no longer calling yourself a rower.
Also, rowing seems to get one exceptionally well aquainted with hurting.