I think it helps here to think of the gains in 3 main ways:
- more aggressive positioning. The body represents about 75% of the overall aero drag, so a lower more aggressive positioning will reduce drag, even if the frame itself isn’t any faster. Aero frames typically have a more aggressive default positioning than endurance frames, but if you are flexible enough, you can probably achieve a similar body position by slamming the stem, etc. And if you are considering moving to an aero bike, it might be worth slamming the stem to see if you can comfortably hold that position over a longer distance, before outlaying money on an expensive new bike.
- wheels. Wheels are a big part of the aero bike equation, and a big chunk of the aero price is often the fact that they come with aero (ish) wheels as default, rather than heavier bombproof shallow stock wheels on an endurance bike. But there is no reason you can’t fit deeper and more aero wheels to an endurance bike.
- frame. After you strip out the above factors, yes, the benefit of the aero frame itself is pretty small.