Still makes a difference. Even in the draft you’re pushing air, so being more aero still saves you watts. And higher group riding speeds means when you are on the front you’re dealing with more wind resistance than when you’re solo. E.g. A “spirited group ride” to me is one where you’re pulling on the front at threshold or higher. Somebody above mentioned a 15-20W saving with an aero bike. That sort of watt saving could well be the difference between holding the group speed comfortably on your turns vs pushing into the red (or slowing the group down).
Whether an aero bike actually saves you 15-20W is a whole different debate of course! And certainly there’s cheaper aero gains to be made in position, clothing, helmet, etc. But if it’s a fast group ride and I want to be one of the stronger riders (or think I’ll be one of the weaker riders and need every advantage I can get) then I’m absolutely going with tight fitting kit (I draw the line at skin suits for group rides, that’s not cool…), aero road helmet, race bike and wheels, etc.