Think of the guys who you know that win races - the guys who are always at the front. As a subset of those, let’s think of just the ones who are riding carbon bikes. As a thought experiment, do you think that all other things being equal - if you swapped their carbon frame out for an aluminum frame, that this person would no longer be at the front?
Let’s try an additional thought experiment. What percentage of benefit do you think that carbon vs. aluminum provides, and what are you willing to pay for that premium? If it provides a 1% increase in performance by some measure, what is that worth? Would you notice it, over the length of a race? People will say things like ‘stiffness’ and ‘chatter’ and ‘feel’ and the like but to be honest, frame material isn’t as important as the interface between your bike and the ground. Wheel/tire combination and tire pressure.
In my humble opinion (I apologize if I stepped on toes) I would opt for the aluminum frame and then perhaps save money make sure you’ve got a good set of wheels and tires on the bike. A good set of bibs to sit on. Good components. Frame material comes after those other things - all else being considered. If money weren’t an object, sure go for all carbon everything.