How fixed are you on your bike set up or are you likely to have a bike fit/ change the set up. Swapping a aluminium stem won’t hurt as much financially if you want to go longer/shorter/change angle.
It’s long time since I had this conundrum (different stem materials) and IIRC when I picked up stems in a shop I couldn’t feel any discernible difference between stems of different materials.