Being more aerodynamic (lower CdA) is about improving the combination of slipperiness (drag coefficient, Cd) and reducing the area (A).
Aero bars typically, but not always, reduce your Cd by putting your arms online with your torso and reduces your area by bringing your head lower. But if your position without the aero bars is already very low, then bolt on aero bars may bring you up enough to increase your area and counteract the effect of bringing your arms in.
Try getting pictures of you from the side and front in both setups to visually compare areas. If you get video footage walking around you, there’s a website to estimate your CdA 20 second Video to CFD (Aero Drag) test is HERE!.
Also, every time you move your head, hands, or arms the airflow can detach and take a while (seconds) to attach again. You’re slower during that time. So try to keep a steady position and if you take position breaks it’s better to make them a bit longer and less frequent.