Giro Aerohead fit

Hey all, ive been rocking a Giro Aerohead helmet for the last few seasons and while i generally love the helmet, i cant decide if i have too small of a size.
I have a size medium. I do wear the helmet kind of on the back of the head so that the visor sits clear of the cheeks and the helmet cuts a small hole while in a tuck position. However, the visor cuts into the cheeks no matter how i position it and the top right magnetic latch always comes undone when i start an effort. I also have long, poofy hair that I tie up and put a hairband around for help with the fit.
If i size up to a large, would it be “too” large? Has anyone else had experience with this? I would go try one on but no other local shops here carry that kind of stuff.
Pics for referance, if it helps.