I think it’s completely understandable. I am still really really new to cycling (2 years). I did a couple of races last year and even then kind of experienced the same feelings. I also strongly noticed the clique-ish feel which is just not for me. I am definitely a bit of a loner and strongly prefer training/riding alone so not really into joining a team etc.
I came into cycling after 10+ years playing amateur soccer as an adult and had grown tired of the cut throat, do or die nature of over-30 games with guys making risky sliding tackles, fights etc. And so switching from that to riding in a group on the edge, with guys bumping shoulders/elbows etc. was no bueno for me.
I’ve just kind of accepted that and have just taken to riding for the sake of riding/training just to see how much I can improve and then potentially picking events that aren’t really races per se but just see how well I can do year over year versus myself