Pretty much what I felt before retiring from my own personal racing last year. That said, yeah, if you don’t have a group, that can certainly help with motivation and making things fun. The only thing I miss about it is the camaraderie of the group and my friends in the field. It’s always fun for me to go out and watch the local races where I’m coaching someone for that reason, but I don’t miss the rest of the grind, personally.
In the end, for me, the friendships are great, but wasn’t worth the constant risk and the logistical challenges of competition as a middle-aged husband and father. That coupled with the fact that I don’t believe training for high-performance endurance sports is a particularly healthy lifestyle in terms of long-term well-being are what led me to leave my personal competition days behind me, and I don’t regret that decision one bit. (Though I do need to get out and do some more casual riding).
You just kind of have to weigh things out for yourself and figure out what it is that you want to do. If you decide that racing isn’t for you anymore, I would definitely have something you’re “quitting to”… another sport, another hobby you can dump your time and energy into. Much like retiring from full-time work, you’ll be a lot happier if you have something else in mind.