I didn’t quit cycling, but I quit motorcycle racing. And yes, it was very hard. I only rode a motorcycle a handful of times over the next near decade and it felt like I was going back home every time.
I imagine I will/would feel the same if I quit riding my MTB.
I picked up a motorcycle for track riding last year, but for track only without racing. It is a little peaceful going out there and riding only for you with no one else caring, and I look forward to that feeling when I quit MTB racing and just ride for pleasure only.
I will just be slower without the dedication. In both sports.