I tend to wear mtb shorts on XC rides too. I like having my phone and a gel in easy reach. Having said that, I bought some Rapha cargo shorts (regular road shorts with a mesh pocket on the thigh) and I’m pretty impressed. I might switch back to being “that old guy in girly shorts” again.
The funny thing is how the mtb guys will all bag on roadies for being uptight about their Lycra, when what they’re really saying is “you’re in the wrong uniform”. Kind of like whining about whiners. 
Our mtb page had one of those posts recently where someone pointed out that “roadies are cyclists, mtb’ers are bikers”. I had to laugh because when I think “bikers” I think Harley Davidson.