Yeah there’s a lot of lighter options too, not sure what you’re using already but I’m having really good luck with my Gore baggies this year. They’re pretty short but do offer a little more protection (I’m off road mainly) and they’re super light and very breathable. On pure XC or road rides I still just wear bibs but I’ve found that over the years I do like a minimal baggy in place for trail riding and for situations like the one you’re in.
I do remember having to get used to baggies in general again once I started wearing them after a long break from them. So there’s that too, also consider using a little anti-chafe cream in the trouble spot or like someone else suggested just pack the shorts with you and throw them on before pit stops.