How do I figure out RPE?

Have you read this?

It comes down to experience. The more you ride, the easier it is to gauge how long you can keep that effort level. As mentioned in the article, the lower ends sort of blur. I think of them as ‘2- easy spin’, ‘5- stead slow cruise’, '6- working, ‘7- spirited’, ‘8- hammering’, ‘10- all out’.

Around here MTB is not great for a structured workout because a lot is either max effort up the hill or coast on the descent.

The differences you describe indoor vs outdoor sound like cadence difference. Higher cadence gets the heart rate up; lower cadence burns the legs faster.