The problem actually occurs all the time, not just on rest weeks. If you’re supposed to ride easy and you ride hard you’re denying your body the chance to get stronger.
Others have stated this in the thread already but not allowing yourself to recover is an extremely common problem for people who are new to training.
Without knowing the precise nature of the mountain bike work you’re describing it is hard to say if it is truly recuperative but it may be just the thing you need. I’d still suggest you learn to ride easier without a course limitation as it will continue to hamper your progress and keep you from reaching your potential