Short answer: Yes
Long answer: That top end fitness is not ever going to stick around. Your body isn’t made to keep it forever, that’s why it’s described as a peak. You’ll definitely keep the gains you’ve made, but it’s ok to let the knife get dull.
When transitioning to “off-season” I always recommend starting with a re-evaluation of your goals for the following year, what do you want to achieve? Take those goals and use them to evaluate what your training should look like. If you have important events, use those to build your plan backwards to where you need to start.
More than likely you’ll go through at the very least a base then build phase. Depending on what and when your events are, you might hit a specialty phase, or you’ll roll back to base/build and then eventually hit your specialty; which is where you’ll get your “top end” back and where you’ll sharpen the knife again.
This is why most successful riders yearly performance charts look like a mountain range, as they’re constantly building then recovering. Here’s what my PMC looks like from 2018 until today: