if this is your only option, go for it. even if you loose a bit of fitness you will be fine. maintain as much as you can during those 3 wk while giving space for your other priorities.
in terms of how to fill your remaining weeks, if you already have base>build>specialty of HD in the plan and still time over why not add some SSB1/2 now before you move on? leave 1-2 weeks as buffer for sickness/unplanned travel or life happening and if you keep them at the end before your 3 weeks off just repeat a week.
see also Long term plan for a 70.3 A race next August
if your 3 weeks are restorative instead of stressfull, maybe consider for the first of the 2 wk before your race some longer training hours or a not-all-out olympic race. this may give you some psychological comfort after the weeks off and take your mind back in.