Living in High Altitude and Traveling To Lower Altitude Before "A" Event

I currently live in Breckenridge Colorado and am training for the Breck Epic, a 6-day MTB stage race ranging from 30-50 miles a day. I have a family event in Wisconsin (way lower altitude) the week before the race where I’ll be spending up to a week there. I’m concerned about “deacclimating” during this period before returning to CO for my race? Does anyone know if this is a thing? How long does it take to deacclimate? How many days would you recommend to return before the race starts? Any other advice or experience with this please let me know.