Similar to me. My weight fluctuations are roughly along these lines:
August/September race season ends ~170 lbs
October base starts ~172-173 lbs
December build starts ~165 lbs
Maintain 165-170 during build and speciality
Season starts and float anywhere from 165-170
So the time to lose weight for me is always base, and the rest of the season I try to maintain power rather than specific weight targets