Follow the plan and rest during week 4. Fatigue accumulates over time. You can be strong as an ox and feel amazing and push yourself too far very easily. At that point your body will force you to rest (sickness, injury, extreme fatigue, etc.), not optional.
Also, the more you do this the more you will find that rest weeks are opportunities for your body to consolidate the gains you’ve made, building a strong foundation for the next layer of gains. If you stack gains on gains without rest, your ultimate level of fitness will be lower than it would have been had you rested. That is because when you are fresh and rested, you will be able to do more (push harder) during your key workouts and doing more during those workouts leads to bigger gains. If you go into your key workouts fatigued, you won’t be able to push yourself as hard and your progression will eventually get stagnant.
So treat rest as a key workout, just as important as any other workout, and stack the gains year over year.
Hope that helps