You sound like you’re doing it right. I’d make your hardest days the first day you come back from rest.
You could experiment with 2 on, 1 easy, 2 on. You could have the hardest days be on the 1st and 4th workout (after the easy day). If you struggle with getting enough volume you can add easy rides in the morning or evening. You can also extend some of those rides before the rest days so that you tack on extra aerobic work.
It sounds like you could turn yourself into a really good 3-5 day stage racer! You’re going to have to be comfortable in being a bit tired during the rest of the week though if you want to do a fair amount of volume.
Good luck!