Yep, the further back in the pack you go, generally the more riders slow down throughout the day. If you can’t make it to twin lakes in 4 hours, you have very little shot of completing the course in 12 with the toughest climbing still to come. Not saying it’s impossible, but it’s a long shot.
But the cuttoffs are there primarily for safety reasons (weather risk at the high points), not designed around finishing time. Letting someone start the columbine climb at noon would be a crap shoot on the weather and it’s extremely exposed for both riders and volunteers at the aid station. The stage race is a good race on the same course where you can finish regardless of how slow your pace is (I rode sweeper this year and I think it took the back of the pack about 5 hours to get up columbine and back to twin lakes).