One factor, that is regularly overlooked in a warm up protocol, is that it helps if you literally warm up. Getting blood and warmth in your muscles is a relevant factor, that every warm up protocol should achieve.
How much a short warm up hinders performance is clearly individual.
This is not science, so if you want to learn what is really going on, I recommend this:
For me, someone who doesn’t like warm ups, a longer warm up has actually helped me. Especially, since legs can feel great or horrible during a warm up, and if you only have 5 minutes, you really have no time to get that sorted.
The harder the effort (not the longer, but the more intense), the more time I need to warm up. I usually ride very easy for 10 minutes, depending on feel. This is the time for my legs, however they feel, to just spin and warm up. Then I do 2*4 minute ramps, the first one easy, the second one up to threshold, and that’s that.