Accepting that threshold for cold varies – compare winter cycling temps in southern Spain with eastern Switzerland, for example – my first thought here is the breathability of your kit, specifically your outer layer trousers. You probably didn’t have a problem on the endurance ride because your quads weren’t working as hard (ie generating as much heat).
My second thought is the trousers are intended for much colder temps, which ties back to the breathability point above. If they are intended for much lower temperatures, this would likely mean they are intentionally less breathable to be fit for purpose (though some breathability is still required since you don’t want that boil in a bag feeling, so it’s relative). Wearing my winter kit at 50º – whether running kit or cycling kit – is simply a no-go. I’ve got kit that I simply cannot wear above 40F.
At 50º, I wouldn’t be wearing knee warmers, just bib shorts but I would probably have lightweight gloves in addition to a long-sleeve, and maybe a gilet. That said, 50F was very cold when I lived and rode and ran in Southern California.