I have a rim brake Allez Sprint, it’s ~7.3kg IIRC (~16lbs) w/bottle cages. Full DA w/Rotor cranks. Roval wheels.
If you’re at 20lbs, you’re doing it wrong 
Ditch the saddle bag or at least some of it’s contents. I only carry 1 CO2 and 1 tube. DA or Ultegra might save you some more weight but is it worth the cost. No shame in going compact if that is what is needed. You will be more comfortable and probably faster from not grinding. I’m riding 52/36 w/11-28 here. I know plenty of guys running 50/34 w/11-30 or bigger.
I live in Taiwan. It’s 85% mountains. I have multiple 1hr climbs 5min from my front door. The difference between my Allez Sprint and SWorks Tarmac SL5 is negligible.
Recently, I have been thinking about SRAM AXS on the Allez Sprint. Not sure how much weight that would save. Hell, it might add weight The attraction, aside from wireless, is the X-Range gearing. 48-35 w/10-28 or 46-33 w/10-33 are both quite appealing with all the climbing here.