Double up.
The more layers there the better. I prefer a tights/pants with no chamois over another bib with chamois.
If you’re wearing two layers you can even put some extra cloth, towel, newspaper, whatever in between the two layers. If you place it correctly you don’t even notice it but keeps your bits warm.
What temps are you riding in? After rides in 30-45F / -1-7C, once inside I notice most of my body is cold. Also like that after skiing outside. I’ve not heard of cold damage to the bits, in fact if you are having trouble with baby making low swimmer counts one thing a doc will say is to switch from boxers to briefs to cool down that area. Not that its easy to find briefs in the stores these days…
All depends how warm you run. I wear shorts down to about 41F. On the top half I’ll wear a mesh vest and shelled micropile such as Rab Vapourise or some form of shelled alpha direct down to about 32F. Below that I might wear a merino layer between shell over the mesh vest and below the shell but no often. I usually stick to two layers in winter with venting options so you don’t get a build up of sweat uphill.
Trick to staying warm is not to overheat to start with, cold sweat in your layers or on the skin can reduce your insulation layers effectiveness significantly. Plus keep a warm layer of air against your skin, hence importance of wind proofing.
In the colder months I tend to also carry an insulated gilet which I’ll throw over the top if I’m stopped for any length of time.
Investing in a good technical shell is huge in my skiing experience. And that is what I did for the bike. A mix of wind stopping and breathable fabrics, strategically placed.
Time to break out the fat bike and cold weather clothes 
For clothing, the key thing is to experiment to see what works for you - as we each have different tolerances for the cold.
And also, chose your clothing so you can adjust in real time if you need to cool down or warm up I.e. Don’t get so hot where you sweat, and don’t get so cold where your extremities start to go numb. Eg zips, removable arm warmers, neck buff, layers to remove/add, etc.
Agree with figuring out what works for you, a selection of layers to be adaptable definitely the way to go.
Other thing not mentioned is that sticking to flattish routes makes it much easier to manage temperature when it’s cold as both intensity and wind chill stay fairly constant. Hilly routes are much harder as you can easily overheat when working harder on the climbs with little or no wind, then have all that residual heat stripped away in seconds freewheeling down a fast descent.
With those bar mitts you can go as high as you want!
Blocking wind chill seems to be the key for me but I rarely go out down here (East of England) if there’s any chance of ice these days as the roads aren’t fully treated (although my garmin say’s I’ve ignored that a few times 
). Back home where the roads are better treated, I’d go out in temps to circa -6deg C when I was younger. I think when I’ve been home since its been -3deg C at it coldest.
Had exactly the same problem…
I was 10kg less than now, at my lightest. I was freezing all the time.
The only thing that really worked was gaining a few kgs, well in my case 5 - 7kgs.
