For my second collarbone fracture (typical non-surgical recovery, I think) I was on the trainer after probably 10-15 days, and back outdoors in maybe six weeks. My doctor prefers a figure-8 brace instead of a sling, and that made it easier to use the trainer. Yes it’s uncomfortable on the trainer, but honestly riding the trainer was much more comfortable than putting on a shirt or trying to sleep.
My first collarbone fracture had been very stubborn and atypical, with zero progress healing for 12 weeks. My doctor suggested using ultrasonic bone stimulation as a last resort before surgery, and within a week of starting that treatment the bones were knitting together beautifully. Needless to say, for fracture #2, I had learned my lesson and started the bone stimulation immediately.