What I can say is…
Mixing an matching lines may be working but is always risky. Crossing lines of the same generation will have more luck (esp if more recent ones.) It just takes a tiny cable pull distance difference on the shifter to make a the derailleur ‘never right.’
Lower line develop more “play” over time. To keep the prices low manufactures use less expensive materials and lower manufacturing tolerances. Unless extremely well maintained they will develop a lot of “slop.” What you are calling ‘rough.’
If the frame isn’t too old, I’d upgrade the group set. If you want new then look at 105. I’d buy everything used. Talking to local shops, you might find a “take off” set — when someone buys a frame with a high spec group set and the shop needs to sell of the “used” but never ridden group set. All bike parts are hard to find these days…but 105 is the most often upgraded to Ultegra — making 105 used pretty common. Keep everything the same series (generation.)
Modern 105 was also designed for a wider gear range to work in the “Age of Gravel.”
Just one persons thoughts.
Collect several more.