There is a long thread about issues like this on shimano drivetrains on the weighweenies forum which appears to be caused by noise from the big chain ring. I’ve posted on there multiple times user this same handle. I would look through that one just to see if it is the issue. I thought it was the RD too but turns out it was the front big chainring causing the noise.
Search for the Ultegra 8050 noise thread and I think you’ll find it.