I use the same Zipp wheels as you and experience the same problem and it really annoys me. I ran the none NSW wheels many years ago and they weren’t as bad.
On another bike I run Roval CL40’s and they’re definitely a harsher ride. I can’t comment on wheel rub as that bike is running disc brakes.
The Zipp wheels are definitely faster and more comfortable than the Roval wheels. They also spin more freely compared to the Roval wheels.