Same here, going from 172.5 mm to 165 mm really makes riding more comfortable for me.
The only thing I’d say to the OP is the (s)he should aim for a difference in crank length of at least 5 mm, but I’d probably go directly to 165 mm. The difference is definitely noticeable. And according to everything I have read, going to a shorter crank length doesn’t really negatives attached to them.