I had the exact same problem with the Rally XC and almost brand new Bontrager Cambion shoes. It was driving me nuts. If you diligently search on line you will find 3-4 other people (it doesn’t seem to be common) with the same problem with SPD cleats and their shoes.
I ended up cutting up an old tire. I settled on using a 1" piece of the tire bead and super glued it onto the bottom of my shoe and it made things much more solid. The hack lasted thru Unbound Gravel, but I have no desire to deal with this kind of issue long term so I’ll be selling the pedals once I get around to cleaning them up. Normally I just use my road shoes and pedals (Favero Assioma) on my gravel bike.
For comparrison I have a pair of Crank Bros Candy pedals on my MTB bike and switch Bontrager Cambion and Rally shoes. With the Rally shoes I can use the Crank brother sole protector plate, but with the Cambion shoes I can’t or I’ll get that loose feeling. The Candy’s come with some rubber spacers you can swap out on the pedals, but I don’t want to hassle with that depending on the shoe so I just don’t use the sole protector on the Cambion shoes.