I did this. Cervélo Aspero GRX 2x11 with 3 wheelsets. First is the stock alloy 700c wheelset that hardly gets used. I might sell them. Then I bought some 650b alloy wheels to maximize my tire width (48 mm) for more volume and comfort vs maxed out 700x42. Then I bought Light Bicycle WR50 (Carbon, 700x25i) as my road wheels, running GP5000 28c (measures 30 mm on these rims) with latex tubes.
I threw on a comfortable carbon seatpost as well. The bike is very comfortable and very fast. My complaint with this bike is the Easton crank doesn’t have the right chainline for GRX, so I have a GRX crank on the way.
I recently took a vacation to an area with great road, gravel and XC MTB trails. I brought the Aspero with my road and gravel wheels and road all three surfaces (sometimes in the same ride). It was nice to not have to bring two bikes.
I also like that I only have to maintain one bike and feel more justified spending money on upgrades to it, since it gets more use.
Go for it