In the end, I set right hand controls to control the rear derailleur: small button to move to the smaller sprocket and big one to move to the bigger sprocket. On the other hand I set left hand controls to control front derailleur: small button to move to the smaller chainring and bigger to move to the bigger chainring.
I have SRAM Rival 1x on the other bike so this is similar to it.