What you are experiencing is likely a fatal combination of all three factors, frame geometry, frame size and wheel size. Wheel size is a matter of taste, and frame size depends on too many variables like arm and leg length in addition to height. I reckon you could make S and M fit, but S might have a geometry that is more to your liking.
Geometry for sure plays a big role. The Trek Fuel is a trail bike, and trail bikes have become significantly longer and slacker than what mountain bikes were about 5-10 years ago. Just think of the difference between a relaxed endurance road bike and an aggressive race bike. Now multiply the difference by a factor of 2 or 3.
Trail bikes are nowadays designed to be stable and capable at fast speeds. Hence, the slack head angle. That also makes them more sluggish in slow-speed maneuvers.
If you want a bike that handles more quickly, you need to look at XC bikes, although even they have gotten quite slack. If I were you, I’d keep the bike for half a season or so and then I’d go to a LBS and try out a bunch of different bikes. If you find one you like, sell the Trek and buy a new one. Mountain bikes — especially trail bikes — are significantly larger and heavier beasts than svelte road bikes. So you need to get used to that. The slack geometry, which you currently perceive as lazy, actually serves a purpose that is anything but lazy. It makes your bike more stable at speed, so in a sense, you may be just going too slowly