Is my setup way off?

Yeah. Gotta be it, which is why my virtual speed and distance seem to parallel outdoor conditions on dumb trainer.
But
The strangest thing I could never figure out on the dumb trainer, before I had the stages crank which calculates cadence, my kirt Kinetic rm with virtual power was tracking cadence virtually and if I was holding 250 watts at 105rpm and switched gears and held 250 watts it would show a drop in cadence to 85rpm. How could it possably know I changed gears? There’s just 1 sensor on the roller the wheel is touching and it’s only counting revolutions. This is contrary to what you are suggesting, albiet on a dumb trainer that’s actually pretty smart. I’m assuming a smart trainer is doing the same. It knows. Some how.