Yes, I'm guilty. I am not in the best shape and it does affect my riding. When you are not "fit" it can make you unbalanced and physically drained. Yes, riding is exercise and it is meant to physically drain you, to a degree but not completely. It is hard work to ride and you use a lot of muscles. You will eventually build all the muscles you will need for riding from actually riding but your riding will drastically improve when you have other muscles to support your effort.
Cardio is also very important. I don't know about you but when one of my horses decides to give me a hard time I can get winded. I have actually gotten to the point with one of my horse, that when we were working on transitions I actually became ill. I had overexerted myself doing a simple ride.
Just like we love our horse, we have to love ourselves enough to take care of our bodies.