Moral Dilemma on the tennis court

Yesterday I experienced a moral dilemma on the tennis court. I was "rallying "(playing for fun) with this old dude. The rest of the people had left the court and he was gracious enough to tell me, "I play with you". He was a foreigner. He was either Asian or middle-eastern but one thing was unquestionable, the dude was old. When God said, "Let there be light," he flicked the switch. You know what I'm saying? The dude was old and could barely move, so I took it easy on him. I hit the ball soft and directly to him. After warming up with him he challenges me to a match. "Wanna play a game?" he asks me. What I wanted to say was, " you are one sneeze away from a coma, I will destroy you!". But I took a more diplomatic approach. " Let's just hit for fun," I tell him. He was relentless. "Let's play a match". Once again, I tell him. "Let's just hit for fun". He challenges me again to a match. I said o.k. It's on like Donkey Kong. Here is my dilemma. Do I "take it easy" on him? Or do I play to win? I was conflicted. Letting him win isn't honest. Plus my philosophy of playing sports is If score is being kept, play to win!. The guy serves super slow and I hit it back and he grunts and misses all my shots. I play to win. I'm killing him. Literally and metaphorically. After 4-0, he sheepishly asks me, "Can we just rally?" I tell him, "that's why I wanted to play for fun." In otherwords this corpse had no chance to win and it was almost insulting to think that he thought, he had a chance to win. We rallyed for another 40 minutes or so and it was good. But I can't help but thinking, was I a dick? I mean, I did give him the opportunity to play for fun twice. Right???