Any good villain wouldn't fight you directly. He'd find someone that is physically stronger then he is, but weaker of mind and use them to fight his battles for him.
Did The Emperor fight Luke himself? No, He had Vader do it. Did Goldfinger go fist-to-cuffs with Bond? No, Oddjob did. Did Sauron go out and try to kill Frodo himself? Nope, the Nazgul did.
If you are fighting a Villain, then you got a shitty Villain. If you want to get a real one, beat up the next guy that tries to get you to help him.