Get angry.

From Create Your Own Story

You feel your blood boiling, and you start to grow angry. "You listen here, Dap, I will not have anybody intruding upon my privacy and property in such a manner!" You push up a sleeve, preparing to punch him. "Get out of my house-"

"Ah ah ah, Chris, ol' pal." He says, putting the tip of his sword up to your stomach. "Wouldn't want to have to end your night on a worse note than we have to. C'mon, take a seat." He motions to the chair by the fire, and jabs lightly with the sword.

You can take him, sword or no. Fight him.

Take a seat, the bastard has a sword, and you don't want to take the chance.