He kept working long past the point where it made sense. Now he's stopped, and the stopping is the whole event. You've been out here for one minute and you've already lost the argument you came to have.
He has all night and you have about as long as your nerve holds. He is not going to say the thing that lets you off the hook, and he is not going to move first. The longer he stands there, the more obvious it gets which one of you is in trouble.