You were lucky enough to have him confront you himself. One of the Carrows was responsible for the other.

I couldn't say which Carrow, though; they're actually quite difficult to tell apart.