Why didn't black people just organize and fight back against the Klan?

I mean most have no problem fighting back against racism today in an aggressive fashion if anyone even dares call them the 'N' word, so why is it they didn't they do the same back then? They were free people that would've been well within their rights to arm themselves and shoot any Klan members that threatened them, just like any other law-abiding citizen. I think it's pretty stupid that they would just leave their homes after all the efforts they put in trying to break away from slavery.