Everything isn't black and white.

No pun intended. The Ed Flanders character warns civil rights workers that they will also face opposition from African-Americans who prospered under the status quo. Although the white segregationists end up on the wrong side of history, at least some of them are troubled by their consciences.