In this case when I hear, "calling the man a racist" I hear, they are unhappy and offended that he used the word. I think that is a reasonable reaction, but assuming it was done from racism is problematic.
I have posted this elsewhere, but I wonder why every time there is a movie about any serious subject, or even some unserious ones that the actors are the ones to speak about the meaning and the history and the whatever ... when it should really be the writers or those who really lived it.
Actors are just cattle as Hitchcock said, or was it furniture?
Viggo should have shut up ... but it doesn't make him racist, and it's still a good movie.
Viggo was obviously intentionally offending anyone, I think he just was not aware of the effect his words would have on people, and the people who were offended or hurt ... what can they say ... it comes out as calling him racist.