The hotel scene

When he walk in there and the manager says "sir either you can leave or we can call the cops and we can have you taken out". Was it the way he looked, the way he was talking, him wearing the underwear on the outside that got him kicked out, which do you think was the main factor for him getting kicked out of there? Or did they know who he actually was?