He cannot control everyone.

He would not be able to control everyone. Some people have Anosmia, or a lack of smell. The perfume would have no effect on him.