Small spoiler ahead:
Don't you think he was quite presumptuous at moments - especially when Sam told him to leave his house and he took his horse even though he's a bad rider which is why horse so dear to Sam broke her leg and Sam had to kill her. Charles liked Henrietta too, but he really got over the line a couple of times and it took him a while to back-off.
It wasn't just Adare. There was a general disrespect of Flusky. He wasn't on their level not only because he was a convict but also he was not of their class. Milly made reference to it many times when she was trying to get under his skin.
They were in a mixed up world. Adare was an aristocrat but it was Flusky who was helping him. His cousin was governor but it didn't really come with any perks the way it would in English society. Flusky invited him to the dinner party at his house hoping it would attract the society women who had always refused to come. In British society, it would have worked but not there.
Adare was also Irish which some looked down on but he was still a gentleman. In Sydney he was there like any other person trying to make a name for himself and to acquire wealth. He had to work hard to achieve it, his family ties wouldn't do it. On some level he resented Flusky and blamed him for what Henrietta had become. It was out of concern for her that he did the things he did but underlying it was his inability to make sense of the new world in which he was living.
Woman, man! That's the way it should be Tarzan. [Tarzan and his mate]