Did Hauser subconsciously know Hillary's character was his daughter?

Yes, he knew it subconsciously.
When he says his daugther was abduced whe she was young, it became pretty clear who she was

I dont think he knew she was his daughter but he felt a fatherly affection for her since she was about the same age as his "dead" daughter but it clicked for him that she was his kid