Did the father have an affair with the sister in law?

In the beginning car crash scene, the mom states she has been betrayed before and he says, "Once, 7 years ago". The little girl is only 5, so it happened before she was born. After he returns from the hospital, the sister is saying, "We never got a chance to talk about the... things that had been keeping us apart". Later in the movie, he tells the sisters husband that his wife hated her sister. I get the impression that maybe the infidelity was with her? She also talks about never having felt love, which is what Kate was battling with also (while having random sex with people, so maybe aunt Elizabeth was doing the same things Kate was?)