There could be a few answers to why Molly's dad let her go out swimming with Ryder...
If they had some kind of 'incestuous' encounters with Ryder's mom, perhaps he felt traumatised by it. Basically, she was the initiator of those games, and Keith, not being bright enough to cope with it all, later blamed his sister for it, although he enjoyed it too. When he grew conscious enough to understand the wrongness of their actions, and not being able to deal with that, he simply put the blame on her. She left, because she "felt better than the rest of them". So THIS was all Keith's twisted plan to get back his sister's Californian family. To involve Ryder and confuse him too, make him feel guilty and dirty, and not "better than them". Just like Keith's sister did to him. Whether Molly's dad planned it or went along when he realised about the chicken-fight repeating with his daughter and Ryder - is another question.
If that's he case, another sub-question here is - does Keith feel bad only because what they did with his sister, or he's hinting (to his sister, his "partner in crime") that their actions made him into a (sorry!) chicken-fight addict of sorts and he started doing it with Molly? In that way it's somewhat (sadly) ironic that it's Keith, not Ryder, who eventually came out.
But I wouldn't want to believe there's this darkness in the movie, simply because it ended so lightly, with Bowie's voice singing Underpressure.
Maybe it's just that Keith noticed Molly's 'weird' (it is to him) behaviour earlier, before they came, and he got a little worried - that his sister's behaviour is repeating with Molly. Imagine Keith's shock when his own daughter would start rubbing against him, just like with his sister in their childhood (or teen years?) And then it's the family reunion and it happens again with Ryder... Perhaps, it's not even "revenge", maybe rather just "now feel the shame I feel and live with it". In that way, of course without caring enough about his daughter who could be traumatised by these weird games, he wants Ryder to fall into that rabbit hole of confusion.
Another theory is that he simply wanted to test Ryder. Perhaps he was sneaking up on them and was in control (anyway, Molly ran away from Ryder and returned to her dad as instructed). He was testing Ryder in the dining room, in Molly's room, and down the river. He made sure before Ryder "trusted" him, as in - trusted that Keith could/would harm him if he found out Ryder's been perversive. He wanted to make sure, to himself, that this was just a weird misunderstanding, that it doesn't correlate with whatever wrong they did with his sister. He needed to know, so this was his awkward Robert de Niro of Meet the Parents method.
Of course, it all could mean far more sinister things. That Keith is abusive with Molly. That he and his sister did more than "chicken-fight". That Ryder could be Keith's son. If latter is true then it explains Keith's "concerned" behaviour when Ryder is drawing Molly a picture and his explosive reaction when he realised his worst fears may have come true and Ryder, his son, could be involved in something sexual with Molly, Ryder's sister.