Everyone out to ruin Christopher?

I admit I have not read the books and I have only caught sporadic episodes of the series, but what I don't understand is why is everyone (his wife, the wife of his best friend, his godfather, his brother, etc) out to ruin the protagonist? Christopher appears to be truly a decent man and everyone in good society in Edwardian London is out to get him. I don't get it and it makes the series unbearable to watch. Why does everyone assume Christopher is having an affair (at the beginning) when there is no iota of evidence that he is??? It's maddening!