Phil is not sleeping with Larry's wife.

As others have explained in other threads.

We don't know who Phil is talking to on the phone at first. When he says "you just missed him", we know it is someone who knows Bob. When he says, "I love you, too" it takes a second to figure out, but then we know he is speaking to Larry (and that he had called to apologize to Bob for grabbing him).

Earlier, Phil asked Larry if he loved him. Larry hemmed and hawed and argued and qualified. Larry simply could not answer with a simple, truthful statement. It isn't in him to say to a man's face, "I love you".

The reason the scene was done that way was to maintain Larry's character. Larry is so cynical and leery of real human emotion it took being pushed to his boiling point to actually admit that he loves and is faithful to his wife.

Larry could only tell Phil he loved him on the phone, not face to face. And to preserve the character's deep sense of privacy, the writer/director did not even let the audience see or hear his confession. We are expected to deduce it (and not that his wife and Phil are having an affair).