I believe he was David's son. In the book, David says that he and Julia were lovers on several occasions. He had proposed marriage in 1927, but she couldn't face the idea of being a headmaster's wife (particularly at Bamfylde, which she hated). He then got a letter out of the blue saying that she had returned to America and married her employer.
When David gets the letter years later telling him that Charles is his son, he mentally acknowledges that the boy looks like his brother who was killed in the mining accident in Wales. Julia also explains in the letter that she had rushed into marriage with her employer to, undoubtedly, give the baby a name and security.
Life isn't about waiting for the storm to pass, it's about learning to dance in the rain.