FOR THOSE FAMILIAR with the novel: [spoiler ahead]

This was supposed to be explained during the first scene where Michael confronts Tom -- unfortunately, in editing the end of the scene got lopped off. During their youth, Tom taught Billy about an old Indian curse called "The Song of the Cicada." While trapped in the basement, Billy invoked the curse upon himself so he could exact revenge. The backstory was spelled out much more clearly in the script, but it's pretty obvious that a lot of footage got left on the cutting room floor.

As for the novel, despite what the credits imply the screenplay was not based on a book. Edward Levy sold the film rights to the title "The Beast Within" before he sat down and composed the book, so when Tom Holland was hired to write the script he had to craft an original story of his own. Levy then used Holland's screenplay as the basis for the novel (which technically makes it a novelization).

Wasn't it mentioned in the movie that the original Billy had Acromegaly and that was why he was deformed?