I haven't read the book, but I know that movies often water down the books. Lack of character development, lack of narrative, and over simplification of the plot are three of the worst examples. Furthermore, it should be noted that this movie was released at a time when studios were pressuring film makers to keep movies to around 90 minutes to get more showings per day. Around the early 90s, that problem was solved as movie theaters routinely started showing blockbuster movies on more than one screen simultaneously.
I suspect that Molloch's fixation with his sister in the movie was a plot device to make the climax in particular, and the last 30-35 minutes of the movie in general, work. For what they had in mind, it was necessary to have someone who Molloch cared about. Furthermore, I think that Claire was revealed to be a lesbian to diffuse any speculation that the two had an incestuous relationship. Ironically, Joseph Maher, who played Molloch, was gay in real life.
I do, however, disagree with you on the casting. While I've read before that Bronson was not a good choice to play Holland as represented in the book - Bronson was in his early 60s at the time, while the character in the book was 28 - I think he does fine for what he had to work with and that the supporting cast was excellent. I consider Molloch to be one of the most chilling bad guys I've ever seen in film.
Most movie bad guys are at least somewhat humorous, at least in a dark way, and often comic book like. But Maher portrays Molloch has being intellectual, calm, and getting great pleasure out of what he does. That's a significant part of the reason that the opening scene is the most disturbing I've ever seen. The electrocution of Hidalgo's nipples and testicles, an his extremely convincing screaming, would be hard enough to endure as it is. But there's even more so right after Molloch articulately describes the physical and psychological aspects of torture. He knew exactly what he was doing. He's one of the very few bad guys I've ever seen who truly was the personification of evil.
The first scene is also maybe the biggest example I've ever seen of a high risk/high reward introduction. I'm sure that lots of people have turned the movie off or walked away during the first scene. But if you get passed that, and the descriptions of torture from some of his other victims, you're probably going to be waiting intently for him to get his.
All in all, not a masterpiece, but a pretty good 80s action movie. If we absolutely, positively had to have another Bronson revenge flick by 1984, this is about as good as could be reasonably hoped for.
And the exchange between Holland and Randolph in the bar is one of my all time favorite examples of hilariously bad dialogue!