was this ending closer to the books ending?

I would say not at all.

very dissimilar in plot and ending

In the book, a couple on their space yacht find a message in a bottle written by a journalist, Ulysse Mérou who accompanied some scientists on a voyage to a planet orbiting Betelgeuse. They find that it is populated by primitive humans and are soon captured by apes. Ulysee is eventually able to learn the ape's language and convinces Zira and Cornélius that he is from another planet and arrived on a spacecraft still in orbit. The apes are at a technological level of Earth's late 1950's and are able to put satellites in orbit. Zira puts Ulysse, Nova and their son on their rocket place of the human test subjects. They are able to board the spaceship and head home to Earth. When they land, they find that Earth is controlled by apes. They escape back into space and release their message in the bottle. It is revealed, that the people at the beginning of the book are chimpanzees who dismiss the idea of intelligent humans as fantasy.

Both films went with advanced apes, but not with space technology. This film has the protagonist on another planet and arriving home to see Earth controlled by apes. Other than the ending and the alien planet controlled by apes, the book and this film really have nothing in common.