Did they really think the reactor would create a meltdown?

I believe that Cohaagen was advised by a reactionary physicist/planetary scientist who believed that it would. At one time, you may recall, before experimenting with nuclear fission, there was a widely accepted belief that a hydrogen bomb explosion would cause a massive hydrogen atom chain reaction which would simultaneously vaporize all ocean water.

Pretty silly, right?

Cohaagen believed that the Turbinium reactor would do the same thing to all the Turbinium on the planet, a belief which also conveniently allowed him to continue his stranglehold on air distribution on Mars (a la President Skroob in "Spaceballs").

-Rod

Would they have studied what was in the ice as it turned out to be oxygen or done simulations on what would happen?

No, I suspect Cohagen WANTED to believe it.