Why didn't Martin kill the Rottweiler at the end.

Because he was still fond of the dog that he befriended as a young child a couple of years earlier -- the dog that Bartok kept alive in that mutant pit. It was showing that even as a gross fly monster, Martin still had some sense of empathy.

I thought he did kill it in the end, smothered it with chloroform or some other chemical to put it out of its misery, the poor thing.