Why Quay brothers decline the cast of the twins

Quay Brothers were in Greenaway’s early film “The Falls” as two brothers. I heard that they were cast in Zoo to play the roles of the twin brothers as originally planned. But Quay brothers declined the invitation of Greenaway. Does anyone know the reason why? Does the movie in anyway insinuate their personalities or life or anything at all?