Surprised Burt Lancaster was never considered

Why the heck wasn't Burt Lancaster considered for the role of Tony Wilson? He had previously worked with John Frankenheimer on 4 movies and they worked well together. Frankenheimer wanted Larry Olivier. I don't understand that. Two years later Burt Lancaster made The Swimmer. He was in his 50's but he looked great in that. He was in such great shape and still looked young. He had the body he had when he made From Here to Eternity. I loved Rock Hudson as Tony Wilson because he was perfect for the part. He was cast probably on looks alone. But then he gave a great performance. It's one thing to look good in a movie, but to give a great performance on top of that, is something else. Rock Hudson was great. But I think Lancaster could've pulled it off.