Walter Matthau's son Charlie claims that Matthau was offered "Dirty Harry."
But it seems that about 40 ranking male actors of the time were also offered the role. Often, scripts aren't sent out "to one actor, one at a time" but to ALL actors, and then things narrow down to who is available and who the director wants.
Frank Sinatra was actually signed to play Dirty Harry(though the movie had a different title), but backed out over a hand injury. Next, Paul Newman turned it down("too right wing.") Those are "actual offers" before Eastwood got the role.
But I have also read that Harry was offered to : Steve McQueen(who found it too close to "Bullitt"), Walter Matthau, John Wayne, Robert Mitchum and Bill Cosby! Frankly, I can see any and all of those guys in the role.
When Sinatra had the part, Harry was an old cop facing retirement. Eastwood considered graying his hair for the role, but had it "younged down" -- a lucky break. Thus he could do all those sequels and grow old in the role. Sinatra, Wayne, Mitchum...all would have been too old for the sequels.
Walter Matthau was a very good, well-respected star during his time. He came up "the hard way" as a character actor, and then was hot from about 1968(The Odd Couple) through 1980. Rather like Michael Caine and Gene Hackman, Matthau's movies didn't always hit, but he had ENOUGH hits(The Sunshine Boys, The Bad News Bears, House Calls) to remain bankable for quite some time.
And he was a good actor, a lot of presence.
"Charley Varrick" was the first of three-in-a-row attempts by Matthau to "break the comedy mold" by making serious crime thrillers: he made "Charley Varrick" "The Laughing Policeman" and "Pelham 123" in order. All three were really good movies(though Policeman was a little weaker). And they didn't do well, and back to "The Sunshine Boys" and "The Bad News Bears" he went.