But wouldn't part of what made him worthy be partly because his being is worthy? And isn't his being part (IIRC) Stark, part Banner, part Mind Stone, part Jarvis and even part Thors' lightning? He's a gestalt being but part of that gestalt is the stone.
Yes, it's a stretch but not too far of one.
Worthiness and power are two different things. In the comics, Captain America has picked it up but the Juggernaut failed. The Juggernaut is a million times more powerful than Cap.
Also, you forget that androids can pick up Thor's hammer. In the comics, Ultron and Airwalker picked it up since they are robots. They just didn't get the power of Thor when they did it.