The real reason you don't like this movie
Yonkers Joe was silly, uncomfortable to watch and poorly made on first viewing. The stakes seemed low, the "full-retard" acting, the pathetic production values (walking into a Rain Man suite that looks like a Holiday Inn). But as I watched it again recently I realized that none of those things factored into why Yonkers Joe is so laughable. The truth is that it's too authentic.
The near rape, the clueless dad, the ridiculous idea of bringing a con movie together with a dad/retarded son story line... they're all a lot more "real" than we see in movies, even independent movies. The grimy locations are all real, too. The bust-out nature of the characters, as well. The problem is that the writer/director took all these very real, hard to look at pieces and tried to make a clean Hollywood movie. (quick aside: he's a terrible director, he shot it like a TV movie -- so boring)
If this movie was shot by an indie filmmaker or dogma guy on 16mm with a handheld camera it would have been a different film. Changing some of the actors out wouldn't hurt, but the big problem is that the look and structure of the film tell the audience it's entertainment while the story is too raw, ugly and human to be given the Hollywood treatment. To pull it off, you need to be extraordinarily talented (Kids are Alright is an example of someone being able to do just that, but they are few and far between). Otherwise, you need visual clues to show the audience that this is going to be real and ugly and tough to watch in patches.
Anyway, the movie's still terrible.