Why did Frank ...*SPOILERS*

"out there is family, in here is...hell."

Joe's not just talking about the bar, he's talking about Frank's being alone, in his own head, with his demons.
(there's a saying in AA - treat your mind like a bad neighborhood, don't go in there alone)

There's a reason the bar is empty that has nothing to do with the time of day
frank stays frozen to his barstool
into the evening as the bar fills up and life goes on around him
and without him

frank turned his back on his family
and cannot stand himself for doing so - he's enraged at his own weakness
and his fury is at his own paralysis
Caesar symbolizes everything he hates about himself

also - i have a theory that Ceasar may have been partly responsible for Frank's attitude - maybe it was a combination of alcohol and Ceasar's 'teaching' that turned the cute kid brother into a violent troublemaker.
Frank's philosophy is to do everything 100 percent - no compromise - and that afternoon in the bar he found out where that philosophy had got him, and regretted it.

'wind the frog'