Why?? And spoilers, maybe.

They were always looking at her menacingly, and that pool boy warned her not to disrespect them anymore because they are bad people. Remember when she refused to give her beach spot to that pregnant lady.

And she didn’t just leave abruptly, someone came up to her and said something along the lines that it is time for her to leave.

That even kind of has some sense, she seemed drunk and making a fool of herself a bit, and she might crashed their party.

But all the other times that they looked at her weird, I always thought they will do something bad to her in the end.

It was a weird movie. Maybe they just looked at her that way because she was acting strange most of the time or it was all in her head. She thought they were looking at her that way? Who knows. Maybe it is better explained in the book.

Spoilers...

They seem to be some kind of "Sopranos" type crime family, a sinister component thrown in for no reason and completely unnecessary to anything the film was trying to establish about the main character. She could have had the same interaction with a young mother from a normal, unintimidating family (except the last scene where the bitch shanks her with the hat pin--also unnecessary).