What's so terrible about aunt Connie going to her room?

Nothing is terrible about it. ...it's just her refuge when things get out of hand. At least that how I interpret it.

I agree with the first person who responded, that her room is her refuge. Also, during that time a woman of her class would likely see to the smooth running of the household and her absence would have a direct effect on Emsworth and Freddies' comforts. There would be snags in everything from the ordering of meals to making sure bills were paid.