were they adlibbing the play at the very end

Myrtle starts to change her lines during the performance and to adlib the text. Obviously she tries to find a way to go beyond what she thinks is shallow and depressing in the text. To change it into something not only more interesting and positive, but also into something she can relate to at all. During the rehearsals she feels like in a cage, so she has the need to adapt the play and the text to her personal problems as an actress. Otherwise she couldn't play it at all. She obviously can only play a part with her complete personality - or she can't play at all.
She's not a stupid method-mistaking actress with an inflated Ego out of control, but on the contrary a very professional actress, so she has to get drunk like hell, to cross this border, to have the courage to leave the road and to grasp the role with full personal risk. So she starts improvising to make ANYTHING out of it she can live with as an actress.
Maurice reluctantly goes along with her, but at last he's courageous enough to do it.

So writer and director (of course) are nearly dying. The audience, not knowing the written play, is just watching something very fresh, intense, funny, serious, surprising in any moment.... and many of them seem to like it.

Well, you could say more (and more complex things) about it, but that's about the way I see it.

Not only were the characters adlibbing....the actors themselves were adlibbing, in front of a live audience of over 1000 people.

The only scripted bits were the director, writer and producer, who as you say, didn't like it.

But the joke was on them, because in the end, the audience loved it.


=================================
http://kingturtle.com/movies.html
=================================

I have to say this isn't one of my favourite Cassavetes films but it's well worth sticking with it for this scene - it's absolutely wonderful. I assumed they were mostly improvising as some of the lines seemed totally out of the blue, and at points it looks like they are trying really hard not to crack up. (On the back of the DVD I have it says that they were real audiences, but I'm not sure about the play). What amazing energy and chemistry they had between them. It must have really been something special to have been in the audience.

I was a bit thrown off by that too at first. Also I couldn't really tell if the producers etc were pissed, in fact at first I thought it was just the writer who was annoyed and the others were just watching in fear that Myrtle might break down. After the film finished however I realised that they must've been improvising that scene, which would explain why its so out of tone with the rest of the scenes we saw (which frankly came across as a bit cliched, but I guess that was the point). Also unless I'm mistaken it's Ben Gazarra's wife who hugs Myrtle at the end, not anyone else. This is presumably because she had silently been in support of Myrtle throughout the film and perhaps could see that Myrtle was finally herself again.

One of the things I liked about the movie the most was how Myrtle's unpredictability made it hard to tell when and at what points she was messing up her performance. Sometimes she pulls it back like a pro, other times she just messes it up. The tension of whether she'll lose it carries the film in my mind.

The (rambling) Cassavetes audio interview on the Criterion edition says that the stage adlibbing was written. That is, it was scripted in the movie, but written to show the actors in the play within the movie adlibbing.