I think audiences wanted a darker story then what they got

Gary Kurtz, who produced ANH and TESB, claims that there was a 'darker' and more nuanced finale planned, but Lucas threw it out and opted for the big happy ending. He claims that toy sales were a big factor as well as the success of Indiana Jones. Lucas' friend John Milius also claims that George had spoken of setting up a huge toy empire to make money, although I suspect Lucas was being ironic (the dude's funnier than folks give him credit for!).

Certainly Lucas was financing his own movies, so he may have felt some serious commercial pressure since Kurtz had gone over budget on TESB.

My guess, and it can only ever be a guess, is that Lucas simply wanted to adhere to his original vision of Star Wars as a fun thrill ride for kids and possibly overcompensated with the Ewoks and the teddy bear picnic. TESB was obviously darker and deeper than its predecessor, but I think George wanted to bring it all back into the light so to speak, rather than opt for the bittersweet finale that Kurtz was hoping for. Lucas wanted to preserve the 'fairy tale' aspect.

Personally I was (and remain) disappointed with ROTJ (I referred to it as 'Return Of the Merchandi' when it came out!). Like you I think it was a missed opportunity, but I guess it's a matter of taste.

I do think Lucas started to let marketing and merchandising affect his creative choices (namely using the Ewoks).

Kid friendly element is a crutch and a oversimplication of all this movies problems

1. Nuanced villiany fell out the window with palpatine. Palpatine is a one dimensional cartoon villian whose only motivation was power. In empire strikes back vader was much more complex and much more nuanced. Vader became secondary and thus less interesting. In gary version not only was vaders redemption much more nuanced and better. Palpatine felt like a lazy backdoor villian to create a quick forgiveness for vaders sins.

2. Skywalker overwhelming the storyline...In empire, new hope and TFA not only were the skywalkers In their proper place...but often times they were merely plot devices. In this film han and leia had a ZERO SUM stake in this fight. This film set the template for *beep* jedi plotlines...one temper tantrum away from being a psychopath apparently.

3. The first half of this film sucks balls. Not only because Jabba palace doesn't really move the characters foward. But because the point of the palace was to save han solo. Han solo contributes nothing to the plot...literally nothing. And yet we are forced to sit through a rescue

From what I've read, the sequences in Jabba's Palace are pretty much what Gary Kurtz had in mind when they worked on the initial outline for Return of the Jedi. Which is why the Jabba scenes feel like a totally different movie from the Ewok scenes.

I don't think they've ever released the entire Gary Kurtz version of the outline. Bits and pieces have come out over the years. For example, no Ewoks. Han would have died (as Harrison Ford wanted). Luke and Leia would not have been siblings. Vader would have died outright fighting Luke, instead if being redeemed.

Most importantly, the story wouldn't have ended with ROTJ. If Gary Kurtz had his way, we would have gotten the "Sequel Trilogy", 30 years ago. It seems that George Lucas was no longer interested in Star Wars at that point and just wanted to put it behind him.

Not to mention George wanted a VII, VIII, and IX at one point and then decided against it. A happier Jedi ending reveals George wanted Darth's storyline to end.

This article pretty much sums it up: https://popcultmaster.com/2015/12/16/body-horror/

It also contains an internet review from 1984, when the closest thing to an IMDB was net.movies (courtesy of Usenet).