Three problems - structure, underdeveloped plot, characters (spoilers)

The structure of the movie is way too episodic. There is no focus on a story, a side story or character development. They just go through time periods with a few jokes to leave along the way. This works in something like Alice, but here people they meet are everything but interesting and they have nothing to do for the most part in their own stories. For instance, after killing the Minotaur, Connery has NOTHING to do for the rest of the movie. Oh, right, except that one pointless magic trick.

That's why the last part where they go after "the most valuable treasure" is the most interesting. You have a pretty tall order as a goal (stop Evil :), a good villain that you can have fun with and some fun henchmen as punch lines.

Plot - go anywhere in time plot has much more potential than this. So, the Bandits bring in heavy guns, space craft, cowboys, archers and knights. Cool. You know what would've been even cooler? To actually see these time periods and their recruitment attempts. Since every Bandit went in different time period to get help this could've been their little side story and a chance to get to know a bit more about them individually. It would've been more entertaining and poignant than the overused "Napoleon is a little person and has a complex" joke.
Also, the idea of a God as kind of a nihilist is barely touched upon and there was so much more room for that plot idea to develop. Instead, it becomes a weird punchline in the "evil microwave" ending.

And finally, the characters. The main kid is so bland, and I don't mean the actor. He has a character development scene at the beginning, then one with Connery and one attempt at a development scene when he meets God.
The Time Bandits have way too similar characters. When the little guy gets crushed in the end, they're all upset and his friend is even out for blood. Powerful scene... or it would have been if I had any idea who the late little guy was. Except for Vermin and their leader, other Time Bandits are pretty much all the same bland character.
Most of the side characters are very uninteresting. The Ogre couple and the Evil being the two exceptions. Other characters show promise, like the giant or God, or Robin Hood but they are just too one note.

And that's why the current imdb rating is very fair. This movie has a huge nostalgic value, phenomenal effects, good villain and some funny moments but it leaves a lot to be desired. It's not a bad movie, it just could've been so much better with little additional effort.