Do the writers think we don't pay attention

No... they don't think we are dumb and forgetful. But they do think that we pay attention and can pick up subtle hints and know the background of the stories we are watching.

This is where it helps to remember that Mjolnir is enchanted and "knows" when a being is worthy to wield it; picking up the hammer has NOTHING to do with strength and everything to do with being considered worthy. This leads to two theories as to why Captain America was able to use Mjolnir in Endgame.

1) Steve Rogers, through all his actions of kindness and self-sacrifice, is finally deemed worthy to wield the power of Mjolnir in Endgame. Now that he is deemed worthy, he is quite capable of picking up the hammer and using it against Thanos.

2) Steve Rogers was ALWAYS worthy of lifting Mjolnir, but did not do so in Ultron to avoid embarrassing Thor by having a mere mortal lift the hammer. Go back and watch the scene again... while no one else is even capable of making the hammer budge, Steve actually has it shift ever so slightly. Even Thor's attitude during the scene changes when he sees this happen. Steve then stops, on purpose, to save face for his friend. This also explains Thor's line in Endgame where, shortly after Steve wields Mjolnir, he shouts, "I knew it!" As if Thor figured Steve was always worthy, but downplayed his ability due to his humility.

I prefer to think that the second theory is the actual reason, but either theory are totally consistent with the universe laid out so far in the MCU. Here's a good article that goes into far more detail than I could here:
https://screenrant.com/captain-america-lift-mjolnir-avengers-endgame-explained/

Or here's a second article that quotes the directors on this subject directly:
https://collider.com/why-could-captain-america-hold-thors-hammer-in-avengers-endgame/#images

So there were plenty of hints leading up to the moment that Steve picked up the hammer in Endgame that he was always worthy, or was CLOSE to worthy, and that events between Ultron and Endgame were the impetus that Steve needed to become worthy.

Maybe Cap didn't want to pickup the hammer then. You can see the worried look on Thor when Cap grabbed the handle like Thor maybe thought he could pick it up.