Was Milner older than the other guys?

That was always my impression as well.

[deleted]

It's stated in the movie that he's twenty-two. His activities (building hot rods, cruising, drag racing, picking up girls) are perfectly appropriate for his age. His participation in them is not "living past glories", but perhaps it looks that way from the immature point of view of his seventeen-year-old friends. I remember that when I was a teenager people in their twenties seemed quite middle-aged.