I never thought Sebastian was in love with Charles
In all three sources (book, mini-series and movie) it seemed to me that Sebastian wanted Charles to be a sort of safe place from all things Brideshead, and that's what Charles never got. Not necessarily a lover.
I'm not arguing Sebastian wasn't gay, but his way of treating Charles was more 'best friends for life' (or, as they would put it, 'us against the world') than 'love of my life, never leave me'.
See how he beheaves with Anthony and Kurt, both unarguably his lovers.
In the book and mini-series he even pushes Charles away not because he couldn't get him to love him back (obviously Charles did), but because Charles stopped backing him up and took his family's side.
I always thought their relationship imploded not because it was a youthful thing (as suggested by Cara), but because it was, for Sebastian, quite serious and Charles did saw it as a sort of summer fling (wheter you think it ever came to be physical or not).
For me, Sebastian wanted to have in Charles someone who would love him for himself (thought perhaps not romantically), a point I think comes across very well in how Sebastian gets many a profoundly personal confession among his seemingly vapid conversation(and they always fly right over Charles' head), while Charles wanted only a charming character to have fun with.