Question about Elinor *Spoilers*

Now, I've read the book, but its been a minute, but I remember the movie very well, because I've watched it so many times. There was just something I didn't understand about Elinor. It was when they were all getting ready to go to the picnic at Delaford right before Colonel Brandon has to cancel. Willoughby shows up and Marianne gets in the carriage with him. Elinor is staring at them like she's unhappy. It seems to me like she didn't really like Willoughby, even before she knew he was no good. She tells her mother, "I'm very fond of Willoughby," but it seems like she never liked him. Am I alone here?