Best musical appearance

I'm going to say it was Elvis Costello and "13 Steps Lead Down" from the season 3 episode "People's Choice".

Runner up would be Shawn Colvin from season 4's "0.409"