It was a baby they threw and here is why...
Granted I haven't seen this movie in years but what I got from it (the song included) is that Billy Joe was gay, I think it was implied even before the mill incident. As a way to quell the rumors and also because I think he really did love the girl in his own way, they did have sex as a way to "prove" his straight status (at least to himself). She then became pregnant, kept it hidden from everyone but Billy Joe, and she lost it early, and THAT was in fact what was thrown off the bridge. With the loss of the baby, the guilt of EVERYTHING, the sex, the fact he knew he was gay, the mill thing, everything is what contributed to the suicide. Thats my take on it anyway...
Most people ignore the strange and unusual, I myself AM strange and unusual...Lydia "Beetlejuice"