The Phone Scam...

The phone scam they talked about - where the caller says, "Can you hear me"? and then by saying "yes", the caller is on the hook to pay something, because the scam consists of taking that "yes" response and attaching it to a different correspondence...How scary is that??

The only thing I don't get though, is how can they prove it is really you who says "yes"? Why wouldn't you be able to dispute that? There is no true authentication to prove that it is really the recipient of the phone call saying anything, so why wouldn't that person be able to argue that they didn't give permission for the caller to charge them for something?