Is Blood and Wine supposed to be this difficult? No silver swords?

I started off at level 35(currently 37) and almost every enemy I encounter is level 40+. I am getting decimated by bandits who are at level 48 not to mention the monsters that make minced meat of me. I do 1500 silver damage and 1100 steel damage and have 400 armour. All of my items are magic and imbued. My melee is all levelled up. I don't get it. Also, can you not buy silver swords in Blood and Wine? Have yet to find a smith that sells them.