He broke your trust
Him being outed as a rapist bothered some many who looked at him as a hero, who have fond memories of Fat Albert and The Cosby Show, his comedy was built around quality family values which makes it feel like he betrayed your trust.
Goes to show never meet your heroes or idolized celebrities because you'll be disappointed.