Significance of Cardinal College

"Cardinal College" was the original intended name for Christ Church (founded by Cardinal Wolsey): this is why Colin Dexter called his fictionalised Christ Church "Wolsey College" in the Morse books. It was changed when Wolsey fell from power.

In other words, this isn't any old college, it's a deliberate reference to one of the richest, poshest and most conservative in Oxford.