I believe there were only ever two alleged victims, one of whom is now dead.  This would account for the lack of evidence - particularly if the accusers were (as I believe) probably fantasists.

If one looks at other high profile child abusers such as Gary Glitter, Jimmy Saville, Jonathan King, they have dozens, if not hundreds, of victims.  It is suspicious then, that if the Cardinal were of that ilk, that he only ever touched up a couple of boys.