"You did not tell us that you were not party to the case. We only got to know from the press," SC tells Subramanian Swamy