I had made it clear that I was on the issue of Fundamental Right to Worship, says BJP leader Swamy