Really? Normally it's just like: "Can we access your contacts list so we can match your 'friends'?" And then you do it and it shows you the list of matches. It doesn't ask each individual "Hey, do you want to be matched to this person?" unless you try to friend them. Most ask you when you sign up and usually somewhere in the settings if you want to enable email matching for your account though. So if that's on it's kinda just part of the system unless you turn it off.
