The only people who can really drain your power effectively are people you are very close to, you are open to, you love, or people you are afraid of, you fear.
All of us grow up in particular realities - a home, family, a clan, a small town, a neighborhood. Depending upon how we're brought up, we are either deeply aware of the particular reading of reality into which we are born, or we are peripherally aware of it.