Not suing others does not mean that others won't sue you. If people are desperate enough to think that they can gain some kind of financial advantage, they'll sue.
Don't worry about your caddie. He may be an irritating little wretch, but for eighteen holes he is your caddie.