Knowing an IP address is kind of like knowing a house's street address. You can't get mail(packets) from someone without giving them your address. Only thing is you want to be resonably sure you lock your doors and bar your windows. Not unlike a suburban development, most ISP owned subnets are populated by row after row of relatively uninteresting denizens. If you don't have a static IP address( an address that never changes) or a known registered domain name(storefront w/ a sign) the odds of being specifically targeted are slim. However, there are those nefarious people out there wandering around rattling doorknobs(script kiddies) but they are usually easily sent looking for easier targets with basic precautions.

Hope this metaphor wasn't too lame...