they could just be scanning for random boxes with the ports open or they may be finding you on irc or another chat prog that you use. It would be highly unlikely for them to randomly guess _your_ ip...