IP address is by default insecure. Anyone can get that. The question is: Is your Operating System secure? Have you got a firewall? Have you got anti-virus software installed, up-to-date dat files and running regularly (including regular full system scans)? Are you cautious about attachments you open?

In addition, if he does "attack" your system, you can file a complaint with the ISP it's from. It is still illegal to break into systems in the US and -- in some cases -- can fall under the Patriot Act (but I don't think that it would in this case).

I suspect he's trying to simply scare you.