Mankind have a great aversion to intellectual labor; but even supposing knowledge to be easily attainable, more people would be content to be ignorant than would take even a little trouble to acquire it.
- Samuel Johnson
If you connect to a server, the server gets your IP address. It needs this because otherwise it can't send packets (or in the case of FTP, files) back.
You could use a proxy server, which would mask your IP address (the server sees its IP address instead), but obviously you would need to find one. Most ISPs these days don't seem to do FTP proxy, only HTTP.
You could try to protect your machine by uninstalling services, or installing a firewall, so that he can't attack your machine even knowing its IP (If indeed that's why you want to hide your IP).