It sounds like port knocking would be great for your network, granted your customers will need to get the proper sequence which could be a pain.

Try change FTP software, even windows FTP server is better than what you are using now.

Try changing the port to something else, at least this way he will have to show some initiative to find your FTP again.

Try Set-up a VPN so that they have to 2 factor authenticate before they can access your FTP