Sounds to me that your hacker was just trying to get your attention, so get to securing it before he comes back to really do some damage.
Well, it seems to me if your router has been compromised it is only prudent to assume that the whole damn system has been as well? You may now be the proud host to all sorts of trojans, bots and backdoors, just waiting to phone home.

You obviously need to secure the router as suggested, but I would also thoroughly scan your PC(s) as well. Winston has a good point, but I feel you need to go further.

The textbook answer would be to reformat and re-install your operating system. Then restore your files from backups.

Cheers