My firewall and bastion host keep thing quite safe, but every day I analyze the syslog from the router. On average there is about one refused connection attempt logged every 30 seconds day and night. They come in bunches of hundreds before all is quiet for a few minutes. Repeat. That said, I do have a web, ftp and mail server at home.

As long as you keep your firewall secure and up to date, you shouldn't worry.