R0n1n's got the right idea with Nessus, it's a great way to find any vulnerabilites that are known. The other major thing is to upgrade any services when a vulnerability is found, don't leave it too long or someone may find out about it, especially if this server is a major server somewhere. If the server is supposed to provide services to a specific network only then look into a firewall of sorts, even if it's only a software based one. They can be set up to block access from certain networks when your regular services don't allow something like that. DoS can be hard to stop without some sort of filter on another computer such as a router as it can be as simple as sending so many connection attempts that the server locks up and stops responding, even if you have something to filter packets it may not be enough as only so much traffic can travel over the network and it can still cause a slowdown if not a complete disruption.