IMO I would say the attacker. They both take skill, but attackers usually have more expertise. What I mean by that is a system administrator might be a specialist in locking down a windows server, but attackers have expertise in many operating systems because they usually do not limit themselves to one specific type of hacking.

The security admins can lock down things by general permissions and knowledge of the OS comes into play in that respect, but a lot of it is staying up on patches. If an attacker goes up against a system that has locked down all known vulnerabilities, it requires a lot more work than people give them credit for.

Another thing is the whole pressure situation. When you attack, especially illegally, the pressure can get to you. A security admin at most can lose their job if they fail at securing the system, which rarely happens, the company just reacts and locks down the system. An attacker fails at an attack and he goes to prison usually. That factors in a lot, especially when your actually in the middle of the attack.

Also attacking isn't just getting in, it is everything. Footprinting, scanning, etc like one of the posters quoted from text. You have to do a lot of work just to get in, then you have to do more work to clean the logs, get admin access if you only got user level access, plant backdoors and other accounts for a return to the system if needed.