I hope this helps......

I had a similar issue and here is how I solved it:

Someone had the NESSUS program installed on RH8.0 so I completely removed it.
I went to nessus.org and got the latest stable release 1.2.7 I believe is the current stable release.

I installed it according to the instructions on the site.
I created a user but chose PASS instead of CERT for authentication
I installed the latest NessusWX client on my W2K box.
I created a new session and added all the plugins and IP addresses I wanted to scan.
Under COMMUNICATIONS, I chose QUICK CONNECT.

Now, I assume this is where you hit a snag right? There is output logged on the bottom of the cosole. Can you post that output if you get stuck here.

Anyway, at this point, I provide a password and it connects to the RH box and I now can conduct my scans.

There are some other things that I do like add the NESSUSD & syntax to the RC.LOCAL file so that the deamon loads automatically at boot time, then I dump the nessus-update-plugins in CRON.DAILY so that I always have the most recent NASLs available.