Port Scanner Challenge

This has been interesting...

It started out as a comparison between nmap and UnicornScan and the recently released PortBunny. The results were interesting... and it spanned a couple of blog posts.

Port Scanner Challenge
And the winner is...
Port Scanner Challenge Update

What made this really interesting was that someone associated with UnicornScan jumped in and jumped all over the results... and then went so far to run his own "unbiased" tests... even though I'm independent and he's not.

Essentially, UnicornScan relies on the user to set the transmission rate... it apparently finds throttling to be too difficult of a task... In order to "properly" use UnicornScan you need plenty of experience and you have to be willing to run multiple scans to tweak the speed. Which apparently every network admin and security admin wanting to quickly scan a box, should have time to do... I'm assuming that some people have never worked in a busy environment. I mean 10 scans at 10 - 60 seconds per scan to get down to that 10 second scan, or 1 scan at 30 seconds (picking arbitrary numbers for comparison here)... the 1 scan is the clear winner. So yeah... it's proved interesting... I've also asked that each software's author provide me with the best string for their product (to optimize it to the max for a single, generic scan) and again I've been told you can't scan with UnicornScan unless you are willing to perform multiple scans to properly tune the speed.

Anyways... enjoy.

I strongly disagree with the methods used in your tests. (I'm Harry on your blog by the way, commented in the 'and the winner is' part)

Scanning with a default scan, with T5 set and no retries is ludicrious and is not indicative of a real world scan - in fact if you were looking foir settings that would guarentee unreliability these would be them.

I've alo read Fydoors, Robert's and your posts on the Sec Focus lists and would be interested in the test you proposed in your last email.

Let me know if you need targets to scan - I can put up a PIX 515E, 7 hosts, oracle, MSSQL,, SOlaris, all flavours of windows and a few Linux and have them on different subnets to put the scaners through their paces - could even through and IDS/IPS in - althouhg couldn't guarentee this in time.

Quote:

---

Originally Posted by Nokia

I strongly disagree with the methods used in your tests. (I'm Harry on your blog by the way, commented in the 'and the winner is' part)

---

:) Good to know

Quote:

---

Scanning with a default scan, with T5 set and no retries is ludicrious and is not indicative of a real world scan - in fact if you were looking foir settings that would guarentee unreliability these would be them.

---

You're right... the nmap settings (for example) were definitely not the best scan methods used... a simple -sS -P0 would have greatly improved the results...

Quote:

---

I've alo read Fydoors, Robert's and your posts on the Sec Focus lists and would be interested in the test you proposed in your last email.

---

Did you see the nmap-dev (I believe it was nmap-dev and not nmap-hackers) list as well? The -T5 and no retries settings came from Fyodor on that list, as he was pointing out how nmap could be improved to operate at the same speed as PortBunny... that was one of the driving factors in using those settings.

Quote:

---

Let me know if you need targets to scan - I can put up a PIX 515E, 7 hosts, oracle, MSSQL,, SOlaris, all flavours of windows and a few Linux and have them on different subnets to put the scaners through their paces - could even through and IDS/IPS in - althouhg couldn't guarentee this in time.

---

Excellent, thanks... I'm still waiting to hear from them... Robert is still fairly insistent that to be fair I run multiple scans and tune UnicornScan's send rate... I may give into this, but if that's the case I will be adding all of the scan times together. If I do get a response from them, I'm actually hoping to use our lab at work... I'll have between 200 and 300 live IPs to scan.

It would actually be good to describe the layout of your network to the relevant authors and ask them what switches they propose will be best for their software....then run them.

Each person would not know what they other programmer is going to suggest, however as the programmers they are suggesting what would work best with thier tools...

This would aim off for any critique from the authors afterwards about not using the right options for the scan..

I am actively testing unicornscan and Nmap myself as I have come across quite a few situations lately during PCI auditing where each tool has produced different results and I have had to go out on a limb when producing a final Pen test report about what services are contactable.

Unicornscan feels like it can do more that Nmap but I think this is because it is so bloody complicated to use it gives you the impression of being able to do everything you want from it.....where as Nmap is so simple to use you would be forgiven to thinking it lacks functionality when compared to unicornscan.......which it most definitely does not...

I've not really used Port Bunny, TBH I've not even heard of it that much.

//edit - IMHO I would really not take scan times into consideration, unless they are absurdly long - personally I don't really care if Nmap completes in 40 seconds and unicornscan completes in 60 seconds; it is reliability that would determine what port scanner I use - if one is more complicated and I have to learn the switches and wait an extra 2 minutes for the scan to complete, then so be it; in my line of work I can't afford to have inaccurate results...

Quote:

---

Originally Posted by Nokia

I am actively testing unicornscan and Nmap myself as I have come across quite a few situations lately during PCI auditing where each tool has produced different results and I have had to go out on a limb when producing a final Pen test report about what services are contactable.

Unicornscan feels like it can do more that Nmap but I think this is because it is so bloody complicated to use it gives you the impression of being able to do everything you want from it.....where as Nmap is so simple to use you would be forgiven to thinking it lacks functionality when compared to unicornscan.......which it most definitely does not...

---

I find that very interesting... most people, that i hear from, think nmap has too much functionality and is overly complicated... and from what I've seen of unicornscan, I find it's functionality to be limited... especially considering how complicated it is.

Quote:

---

I've not really used Port Bunny, TBH I've not even heard of it that much.

---

It's a kernel-based port scanner that was released recently.

Quote:

---

//edit - IMHO I would really not take scan times into consideration, unless they are absurdly long - personally I don't really care if Nmap completes in 40 seconds and unicornscan completes in 60 seconds; it is reliability that would determine what port scanner I use - if one is more complicated and I have to learn the switches and wait an extra 2 minutes for the scan to complete, then so be it; in my line of work I can't afford to have inaccurate results...

---

You're right... in the long run.. a few seconds doesn't matter... neither does learning the switches... For me, with unicornscan, the biggest issue is still that the authors are telling me I have to run multiple scans to fine tune it... That to me reeks of bad design... One scan should be sufficient.. maybe I missed ports that I didn't expect to be open.. How do I know I didn't have it tuned properly and that I need to scan again.. I'd rather unprecedented accuracy... and beyond speed..nmap also had the accuracy in my testing.

@HTRegz:

Of the three portscanners mentioned (nmap, portbunny, unicornscan), only portbunny is designed without tuning options available. Nmap has default settings which would make for very slow scans in filtered environments. Unicornscan stays at a steady 300 packets per second by default in the unicorn.conf configuration file. It is expected that the end user will know what line speed they can test at.

Point is, with nmap or unicornscan, you will likely run it several times with different options before finding that optimal setting for quick scans and high accuracy.

Also, please study up on the design decisions made by the various authors. By understanding the problems the authors were trying to solve, you will find test situations that should favor each particular tool. Running one round of tests against a flat unfiltered network does not represent what professional testers see today.

I meant for the testing I've done at http://loquens-caesu.blogspot.com/ to encourage others to begin testing for themselves. End users are the ultimate judge of what works for their environment. Don't take my word for it. Set up a similar environment. Use the options I did. You will find similar results.

If you're going to go through the effort to run a test, at least try to understand the tools you are evaluating. Otherwise you come off favoring a car with automatic transmission because you don't understand the efficiency gains of stick shift.

Welcome :) A simple post brings in new membership... I should ask for a cut of the advertising fees.

Quote:

---

Originally Posted by jeneral

@HTRegz:

Of the three portscanners mentioned (nmap, portbunny, unicornscan), only portbunny is designed without tuning options available. Nmap has default settings which would make for very slow scans in filtered environments. Unicornscan stays at a steady 300 packets per second by default in the unicorn.conf configuration file. It is expected that the end user will know what line speed they can test at.

---

I agree that nmap has to be tuned... that being said... nmap has some "fairly standard" argument sets. With unicornscan, you were telling me that I had to run several scans to ensure that I had the best possible speed... That becomes a fairly different tuning that nmap generally requires.

Quote:

---

Point is, with nmap or unicornscan, you will likely run it several times with different options before finding that optimal setting for quick scans and high accuracy.

---

See above

Quote:

---

Also, please study up on the design decisions made by the various authors. By understanding the problems the authors were trying to solve, you will find test situations that should favor each particular tool. Running one round of tests against a flat unfiltered network does not represent what professional testers see today.

I meant for the testing I've done at http://loquens-caesu.blogspot.com/ to encourage others to begin testing for themselves. End users are the ultimate judge of what works for their environment. Don't take my word for it. Set up a similar environment. Use the options I did. You will find similar results.

If you're going to go through the effort to run a test, at least try to understand the tools you are evaluating. Otherwise you come off favoring a car with automatic transmission because you don't understand the efficiency gains of stick shift.

---

Here's the problem... I can test drive a car with an automatic transmission and a car with a manual... and I can see the difference for myself... I can see pick-up, I can see mileage... That's without racing, without being an engineer... just by driving.

I took the port scanners out on a drive... I don't see what you are talking about with a simple drive... You are telling me that I need to be a stock car driver, or an automotive engineer to see your benefits because a simple drive isn't a proper test...

I think the analogy he was using is that if you only know how to drive an automatic car and then jump in to a manual you may not have a very good experience of using it.

Similar to if you don't understand how to use unicorscan properly then you will not have a good experience of this either ( and could be tempted to give it an unfavorable write up due to lack of knowledge?)

Quote:

---

Originally Posted by Nokia

I think the analogy he was using is that if you only know how to drive an automatic car and then jump in to a manual you may not have a very good experience of using it.

Similar to if you don't understand how to use unicorscan properly then you will not have a good experience of this either ( and could be tempted to give it an unfavorable write up due to lack of knowledge?)

---

I don't think that lack of knowledge has anything to do with it... Could I do a scan with several options... sure no problem (it's not like there are that many to consider)... is that what I was demoing? Nope... I've never once called my testing the be-all-end-all or even said it was definitive.

Unicornscan on default settings was sub-par to the other products... end of story... That doesn't apply to everyone's use case, that's fine... I know plenty of people for whom that is a very standard use case.

Quite honestly Robert's original emails to me were what was detrimental to my view of UnicornScan... prior to that I was interested in investigating it further... However multiple emails telling me I need to scan, tune and scan again until I find the ultimate settings for the network / device... that tells me that the tool isn't conducive to my general requirements of a quick scan. It might be great for people doing IDS/IPS evasion.. that wasn't what I was doing, nor is it something I have to do regularly, so it wasn't in my scope.

DISCLAIMER i am the author of unicornscan.

Quote:

---

Originally Posted by HTRegz

What made this really interesting was that someone associated with UnicornScan jumped in and jumped all over the results... and then went so far to run his own "unbiased" tests... even though I'm independent and he's not.

---

speaking for myself, i was confused about what your tests were actually
trying to measure. If you were trying to measure accuracy, I think your
measurements were too coarse. I also think your test scenario was
unrealistic compared to what a pen tester actually sees.

Quote:

---

Originally Posted by HTRegz

Essentially, UnicornScan relies on the user to set the transmission rate... it apparently finds throttling to be too difficult of a task...

---

Ignoring your mildly offensive tone here, i will point out that unicornscan
had a different design goal of being asynchronous and stateless, so i will
chalk up your misunderstanding here as ignorance of why stateless systems
do not handle retransmissions in a statefull way.

Quote:

---

Originally Posted by HTRegz

In order to "properly" use UnicornScan you need plenty of experience and you have to be willing to run multiple scans to tweak the speed.

---

this is true, it is designed for pen testers or other professionals who have
extensive understanding about networks, and experience measuring them.
But unicornscan is opensource, so if you have great ideas or specific things
that you dont like, i am more than happy to try and help. I have noticed
that some people really like unicornscan and others hate it, and i am OK
with that too.

Quote:

---

Originally Posted by HTRegz

I've also asked that each software's author provide me with the best string for their product (to optimize it to the max for a single, generic scan)

---

Minor correction, you did not ask me anything, but I may have told you the
same exact thing anyhow.

Quote:

---

Originally Posted by HTRegz

I don't think that lack of knowledge has anything to do with it...
Unicornscan on default settings was sub-par to the other products... end of story...

---

As I pointed out on your main blog, the real fault here was in your CPU's TSC (a common problem on systems with power management like laptops). If you had used -d2 (GTOD timer) instead, you would have had 0 missed ports, and scan times that matched the default pps of 300. Instead your results showed non-deterministic scan times and missed ports. The fact that you went on and published the results as such without doing any fact finding to understand the tool shows that you completely lacked any knowledge of the tool.

If you haven't seen it yet, go out and rent Buffalo '66:
[Trying to start Layla's car]
Billy Brown: Is this a shifter car? I cannot drive a shifter car, alright, so we got a little situation here. I can't drive these kinda cars! What the **** is goin' on! You think that's funny? Would you like to know, smartass? Would you like to know why I can't drive this kinda car? I'll tell you why, I'm used to *luxury* cars. Have you ever heard of a luxury car? You know what luxury means? Ever heard of Cadillac, Cadillac Eldorado? That's what I drive. I drive cars that *shift* themselves.

Quote:

---

When I started this challenge, I wasn't sure what the outcome would be... the only prediction I had was that unicornscan would be defeated by both PortBunny and nmap. This proved to be true...

---

I figured you were well intentioned and wanted help from the various projects. Now I'm just confused by your close minded nature. As I've said before, if you use the options I did on a similar network, you will find the same results. One can't say the same about your "test".

Quote:

---

Originally Posted by jeneral

As I pointed out on your main blog, the real fault here was in your CPU's TSC (a common problem on systems with power management like laptops). If you had used -d2 (GTOD timer) instead, you would have had 0 missed ports, and scan times that matched the default pps of 300. Instead your results showed non-deterministic scan times and missed ports. The fact that you went on and published the results as such without doing any fact finding to understand the tool shows that you completely lacked any knowledge of the tool.

---

Forgetting about the rest of it... and just looking at this.. You're right.. you did tell me to use -d2, and I did redo the tests with -d2... it didn't provide any additional assistance, the results were no better and in one case the results were worse... I didn't post the results since they didn't benefit unicornscan in anyway.

Personally I think the 'tests' where fundamentally flawed from the outset due to your lack of knowledge of when to employ the two very different scanners and how to use unicornscan competently.

I have been testing both Nmap and unicornscan for the past few weeks in work after noticing anomaly's in Nmaps' results, which is why I took an interest in your 'tests'.

I aim to post the results of these tests, ran on a variety of different networks and through some IDS's/IPS's (mainly snort) within the next 10 days or so, after making the results available to both authors if they so wish me to.