NMAP 3.70 Core Re-write is Out!
Yea man, just got the news....
Nmap hackers,
Nmap's 7th birthday is tomorrow (Sept. 1). Since the proper way to commemorate it is with a major new release, I am pleased to announce the immediate availability of Nmap 3.70. This release brings dramatic changes. The main port scanning engine has been rewritten from scratch to be faster, scan many hosts in parallel, and be gentler against target hosts and networks. Service/version detection also now functions against many hosts in parallel. The UDP system has been overhauled to work in conjunction with version detection and a new "open|filtered" state to avoid false open reports against filtered systems. Nmap now estimates completion time for port/service scans in verbose mode (-v) when they are expected to take at least a couple minutes. A "port scan ping" system can dramatically improve scan times against heavily filtered hosts. There is also a new --exclude option that allow you to skip given hosts or networks in a broad scan. For example, a host may be too sensitive or critical to scan at a given time, or a security admin may not be responsible for certain subnetworks. Or maybe you want to cease scanning CW.Com IPs because they keep sending abuse reports to your ISP :). There is a workaround for the Windows SP2 problems (improved from the workaround in 3.55-SP2), and MAC address reporting now works on Windows. There are dozens of other changes, which you can read about in the Changelog entries below.
Making Nmap faster was one of your top 5 priorities in the last Nmap user survey, and I hope 3.70 will not disappoint. Timing varies dramatically based on network/firewall characteristics, but almost all of the pre-release feedback has been positive. For example, Bill Peterson, an Information Security Analyst at Alcatel, regularly scans a million IP addresses over the Internet to keep the company secure. He reported that with 3.55, "my scans were running for more than two weeks". He switched to a 3.70 pre-release (incidentally on a somewhat beefier machine), optimized the option flags a bit, and was soon finishing the scans in under a day. I've been doing my own testing against thousands of machines as well. The time taken for the command "nmap localhost" on my primary development machine improved from 3 seconds to less than three TENTHS of a second. The command "nmap -T4 scanme.insecure.org" (a filtered-by-default machine) over my home aDSL line improved from 31.8 seconds to 19.7. Results involving multiple machines or UDP scanning are often far more dramatic than these. The official motto for this release is: So fast it deserves a CERT Advisory :).
If you do find a case where the UNIX version 3.70 is slower than 3.55, let me know. I'm afraid that speeds of the Windows version of Nmap may not have improved as dramatically as on Linux/BSD/Mac OS X. I had to spend time working around MS SP2 nonsense rather than focusing on optimizing for that platform.
You might expect that such dramatic changes to the core of Nmap would load the new release with bugs. There are always some, but I hope you will be pleasantly surprised by 3.70's stability. The nmap-dev list has admirably tested many pre-releases over the last few weeks. I would particularly like to thank Gisle Vanem, Eric of Catastrophe.net, Andy Lutomirski, Dana Epp, Mark-David McLaughlin, William McVey, Arturo "Buanzo" Busleiman, Bill Petersen, and Tom Duffy for major contributions to 3.70.
Here is the full list of significant changes:
o Rewrote core port scanning engine, which is now named ultra_scan().
Improved algorithms make this faster (often dramatically so) in
almost all cases. Not only is it superior against single hosts, but
ultra_scan() can scan many hosts (sometimes hundreds) in parallel.
This offers many efficiency/speed advantages. For example, hosts
often limit the ICMP port unreachable packets used by UDP scans to
1/second. That made those scans extraordinarily slow in previous
versions of Nmap. But if you are scanning 100 hosts at once,
suddenly you can receive 100 responses per second. Spreading the
scan amongst hosts is also gentler toward the target hosts. Nmap
can still scan many ports at the same time, as well. If you find
cases where ultra_scan is slower or less accurate, please send a
report (including exact command-lines, versions used, and output, if
possible) to Fyodor.
o Added --max_hostgroup option which specifies the maximum number of
hosts that Nmap is allowed to scan in parallel.
o Added --min_hostgroup option which specifies the minimum number of
hosts that Nmap should scan in parallel (there are some exceptions
where Nmap will still scan smaller groups -- see man page). Of
course, Nmap will try to choose efficient values even if you don't
specify hostgroup restrictions explicitly.
o Rewrote TCP SYN, ACK, Window, and Connect() scans to use
ultra_scan() framework, rather than the old pos_scan().
o Rewrote FIN, Xmas, NULL, Maimon, UDP, and IP Protocol scans to use
ultra_scan(), rather than the old super_scan().
o Overhauled UDP scan. Ports that don't respond are now classified as
"open|filtered" (open or filtered) rather than "open". The (somewhat
rare) ports that actually respond with a UDP packet to the empty
probe are considered open. If version detection is requested, it
will be performed on open|filtered ports. Any that respond to any of
the UDP probes will have their status changed to open. This avoids a
the false-positive problem where filtered UDP ports appear to be
open, leading to terrified newbies thinking their machine is
infected by back orifice.
o Nmap now estimates completion times for almost all port scan types
(any that use ultra_scan()) as well as service scan (version
detection). These are only shown in verbose mode (-v). On scans
that take more than a minute or two, you will see occasional updates
like:
SYN Stealth Scan Timing: About 30.01% done; ETC: 16:04 (0:01:09 remaining)
New updates are given if the estimates change significantly.
o Added --exclude option, which lets you specify a comma-separated
list of targets (hosts, ranges, netblocks) that should be excluded
from the scan. This is useful to keep from scannig yourself, your
ISP, particularly sensitive hosts, etc. The new --excludefile reads
the list (newline-delimited) from a given file. All the work was
done by Mark-David McLaughlin (mdmcl(a)cisco.com> and William McVey
( wam(a)cisco.com ), who sent me a well-designed and well-tested
patch.
o Nmap now has a "port scan ping" system. If it has received at least
one response from any port on the host, but has not received
responses lately (usually due to filtering), Nmap will "ping" that
known-good port occasionally to detect latency, packet drop rate,
etc.
o Service/version detection now handles multiple hosts at once for
more efficient and less-intrusive operation.
o Nmap now wishes itself a happy birthday when run on September 1 in
verbose mode! The first public release was on that date in 1997.
o The port randomizer now has a bias toward putting
commonly-accessible ports (80, 22, etc.) near the beginning of the
list. Getting a response early helps Nmap calculate response times and
detect packet loss, so the scan goes faster.
o Host timeout system (--host_timeout) overhauled to support host
parallelization. Hosts times are tracked separately, so a host that
finishes a SYN scan quickly is not penalized for an exceptionally
slow host being scanned at the same time.
o When Nmap has not received any responses from a host, it can now
use certain timing values from other hosts from the same scan
group. This way Nmap doesn't have to use absolute-worst-case
(300bps SLIP link to Uzbekistan) round trip timeouts and such.
o Enabled MAC address reporting when using the Windows version
of Nmap. Thanks to Andy Lutomirski (luto(a)stanford.edu) for
writing and sending the patch.
o Workaround crippled raw sockets on Microsoft Windows XP SP2 scans.
I applied a patch by Andy Lutomirski (luto(a)stanford.edu) which
causes Nmap to default to winpcap sends instead. The winpcap send
functionality was already there for versions of Windows such as NT and
Win98 that never supported Raw Sockets in the first place.
o Changed how Nmap sends Arp requests on Windows to use the iphlpapi
SendARP() function rather than creating it raw and reading the
response from the Windows ARP cache. This works around a
(reasonable) feature of Windows Firewall which ignored such
unsolicited responses. The firewall is turned on by default as of
Windows XP SP2. This change was implemented by Dana Epp
(dana(a)vulscan.com).
o Fixed some Windows portability issues discovered by Gisle Vanem
(giva(a)bgnett.no).
o Upgraded libpcap from version 0.7.2 to 0.8.3. This was an attempt
to fix an annoying bug, which I then found was actually in my code
rather than libpcap :).
o Removed Ident scan (-I). It was rarely useful, and the
implementation would have to be rewritten for the new ultra_scan()
system. If there is significant demand, perhaps I'll put it back in
sometime.
o Documented the --osscan_limit option, which saves time by skipping
OS detection if at least one open and one closed port are not found on
the remote hosts. OS detection is much less reliable against such
hosts anyway, and skipping it can save some time.
o Updated nmapfe.desktop file to provide better NmapFE desktop support
under Fedora Core and other systems. Thanks to Mephisto
(mephisto(a)mephisto.ma.cx) for sending the patch.
o Further nmapfe.desktop changes to better fit the freedesktop
standard. The patch came from Murphy (m3rf(a)swimmingnoodle.com).
o Fixed capitalization (with a perl script) of many over-capitalized
vendor names in nmap-mac-prefixes.
o Ensured that MAC address vendor names are always escaped in XML
output if they contain illegal characters (particularly '&'). Thanks
to Matthieu Verbert (mve(a)zurich.ibm.com) for the report and a patch.
o Changed xmloutputversion in XML output from 1.0 to 1.01 to note that
there was a slight change (which was actually the MAC stuff in 3.55).
Thanks to Lionel CONS (lionel.cons(a)cern.ch) for the suggestion.
o Many Windows portability fix and bug fixes, thanks to patch from
Gisle Vanem (giva(a)bgnett.no). With these changes, he was able to
compile Nmap on Windows using MingW + gcc 3.4 C++ rather than MS
Visual Studio.
o Removed (addport) tags from XML output. They used to provide open
ports as they were discovered, but don't work now that the port
scanners scan many hosts at once. They did not specify an IP
address. Of course the appropriate (port) tags are still printed
once scanning of a target is complete.
o Configure script now detects GNU/k*BSD systems (whatever those are),
thanks to patch from Robert Millan ([email protected])
o Fixed various crashes and assertion failures related to the new
ultra_scan() system, that were found by Arturo "Buanzo" Busleiman
(buanzo(a)buanzo.com.ar), Eric (catastrophe.net), and Bill Petersen
(bill.petersen(a)alcatel.com).
o Fixed some minor memory leaks relating to ping and list scanning as
well as the Nmap output table. These were found with valgrind (
http://valgrind.kde.org/ ).
o Provide limited --packet_trace support for TCP connect() (-sT)
scans.
o Fixed compilation on certain Solaris machines thanks to a patch by
Tom Duffy (tduffy(a)sun.com)
o Fixed some warnings that crop up when compiling nbase C files with a
C++ compiler. Thanks to Gisle Vanem (giva(a)bgnett.no) for sending
the patch.
o Tweaked the License blurb on source files and in the man page. It
clarifies some issues and includes a new GPL exception that
explicitly allows linking with the OpenSSL library. Some people
believe that the GPL and OpenSSL licenses are incompatable without
this special exception.
o Fixed some serious runtime portability issues on *BSD systems.
Thanks to Eric (catastrophe.net) for reporting the problem.
o Changed the argument parser to better detect bogus arguments to the
-iR option.
o Removed a spurious warning message relating to the Windows ARP cache
being empty. Patch by Gisle Vanem (giva(a)bgnett.no).
o Removed some C++-style line comments (//) from nbase, because some C
compilers (particularly on Solaris) barf on those. Problem reported
by Raju Alluri <Raju.Alluri(a)Sun.COM>
As usual, 3.70 is available from http://www.insecure.org/nmap/nmap_download.html, including Windows (.zip format) binaries.
