The purpose of this paper is to try to enumerate and briefly describe all applications and technics deployed for defeating Nmap OS Fingerprint, but in any case, security by obscurity is not good approach; it can be a good security measure but please take into account that is more important to have a tight security environment (patches, firewalls, ids, ...) than hiding your OS.
Learning which Operating System is running in a remote system can be very valuable for both the pen-tester and the black-hat. Suppose that they find an open port in their (approved or not) penetration; knowing the OS makes easier to find and execute an exploit against that service, because often an exploit is OS version specific, and an exploit for Sendmail running on HP-UX won't work for Sendmail running on AIX, or being more accurate, an AIX 4.3.3 exploit could not work in a system running 4.3.3 with the latest maintenance code applied. Fyodor (Nmap's author) has written a detailed article about remote OS Fingerprint, describing some different methods to successfully detect the remote OS, from the basic ones, to the more powerful ones.
In the beginning, guessing the remote OS was done grabbing the banner that a specific service was serving. For example, a typical telnet or FTP banner was always shown to the entire world, telling which OS was running, or if the banner has been changed or removed, some service commands could be executed to know the OS (remember the SYST in the FTP). Other basic ways to know the OS could be searching for HINFO entries in the DNS server, or trying to get information using snmp (lot of devices have enabled by default snmp access using the 'public' community string). Even searching for the target company jobs posting in the Internet, dumpster diving looking for OS manuals, or social engineering are valid methods for trying to know the remote OS.
Then, some more advanced network solutions were deployed, taking advantage of each OS vendor TCP/IP stack implementation. The idea is to send some crafted packets to the remote OS and wait for its answer. Those packets are "nasty" packets, crafted with uncommon TCP options or with 'impossible' options. Each OS has its own TCP/IP stack implementation, there isn't a common stack implementation for every OS and this issue allows to create a classification of different OS and versions according to their answers. Playing around with those tricky packets is how remote OS Fingerprinting tools work; some of them using the TCP/IP protocol, and others using the ICMP protocol.
There is a paper about '
Defeating TCP/IP Stack Fingerprinting' that describes in high level the design and implementation of a TCP/IP Stack fingerprint scrubber. That paper outlines why and how you can defeat TCP/IP OS Fingerprinting, so I am not going to talk too much about that; therefore I will focus on the solutions available out there.