Here is some good info from the man pages of nmap:
-sF -sX -sN
Stealth FIN, Xmas Tree, or Null scan modes: There are times when
even SYN scanning isn’t clandestine enough. Some firewalls and
packet filters watch for SYNs to restricted ports, and programs
like Synlogger and Courtney are available to detect these scans.
These advanced scans, on the other hand, may be able to pass
through unmolested.

The idea is that closed ports are required to reply to your
probe packet with an RST, while open ports must ignore the pack-
ets in question (see RFC 793 pp 64). Filered ports also tend to
drop probes without a response, so Nmap considers ports
"open|filtered" when it fails to elicit any response. If you
add version detection (-sV), it will try to verify whether the
ports are actually open and change the state as appropriate.
The FIN scan uses a bare (surprise) FIN packet as the probe,
while the Xmas tree scan turns on the FIN, URG, and PUSH flags.
The Null scan turns off all flags. Unfortunately Microsoft
(like usual) decided to completely ignore the standard and do
things their own way. Thus this scan type will not work against
systems running Windows95/NT. On the positive side, this is a
good way to distinguish between the two platforms. If the scan
finds open ports, you know the machine is not a Windows box. If
a -sF,-sX,or -sN scan shows all ports closed, yet a SYN (-sS)
scan shows ports being opened, you are probably looking at a
Windows box. This is less useful now that nmap has proper OS
detection built in. There are also a few other systems that are
broken in the same way Windows is. They include Cisco, BSDI,
HP/UX, MVS, and IRIX. All of the above send resets from the
open ports when they should just drop the packet.

And here is some more from another source, good reading :

FIN (-sF), NULL (-sN) and XMAS (-sX) scans are all similar. They all rely
on RFC-compliance and as such don't work against boxes like Win95/98/NT or
IRIX. They also work by getting either a RST back (closed port) or a
dropped packet (open port). Of course, the other situation where you
might get back a dropped packet is if you've got a packet filter blocking
access to that port. In that case you will get back a ton of false open
ports. A few years back these kinds of scans might have been stealthy and
undetectable. These days they probably aren't.

There is a ton of good reading on scanning methods located here:

http://www.insecure.org/nmap/nmap_documentation.html