I'm posting this because well, quite frankly, I can.

I'm going to start by posing and answering a few questions.
If you're trying to break into a computer over a network, what's the first thing you need to know?

Well, for starters, knowing what OS you're dealing with will enable to you more specifically target your attacks.
There are various ways of finding out what OS a machine is running, and these techniques are collectively referred to as OS fingerprinting.

The first step, is to make sure that you can't get this info from banners. Sometimes, if a computer is running an FTP, Telnet, POP3, SMTP, or HTTP server, they will give you a banner that you could use to determine what OS it's running. Take the Apache webserver for instance. You can simply telnet to the webserver, request the header using GET / http/1.0 and you might see a line like:
Server: Apache/1.3.19 (Unix) (Red-Hat/Linux) mod_ssl/2.8.1 OpenSSL/0.9.6 DAV/1.0.2 PHP/4.0.6 mod_perl/1.24_01
That tells you right there that it's running on RedHat Linux.

When attempting to fingerprint a computer on a network, usually you need a combination of things to be certain, which brings me to the next section.

There are two main groups of OS Fingerprinting that revolve around two commonly used network protocols: ICMP and TCP.

ICMP OS Fingerprinting deals with examining how a computer responds to various ICMP messages.
One quick example of this is pinging the broadcast address. The broadcast address is (assuming you're using 10.0.0.x as your home LAN's IP space) 10.0.0.255. You can tell *nix boxes apart from Windows PCs and vice versa by pinging the broadcast address because computers running Windows don't respond to the broadcast address. Combine this with something like nmap's -sP option and you can ferret out all the Windows boxes on a network.

An example from a RedHat Linux 7.1 box:
[chsh@compname dir]$ ping 10.0.0.255 -b
WARNING: pinging broadcast address
PING 10.0.0.255 (10.0.0.255) from 10.0.0.132 : 56(84) bytes of data.
Warning: time of day goes back, taking countermeasures.
64 bytes from 10.0.0.132: icmp_seq=0 ttl=255 time=100 usec
64 bytes from 10.0.0.120: icmp_seq=0 ttl=255 time=100 usec
64 bytes from 10.0.0.117: icmp_seq=0 ttl=255 time=100 usec
64 bytes from 10.0.0.132: icmp_seq=0 ttl=255 time=100 usec
64 bytes from 10.0.0.120: icmp_seq=0 ttl=255 time=100 usec
64 bytes from 10.0.0.117: icmp_seq=0 ttl=255 time=100 usec


As you can see, by pinging 10.0.0.255 in the example, we get a response from 10.0.0.132, 10.0.0.120, and 10.0.0.117. Now let's say that we've scanned the entire 10.0.0.1-254 range, and we know that there are also two other computers at 10.0.0.119 and 10.0.0.130. We know they're obviously a different OS than the three that did respond, and Windows does ignore the broadcast address.

From here, you can use some other ICMP or TCP techniques to determine which machines are which OS, but just this one step can cut down your list of possible OSes from about 30 to 2 or 3 and it's a very simple step. If I told you what to do next, it wouldn't be a mini-tutorial, would it?
Keep in mind that many network devices (routers, network printers, etc.) respond to a broadcast request as well.

TCP Fingerprinting is a technique whereby you examine the response (or lack thereof) you get from a computer when requesting a TCP connection.
In my travels, I have found several key differences between the different OSes. Take Windows and Linux as an example. When a port is closed, Windows responds to a connection request with TCP-RST. Linux responds with ICMP-Destination Port Unreachable. This key difference can also help you narrow down what OS you're dealing with.

My goal here was to raise interest and give some slight education on how to determine which OS you're dealing with. All of this information is from my experiences, and if I'm out of date or wrong of any of this, please correct me.

If you want some recommended reading, this is a good start.