The problem with your question is that it's wayyy to broad.
First of all, a buffer overflow wouldn't work on a properly written telnet (or any other service) server. Otherwise the internet would be nothing but a bunch of exploited websites etc.
Another thing is that windows doesnt have a default telnet server, and if your talkin about hacking a BSD or Linux box then your practically into a whole different subject.
I'll try to give you the low-down without telling you "How to hack into Hotmail", lol...

What I do:
- Check for the really basic ****: SMB/ Netbios Network shares, Telnet Servers, FTP Servers, etc. If there are any then I write down the name of the server software (usually says during login) and then go on the interweb and check for any known vulnerabilities. Unless of course... They have SMB network shares, in which case they're dead.
Next:

- Full range portscan: I wouldnt suggest this against a non-windows box, cause its the electrical equivalent of smashing down the bank door with a sledge hammer. This sound ridiclious, but over high speed DSL (Mine= 1.5mbs up 6.0mbs down) it actually doesnt take that long. If you dont have a good connection aka <512k then just scan the first 1024 ports, all the most common services are there.
When I get a list of ports I try to directly connect to each one with telnet, because the login messages sometimes let you know what service it is. If I cant find out what they are from that, and i dont already know (aka really common services) then I go back to the interweb and do a search for that port # and see what it is.
[glowpurple]YOU CAN ALWAYS FIND OUT WHAT A PORT DOES[/glowpurple]
If you cant your not trying hard enough.
Then if i dont already know some exploits for that service I go to the interweb and try to find some, or I Download the software and try to find some myself. Oh yeah, and if I want a username to try then I try to grab the remote NAMETABLE - very good method
Stupid windows tells u who is currently logged on, and they sometimes use that username 4 other stuff. Do this at the command prompt by: NBTSTAT -A ipaddresshere
The A has to be capital.

- Last effort: If the first two options dont work, then I usually get desperate and start trying the weird ****. Examples:
SNMP Walking
Try to buffer overflow every service they got... lol... <- this gets u BUSTED
And one more crazy thing that I havent heard of anyone else trying:
I ping them, then i look at the reply in code form.
This can tell u what OS they are using, windows sends the alphabet.
SOME OS's SEND RANDOM DATA FROM RAM! <- This has possibilities

Anyhow, thats roughly what i do, or most of it.
That should be enough 4 a report without teachin u to hack ur friends hotmail acc.

- Trak