What you want to do is publish your algorithm so others can test it. Otherwise it's a mix between obscurity and security, and generally it won't get you anything good. Unless you're the NSA and have some of the best cryptographers/cryptologists in the world working for you, you need peer-review.

But supposing you've just gotten across a ciphertext and you'd like to break it...

First off you look at the types of characters involved. If it's alpha-numeric, you try out a bunch of things, like character frequency [different for different languages] or likelihood of coincidence. Also splitting into groups [blocks] or pattern analysis. If it's numbers, you could also try some of these things if you suppose it's a transposition cipher.

If it's binary data it'll be hard [I know everything is binary, but I mean something outside of characters and numbers] because you have to take into account complicated mathematical modelling that can only have an application in computers etc.

Some algorithms also leave a 'mark' of what they are. If the data is valuable, believe me that a cryptanalyst would spend time trying out possibilities of multiple ciphers/algorithms. Even a polyalphabetic substitution cipher can have you coding for days if you have no clue about what you're doing, but in general cryptanalysts know where to start looking.

Read Bruce Schneier's "Applied Cryptography" if you're really interested in the subject. I just got it in the mail yesterday [after repeatedly browsing it in bookstores and libraries] and it's an excellent read.