Okay, I'm no professional cryptographer, but, I've done my fair share of crypto work and i'm a math geek.

After looking over the algorithm a bit, i managed to somewhat simplify it and explain it. First off, whever you have ch+=key2*key;ch+=key; you can simplify that to ch+=(key2+1)*key. Now, I'm not up to par on C++, but, I'm pretty sure that when you add a long to a char, it stays within the char's domain (0-255). So, add a % 256 into that then. You also flip-flop the two characters you're working with in the output file. So, a simplified pseudo-code version of the algorithm would be:

encrypt:
get 2 characters
character 1 = (character 1 + (key2+1) * key) % 256
character 2 = (character 2 - (key2+1) * key) % 256
exchange characters
write to file

decrypt:
get 2 characters
character 1 = (character 1 - (key2+1) * key) % 256
character 2 = (character 2 + (key2+1) * key) % 256
exchange characters
write to file

Okay, now that we've got what's going on laid out pretty plainly, we can start to decide how we want to crack the system. Well, as I look at it, I see that the file which once was something like:

1234567890

is now

2143658709

So, my first step would be to just put the characters back in order to simplify life. From there, I'd see that all the odd characters had something added to them and all the even ones had something subtracted from them. So, I split the file into separate files, one containing all the even characters, one containing all the odd characters. Then, I'd do frequency analysis of it, hoping for some kind of recognizable pattern. If there was a recognizable pattern, I could start mapping certain characters to others, which would then give me a list of possible keys. For example, if I can tell that the character 'e' has been mapped to 'f', that's a single unit of difference. Let's say it was in the odd set, which i know had things added to it. The keys could be represented by:

256^n+1

Where n can be any whole number from 0 to log(32768)/log(256) (read that as log base 256 of 32768, it'll make more sense). Now, I KNOW that the same key was used to encode each character in the odd set. So, as I find other mappings, I can deduce the actual key. HOWEVER, I don't think it's actually necessary to. Since the key 1 will encode the set the same way a key of 257 will, there's really no sense in checking for keys that are greater than 255 (256 maps to 0, remember?).

Note that when I'm referring to key, I'm referring to the (key2+1)*key combination because that's the only outwardly expressed thing in the encrypted file.

Now, let's say that there was no recognizable pattern in the file. I could simply try all the numbers from 0 to 255 and see if adding that to half the set and subtracting from the other half would make a sensical file.

The algorithm seems to be fairly weak to me. Here's pseudocode for cracking it:

get file
for each pair of characters
flip them
end for
for i=0 to 255 do
subtract i from all odd characters
add i to all even characters
test to see if the file makes sense
end for

Because you're using % as the final operand, you're directly reducing your key space. I'll post some suggestions later as to how to improve it and some other general techniques, I must leave for now.

[edit] I'll have some free time tonight to code. If you could attach an encrypted file without giving me the keys, I'd like to write a short program to crack the file and then post the decrypted version. I think it'd prove useful and insightful. [/edit]

[edit2] Fixed a technical error [/edit2]