time for a few observations:
given an exe, it is always possible to convert the binary back to assembly language, as it's a one-to-one correspondance, from there any good compSci with enough time can convert it up to higher level code, or work out the algorithm from the assembler. It may not be a fast job, but I'm sure there is software around that can speed it up a lot.
hence, as soon as the exe is released, and there is any worthwhile reason at all to break it, the source/algorithm will be available within the day, whether the developers like it or not.
as for the security of the algorithm itself:
as I understand it, the output of your algorithm depends on the current letter, last encrypted letter, and the key.... nothing else. so given the current letter & last encrypted letter, we can work out the key, assuming we've already broken the exe (see above)
for the first letter, I don't know what you consider the "last encrypted letter" to be, I'll assume you use a null (0), or this is preset somewhere else in the exe (which we've already broken), hence for letter #1 the last letter is no longer a variable. making the first chunk of each message a weak point. and because letter 2 depends on 1, the whole cryptosystem breaks down when the first letter goes. this can be made even faster by guessing the start od a message based on probabilities, eg a lot of messages start "hi, [name]" or "dear", or perhaps with a sender ID.
also, as the algorithm is symmetric (same key for en/de - cryption) given the input, output and algorithm, the key can be obtained. take the first letter again, we have the output & algorithm, input is just one of a small range of brute-forceable values. once you have this key, you can work on letter #2, again you have everything you need for the exact same process, if the two key;s you generate match.... bingo, key broken (probably, would be an idea to keep going just in case, but in principle the message is done for).
a frequency analysis on large chunks of ciphertext is still highly possible. you just look at pairs rather than single characters.
for example every char following ascii 0 in the ciphertext will be encrypted in exactly the same way, hence given a large volume of ciphertext you will get a large number of frequency analyses. This sounds complicated but this kind of analysis was being done before the computer was smaller than a room (maybe before it even existed) and is a highly valid method today. The system would therefore only be secure for short messages, or where the keys get replaced on a regular basis, which for secret key algorithms is quite a hassle.
the algorithm is fine for small, low-priority usage, but to be honest a little pointless as plenty of incredible systems exist today. and as I've seen in a number of sources, cryptography is already the strongest link in the security chain.
and by the way, secrecy by obscurity != a secure algorithm. nor is shoving some extra convolutions in your code without thinking about them carefully, as shown by XSquared, you can make the algorithm a lot less secure than you think.
-mark