How I broke Queatrix' encryption:
- I started by writing up a small program to do frequency analysis on varying chunk sizes. Since the size of the file (951 bytes) is only divisible by 3 and 317, I checked for chunk sizes of 1 and 3 bytes. There were a total of 57 1-byte patterns, and 260 3-byte ones, so I assumed that it was a relatively simple substitution/modification cypher which only operated on single bytes with a one-byte key.
- Looking at the output from my frequency calculator, I noticed that all of the results had one of the following hex characters in their four most least significant bits: 2, 3, 6, a, b, e and f. Looking at the binary representations of these numbers revealed that in every case, the 2nd least significant bit was set. I presumed this to mean that the data encoded was 7-bit ASCII, with a 1 inserted there for no apparent reason.
- Then I looked at the most frequently occuring character: 0xFB with 129 occurences. Since I was working under the assumption that I was dealing with an english-language plaintext, it was a safe bet that this was the space character.
- Examining the binary representation of 0xFB, I noticed that it is an inverted mirror image of the space character (0x20). Thus, my first guess at your encryption method was that you were simply NOTing the bytes and reversing the order of bits.
- I then turned my attention to the next most frequent characters, 0x9A, 0xBA, 0x4A and 0xAA. Still under the assumption that the plaintext was in English, I assumed that three of those four represented the three most common letters in English: 'e', 't', and 'a'.
- I applied my first guess at your encryption algorithm to these characters, and realized that they didn't match up, which showed me that I was incorrect in my guess.
- Looking at the binary representations of all of the above (plaintext and NOTted values), I realized that 'a' and 't' differ by one bit, as do 'e' and 'a'. Looking at the encrypted values, I realized that 0x9A and 0xBA differ by one bit, as do 0xBA and 0xAA. I thus assumed that 0xBA was the encrypted version of 'a'. With that assumption, I also assumed that the more frequent of the two remaining (0x9A) represented 'e', while 0xAA represented 't'.
- Upon realizing that the NOTed versions of the encrypted characters had the same number of set/unset bits as their plaintext equivalents, I realized that what was most likely happening was that the bits were being swapped around.
- To determine where the swapping occured, I wrote all four plaintexts in binary form one above the other, and then did the same thing with the NOTed encrypted values, to get the following:
- I then looked to see which columns were unique, and where they appeared on each side. I found that column 4 in the encrypted table moved to column 4 in plaintext, and that columns 3 and 6 switched places. Columns 1, 2, 7 and 8 seemed to stay in place.
- I then wrote a quick program to test my theory by having it swap columns 3&6 and 4&5 in the file, and realized when I looked at the output that only some characters were decrypted (such as 'a', 't', etc.).
- After modifying my application to swap colums 1&7 and 2&8, I ran it again and got the plain-text. Voila.