Thread: Encrypting and knowing if it's encrypted

Hello there!
Well i'll present my problem: As a part of a bigger program I was trying to encrypt a file using basic xor encrypting, I can encrypt and decrypt files easily. The problem is in my "bigger" program I would like to open the file and read its data but what if its encrypted? well I went over this thinking about using a magic key at the start of the encrypted file for example "EN\n" then if I read the first line and the program found the magic key it would know that the file is encrypted, decrypt it and read the data. Problem comes when trying to decrypt and depending on the key even trying to encrypt since sometimes I experienced that \n wasn't being printed to the file... Here's a sortened version of a small encrypt program I made to test the problem...

Code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAXLEN 256

#define MAGICKEY "EN"

void encrypt_file(FILE *src, FILE *dest, char *key);
void copy_file(FILE *src, FILE *dest);

int main()
{
    FILE *f, *tmp, *tmp2;
    char s[MAXLEN], key[MAXLEN], magic[MAXLEN];

    printf("Enter file name: ");
    fflush(stdin);
    gets(s);

    f = fopen(s, "r");
    if(f == NULL){
        printf("ERROR: Couldn't open file %s\n", s);
        return -1;
    }
    tmp = fopen("tmp.txt", "w");
    if(tmp == NULL){
        printf("ERROR: Couldn't write temp file\n");
        return -1;
    }

    printf("Enter key: ");
    fflush(stdin);
    gets(key);

    fscanf(f, "%s", magic);
    if(strcmp(magic, MAGICKEY) == 0){
        fgetc(f); // get rid of '\n'
        copy_file(f, tmp);

        tmp2 = fopen("tmp2.txt", "w");
        if(tmp2 == NULL){
            printf("ERROR: Couldn't write temp file\n");
            return -1;
        }

        encrypt_file(tmp, tmp2, key);
        fclose(tmp2);*/
        fclose(tmp);
        fclose(f);

        remove(s);
        remove("tmp.txt");
        rename("tmp2.txt", s);

    }else{
        rewind(f);
        encrypt_file(f, tmp, key);
        rewind(tmp);
        fprintf(tmp, "%s\n", MAGICKEY);

        fclose(tmp);
        fclose(f);

        remove(s);
        rename("tmp.txt", s);
    }

    return 0;
}

void encrypt_file(FILE *src, FILE *dest, char *key){
    int i, b;

    i = 0;
    while((b = fgetc(src)) != EOF){
        fputc(b ^ key[i], dest);

        ++i;
        if(i == strlen(key))
            i = 0;
    }
}

void copy_file(FILE *src, FILE *dest){
    int c;
    while((c = fgetc(src)) != EOF)
        putc(c, dest);
}

I tried binary reading mode but doesn't seem to work cause I have to remove the \n and since this varies from OS to OS as I understood, I'm having a headache and don't know how to go about it...
Thanks in advance!

Are you sure you want to encrypt the "magic key"?

@CommonTater No I don't want to encrypt the magic key, magic key is added after encryption is done, as you can see in my code,
@Fordy About the extension changing it's a great idea the problem is files are read with txt extension always I mean the option is like Open File and I don't distingish between Open File and Open Encrypted File, so maybe the length of the key is a good idea, thanks
Anyways, I think I should open the file as binary because some characters can get encrypted as EOF as I read while searching, in other post. But... any other solutions to my problem?
Thanks in advance! And thanks again both of you :P

Thanks commontater that seems very interesting and makes sense
Anyways I didn't experience any overwriting (at least non that one can see), my question now is when you say process the file here you mean processing it character by character like shifting all the characters for example two places and then adding magic key 'EN' and then writing it to file?
Thanks in advance!

Originally Posted by beta3designs

Thanks commontater that seems very interesting and makes sense
Anyways I didn't experience any overwriting (at least non that one can see), my question now is when you say process the file here you mean processing it character by character like shifting all the characters for example two places and then adding magic key 'EN' and then writing it to file?
Thanks in advance!

In the interests of efficiency I would do a little check.

If you've loaded an encrypted file the magic key should already be there so you can test for it. If it is there take a starting point 2 bytes over and decrypt the file... If not the file is not encrypted so you don't take an offset... the resulting data can be passed to your application as a memory pointer.

If you are writing encrypted data... write the magic key to disk (this will advance the file pointer for you) encrypt the memory data and write it out as one big blob. If you are writing a non encrypted output... just write the data.

Alright thanks I was also using that kind of check in my first code so I was aware of doing that but I was struggling with the last part you wrote so yeah writing the magic key to file and then adding the encrypted blob to it makes much more sense than trying to encrypt the blob then adding the magic key to it then writing to file... Thank you very much I wasn't at home but the problem should be gone so I'll mark this as solved! Thanks again! EDIT: One last question, if I test for the magic key, should I take that starting point to bytes over? I mean if I'm reading to test for the magic key the pointer should be positioned correctly shouldn't it? I think I should rewind or fseek to SEEK_SET when the magic key is not there am I right? Thanks

what if the "blob" of data starts with "EN" the same as magickey ? a small possibility but a possibility none the less. meow. think you need a better system.

Originally Posted by kryptkat

what if the "blob" of data starts with "EN" the same as magickey ? a small possibility but a possibility none the less. meow. think you need a better system.

Good point... but I'm sure the "EN" was just an example.

Originally Posted by beta3designs

Ok thank you very much! ^^
EDIT: BTW, in your code

Code:

fread (fm,fs,1,lf)

should be;

Code:

fread (fm,1,fs,lf)

Sorry about that...

Originally Posted by AndrewHunter

Regardless of what 'key' you use, this is always a possibility. This is where the user is expected to use the program on the right files. I could have a "blob" of data, I'll say the andyFileFormat, that begins with the same signature JPGs use to indentify them. If the user decided to open it in a picture viewer and complained because they saw garbage, I would just inform them to throw their computer away because they were clearly not smart enough to use it.

His key could just as easily be a DWORD or QWORD value.... nothing says it has to be ascii.