As always, john.c codes are flawless. But I would insert two more checks on encrypt() - to see if both files are different and to see if the output file was opened. Consider the case where 'text.txt' is said to be the input and output file... Someone can think of testing only the filenames... But consider "./text.txt" and "text.txt" (different strings) or another combination of paths... Well, on Unix systems you can use stat() and check if st_ino are equal (hardlinks and same files have the same inode) -- this doesn't work on Windows! On Windows you can open the files with OpenFile() and use GetFileInformationByHandle() to check nFileIndexHigh and nFileIndexLow struct members:

Code:
// Gets the inode of a file (<0 if error or file doen't exist)
long long getfile_ino( char *path )
{
#ifdef WIN32
  OFSTRUCT ofstruc;
  HFILE h;
  BY_HANDLE_FILE_INFORMATION fi;

  h = OpenFile( path, ofstruc, OF_READ );

  if ( h == NULL )
    return -1LL;

  if ( ! GetFileInformationByHandle( h, &fi ) ) 
  { 
    CloseFile( h );
    return -1LL;
  }

  CloseFile( h );

  return fi.nFileIndexLow + (long long)fi.nFileIndexHigh << 32;
}
#else
  struct stat st;

  if ( stat( path, &st ) )
    return -1LL;
  
  return st.st_ino;
#endif
}
So, to check if both input and output file are the same:

Code:
long long ino1, ino2;

ino1 = getfile_ino( path1 );
ino2 = getfile_ino( path2 );
if ( ino1 != -1 && ino1 == ino2 )
{ /* same file! */ }
As for the check of openess of the output file, you should test for NULL after fopen, for fout...

In case of success or error, encrypt() can return an int.