Replying to segmentation fault with fscanf and encrypt which seems to be hard to touch at the moment.
> else if(!fopen(argv[1],"r"))
But if you DO open the file, you leak the file handle and remove the possibility of opening the file again (particularly when trying to open the file for writing).
> while(line[x]!='\n'||line[x]!=EOF)
Lines read with your current fscanf will never have '\n' or EOF characters. So you blindly wander through memory until it either segfaults or you luck onto one of the two magic values.
Try checking for '\0' instead.
> ciph[x]=line[x]+shift;
1. Does this always guarantee a printable character?
2. Can it ever result in a value of \0?
Speaking of \0, you don't append one, so printing will result in more difficulty.
> int calckeylen(char *ki)
Not sure why you couldn't just use strlen() here.
