First thing I noticed was line 26. That should be giving you an error when you try to compile (or at least a warning).
Code:
scanf("%s", isbn[10]); //error
/* isbn[10] is ONE element, and it is outside the isbn array of ten elements. In C, array elements are numbered FROM 0 TO n-1, (so here, from 0 to 9, instead of 1 to 10.
Also, the %s format in scanf("%s"...), requires a pointer to the isbn array (which in C, is the name of the array itself: "isbn").
So the correct scanf would be: scanf("%s",isbn);
AND, if you want to hold the isbn array values as a string, then you should use %s and the isbn should be a char array with 11 elements in it, instead of 10. Why? Because in C, chars are just char's, and NOT strings, unless they have an end of string char marking the end of the string: '\0'.
Integer arrays are not marked with an end of digit marker of any kind. So an array of 10 integers (numbers of any kind), would need 10 array elements in it (for it's size). A char array NOT a string, would also need just 10 elements for it's size, to hold 10 char's: array[10]. But an array of chars TO BE A STRING, needs 11 elements, because it needs one extra char to go on the end of it: array[11].
th1 = checkx(isbn[10]);
th2 = checko(isbn[10]);
You need two nested for loops for your "find the answer" block of code. The outer for loop will control the position (which digit of the number), that is currently being worked on to find the answer, and the inner for loop will actually try the various values. If your other functions take the place of the outer for loop, then only one for loop will be needed for the testing loop.
Testing should be done ONLY inside the inner for loop. No while loop is needed (could be used, but they generally make things more difficult for beginners). For loops give
you extra structure to build a correct loop.
Most of what you have in your program, you don't need. This is common with good students, however. It's one of the things you will find programmers always need to be on guard against -- going "Rube Goldberg".