When the preprocessor processes your program, it becomes:
The A*A is replaced with whatever you have in the parentheses after SQRT. In this case x-3 replaces A in both places, so x-3*x-3. 4*5-3*5-3 == 20-15-3 == 2.
/* a bunch of included header file code from stdio.h here */
If you want to use a preprocessor macro for SQRT, you can not put anything except a single value as the parameter or "strange" results like this will occur. If you want to have a sqrt() function that you can call with x-3 (or any other expression), you need to make it a function.