Are you sure the correct answer is 3? What did you use to calculate (38^37) % 77?
Here is some code and answers I got.
And here is the output i got
double dn1 = 38, dn2 = 37, dmod = 77 , dresult,dtemp;
long double ldn1 = 38, ldn2 = 37, ldmod = 77, ldresult,ldtemp;
cout << "Using doubles " << endl;
dtemp = powl(dn1,dn2);
dresult = fmodl(dtemp,dmod);
cout << dresult << endl;
cout << "Using long doubles " << endl;
ldtemp = powl(ldn1,ldn2);
ldresult = fmodl(ldtemp,ldmod);
cout << ldresult << endl;
Using long doubles
so there is a problem here. Either the doubles are over flowing, or using long doubles results in wrong answers.
Using limits.h and DBL_MAX on my system doubles max out at 1.79769e+308. So i think its just an error due to trying to use long doubles.