Return a value from EndDialog() which specifies that the application should be closed.
Code:
nRet = DialogBox( ... );
if (MY_CLOSE_VALUE == nRet) PostMessage(hwndMain, WM_DESTROY, ...);
OR

Try sending WM_CLOSE or WM_DESTROY to the main window after calling EndDialog().
Code:
EndDialog( ... );
PostMessage(hwndMain, WM_DESTROY, ...);