Your dialog resource is measured in dialog units, which are proportional to the font being used in the dialog (which can be the default). So in the first case, the size is 640x480 DLUs.
In the second case, your popup "dialog" (I think it's a window but I can't see your code) is measured in pixels.
If you want the main dialog to be 640x480 pixels, try using a SetWindowPos call when it receives a WM_INITDIALOG message, like so:-
The advantage of dialog units is that dialogs will take up similar amounts of space at a variety of resolutions (because the font their based on is scaled).
SetWindowPos(hwndDlg, NULL, 0, 0, 640, 480, SWP_NOMOVE | SWP_NOOWNERZORDER);