I'm reasonably certain that your error message with that function doesn't make sense. waveOutOpen(...) doesn't take a file handle or name at any point, given that it provides a "simple" interface to a sound card. Its parameters are:-
Perhaps you have a different problem? Could you post some code?
LPHWAVEOUT phwo, //Pointer to where the device handle is returned
UINT_PTR uDeviceID, // ID of the device to open (WAVE_MAPPER unless you have a reason for being specific)
LPWAVEFORMATEX pwfx, // Pointer to WAVEFORMATEX structure describing the output buffer format
DWORD_PTR dwCallback, // Pointer to callback function or window handle
DWORD_PTR dwCallbackInstance, // User-defined data to pass to callback
DWORD fdwOpen // Flags