I'm pretty sure all kinds of wonderful things can happen, which may or may not be limited to short term desync and long term desync.
Remember: you have absolutely no control over the network, and no existing protocols guarantee timing constraints. Furthermore, if you use UDP, frames can arrive out-of-order.
I think you are in for much less pain if you just put the audio and video together (interleaving) and push it out together in one stream. Though, I am no expert in the area.