I am currently working on a remote desktop program that uses a UDP connection for streaming video and audio only, and a TCP connection for everything else (including mouse input, etc).
The TCP connection is currently secured using AES-256-CBC (pre-shared key), with half the initialization vector generated by the server, and the other half generated by the client (to prevent replay attacks on either side). There is also SHA-256-HMAC encrypted with each message for authenticity and integrity.
1. Does anyone see any vulnerability with that scheme? Network security is not really my thing, but unfortunately is important for this project.
2. How do I secure the UDP channel (uni-directional from server to client)?
Since the UDP "connection" is always "established" after the TCP one, it can use the same IV and key as the TCP connection.
The problem is datagrams can be dropped or duplicated or come out of order, or corrupted, which breaks most block cipher modes.
A simple scheme would be to randomly generate an IV per datagram. That will make the same plaintext encrypt to different ciphertext, but doesn't prevent replay attacks, unless the client remembers all IVs that have ever been used (not practical).
One solution I can think of is to have the server request, say, 512 randomly-generated one time initialization vectors from the client, and the client keep track of which ones have been used, and not allow any IV to be used twice.
But that's pretty complicated. Is there an easier way?