Video-based remote desktop

[Elysia]

I don't know about statistics.
What I do think is that it makes sense to buffer future packets (you're not going to resend them anyway), but obviously not past packets.

[cyberfish]

If I understand things correctly, it looks like H.264 can tolerate lost packets, but not out-of-order packets (except for baseline, strangely. Baseline can tolerate out of order packets, but using baseline means I'll miss out on a lot of the fun stuff).

So if I receive packets, for example in this order, "1, 2, 3, 5, 4", I'll have to decide, after receiving packet 5, whether I should hold on to it and wait for packet 4 to show up, or should I just feed 5 into the decoder, and declare 4 lost (since out of order decoding is not supported, once I fed 5 into the decoder, 4 is not useful anymore).

If out of order packets are a more common than dropped packets, waiting would be a better idea. If dropped packets are more common than out of order packets, not waiting would be a better idea (for better latency). I just don't know statistically which one is better.

[cyberfish]

Also, when a packet is dropped, the client can immediately request an intra-refresh, so there would only be a few frames with possible visible error (before the intra refersh is done). I'm leaning towards just dropping the packet since the cost (in visual perception) is not much, and makes things a lot simpler.

In fact, it looks like this is the example use-case used in x264.h

/* x264_encoder_intra_refresh:
 *      If an intra refresh is not in progress, begin one with the next P-frame.
 *      If an intra refresh is in progress, begin one as soon as the current one finishes.
 *      Requires that b_intra_refresh be set.
 *
 *      Useful for interactive streaming where the client can tell the server that packet loss has
 *      occurred.  In this case, keyint can be set to an extremely high value so that intra refreshes
 *      only occur when calling x264_encoder_intra_refresh.
 *
 *      In multi-pass encoding, if x264_encoder_intra_refresh is called differently in each pass,
 *      behavior is undefined.
 *
 *      Should not be called during an x264_encoder_encode. */
void    x264_encoder_intra_refresh( x264_t * );

[cyberfish]

Also, new rate control settings.

	mParam.rc.i_rc_method = X264_RC_CRF;
	mParam.rc.f_rf_constant = 22;
	mParam.rc.i_vbv_max_bitrate = 2000;
	mParam.rc.i_vbv_buffer_size = 2000 / 30;

This way, each frame will be constrained to a maximum of 2/30 megabits (for a ~2mbps connection. The bitrate will be changed on the fly by observing how fast the network is), and on frames that are easy to encode (desktop screen), it will use a minimum (maximum quality) of 22 to not waste bandwidth.