Video-based remote desktop

[Elysia]

Hmm that's interesting. Are you talking about short term out of sync (a few frames) or long term drift?

I don't think long term drift will be a problem because the source is inherently synchronized. If they somehow get out of sync, that means one of them has very high latency, in which case I'd have big problem anyways. If the (very small) buffers get full, I'd be dropping frames/samples anyways for the network to keep up.

Not really sure how short term syncing can be done (every frame?). I'm not sure if I want to, for example, delay a video frame if audio is not here, yet.

I'm pretty sure all kinds of wonderful things can happen, which may or may not be limited to short term desync and long term desync.
Remember: you have absolutely no control over the network, and no existing protocols guarantee timing constraints. Furthermore, if you use UDP, frames can arrive out-of-order.
I think you are in for much less pain if you just put the audio and video together (interleaving) and push it out together in one stream. Though, I am no expert in the area.

[cyberfish]

I'm pretty sure all kinds of wonderful things can happen, which may or may not be limited to short term desync and long term desync.
Remember: you have absolutely no control over the network, and no existing protocols guarantee timing constraints. Furthermore, if you use UDP, frames can arrive out-of-order.
I think you are in for much less pain if you just put the audio and video together (interleaving) and push it out together in one stream. Though, I am no expert in the area.

Hmm I will look into that. I don't really know how to do the audio part, yet. I have very little experience with audio/video stuff, and most of what I know now is from what I read in the past 2 days.

Audio doesn't really have concept of frames (unless you want to call each sample a frame, but then it's not possible to separate the stream into "frames" due to compression), so I'm not really sure how to interleave it with video.

The way I think of it, for audio compression to be effective, each "segment" must be at least a few seconds long, so that's the "syncing" resolution. But I'm not sure if I'll encounter desync longer than a few seconds long anyways, and if I do, I'll probably have bigger trouble with responsiveness anyways. In the case that audio lags behind a few seconds for whatever reason, I don't think I'll want to pause video to let audio catch up.

I think I'll try to get the video part working first, and worry about audio later.

[phantomotap]

Not really sure how short term syncing can be done (every frame?). I'm not sure if I want to, for example, delay a video frame if audio is not here, yet.

The natural sync will erode because of a lot of reasons. One of the biggest (I'm winging this from experience. I may be guessing wrong about this being the biggest.) problems with sync will be frame duplication. You aren't going to get a perfect (one snapshot exactly on refresh for each and every frame) video feed but may manage to pull enough samples to manage a near perfect audio feed. The "H264" doesn't naturally duplicate identical frames. (This is, of course, an option that "x264" is happy to do for you, but the cost would be sending identical frames for no reason or checking them at a higher level which you almost certainly can't do as well as "x264".) Because you are manually managing the stream you'll probably have to be ready to adjust samples of audio every consecutive couple of identical frames.

We aren't talking a lot here. It will probably never be worse than half a second. I know from experience how grating a half second sync problem can be when watching moves or playing games.

*shrug*

Anyway, I would sync audio to video exclusively. I've done exactly that with good result. If you were streaming for archival (which is unbelievably a thing) you would need to deal with syncing across both streams. You aren't doing that so you have the option of "lossy transmission".

So, say you have a frame (here meaning a slice of time) of video but not enough audio to go with it you duplicate the last samples (of the current frame) with diminishing echo (or some other fade or just silence if this is too costly) or you have a frame of video and too much audio you can cache a little or just burn it off assuming that fresh data will be ready "soon enough". This can keep you nicely synced up without introducing much in the way of extra latency.

I don't want to use hardware acceleration (at least for now) because there is (AFAIK) no cross-platform way to do it, and I want to keep the client cross platform because there's no reason why it shouldn't be.

I don't know where it went, but an "OpenGL ES" implementation of "H264" main profile decoder using "GLSL" was a thing. That would probably be at least as portable as anything else you plan on using.

Copying to SDL should be at most just a ~6MB memory copy + a colorspace conversion.

My crack was more targeted at the latency of embedded displays in telephones.

Soma

[cyberfish]

Ah! Variable frame rate. I was just reading up on that yesterday. Right now I have x264 set to constant frame rate, and just feed it frames at 24 fps. There's a lot of duplicates on desktop (as opposed to in games), but I don't think that will affect compression rate much because 1) I'm not using B frames for latency reasons, and that's what duplicate frames hurt the most from what I heard, and 2) I set the intra-frame interval to be very high (default 250 frames or around 10 seconds), because the client can simply request an intra-frame to be injected if there is packet dropping, and 3) desktop applications are highly compressible anyways (for intra).

This should also solve the sync problem right?

I don't know where it went, but an "OpenGL ES" implementation of "H264" main profile decoder using "GLSL" was a thing. That would probably be at least as portable as anything else you plan on using.

I believe I'm still using high profile (x264's "veryfast" preset).

[Elysia]

B frames help compression when there are many duplicates because they encode only the difference between two frames (future or past), plus they can use other B frames as reference (P frames can only use I frames as reference).
What you really need to be careful of is the streaming latency. Assume the worst. Very likely you'll get into problems you or we never thought of. This is unexplored territory, right? That's a given in such territories.

[cyberfish]

B frames help compression when there are many duplicates because they encode only the difference between two frames (future or past), plus they can use other B frames as reference (P frames can only use I frames as reference).
What you really need to be careful of is the streaming latency. Assume the worst. Very likely you'll get into problems you or we never thought of. This is unexplored territory, right? That's a given in such territories.

Hmm I thought the only difference between B frames and P frames is B frames can use frames in the future or past for motion estimation/compensation, while P frames can only use frames in the past (which is better for latency since all frames can be displayed immediately once received).

From what I heard, duplicate frames screw up (makes them less effective) B frames because B frames can only use frames, for example, 2 before and 2 after itself, and having duplicate frames eats up those precious slots.

[Elysia]

There is a command line which controls how far into the past or future x264 will look for good candidate frames.
I think it is --rc-lookahead.
--ref also limits how many consecutive B or P frames you can have.
Regardless of all that, the best way to see is to test. Just test with and without them and see what gives the best quality/filesize ratio.
And don't forget that zerolatency will also disable some features.

[cyberfish]

lookahead is completely disabled by zerolatency, to ensure that as soon as a frame is received, it can be completely encoded, decoded, and displayed.

[cyberfish]

Made a SourceForge project for it DesktopStreamer | Free software downloads at SourceForge.net

And switched to MinGW-W64 toolchain (because the client has to be on MinGW/GCC since it has to be cross-platform, and I don't want 2 parts of the project on 2 different development environments).
The toolchain I'm using is here Mingw-w64 ( for 64-bit windows x64 / Win64 ) @ drangon.org (mingw-w64-bin-x86_64-20120525.7z)

[cyberfish]

Looks like the zerolatency tune does a few things -
Disables B-frames
No sync-lookahead (I think this is only relevant for B-frames anyways?)
No rc-lookahead (how is this different from sync-lookahead?)
Use sliced threading (each thread encodes part of a frame, instead of alternating frames)

More details: Diary Of An x264 Developer » x264: the best low-latency video streaming platform in the world

In addition I added intra-refresh to reduce bandwidth spike (so instead of an intra-frame once in a while, the I-frame is spread across multiple frames, so each frame would have a little intra-part).

[Elysia]

X264 Settings - MeWiki

[cyberfish]

Yeah I have been staring at that page for a very long time.

[cyberfish]

Next step - do latency testing on same computer.

I plan to write a SDL and libavcodec based client that will initially run on the same computer, but receive UDP packets from the server (localhost).

Then, the server will log the time it captured each frame in a file, and the client will log the time it finished blitting the frame onto the screen (this is possible because they are on the same machine, so time should be relatively reliable). This should measure the encode+pack+unpack+decode+blit. Then the real world network latency (1/2 ping) would just have to be added to it.

[Elysia]

Beware that UDP packets may not arrive in-order. I'd be surprised if they weren't on the same computer, but once on the network, all bets are off.

[cyberfish]

Yeah I'm thinking about adding a packet number to detect out of sequence and dropped packets.

Do you know how much it happens in real life on the internet typically? 0.1%? 1%? 10%?

Is it worth it to attempt reorder? Or should I just declare the packet lost?