Thread: Video-based remote desktop

  1. #1
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229

    Video-based remote desktop

    Has anyone heard of any video-based remote desktop system? I can't seem to find any.

    What I mean is, it looks like all remote desktop systems nowadays (rdesktop, VNC, etc) are all image based, with some simple ad-hoc optimizations (dirty rectangle, cursor overlay, JPEG, etc). Due to their inefficient use of bandwidth, very high speed connection is required even for just mediocre performance.

    I can't seem to find one that's based on, for example, H.264, which should lower the bandwidth requirement quite a lot (especially with inter-frame compression). This is obviously technically possible since that's what sites like OnLive ("cloud" gaming service) do, and they work quite well if you have good internet.

    It probably wasn't possible a few years back because PCs could not do H.264 encoding in real time. Should not be a problem anymore, though, with fast multi-core CPUs and GPUs. I know my computer can encode H.264 1080p main profile at around 120 fps.

    It looks like hardware has improved but software hasn't caught up.

    Anyone knows if something like this exists?

  2. #2
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Due to their inefficient use of bandwidth, very high speed connection is required even for just mediocre performance.
    In some ways it is a necessary evil for what they are generally designed to do.

    You can't devote the host machine to encoding. If the host does nothing but encode you can't really do anything with it.

    That said, if you setup the host machine properly, you can do a lot with a little. I have a "rdesktop" installed to a virtual machine setup with 800x600x16 running over native. It is perfectly responsive over a 768k pipe. I submit that you may be shooting to high if you want good use of all resources in the chain.

    This is obviously technically possible since that's what sites like OnLive ("cloud" gaming service) do, and they work quite well if you have good internet.
    Services like this aren't creating the H.264 stream on the same device that generates the feed.

    These sorts of services are taking the raw video output and putting it through different algorithms living on different hardware depending on the size of the feed, the latency requirements, and the bandwidth available.

    Generally speaking, they get the stream generated by arbitrarily splitting the video into blocks with hardware based splitters, feeding those blocks into dedicated H.264 encoders, assembling the resulting streams into a single package, and finally converting the package into a useful stream. This process allows higher compression with better motion recovery with simpler hardware requirements while only costing a little in stream integrity and overall compression level.

    From my resources, which may or may not know what they are talking about, "OnLive" strips the feed into a row of blocks and sends these rows off into a process similar to what I've described above. Essentially, they've added another layer in the chain which gives them a little more on the same hardware but causes "artifacts" at every row seem.

    I know my computer can encode H.264 1080p main profile at around 120 fps.
    O_o

    A lot of decoders can't handle "raws" in the main profile.

    You'd need a package and a stream that the target can consume or the speed is irrelevant.

    Is your target just general desktops?

    My telephone handles "RDP 4" okay, but it can't do "raws" H.264 at all.

    It looks like hardware has improved but software hasn't caught up.
    That's not entirely correct.

    The dedicated hardware encoders in a lot of consumer products produce low quality streams. They make "guesstimates" that you would not want in an archival streams. Look at the stream generated by the hardware in the newest video cards (expressed through various "cinema" software). If you are willing to sacrifice stream quality for speed the hardware is great. It can encode a stream about 30 times faster than my server. The stream simply isn't worth storing.

    It actually works pretty great for streaming "on the fly". That's what a lot of services do with them. (Any content generated for "live" streaming is processed with these components while the unmolested video is saved for later processing by better components for archival.)

    I just don't think anyone has applied these components to desktop services.

    Soma

  3. #3
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    Thanks for the insight!

    I don't really see how the encoding can not be done on the same machine, though, given enough computational power.

    Like you said, quality is sacrificed for speed. But for things like remote desktop, responsiveness (due to lower bandwidth usage) will be preferable to absolute best image quality in most cases. I'm still doing research, but it looks like modern video cards can do H.264 encoding extremely fast, with only slight loss of quality compared to x264 at the same bitrate.

    Dedicated hardware encoding sounds like a good idea. I was looking into using a FPGA based encoder (someone already wrote a H.264 encoding core) for something totally unrelated. Doesn't look like it will be necessary, though, since they probably aren't much/any faster than video cards, due to the huge clock rate difference (for cheaper FPGAs).

    It actually works pretty great for streaming "on the fly". That's what a lot of services do with them. (Any content generated for "live" streaming is processed with these components while the unmolested video is saved for later processing by better components for archival.)

    I just don't think anyone has applied these components to desktop services.
    That is what I'm talking about. It's for streaming for remote desktop, and nothing will be stored. Ideally, with enough upload speed, it can even be used for gaming.

  4. #4
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    But H.264 is a lossy compression (it's just MPEG), so the crisp lines and clear fonts on your desktop would be re-rendered as a fuzzy mess at the other end.

    I would suggest you make a local MPEG of your screen to begin with, before trying to implement the rest of the system.
    My guess is that you won't like the quality of the result.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  5. #5
    Registered User
    Join Date
    Oct 2006
    Posts
    3,445
    there's no reason why you couldn't make a relatively simple codec that breaks up the screen into 16x16 blocks or whatever, and only send the blocks that have changed. for remote desktop, most of the content is often pretty static, so the actual fps could be quite high. I believe that this is how services like logmein and teamviewer send screen data, along with UDP pinhole punching.

  6. #6
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    especially with inter-frame compression
    This is a thing I forgot to cover.

    The interactive nature of the thing makes interframe compression a little problematic because the common motion estimation algorithms (recommended by the standard) break with isolated burst of movement such as a cursor.

    You'll probably get severe distortion trying to do this live. You probably can't actually. The system will almost certainly generate a bit of latency due to this.

    I'm still doing research, but it looks like modern video cards can do H.264 encoding extremely fast, with only slight loss of quality compared to x264 at the same bitrate.
    The last stream I investigated was produced May 22.

    The source was from a friendly "Turbine Airship" scene rendered by "Blender" into native 1080p (30 FPS) and converted into "YV12" color space in advance.

    The investigated stream was produced by (an associates) GeForce GTX 680.

    The speed was incredible. (I was told it hit 198 FPS.) The quality was not promising.

    I tried several different encoding schemes with "x264". The quality for the "x264" was speculatively (PSNR and SSIM) better globally and for individually for most frames. I obviously can't watch that much video all by myself so I only have volunteers to go on for a lot of the videos, but the videos I did watch the "x264" stream was visibly better.

    The slowest options passed to "x264" (that was tried) resulted in only 12 FPS. (Overclocked Phenom II 955)
    The fastest options passed to "X264" (that was tried) resulted in a very impressive 84 FPS. (Overclocked Phenom II 955)

    The slowest options passed into my distribution software (using "x264" as the actual encoder) living on my 4 machine (10 core on various processors) farm resulted in about 30 FPS with obvious problems at the scene breaks.

    there's no reason why you couldn't make a relatively simple codec that breaks up the screen into 16x16 blocks
    This is how most already work.

    Well, not exactly, but in general remote desktop software already use techniques to separately send update rectangles.

    Soma

  7. #7
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    But H.264 is a lossy compression (it's just MPEG), so the crisp lines and clear fonts on your desktop would be re-rendered as a fuzzy mess at the other end.

    I would suggest you make a local MPEG of your screen to begin with, before trying to implement the rest of the system.
    My guess is that you won't like the quality of the result.
    That's the plan. I'm trying to figure out exactly how to take snapshots in Win API. Never used it my entire life...

  8. #8
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    The interactive nature of the thing makes interframe compression a little problematic because the common motion estimation algorithms (recommended by the standard) break with isolated burst of movement such as a cursor.

    You'll probably get severe distortion trying to do this live. You probably can't actually. The system will almost certainly generate a bit of latency due to this.
    Hmm that's good to know. Will keep that in mind.

    The last stream I investigated was produced May 22.

    The source was from a friendly "Turbine Airship" scene rendered by "Blender" into native 1080p (30 FPS) and converted into "YV12" color space in advance.

    The investigated stream was produced by (an associates) GeForce GTX 680.

    The speed was incredible. (I was told it hit 198 FPS.) The quality was not promising.

    I tried several different encoding schemes with "x264". The quality for the "x264" was speculatively (PSNR and SSIM) better globally and for individually for most frames. I obviously can't watch that much video all by myself so I only have volunteers to go on for a lot of the videos, but the videos I did watch the "x264" stream was visibly better.

    The slowest options passed to "x264" (that was tried) resulted in only 12 FPS. (Overclocked Phenom II 955)
    The fastest options passed to "X264" (that was tried) resulted in a very impressive 84 FPS. (Overclocked Phenom II 955)

    The slowest options passed into my distribution software (using "x264" as the actual encoder) living on my 4 machine (10 core on various processors) farm resulted in about 30 FPS with obvious problems at the scene breaks.
    That's interesting! I was planning to try NVIDIA's CUDA H.264 library as well as x264 and see what happens. I have a Core i5 at 4.5 GHz, and 2 GTX 560 Ti's, so I don't think computing power will really be a problem (at least at low settings).

  9. #9
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Video encoding with H264 is just an extremely tough nut for video cards. There just isn't enough parallelism and latency overhead is horrible.
    Offloading to GPU has been tried with x264 several times, but all attempts so far has failed AFAIK.
    We may get different results with the recent APUs, though.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  10. #10
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    I was planning to try NVIDIA's CUDA H.264
    ^_^

    That is a completely different beast.

    We were missing each other a little.

    When I was talking about the encoder from modern video cards I was talking about the video cards that literally have a dedicated embedded processor for exactly that purpose.

    I was not talking about using things like CUDA or OpenCL.

    Of course using the massive number cruncher that is a modern GPU for a good software implementation for the relevant filters is fine. I've tested these as well and the quality can be equal to "x264".

    Video encoding with H264 is just an extremely tough nut for video cards.
    It is not.

    Some of the different parts of the H.264 chain aren't really a chain at all. Many parts of the algorithms can be easily done in parallel. The motion estimation and parts of the block compression are completely independent.

    Many filters necessary to get good compression of different types of content can't easily be made parallel. On the other hand, video cards happen to be extremely good at exactly that kind of process completely crushing anything done in software on a general purpose CPU.

    Offloading to GPU has been tried with x264 several times, but all attempts so far has failed AFAIK.
    "x264" is arguably the best software encoder, but it isn't the only game in town. Other encoders live on the GPU and do admirably especially when speed is favored over compression level or quality.

    Soma

  11. #11
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Quote Originally Posted by phantomotap View Post
    It is not.

    Some of the different parts of the H.264 chain aren't really a chain at all. Many parts of the algorithms can be easily done in parallel. The motion estimation and parts of the block compression are completely independent.

    Many filters necessary to get good compression of different types of content can't easily be made parallel. On the other hand, video cards happen to be extremely good at exactly that kind of process completely crushing anything done in software on a general purpose CPU.

    "x264" is arguably the best software encoder, but it isn't the only game in town. Other encoders live on the GPU and do admirably especially when speed is favored over compression level or quality.

    Soma
    That is precisely what I meant by "tough nut."
    Precisely because everything is not fit for the GPU, there is a lot of communication overhead between the CPU and GPU, making it difficult to write software that takes advantage of the best of both worlds.
    Encoders that produce high-quality encoding on modern cards is very difficult. Most GPU encoders that I've heard about produces sub-par quality, but high speed. My guess is that they dumb down the sequential algorithms while taking full advantage of the parallel ones.
    Dunno about how much special chippery in the cards help, though. Never really have done much research into this area; I know mostly just hearsay.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  12. #12
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    O_o

    Really? Okay.

    It sounded to me like you were saying that it didn't work at all.

    Soma

  13. #13
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    So I'm still having trouble capturing the screen.

    Can someone experienced with Win32 API help me out? I imagine it would just be 2-3 lines of code. Capturing the whole frame buffer into system RAM in RGB (or even better, YUV) is all I need.

  14. #14
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    ^_^

    That is a completely different beast.

    We were missing each other a little.

    When I was talking about the encoder from modern video cards I was talking about the video cards that literally have a dedicated embedded processor for exactly that purpose.

    I was not talking about using things like CUDA or OpenCL.

    Of course using the massive number cruncher that is a modern GPU for a good software implementation for the relevant filters is fine. I've tested these as well and the quality can be equal to "x264".
    Ah! I didn't know video cards have dedicated encoding circuitry. I thought they only have decoding circuitry. Maybe in workstation cards?

    But yeah I was talking about, for example, NVIDIA's reference implementation of H.264 using CUDA (NVIDIA GPU Computing Documentation | NVIDIA Developer Zone).

  15. #15
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    Looks like H.264 hardware encoding was added to Kepler (NVIDIA 600 series).
    NVENC
    All Kepler GPUs also incorporate a new hardware-based H.264 video encoder, NVENC.
    Prior to the introduction of Kepler, video encoding on previous GeForce products was handled by encode software running on the GPU’s array of CUDA Cores. While the CUDA Cores were able to deliver tremendous performance speedups compared to CPU-based encoding, one downside of using these high-speed processor cores to process video encoding was increased power consumption.
    By using specialized circuitry for H.264 encoding, the NVENC hardware encoder in Kepler is almost four times faster than our previous CUDA-based encoder while consuming much less power.
    27
    It is important to note that an application can choose to encode using both NVENC hardware and NVIDIA’s legacy CUDA encoder in parallel, without negatively affecting each other. However, some video pre-processing algorithms may require CUDA, and this will result in reduced performance from the CUDA encoder since the available CUDA Cores will be shared by the encoder and pre-processor.
    NVENC provides the following:
     Can encode full HD resolution (1080p) videos up to 8x faster than real-time. For example, in high performance mode, encoding of a 16 minute long 1080p, 30 fps video will take approximately 2 minutes.
     Support for H.264 Base, Main, and High Profile Level 4.1 (same as Blu-ray standard)
     Supports MVC (Multiview Video Coding) for stereoscopic video—an extension of H.264 which is used for Blu-ray 3D.
     Up to 4096x4096 encode
    We currently expose NVENC through proprietary APIs, and provide an SDK for development using NVENC. Later this year, CUDA developers will also be able to use the high performance NVENC video encoder. For example, you could use the compute engines for video pre-processing and then do the actual H.264 encoding in NVENC. Alternatively, you can choose to improve overall video encoding performance by running simultaneous parallel encoders in CUDA and NVENC, without affecting each other’s performance.
    NVENC enables a wide range of new use cases for consumers:
     HD videoconferencing on mainstream notebooks
     Sending the contents
    http://www.geforce.com/Active/en_US/...aper-FINAL.pdf

    They advertise "8x real-time 1080p encoding" on GTX 680. Which means, a dedicated low end card (GT 630 or something) should be able to do real time encoding.

    Now I want to get a Kepler to try it out (hopefully getting one for free at the end of my internship at NVIDIA in 3 months...).

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Desktop video program
    By jmd15 in forum Tech Board
    Replies: 3
    Last Post: 08-30-2006, 10:23 AM
  2. Replies: 3
    Last Post: 04-24-2006, 07:45 PM
  3. Desktop of remote PC
    By Unregistered in forum Windows Programming
    Replies: 11
    Last Post: 10-10-2001, 05:26 PM