Video-based remote desktop

[phantomotap]

Can someone experienced with Win32 API help me out?

I hate the "WIN32" API, but I do have this done in my example source tree.

    PAINTSTRUCT ps;
    HDC hdcScreen = GetDC(NULL);
    HDC hdcTarget = BeginPaint(hwnd, &ps);
    BitBlt(hdcTarget, 0, 0, 640, 480, hdcScreen, 0, 0, SRCCOPY | CAPTUREBLT);
    EndPaint(hwnd, &ps);

Soma

[cyberfish]

Thanks!

I'll try to hack something together tonight, or something.

[cyberfish]

I hacked together something to see how fast the BitBlt is, and it's pretty disappointing .

// sstest.cpp : Defines the entry point for the console application.
//

#include <Windows.h>

#include <iostream>

#include <time.h>

#include "stdafx.h"

static const int WIDTH = 1920;
static const int HEIGHT = 1080;

int _tmain(int argc, _TCHAR* argv[])
{
	HDC hdcScreen = GetDC(NULL);
	HDC hdcTarget = CreateCompatibleDC(hdcScreen);

	HBITMAP bmp = CreateCompatibleBitmap(hdcScreen, WIDTH, HEIGHT);

	SelectObject(hdcTarget, bmp);

	BITMAPINFO bminfo;

	time_t begin = time(0);
	for (int i = 0; i < 100; ++i)
	{
		BitBlt(hdcTarget, 0, 0, WIDTH, HEIGHT, hdcScreen, 0, 0, SRCCOPY | CAPTUREBLT);
	}
	time_t end = time(0);

	std::cout << "Frame rate: " << (100.0 / static_cast<double>(end - begin)) << std::endl;
}

I'm getting 30 fps idle on my laptop (low end Quadro + 2.7 GHz i7), and 10 fps with just a youtube video playing.

Am I doing it wrong? Or is there something with a little higher performance?

[cyberfish]

Just did a lot more research... it looks like fast screen capturing is just not possible. That's very strange. Definitely not what I expected the bottleneck to be. I guess there's no sense continuing if I can't get fast screen capturing.

Looking into hardware solutions now (capture DVI). Looks like they are all insanely expensive, though.

[phantomotap]

O_o

I have no idea what you are doing or why your results suck so bad, but I image it is because you are running Windows 7 and the `bitblt' function has to jump through extra hoops because of some flaky crap they did with "UI" elements. I suggest you try turning off those "UI" elements long enough to run the test. Alternatively, you'll also seem some slowness when video plugins run through a "DirectShow" filter. (I've never understood why.)

I ran your test; it failed. I changed the frame count to 1000; it ran fine and told me I had over 200 frames per second.

I built a simple demo application with a "flip chain" to further test the idea. It wasn't extremely fast, but I was able to run four instances of the application with a "Youtube" stream in the background. I even layered them (I've looked into the "Youtube" abyss.) and still had enough juice I had to insert a yield to keep the thing from breaking.

The "GDI" API isn't the fastest. The `bitblt' function isn't the fastest part of "GDI". The function is used everywhere in the GDI and so has seen some heavy investment in building performance. With the machine you say you have (faster than my development laptop by a lot) you should see a faster FPS than that.

[Edit]
^_^;
[/Edit]

Soma

[cyberfish]

That is very strange. This is on my work laptop, so there is the UI crap going on.

I disabled Aero and am now getting 270 fps idle and 300 (!) with Youtube.

My eventual goal is remote gaming (like OnLive), which I'm assuming (may be wrong?) will be too much for GDI, but that can wait.

On to H.264 encoding tonight!

[cyberfish]

In the mean time, I've looked into hardware capture cards (which of course won't care whether it's Solitaire or a 3D game running). Something like this -
Amazon.com: StarTech.com PCI Express HD Video Capture Card 1080p - HDMI/DVI/VGA/Component TV Tuners/Video Capture (PEXHDCAP): Electronics
looks like it should work. No idea about latency or software support, though.

Another brand -
http://www.amazon.com/Blackmagic-Int...8942443&sr=1-3
Amazon.com: Blackmagic Design Intensity Pro - HDMI and Analog Editing Card: Electronics
is more expensive but has good reviews and good support from what I heard.

Will hold off on those for now though, until I can prove the concept works with BitBlt, AND that BitBlt is too slow (for 3D gaming).

[Elysia]

I'm guessing it is because GDI is legacy stuff that's non-accelerated. You'd be better off with modern APIs, such as GDI+ (I think?) or just D2D or D3D, if it is possible via them.
Heck, if it's gaming, then isn't that all the more reason why D3D/D2D wouldn't work? I mean, there are already capture programs such as Fraps that does this with reasonable framerate already (remember, they compress the video, as well).

[phantomotap]

I'm guessing it is because GDI is legacy stuff that's non-accelerated.

The GDI system on "Windows Vista" wasn't accelerated; it has some hardware acceleration on "Windows XP" and "Windows 7".

You'd be better off with modern APIs, such as GDI+ (I think?) or just D2D or D3D, if it is possible via them.

The GDI+ UI elements are generally part of the reason that `bitblt' for capture is slower when those elements are used. The GDI+ system employees are more abstract view of the world than GDI allowing certain operations to be performed transparently by the system better than GDI+; the more abstract view gets in the way of raw manipulation making it slower for most things that GDI does than GDI.

Soma

[SMurf]

I would wager that you're going to need to make a video driver to get the desktop capture performance that you desire.

The main issue is that all of the graphics operations performed by Windows writes into VRAM in various ways (straight blitting, hardware acceleration, etc.), then you're asking it to read it back. Unless things have changed recently, VRAM has never been quick to read back into system memory.

You are likely to need a driver that presents a "virtual" display which is just a region of system memory. Naturally this will negate the possibility of hardware acceleration (no Aero for you). Being at the end of the rendering chain, all you then need to do is pass the region to an encoder.

[cyberfish]

Didn't get around to posting about it last night, but I did some experiments ("feasibility study") with screen recording at different bitrates.

Turned out I didn't have to write any code at all. Someone already wrote a virtual DirectShow source using bitblt (I checked the source code - it basically does the same thing). https://github.com/rdp/screen-captur...o-windows-free

Coupled that with FFmpeg (x264 backend), I was able to get very good results.

ffmpeg -f dshow -rtbufsize 2048m -i video="screen-capture-recorder" -f dshow -i audio="virtual-audio-capturer" -r 20 -t 30 -b 2m screen_capture.mp4

http://matthewlai.ca/download/screen_capture.mp4
(for some reason WMP won't play it, even though it plays most other H.264 files. VLC works, though)

This is ~1mbps average. Definitely very usable. Especially if you look at the scrolling performance. Scrolling has always been a big problem with remote desktop solutions, but I suppose motion estimation and compensation help greatly here.
I have not seen any remote desktop system that can do anything like this with just 1mbps.

There is definitely some frame dropping going on, and my CPU (this is my work laptop - dual core i7, 2.7GHz) usage was at 100%, so it just couldn't encode fast enough. Should not be a problem using CUDA or a faster CPU, though, since even this one, which is very slow by modern gaming computer standard, can almost do it.

I tried some gaming yesterday, too. With a not-so-demanding game (Mass Effect 2), I was able to record and encode fairly smoothly. Definitely some frame dropping going on there as well, though, but quality was excellent for 1 mbps.

[Elysia]

You are likely to need a driver that presents a "virtual" display which is just a region of system memory. Naturally this will negate the possibility of hardware acceleration (no Aero for you). Being at the end of the rendering chain, all you then need to do is pass the region to an encoder.

Clearly it is possible to do capturing without killing Aero. There are plenty of software out there that does this already!

There is definitely some frame dropping going on, and my CPU (this is my work laptop - dual core i7, 2.7GHz) usage was at 100%, so it just couldn't encode fast enough. Should not be a problem using CUDA or a faster CPU, though, since even this one, which is very slow by modern gaming computer standard, can almost do it.

I tried some gaming yesterday, too. With a not-so-demanding game (Mass Effect 2), I was able to record and encode fairly smoothly. Definitely some frame dropping going on there as well, though, but quality was excellent for 1 mbps.

Try faster x264 settings.
Heck, why don't you try x264 directly?

[phantomotap]

for some reason WMP won't play it, even though it plays most other H.264 files.

Using "ffmpeg" to produce a MP4 package on "Windows" is somewhat broken in a way. It probably will not be an issue for you but if it is try a different package format.

Heck, why don't you try x264 directly?

Using "x264" directly can't get him a mixed stream without also running through another program.

Soma

[cyberfish]

I would wager that you're going to need to make a video driver to get the desktop capture performance that you desire.

The main issue is that all of the graphics operations performed by Windows writes into VRAM in various ways (straight blitting, hardware acceleration, etc.), then you're asking it to read it back. Unless things have changed recently, VRAM has never been quick to read back into system memory.

You are likely to need a driver that presents a "virtual" display which is just a region of system memory. Naturally this will negate the possibility of hardware acceleration (no Aero for you). Being at the end of the rendering chain, all you then need to do is pass the region to an encoder.

I believe that's what programs like Fraps does. Unfortunately, it looks like it can only record D3D applications (or the desktop if Aero is enabled). There's no open source implementation for this technique, however, and I know way too little about Windows API/DX to do something like this (unless someone wants to jump in!)

Try faster x264 settings.

I would, but this is just a proof of concept at the moment. I'd just throw more computing power at it . This experiment is just to prove that 1mbps desktop streaming is definitely usable and way better than any remote desktop solution I've tried.

Next thing to prove is latency, but that's a lot more hardware and network dependent.

If there's no solution to capturing flakiness, the only option seems to be looping the video card's output back through a capture card. That is not very efficient, however, and I'm not sure what that will do to latency.

[Elysia]

Well, CamStudio is open source and can capture your desktop, so I wager it's worth taking a look at if you want better or different capturing if you aren't happy with what you have now.
For latency, I'd wager x264's zero-latency feature(s) could come in handy here, if you can access them via ffmpeg (dunno how they work, though). I'd also use crf and not bitrate, if possible.