Thanks. I'm looking into it now. 20 minutes of browsing source code and still can't find the capture function. Ugh.Well, CamStudio is open source and can capture your desktop, so I wager it's worth taking a look at if you want better or different capturing if you aren't happy with what you have now.
I do understand it's usually better to use crf, but for network streaming the restriction is really bitrate.I'd also use crf and not bitrate, if possible.
Some gaming footage. Not a very demanding game. Microsoft Flight. There is noticeable lagging, though (in the video, not game).
http://matthewlai.ca/download/screen_capture2.mp4
And also, when I tried full screen it would only record a black screen.
Looks like a capture card is really required for gaming.
PS. Don't worry. I don't fly like that in real life.
Wrote some code to test it out (see attached). Visual Studio 2010, x86-64 only (because I didn't bother building 32-bit x264), all libraries included.
It captures the primary screen using the GDI BitBlt method, and encode them in real time using libx264 ("veryfast" preset, "zerolatency" tune), and store the raw bitstream in test.h264 for 10 seconds. Not many players can play the raw stream. ffplay can, or you can mux it into MP4 with MP4Box. VLC reportedly can, but just crashes for me.
It looks like on my machine (Intel Core i5 Quad at 4.6 GHz), with 1680x1050 res, I can get about 90 fps with 4 threads (~60% utilization across all cores), and 45 fps with 1 (assuming > 30 fps at 1080p).
So it's totally possible to just dedicate 1 core to encoding, and still have 3 cores for doing whatever the computer needs to do. Can probably do better with CUDA.
Capturing performance is ~300 fps.
Latency has not been verified, yet. Going to build a real time viewer to verify it. Time to read up on decoding libraries...
EDIT: looks like CBoard doesn't like 4MB archives. Uploaded here -
http://matthewlai.ca/download/DesktopStreamer.zip
Last edited by cyberfish; 06-10-2012 at 05:27 PM.
Looks like OnLive is experimenting with something like this already - OnLive Desktop - Simple PC App Access from Anywhere
But AFAIK, still no open source, or even publicly-released solution. Maybe I'll be the first!
Mkv is excellent for streaming. You should try it.
Originally Posted by Adak
O_oMkv is excellent for streaming. You should try it.
What are you trying to get at?
"MKV" is just a container format; one that is less widely implemented than simple "MP4".
If he is set on using "H264" for the stream properties changing the container will not do much of anything beyond that.
Soma
It depends on the implementation, but a typical MP4 file produced by x264 won't be streamable (can't watch it while downloading), but that's not the case with MKV. That's all I'm implying.
Ah I won't be using any container. The NAL stream will be generated in memory, transmitted over UDP, decoded in memory, and displayed. It shouldn't hit the disk any time.
I only made it write to disk to verify that I have the encoder set up correctly and it's working correctly.
That's completely wrong.It depends on the implementation, but a typical MP4 file produced by x264 won't be streamable (can't watch it while downloading), but that's not the case with MKV. That's all I'm implying.
The "H264" video stream produced by "x264" is generally stream ready; some options produce a stream that is undesirable for streaming. (To few key frames, for examples, makes seeking horrible broken.)
The "MP4" file (container with "H264" produced by "x264") produced by "mencoder" or "ffmpeg" can be broken in a lot of ways; this "brokeness" affects almost every other player whether a file or streaming.
The "H264" (as a "raws" file) produced by "libx264" only needs an atom (essentially a tag) moved to the first of the file in order for it be streamable. (This is because it naturally exists at the end of the file produced by "libx264" where players would need the entire file to read it.)
The "MKV" container is designed as a generic container. You can put the stream information at the beginning of the file, but you can also put it at the end where it would be unsuitable for streaming.
Be aware that you are responsible for syncing audio and video. A lot of embedded players simply aren't smart enough to request different frame identifiers if the stream gets out of sync.The NAL stream will be generated in memory, transmitted over UDP, decoded in memory, and displayed.
Soma
It's not completely wrong.
As far as x264's implementation goes, streams muxed into MP4 aren't streamable. You need to remux the stream afterwards to make it streamable.
MKV produced by x264 when encoding can be both streamed and watched while still being encoded.
I don't know how much the standards for these formats specify about this issue, though. Nevertheless, these are the limitations imposed by x264's implementation and whatever 3rd party libraries it may be using.
I was planning to just stream them independently. If the latency of video and audio are similar, they should be inherently in sync already, due to the real time nature.Be aware that you are responsible for syncing audio and video. A lot of embedded players simply aren't smart enough to request different frame identifiers if the stream gets out of sync.
Also, I'm probably not going to use an embedded player. I'll just use libavcodec to decode the H.264 stream, and blit the RGB images onto a SDL surface or something for display.
If you're streaming over a network, you can't guarantee that. You are intending to stream this over a network, right?
That is a terrible mistake waiting to happen all over you. We are talking "10 gallons of blood under higher pressure (*)" Japanese martial arts films "happen".I was planning to just stream them independently. If the latency of video and audio are similar, they should be inherently in sync already, due to the real time nature.
That you are streaming the (>_<) streams independently allows you all kinds of wonderful opportunities for an occasional sync fix.
^_^I'll just use libavcodec to decode the H.264 stream, and blit the RGB images onto a SDL surface or something for display.
I got $5 USD that says you get greater latency from "libavcodec->SDL->screen" than from "desktop->router->wireless access point->wireless nic".
*shrug*
That aside, again, that you are going to code the client side yourself gives you a lot of opportunities for a sync. Heck, if you are going to add an API layer over "HTML" or even just raw "TCP" you can easily plan on the occasion of needing mutual sync commands even if you never get so far as implementing them.
Soma
(*) I can't recall where this is actually from. Apologies.
Hmm that's interesting. Are you talking about short term out of sync (a few frames) or long term drift?That is a terrible mistake waiting to happen all over you. We are talking "10 gallons of blood under higher pressure (*)" Japanese martial arts films "happen".
That you are streaming the (>_<) streams independently allows you all kinds of wonderful opportunities for an occasional sync fix.
I don't think long term drift will be a problem because the source is inherently synchronized. If they somehow get out of sync, that means one of them has very high latency, in which case I'd have big problem anyways. If the (very small) buffers get full, I'd be dropping frames/samples anyways for the network to keep up.
Not really sure how short term syncing can be done (every frame?). I'm not sure if I want to, for example, delay a video frame if audio is not here, yet.
Why is that? I thought libavcodec would be very fast (much faster than encoding) on modern CPUs. I don't want to use hardware acceleration (at least for now) because there is (AFAIK) no cross-platform way to do it, and I want to keep the client cross platform because there's no reason why it shouldn't be.I got $5 USD that says you get greater latency from "libavcodec->SDL->screen" than from "desktop->router->wireless access point->wireless nic".
Copying to SDL should be at most just a ~6MB memory copy + a colorspace conversion.
