Thread: Video encoding questions

  1. #1
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229

    Video encoding questions

    I'm thinking about building a device that will stream realtime VGA (640x480) compressed video, over a fast serial link (SPI).

    The processor will be embedded, so computing power is the major constraint (best if it can be handled by an ARM at 72MHz, but I can potentially use a 400MHz DSP). Bit rate is the second concern, because the data will be piped through lower speed chips, and maybe even wireless.

    What encoding algorithm would you recommend?

    I've done about an hour of research, and found out that the most common algorithm for things like USB webcams and digital cameras is MJPEG. Basically a stream of independent JPEG images.

    That should be computationally cheap, at the expense of size, vs something like MPEG, but I'm pretty sure something like H.264 will be too expensive.

    Now my more concrete questions -
    How much space saving is there generally between inter-frame compression algorithms like H.264, and MJPEG? (at similar quality) Surprisingly I couldn't find any benchmark on this.

    Are there any popular algorithm that's computationally cheaper at the cost of slightly bigger file size?

    Can I do my own simple inter-frame encoding? Eg. running binary diff on BMP files of 2 consecutive frames?

    Theoretically I can also throw in an FPGA and do some hardware acceleration, but I couldn't find any free video encoding IP.

    I know the requirements are really vague, that's because I don't know what I am doing .

    The device I'm trying to build will have a cheap camera (cellphone camera style), connected to a processor/microcontroller that will compress the video stream, and send it to the "rest of the system", which will just merge the stream with its own low bandwidth data, and send it over a wireless link.

    Thanks

  2. #2
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    Something like this -
    SparkFun Electronics - JPEG Color Camera - UART Interface

    Except higher frame rate, on a faster data link (UART/RS232 goes up to about 100kbit/s, SPI can do 40mbit/s).

  3. #3
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    How much space saving is there generally between inter-frame compression algorithms like H.264, and MJPEG?
    O_o

    It can be huge. (I've seen an easy %50 for a recorded graduation ceremony.) It can also be none. The idea is, if you can afford it you absolutely should use an interframe compression algorithm. Even if nothing is shared between target frames the overhead of the interframe compressed stream is generally negligible compared to the same data using simpler intraframe algorithms.

    Can I do my own simple inter-frame encoding?
    Yes. I may be willing to discuss this with you a little more to get you started, but I don't know where in the world you exist. It would have to be private communication.

    Eg. running binary diff on BMP files of 2 consecutive frames?
    That's actually a really bad idea.

    The human eye just isn't that great. A dumb scheme for binary comparison and delta recording is guaranteed to manage a lot of redundant data when you realize that we can't tell the difference between subtle variations of color and intensity.

    If you aren't careful with your packages, you may lose seconds of video for a virtually instantaneous transmission problems. The overhead of getting this right with a simple scheme can be a lot more than the necessary overhead for more complex encoding. (In other words, in my opinion you would be using precious resources in the wrong way.)

    Soma

  4. #4
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    H.264/MPEG-4 AVC - Wikipedia, the free encyclopedia
    Your ARM CPU seems typical of current crop of non smartphone handsets. Of those that are capable of video telephony, the performance is down at level 1 for a small display.

    You may need that DSP if you want the image size and frame rate. The bit-rate won't be an issue for your interface if you can hack the math in the CPU(s)

    Motion JPEG - Wikipedia, the free encyclopedia
    Like it says, less CPU, but less compression. But probably still within the bandwidth of your SPI interface.

    It should be relatively easy to benchmark this approach to see if it's worth pursuing.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  5. #5
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    It can be huge. (I've seen an easy %50 for a recorded graduation ceremony.) It can also be none. The idea is, if you can afford it you absolutely should use an interframe compression algorithm. Even if nothing is shared between target frames the overhead of the interframe compressed stream is generally negligible compared to the same data using simpler intraframe algorithms.
    I see! Yeah the main problem is computing power. Also memory (assuming interframe would require storing several uncompressed frames in memory).

    Yes. I may be willing to discuss this with you a little more to get you started, but I don't know where in the world you exist. It would have to be private communication.
    Thanks for the offer, but I would really want to keep this public. Also, after doing some more research, I don't think it's really necessary to do my own encoding.

    That's actually a really bad idea.

    The human eye just isn't that great. A dumb scheme for binary comparison and delta recording is guaranteed to manage a lot of redundant data when you realize that we can't tell the difference between subtle variations of color and intensity.
    That's true. I really want something lossy.

    Your ARM CPU seems typical of current crop of non smartphone handsets. Of those that are capable of video telephony, the performance is down at level 1 for a small display.

    You may need that DSP if you want the image size and frame rate. The bit-rate won't be an issue for your interface if you can hack the math in the CPU(s)
    Yeah I don't think the ARM can do it. ~100MHz is about as fast as they go. Anything faster (eg. Cortex-A8) would be a lot more complicated because it would just be a naked processor, and require a bunch of supporting circuitry, instead of a nicely packaged microcontroller that includes RAM, flash, and peripheral.

    Motion JPEG - Wikipedia, the free encyclopedia
    Like it says, less CPU, but less compression. But probably still within the bandwidth of your SPI interface.

    It should be relatively easy to benchmark this approach to see if it's worth pursuing.
    I chose SPI just so the bus bandwidth won't be the limiting factor. But storage requirement and wireless streaming bit rate still are. The data has to go somewhere eventually.

    With 40Mbps I can probably even do uncompressed stream.

    Benchmarking is a good idea. I didn't want to do it because I'll have to build the hardware first. But I guess I can do it on a PC, and assume it will scale relatively linearly.

  6. #6
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    And I just found VHDL (hardware) implementations of both MJPEG and H.264.
    Video compression systems :: Overview :: OpenCores
    JPEG codec library based on microblaze :: Overview :: OpenCores

    Will need to give them a closer look, but they look promising.

    A compatible FPGA -
    Digi-Key - 122-1340-ND (Manufacturer - XC3S200-4TQG144C)

    It's similarly priced to the DSP, but I'm not sure if the DSP can actually do it.
    Digi-Key - ADSP-BF531SBSTZ400-ND (Manufacturer - ADSP-BF531SBSTZ400)

    an FPGA running at 100MHz should have much higher performance than the DSP at 400MHz, and designing a 400MHz PCB would be challenging.

    EDIT: looks like Blackfin has an implementation http://www.analog.com/en/embedded-pr...s/product.html

    It says at 600MHz it can do 720x576 at 15fps. So that's around 10fps at 400MHz. I guess that's acceptable.
    Last edited by cyberfish; 10-28-2010 at 12:09 PM.

  7. #7
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    assuming interframe would require storing several uncompressed frames in memory
    This is something I probably shouldn't discuss in public. ^_^;

    I would really want to keep this public
    I appreciate that. Let's just say I have a little paid experience that I believe can't legally be exported outside of the United States. I could give some guidance in private, but I'm better off not getting involved in a too technical discussion in public view. I'd likely describe a process I shouldn't even be discussing.

    Yeah I don't think the ARM can do it.
    You need to understand that my statement about the differences was related to the target video size and an assumed framerate. At a low resolution the gains from interframe compression can be significant, but as the resolution and the bitrate for a given quality grows very high the expense of interframe compression doesn't provide a very good balance.

    As an example, I've measured an MJP2 file (rendered scene) as being well over twice the size of an equivalent h264 file for a film NTSC stream but the MJP2 was only a few tens of MiB larger than the h264 for an 8k stream.

    This is where the highly adjustable h264 baseline actually shines when compared to some other encoding. You can literally ignore parts of the production algorithm and still produce a compatible stream. (It isn't uncommon for main profile encoder implementations to completely ignore "B Slices".)

    I seriously doubt that a 72MHz ARM will get you a level 3 stream, but my point is, if you are going to spend a little extra money on a DSP or FPGA to process video at that resolution you are really best served making sure that it can handle the h264 stream and not just MJPEG even if it costs you a few extra dollars.

    I didn't want to do it because I'll have to build the hardware first. But I guess I can do it on a PC, and assume it will scale relatively linearly.
    I'm just guessing, but I think the suggestion was to compare various videos using both codecs while simply assuming that you could hit your target FPS and resolution with either implementation. You can do that easily with a few commands to MEncoder. Knowing those results you would know if the MJPEG stream would fit your bandwidth requirements. If the simpler MJPEG streams fits your bandwidth requirements you don't need to go after the h264 implementation.

    It says at 600MHz it can do 720x576 at 15fps. So that's around 10fps at 400MHz. I guess that's acceptable.
    I obviously don't know the kind of feed you'll be getting or the nature of the codec implementation you will ultimately use, but you may be able to simply drop one of the "values" (saturation would be a favorite) to get a higher FPS low quality video for maneuvering switching to a normal feed for when you want to record what you are actually doing.

    Soma

  8. #8
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    Yeah I understand what you mean. I wouldn't want you to get into trouble for breaching NDA.

    Thanks for the insight on intraframe/interframe. I've been looking for information like that for hours, and couldn't find any from Google.

    I'm pretty sure if I'm going DSP or FPGA, I will want H.264. MJPEG would be if I decide to stick with just the lowly ARM (even then, I'm not sure if the ARM can handle that). Microcontrollers are really not suitable for this kind of heavy computations.

    I obviously don't know the kind of feed you'll be getting or the nature of the codec implementation you will ultimately use, but you may be able to simply drop one of the "values" (saturation would be a favorite) to get a higher FPS low quality video for maneuvering switching to a normal feed for when you want to record what you are actually doing.
    This would be more for something like pure live-streaming. Feed will be from a webcam mounted on a vehicle or aircraft for example, so there should be a lot of room for interframe compression.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. A few encoding and replacement questions...
    By NaartjieDude in forum C Programming
    Replies: 7
    Last Post: 05-01-2010, 03:59 AM
  2. Question about Video Encoding
    By merlin2011 in forum C Programming
    Replies: 2
    Last Post: 11-17-2008, 02:54 AM
  3. A very long list of questions... maybe to long...
    By Ravens'sWrath in forum C Programming
    Replies: 16
    Last Post: 05-16-2007, 05:36 AM
  4. OpenGL .dll vs video card dll
    By Silvercord in forum Game Programming
    Replies: 14
    Last Post: 02-12-2003, 07:57 PM