Where can I find information about segmented downloading?

[Once-ler2]

I am wanting to do segmented video encoding and thought that this would be a good thing to look up. I mean if download managers can put the file together exactly as it is supposed to be shouldn't I be able to do the same with video or audio files? I mean have separate threads working on different parts of the video and piece them all together at the end. Thanks

[rcgldr]

My understanding of video encoders or at least some of them is that they separate each video frame into segments (2 segments for 2 cores, 4 segments for 4 cores, ...), as opposed to working on different parts of the video stream. I'm not sure how they deal with the borders of each segment.

For audio, the left and right channels could be worked on in parallel. If more channels are used in an audio stream, like surround sound, then more cores could be used.

[phantomotap]

I mean if download managers can put the file together exactly as it is supposed to be shouldn't I be able to do the same with video or audio files?

O_o

Whether or not you can "segment" encoding/decoding of an image depends entirely on the algorithms and data on which the "codec" operations.

For example, if you are doing encoding of macroblocks for intra-frame compression, you can easily give each thread a block of its own; as each block always requires and represents the same number of pixels, the logical is trivially split over multiple threads.

However, if you are doing variable bit encoding for inter-frame compression using some form of motion prediction and encoding, you would probably be worse off for using threads because you need to consume and produce heavily varied amounts of data; a better approach might be splitting each step of the process into a thread from a pool and producing a queue of "work" to be used in each successive step.

Of course, everything I could say would be a guess without knowing which "codec" you are using. The point is, generating a new video stream isn't the same as downloading an existing file.

Soma

[vart]

H264 codec uses NAL-Units to "segment" the video
NAL-Unit could be as big as one frame...
Or as small as one line of macro-blocks

Intel IPP encoder when run in multi-threaded mode encodes each NAL-Unit separately as a Task. Each task is assigned to the free working thread.
(720p frame has dimension of 80x45 macroblocks, so each frame could be split into 45 NAL-Units, so 45 working threads could work in parallel on one frame encoding...)

Of course all NAL-Units should be concatenated in write order into frame before next frame encoding could start.

Intel IPP library is free for personal use on linux - so you can downloaded it as well as sample H264 encoder/decoder application and see the code by yourself.

[Once-ler2]

Where can I find some C source for segmented downloading? I am really interested to see how this actually works. Surely there must be some source out there, so far google has turned up a bunch of nothing. Thanks

[jwroblewski44]

Do some research on Linux ultra fast command line download accelerator. It's not actual C source, but it will give you an idea of where to start.