Distributed Video Compression Idea

[Emp3r0r]

With video coding complexity increasing (plus prefiltering), and HDTV resolution video becoming more popular, today's high-end machines are still way too slow for the job at hand.

Compressing a 1920x1080 source with avisynth and xvid is a long process when using VHQ, AQ, etc. So I was thinking, wouldn't it be nice if I could harness multiple PCs to decrease the time required.

I know there are many bandwidth issues involved with a distributed enabled codec, so why not just hack the job up into pieces and let each client machine do a portion of the encode.

Say you have 7 client machines all with avisynth (and plugins), dgindex, xvid, and avs2avi. You have a server that has around 70 256meg mpg files created using HDTVtoMPEG2 from a 18gig transport stream. The server starts sending each machine a mpg file which would then dgindex and generate an avsscript from a template and encode to xvid with avs2avi. It would send back the resulting avi to the server. Each client would have only 10 MPG files to index and encode thus greatly decreasing the overall compression time.

What do you think? Is it feasible?

edit: network would be gigabit

[DoC hEx]

A year or so ago I was looking into doing something similar. Theoretically it would be possible. From my limited understanding of XivD I believe it looks at the first pass using best quality compression. Then at the second pass calculates all the fluctuations in bitrate needed to achieve the desired total file size (assume working for total filesize). The problem would be if during the compression it were to dynamically change the originally calculated amount of bits for a series of frame, because it was unable to get this compression. If the file were split up into multiple jobs and this happened the other codecs wouldn’t know to adjust. This assumes filesize limits are a priority.

But if you just wanted a constant bitrate it shouldn’t be a problem to setup a couple of AVS scripts that cut up the movie and then at the end join them all back together.

7 PCs, Movies Length = 145410 frames
145410 / 7 = 20772 frames for each job and 3 frames extra for one client.

Issues:

Cutting the files for transfer means the files need to be processed in some manner to ensure that they always have a key frame for the first frame in a sequence. That’s assuming you’re going to create 70 individual files for transferring to clients. Then each of these files still needs to be transferred to the clients, try getting 7 clients copying all those files at once and see how good our IO system is.

How long will it take the one PC to create these 70 files and then transfer them?

What about the Audio track?

Would it be quicker to have one PC create a single large dgindexed file that will be used to stream the video as required to each PC? So you have one file on one fileserver with a high-speed file system to fuel your Gb net, i.e. (RAID-0 Stripped Drives to get max performance. Most motherboards have a 120MB/s PCI BUS speed limit depending on the setup so this might turnout to be your bottleneck, then there’s the 33Mhz frequency PCI... sucks!) But it should still easily be able to maintain 5MB/s transfers to all the clients without having to go SCSI or 10,000rpm SATA. The client PCs CPUs should be the bottleneck to encoding speed.


It should be simple enough to setup a test example of the above: one big file, file serving only the required parts per AVS Script client requests. Then kick off all the encoding at once on each PC to get an idea if there is enough IO bandwidth. If you’re into programming you could code this in a day if you already knew your way around VB or Delphi, as all the code that would be required is available.

I’ve a server that acts both as a fileserver and an encoder as it has to be on 24/7 anyway. I queue up a job I want done and it’s ready the next day. I could only see digital studios or broadcast companies needing to be able to compress HDTV signals in very short time spans, due to quantity or turn around of material. If you need quantity then 7 PC doing 7 files would workout to have a similar total speed. Also if one PC does something it shouldn’t you don’t have to worry about the whole process getting messed up, as you’ve just increased the chances of a crash by a factor of 7.

[Emp3r0r]

interesting thoughts, i wish i had some spare time to hack someting together. it would be very cool to have a networked encoding queue.

[Selur]

haven't tried it in a long time but www.vidomi.com offers a network encoding mode, may be this helps,..

there's also a modified vidomi, which should work with normal xvid builds, see:
http://www.alexander-oelmaier.de/

----

could all be bogus, just thought I mention it, since they support some sort of network encoding.

Cu Selur

[pieter1976]

I have a idea of improving mjpeg.
With the normal mjpeg encoding 8x8 dct blocks are created at the same position every time. What if you changed the start position for every frame wouldn't that give a better moving image?

With the start position I mean then x,y point.



x
-------|---
|
|
y- *ooooooo
| oooooooo
oooooooo
oooooooo
o8x8dcto
oooooooo
oooooooo
oooooooo

* = start position

[DoC hEx]

Why should it give a better picture???

[DaveEL]

2 pass encodes would be easy to split across multiple computers if you can get them the source data fast enough.

First pass just give them half the video each and append the two stats files at the end. Ok you will get an extra keyframe where the videso join but it won't effect the quality much at all. Then for the second pass give the machine doing the first part of the movie the whole stats file but only the first half of the frames. The second computer should be given all the data in the stats file but the data for the frames being processed need to be moved to the begining of the stats file. Then append the two videos. All xvid settings should be normal each computer should be asked for a file the size you want the video to finish once both are joined.

DaveEL

[Dark-Cracker]

the fact the rejoin the pass file in a big one is a great idear but how to ask the codec to stop (or start) the second pass at a given frame ?

[pieter1976]

Why should it give a better picture???


Because the 8x8 dct block patterns would be less visible.
This is when the start position is randomly.

I haven't tested it yet but I think it should have a positive effect.
Can this be tested with a avisynth script?

[DaveEL]

Originally posted by Dark-Cracker
the fact the rejoin the pass file in a big one is a great idear but how to ask the codec to stop (or start) the second pass at a given frame ?

Only give it the frames you want to encode for the stop point for the start point same again but you reorder the frames in the stats file.

requires no nodification to the codec

DaveEL

[Dark-Cracker]

hum but in this case why join the 2 pass files ?

why not made the 2 pass given them the average bitrate desired and join the 2 final .avi file ?

i was thinking the fact to join the 2 pass file was used to allow the codec to made a better scale.

[DaveEL]

Originally posted by Dark-Cracker
hum but in this case why join the 2 pass files ?

why not made the 2 pass given them the average bitrate desired and join the 2 final .avi file ?

i was thinking the fact to join the 2 pass file was used to allow the codec to made a better scale.

You give the codec a full stats file so it does correct curve scaling and you get file size predictability. The codec will assume its going to get the same number of frames in the second pass and calculate the frame sizes

DaveEL

[morsa]

Pieter,
Wouldn´t be better to implement the overlapping blocks method instead?

[pieter1976]

morsa ,I did read something on the internet about that.
It uses mdct for images like mp3 does for audio.
But this would complicate things and make it slower.

Do you think that my idea would work?

[Dark-Cracker]

ok i now understand.
for example in a 2000 frames movie u encode it on 2 computers (encode each 1000 frame on each computer).

and for the first file u create the logfile of the 2000 frames with the 0 - 999 at the top of the pass file , and u encode the second file with the same pass file but u move the 1000-2000 frame at the top of the pass file.

but i think even if u gave at the codec a stat file with 2000 frames and u encode a .avs file with only 1000 frames using this pass file , i think it will not work (the developpers have surely made a check to see if the number of frame is equal even if in c++ the array check is not really common).

it will be interesting to have the opinion of someone who have some good knowledges on curve compression to understand which way is the best to avoid to disturb the curve adjustement.

++

[DaveEL]

Originally posted by Dark-Cracker
ok i now understand.
for example in a 2000 frames movie u encode it on 2 computers (encode each 1000 frame on each computer).

and for the first file u create the logfile of the 2000 frames with the 0 - 999 at the top of the pass file , and u encode the second file with the same pass file but u move the 1000-2000 frame at the top of the pass file.


Exactly.


but i think even if u gave at the codec a stat file with 2000 frames and u encode a .avs file with only 1000 frames using this pass file , i think it will not work (the developpers have surely made a check to see if the number of frame is equal even if in c++ the array check is not really common).

No the codec has no idea how many frames it is going to be given until it gets the last one, so it cannot check even if it wanted to. Xvid assumes that it will get the number of frames in the stats file and scales the curve accordingly so each frame is the right size to get the desired output size. This will introduce a small error due to not being able to use overflow from one half in the other but as long a the clips are reasonable long (ie films) and not in too many parts (3-4 max i guess) it shouldn't present too much problem.


it will be interesting to have the opinion of someone who have some good knowledges on curve compression to understand which way is the best to avoid to disturb the curve adjustement.

++


Trust me ive done this before (this was certainly true for xvid with old nandub style stats files i see no reason this will be different now)


Beyond this method the codec would have to be made network aware so over/underflows could be communicated between machine and unless working with lots of machines this really won't be worth it.

DaveEL

[Dark-Cracker]

yes it's a good idear , the only *problem* is to wait each computer finish is first pass to start the second pass in the other computers.

PS: it seems to me the overflow redistribution is generaly based on a short amount of frame (250 for exemple) so i think even if sample on each computer are small the redistribution shouldn't be too much disturbed.

Bye.

[DaveEL]

Originally posted by Dark-Cracker
yes it's a good idear , the only *problem* is to wait each computer finish is first pass to start the second pass in the other computers.


You would probably want to do some speed testing before you start and then pass appropirate numbers of frames to each machine so they take approximatly the same time to complete.

DaveEL

[pieter1976]

I tested the new Mjpeg methode with x and y offsets.
Here are the results (http://home.zonnet.nl/pieterboots/compare.zip)

I see some more detail in my version.