Distributed Encoder

[ChrisBensch]

I'm making a tool similar to ELDER4x264 so split my encodes over multiple machines. After I setup the encode, I have a temp AVS as the source and I need two things: How do I easily get the total number of frames in the source? How do I smartly split those frames over the user defined number of nodes? Basically I'll point to the AVS I've created (all the cropping/resizing), then it'll take that and figure out the total frames and write new AVS files for each node that I configured. I just need to know how to get the total frames and then Ii need an algorith to split them over the nodes. For example, I have a source with 143837 frames and 4 nodes. How do I smartly split this up?

[foxyshadis]

What language? The avi-handling tools available in different languages differ greatly. Otherwise you can use command-line software and string parsers.

[ChrisBensch]

I think I have it figured out...but in case not, I'm trying this app in C#. I normally use Java but I'd like to try this stuff out.

[Mug Funky]

no need to get the framecount if you do the splitting inside avisynth.

avisynth is aware of the number of frames, and trim can accept an expression instead of a number. so all you need to know is how many nodes you want.

here's the function i use (in an autoloaded avsi file in my plugins directory):

function split (clip c, int chunks, int chunknumber, int "GOP")
{
GOP = default(GOP,12)
chunksize=(c.framecount/chunks)-((c.framecount/chunks)%GOP)

return chunknumber==1? c.trim(0,chunksize-1) :
\ chunknumber==chunks? c.trim(chunksize*(chunks-1),0) :
\ c.trim(chunksize*(chunknumber-1),(chunksize*chunknumber)-1)

}

this will split on multiples of your GOP size (default 12 for me).

so create 4 scripts (using a batch file? it's quite possible) and put this in them:

...source(...)

...filters...

split(<number of nodes>,<current node>,<gop size>)

just "split(5,1)" would be enough for the 1st part of a 5 node encode.

although, be aware that this will make 2-pass ratecontrol not quite optimal - we'd need to have some kind of distributed encoder-server running that collates the 1st-pass stats and merges them in before the 2nd pass runs... not a clue how to do that.


[edit]

if you need to output the framenumber, avsutil.exe will return it when you run any script. you can pipe this to a text file or anywhere else.

[Sharktooth]

Uhm, ratecontrol wont be screwed until you split too much...
Im already making some C# code for distributed encoding with MeGUI.
It's all based on a split function similar to the Mug Funky one.
The only difference is i connect the encoding stations BEFORE i split the file so it knows exactly how many chunks it should create.
The avisynth script is the same for every encoding station. It gets parsed and the #MEGUI_STATION_ID# comment is replaced by the current encoding station number.
It is also able to start single or multiple jobs simultaneously.
Lets say you want to run 4 different jobs (2 videos and 2 audios), you can select what station will run what (even all 4 jobs on one station)... So if you have slow computers you can just run Audio encodings or Mux Jobs (thru samba shares) on them.
If total video frames / number of encoding stations < 30000 (to be tweaked) it will automatically assing other jobs to the exceding encoding stations coz too much chunks can hurt ratecontrol. Also it wont split sources smaller than 30000 (to be tweaked) frames. Obviously that is valid only for a ABR or Nth Pass encodings. CQ encodings wont have those restrictions.
Once an encoding station becomes idle (finished encoding) it gets automatically moved to the "Idle stations" pool and assigned to another job (if it's in the queue). You can also connect another encoding station during a job processing... it will join the "Idle stations" and it will get assigned to something if the queue is populated (next video job to be splitted will take the new encoding station into consideration).
When chunks are encoded they get joined by the "main" workstation (the one that runs the full megui) with the appropriate tool (depending on the container), and audio and other streams get muxed.
Hope i gave you some ideas.
The source code will be available as soon as i get it working properly ;)
I still have to code the error management (what happens if an encoding station fails?), to refine (probably even rewrite to support job priorities) the algorithm for jobs distribution and to improve the connection protocol between the main workstation and the encoding stations (actually it is a sort of passive authentication... ugly... and insecure).

EDIT: On linux, instead of avisynth, i use the encoder ability to start and stop encoding at given framenumbers but maybe i could revert to that method for windows too. However the logic is practically the same as described above.

[ChrisBensch]

Thanks so much. Even though my favorite encoder suite (MeGUI) may get distributed soon, I appreciate the help in getting this to work for me too. It's just something to help out with me learning C# and if I can help the Doom9 folks too, then great. Again, thanks a lot.

[frodeste]

@Sharktooth
Fantastic work! I can't wait to see the final product. :thanks:

[Pookie]

MVdegrain2 for everything, here we come :) Distributed transcoding will really boost the usage of x264 and HD resolution material. As an OT footnote, the newer FFmpeg builds allow for direct AVS input and MPEG2 output of 1280x720 and 1920x1080 resolutions - perhaps something to consider for the distributed Megui version.

[Sharktooth]

1 step at a time...
im only 1 person and megui linux port is already drawing me too much time

[ChrisBensch]

Uhm, ratecontrol wont be screwed until you split too much...

I plan on 4 nodes to start, probably not more than 6, do you think that will be OK? I guess I'm asking, how many is too many?

[ChrisBensch]

this will split on multiples of your GOP size (default 12 for me).

Most of my work is HDTV captured from a cablebox, then I encode to x264, is there a "safe" "default" GOP length I could use for all of my encodes?

[foxyshadis]

The biggest inefficiency comes from assuming every chunk will have the same bitrate. The inefficiency of a few more I frames pales compared to that; so if you can attempt to determine bitrate needed for each chunk after the first pass, presumably based on the "average QP" x264 outputs, or by doing the first pass as a crf instead of abr. You still can't really get around having to do the entire first pass, collecting all the information, and then calculating the bitrates to run each second pass chunk with.

The GOP size isn't a big deal; with DVD and HD-DVD you have a maximum gop size of half a second, but PC-based codecs let you have a whole 10-15 seconds. That means each spurious I-frame is a harder hit, so down the road you might look into avisynth-based metrics for optimizing the split.

Tobias was working on juggling all this in a single pass, and it works pretty well for xvid, it just never got finished for x264.

[Sharktooth]

Yes, i know... that will come in a second phase...
chunks of 30000 frames wont screw the ratecontrol for now...
what i need is to get it working... then i will do the refinements.

[Hellworm]

I'm trying to do an x264 distributed encoding tool too( time for yet another anything :)
I want It to completly ignore the initial number of nodes. The first node simply starts at the beginnning and additional nodes look at what is still left, look at the speed the first node is progressing and the own benchmarks and then decide on a good starting point. Then they'll do a fast first pass around the desired starting point and, look at that output and then decide on the exact starting point.

Well the theory is very nice, the Problem I have is that I don't know how to exactly stop x264 from continuing when it reaches the start point of the next node. If I simply abort it can i be sure that it doesn't write addidtional frames, and that all frames till the last reported are written?

[Sharktooth]

I dont think it's possible to stop it at the "right time".
But i think you may continue to encode some more frames and discard the frame in excess.
At that point i think the rate control gets a bit screwed though...

[Hellworm]

But if the input file ends the last complete frame is written? if yes, does this cause any non-optimal results (p/b-frame placement)?

[Mug Funky]

it might be possible for x264 (or any encoder that generates 2-pass stats) to keep the same ratecontrol accross several machines (save for the extra i-frames... boo hoo).

it'd require some rewriting i think.

say you have one source (an avs or whatever). the program will know it's length, and you'll tell it how many nodes are running, and which node it is. it can write it's own 1st pass stats, and when all nodes have finished their bit (here's an inefficiency. oh well), the stats are all collated together and the 2nd pass performed.

the encoder would internally do the "split" thing, and with access to complete 1st pass stats it'd be quite simple for it to just start in the middle.

[Dark-Cracker]

for the second pass you just need to paste all the .pass file of each part and starting with the pass file of which part you want to encode, i think before the paste you need to remove at the end of the log file the keyframe generated for the last frame, (changing the keyframe flag, quantizer and size).

the encoder will now calculate correctly the bitrate curve adjustement, and there is generaly no verification of the number of frame in the .pass file so you can start to encode each part of the file with a complete .pass file.

However you need to wait until each node is finish before starting the second pass.

Hope it's enought clear with my poor english skill.

PS : RV10 also use .pass file and i have start to update an old tool who made distributed encoding. (http://forum.doom9.org/showthread.php?t=111530) perhaps i will update it later for x264 if there is a need.

Bye.