Distributed encoding - avisynth filtering

[Dark-Cracker]

Hi,

I want to know how to made an optimised distributed encoding method.

there is 2 ways :
- one is to share a folder (the server) and access the file to encode part of the video material on the other computers (the clients).

- the second is to use tcpserver to open the video stream on the server and encode it by parts on the others clients.

but i have a doubt about the filtering, on which side i should made the filtering and for these 2 ways which cpu is used for the filtering ?

for exemple if i want to use the avisynth's tcpserver plugin,

ex: i have 2 computers :

computer A have a avs script with the filtering and the script is open with the tcpserver plugin, and the computer B is connected to the computer A to read and encode the avs script, which CPU will be used to apply the filter ?

RMQ : if it's the computer A i suppose it's better to open a *basic* avs script on the computer A and when you open it on the computer B you apply the filters.

is it the same if you share a folder and open the avisynth script using the UNC path (ex : "\\mycomputer\share\patr1.avs") ?

any remarques, suggestions are welcome.

PS : Sorry for my poor english, i hope my post is enought clear, else feel free to ask for a more clear message.

for the tcpserver exemple sh0dan have answer :


The best way is to evenly distribute load between the two computers. But remember that both decoding and encoding also takes up various amount of CPU. I you have CPU-demanding encoding, like h.264 it might be faster to have all filters on machine A.

[Mug Funky]

encoding on the network with \\blah\d$\blah\blah.avs is no slower than going off a local drive (especially if you have gigabit ethernet :)).

TCPserver has considerable overhead though, and is slower than loading the file directly. but if you have a lot of filtering to do, do it on computer A, then encode on computer B. otherwise there's no point.

i find the quickest way is to make 2 avs's with half the feature in each, then just set them off and join later.

[edit]

renderfarming would indeed be a good thing. more something to be done encoder-side i think. and it'd only be useful for urgent stuff, otherwise just let it go overnight or in the background while you do other things (provided they're not nearly as CPU intensive, which is almost everything). having an encode going with spruce and DVDdecrypter and firefox running works pretty good, especially on a HT machine.

[Dark-Cracker]

a bit out of topic but if i want to separate the avs input file in different part to be encoded on differents computer i am not really sure if i should relay on the cpu frequency to separate equaly the different part. (i want to get the frequency of each client and separate the input file in part proportional to the frequency of the client toward all the frequency of the clients)

for exemple should i consider an CPU :
- amd xp 2000+ like a 2 ghz cpu or like a real 1.6 ghz ?
- does a biprocessor 2ghz should be considered like a 4ghz ?
- same for a HT cpu

if someone have clue, feel free to post your opition.

Bye.

[foxyshadis]

Bandwidth is really your most important consideration, unless you're using very slow filtering. Then try to match up the framerates of the two sides. It doesn't really matter what cpu they have, just that both sides have a nearly equal role. If the avisynth machine gets 6 fps and the encoder can do 18 off the same source w/o filtering, bring some filters over to the encoder, because the slowest side will limit the faster.

To add more machines, you can chain filtering, but that works best if you have GbE all over and offloaded network cards. You can also split frames in half, like MT(), with some overlap, but you have to make sure you ivtc/deinterlace on full frames only. (A hybrid of chaining (for ivtc) and parallel (for other filtering) could work, but I've never had enough machines to try it.)

HT and SMP only matter if you're using tsp's MT, or running several tcp servers on one machine (for smp).

[Longinus]

A little bit OT, but does tcpserver have some kind of buffer?

Imagine this (I'm probably wrong, and it doesn't work like this, but...):

2 computers with the same speed. Your script takes 1 second to process each frame, and 1 second to encode it; in a single machine. So in the end it will take 2 seconds per frame.

So you try to split this. The first computer process it, and the second encode. BUT, the one encoding will always have to wait for the first to finish, won't it? It will request frame X, wait 1 sec for the first computer to process it, and then it will encode it. Tanking 2 secs in the end!

The avisynth docs don't mention if the frame is processed on the server machine when requested by the client. If it's, I don't see a point in this for distributed processing.

[Dark-Cracker]

The avisynth docs don't mention if the frame is processed on the server machine when requested by the client. If it's, I don't see a point in this for distributed processing.


Hum you are right for exemple if i have 4 client and 1 server, i think it's the server who is providing the frame filtered. So if i try to encode using the 4 client they will all request a different frame and the server will need to filter all the 4 frames simultanely. So the cpu of the server will be really busy, while if you access the input avs throw an UNC path each client will use it's own CPU to access the frame and apply the filter. i am right ? could someone confirm this ? (perhaps sh0dan).

Bye.

[Pookie]

Seems like a "per frame" method of distribution would be inefficient, unless you're looking at a frame that takes minutes to process.