Log in

View Full Version : Automatic load balancing across all nodes in a network?


CAFxX
14th January 2007, 19:26
Just a quick feasibility question. Could it be possible to write a plugin that automagically does load balancing on all the available nodes in a network (instead of doing it manually via the current TCPDeliver system)?
I've been wondering about this for a while and the only real problem I can spot (apart the intrinsic complexity of the scheduling) is: does avisynth currently allow a plugin to create/modify the "internal avisynth filter graph" (I don't know how you guys call it). IIRC it is allowed to call other functions from inside a plugin: this should suffice, even if it ain't the best way to do it.
Can anyone spot some other potential problem that may prevent such a plugin to work? Just because, even if ATM I don't really have enough time to code it, it may be something funny to play with.

CAFxX
16th January 2007, 21:34
Nothing at all, huh?
I started to take a look at the sources of avisynth (thank you, doxygen!) and until now I couldn't find major problems with this, even if there's something that is not clear to me.
I was planning to let TCPDeliver do the job of moving data between nodes, but in the TCPDeliver docs I found this:
Using commands after TCPServer is also a bad idea:

AviSource("avi.avi")
TCPServer(1001)
AviSource("avi2.avi") # Do not do this, this will disable the server.

AviSynth detects that the output of TCPServer isn't used, so it kills the Server filter. TCPServer should always be the last filter.
I can't understand where the "it kills the Server itself" is enforced. Is it done during parsing, or later (eventually even after the graph has been created)? In the latter, is it done on a frame-by-frame basis or just once at the beginning?
In fact, my idea was to allow the filters to "migrate" from one node to another dynamically, e.g. (just two nodes, for the sake of clarity):
Script:
Server:LBFServer* -> FilterA -> FilterB -> AviSource
Client:LBFClient
When started, the clients listen for incoming connections and the server searches for active nodes while doing the frameserving as usual. Then (by means of some benchmarking, filter instances timing and other dirty hacks) the server decides that FilterA should migrate on the Client. The graphs therefore becomes*:
Server: LBFServer -> TCPSource2 -- TCPServer1 -> FilterB -> AviSource
Client: LBFClient -- TCPServer2 -> FilterA -> TCPSource1

(*) notes:
- LBF simply means load balancing filter
- I perfectly know that in this way the data will travel at least twice on the network instead of once of a "traditional TCPDeliver client-server approach". And yes, I understand what that means.

tsp
16th January 2007, 21:59
Nothing at all, huh?
I started to take a look at the sources of avisynth (thank you, doxygen!) and until now I couldn't find major problems with this, even if there's something that is not clear to me.
I was planning to let TCPDeliver do the job of moving data between nodes, but in the TCPDeliver docs I found this:

I can't understand where the "it kills the Server itself" is enforced. Is it done during parsing, or later (eventually even after the graph has been created)? In the latter, is it done on a frame-by-frame basis or just once at the beginning?
.
I think it happens when the script is opened. The application that request a frame (ie virtualdub) gets it from the last filter in the script in this case avisource as avisource doesn't take a clip as an input tcpserver is never run. And yes it is possible to modify the filter chain. Try look at the conditional filters (ConditionalFilter / FrameEvaluate / ScriptClip / ConditionalReader) for examples.

CAFxX
16th January 2007, 22:28
Thank you tsp, your suggestion is very appreciated. I'll have a look at them soon.
Last question for tonight: does avisynth currently provide some facility to measure the execution times of each filter instance in the graph (i.e. some kind of profiling)?

tsp
16th January 2007, 22:48
not buildin. You can use avstimer (http://avisynth.org/warpenterprises/files/avstimer_25_dll_20050809.zip) to do it (or use the sourcecode from it).