AviSynth speed and memory requirements?

[redfordxx]

There is an answer on this in post 4, but I added new Q in post 5 (I thought it is better than starting new thread on similar topic)



Hi,
I wanna make sure one thing, which I hope I assume correctly.

Let's have a scripto=AviSource(...)
b=o.SomeTimeOrMemoryOrWhateverConsumingFilter()
b=o
bThis script has not much sense but for explaining the question is perfect.

Is the second line ignored and AviSynth does not waste time or memory or whatever uselessly?


How about this?o=AviSource(...)
s=0
b=Select(s,o,o.SomeTimeOrMemoryOrWhateverConsumingFilter())
b
Same can apply to ? :


Thanx

[danpos]

Are you sure that this line

s=0
(s==zero)

shouldn't be

s=o
(s==letter "o")

???

[redfordxx]

Are you sure that this line
s=0
(s==zero)?
Yes, I am sure, it is index for nonconditional switching among more clips.
But there was other mistake: Switch/Select function. I corrected it in the post.

[stickboy]

Hi,
I wanna make sure one thing, which I hope I assume correctly.

Let's have a scripto=AviSource(...)
b=o.SomeTimeOrMemoryOrWhateverConsumingFilter()
b=o
bThis script has not much sense but for explaining the question is perfect.

Is the second line ignored and AviSynth does not waste time or memory or whatever uselessly?It is not ignored. The filter will be constructed. If the filter consumes a lot of time or memory just to create it, then that's a problem. If it consumes time and memory only when you request frames through it, then it's not.
How about this?o=AviSource(...)
s=0
b=Select(s,o,o.SomeTimeOrMemoryOrWhateverConsumingFilter())
bSelect() is an ordinary function, so again evaluating that line requires evaluating all of its arguments. If this is a problem, you can thunk the arguments by making them strings and then using Eval(). Also see my SelectByString (http://www.avisynth.org/stickboy/) plugin.
Same can apply to ? :In that case only one of the paths will evaluated.

[redfordxx]

I am working on functions, where are many clip manipulations, and function calls.

Are there any rules to follow to minimize avisynth's memory (maybe speed too) requirements?

for instance, are these same?:
o=source
o1=o
o2=o1.filter(...)
o2
--------------or
o=source
o.filter(...)


another example

o=source
a=o.filter1
b=o.filter2
c=TwoClipsFilter(a,b,...)
--------------or
o=source
c=TwoClipsFilter(o.filter1,o.filter2,...)


o=source
o=o.filter1 #reusing variable instead of creating new one
o=o.filter2
--------------or
o=source
p=o.filter1
q=p.filter2
--------------or
o=source
o
filter1
filter2


Filter(clip)
--------------or
Eval("Filter(clip)")

using global clip variables instead of passing them by function call (esp. by recursion)

I think you might get an idea what I mean...
... some unnecessary frames copying or storing. Or, otoh where it can be good to speedup.

Or, AviSynth optimizes the mess I write and I should prefer to write the code readable instead of optimized.

[foxyshadis]

It really depends on the filter. Many filters will call MakeWriteable(), which will return the original frame written over (however many aliases you go through) if nothing else asks for it, or make copies if it turns out to be extra popular. I'm not sure just how smart this is, whenever I get out-of-memory errors I just give it another half-gig. :p Some copies and operations will make extra frames anyway, even if unnecessary (can't help that but to rewrite them), but avisynth's management is pretty good about handling that part. It's also pretty good about dumping unneeded frames once they have no more references (directly or temporally) when the cache fills up, but tritical's are even better.

It will never even initialize filters that aren't discovered in the walk back from the return to the sources, afaik, except for Source filters (I'm unsure why they get special treatment).

Tritical's builds include some aggressive cache tweaks that may or may not make it into 2.5.7 mainline, that would improve handling of so many clips (they're a godsend when using Didee-scripts) but they're missing a few of the later cvs bugfixes, so ymmv.

[redfordxx]

OK, conclusion: I won't care, because I can't help that much. And hope Tritical's improvement's will be included in mainline.

It will never even initialize filters that aren't discovered in the walk back from the return to the sources, afaikHmm, sometimes it seemed to me they eat resources...

[IanB]

...And hope Tritical's improvement's will be included in mainline.For 2.6 but not for 2.5.7

2.5.7 is supposed to be a bug fix only release. The few features it has grown are due to poor self control and self discipline by myself ;).


For filters which have a create method that ends up doing return arg[0].AsClip();style there is no effective overhead for including the associated statement in your script, just the few cycles to parse the syntax. Typically these are ConvertTo....(), etc, where the pixel format is already as required.

For filters which the Create method end up doing return new Plugh(arg[0].AsClip, ..., env);The filter will be constructed and a link will be added to the child->GetFrame()/GetAudio() chain. So there is the overhead of at least a proceedure call.

Decisional filters like "? : " and ApplyRange() make a choice in there GetFrame method about which subsequent chain to call this time. So although all chains will be constructed only one chain is executed for each GetFrame call.

The parser does no optimization other than what each individual Create method decides.

[foxyshadis]

I know it seems like a good idea to have 2.5.7 a bugfix release, since you don't want to perpetuate 2.5 too long, but as long as there's viable and efficient code patches for it while the new platform is still raw and unfinished, why not just include them? If nothing else, have 2.5.6b (mostly bugfix) and 2.5.7 (new features) released semi-concurrently. This goes for your roadmap for the vs2k5 transition in 2.6 laid out in the other thread, instead of bumping the version just bump the letter for bugfixes, not that I want it to become a mozilla-like maze of branches.

Also, I didn't know about ":?" behavior. So that means that:

false ? fft3dfilter.addgrain().mt_lut("x") : last

still constructs all of the filters? Hrm, that would explain some of my crazy slow startup times with complex & recursive scripts. I guess I could move some of the logic into the first getframe call, but that kind of defeats the purpose of the constructor.

[redfordxx]

The last thing foxyshadis mentioned happens to me as well. Some scripts start up very long time. Much longer than the real processing of one frame...
1) So, is the recursion the cause (are there alternatives?)?
2) In case of recursion or function calls generally, do global clip vars make difference to normally passed vars?
3) Is that then so, that as Stickboy says, ? : is better than Select() but still costs something, and better is his SelectByString.
4) How about false ? Eval("""fft3dfilter.addgrain().mt_lut("x")""") : last?

[foxyshadis]

Wait, "false ? filter() : last" can't be instantiating the filters, because it will work even if you don't have the filter installed at all! Real functions on the other hand, like default() and select(), definitely do (and I'm pretty sure eval treats them the same way, initting everything at startup however a normal script would).

It'd be nice to have a function_exists() so one could give more useful error messages, but it's a real pedantic corner case so meh.

[Didée]

Also, I didn't know about ":?" behavior. So that means that:

false ? fft3dfilter.addgrain().mt_lut("x") : last

still constructs all of the filters? Hrm, that would explain some of my crazy slow startup times with complex & recursive scripts.
It's the assignments that kill, you have to be careful with them. Just a matter of discipline, though.
Taking a random recent script of mine, calling "RessourceHog(mode="snowball")" may take a minute to construct the full filterchain. So, the crux is
fast = true
a = something
b = a.FastFilter(mode="cheap")
c = a.RessourceHog(mode="snowball")
fast ? b : c still takes a minute (since clip c once *is* assigned, even though not needed),
whereas
fast = true
fast ? a.FastFilter(mode="cheap") : a.RessourceHog(mode="snowball") loads in a second. This, but the fact that this way of arranging not only is combersome and hurts readability, but in the first place comes so bulky during script creation, is why lately I find myself typing the "dummy" word more frequently ...
fast = true
dummy = blankclip(something, width=16, height=16)
a = something
b = fast ? a.FastFilter(mode="cheap") : dummy
c = fast ? dummy : a.RessourceHog(mode="snowball")
fast ? b : c loads in a second, since c won't wear the kitbag until the alarm truly rings.
Especially in scripts that split many parallel of long branches (motion compensation for temporal processing over a multiple of frames, anyone?), ressource load may drop considerably when keeping an eye on this strictly.

Reminds me ... mf once scolded me for my unreadable, disorganized and whatnotelse (in short: ugly) scripts , defeating the DoSomething()
Var1 = last
DoSomethingElse()
Var2 = last
Decision ? Var1 : Var2 road. I had my reasons that time back, and times haven't changed in this respect.

[IanB]

Re: ? :

Yes I am wrong. Sorry! The condition part is fully resolved at compile time and only one of the true or false branch is subsequently constructed. Likewise Select() should do the same and only construct the single branch matching the index. Eval(), Apply() and Import() also fully resolve at compile time. As Didée points out expressions evaluated outside of the scope of a conditional element will be constructed (but may remain unused).

[redfordxx]

so, does it mean, that I can use Select() without fear of useless resources consumption? ... and stickboy was mistaken?

(compile time=when I load it eg to VDM?)

[IanB]

so, does it mean, that I can use Select() without fear of useless resources consumption? ... and stickboy was mistaken?No, I've still got it wrong. Select() goes for a run thru env->Invoke() so the arguments all get resolved and constructed before Select() gets to pick one, just like you wentA1=plugh()
A2=xyxxy()
A3=guano()
S=2
Select(S, A1, A2, A3)Hey, how about somebody else read and interprete the code, my brain is starting to hurt.:rolleyes:(compile time=when I load it eg to VDM?)Effectivly yes, the time between opening the file and being ready to serve the first frame.

[redfordxx]

Hey, how about somebody else read and interprete the code, my brain is starting to hurt.:rolleyes:
Thanks much for your effort for the ones, who do not understand the code.

I think or hope I start to understand some parts of how avisynth works:

In case of recursion, all the milion of filters is prepared in one chain or tree and then when serving, it is already same like written in one line?
Then it seems to me, that conditional recursion is not possible, because an infinite number of filters would be constructed.

Once had the idea to create recursion script like:
MaxLuma-MinLuma<threshold ? Perform operation : split clip into 4 parts and try again.
Well, let's say, unsuccessful. And now I see that in different light.