Complex Script-> Avisynth very slow

[krieger2005]

Hi,

when one write a script which is more complex Avisynth get slow. In such cases i divide the script into 2 or 3 parts and decode them sequentially.
So i get for example for the hole script a coding time of 10 days... and for the every part some 14 hours (3*14 hours ~ 2 days).

Because for every part of the Script i must have a lossless quality-outcome of the movie i get with this method GBytes of temporary movies. So now i'm searching for a workaround or maybe someone know an other way...

Can someone tell his override of such problems? Is threre one?

greets
krieger2005

[Soulhunter]

Only thing I could think of...

Use the full script, but splice the source into parts !!!


Bye

[Manao]

Could you post the script, perhaps you're using filters which are known to have memory leaks ?

[krieger2005]

@Soulhunter: As I have discribed above... One must save the parts (of every Script) lossless. And if you have a movie with 2 hour playtime you get sometimes Files above 20GB. Thats why i'm searching for an other method.

@Manao: This is not a special script. The last time i have many movies with very bad quality. When i try to remove the artifacts the script get longer... And the encodetime sum exponational.

Most useable Filters are:
- RemoveDirt
- MaskTools
- Edge-Script (which i get from somewhere here):

edges = Overlay( DEdgeMask(0,255,0,255, "4 10 4 0 0 0 -4 -10 -4", setdivisor=true, divisor=2)
\ ,DEdgeMask(0,255,0,255, "4 0 -4 10 0 -10 4 0 -4", setdivisor=true, divisor=2)
\ ,mode="lighten" )

I have noticed if i clean a movie before deinterlacing and after i must split the script at the deinterlacing-Moment.
For Deinterlacing i use KernelDeint, FieldDeinterlace or last time (with the edge-Script) SangNom.

I think this effect is extremly noticeable when using Temporal-Filter (TemporalCleaner, FluxSmooth, RemoveDirt...). Maybe Avisynth drop some calculations and calculate so some things twice...

i don't know. But i searching for a solution... Maybe someone know one ;)

PS: @Manao: If you want i can post my last script and show you when it get very slow...

[Manao]

I'm still interested with the whole script ( mainly to try on my computer ). I think this effect is extremly noticeable when using Temporal-Filter (TemporalCleaner, FluxSmooth, RemoveDirt...). Maybe Avisynth drop some calculations and calculate so some things twice...How much RAM do you have ? Did you try to play with the SetMemoryMax() function in order to increase the memory used by Avisynth ( on complex scripts, it can help ). If you don't have enough memory, it may happen that the cache becomes progressively misused. In that case, frames could be computed twice, as you were thinking.

[krieger2005]

I have 512 MB RAM. Does not played with "SetMemoryMax". I will try it (Now VDub show, that the sript need 160 MB).

Here my last script:


von=136
bis=1305


mpeg2source("D:\VTS\Çèìíÿÿ âèøíÿ 3\D2V\film.d2v").UnDot
#trim(von,bis)
YV12LUT(Yexpr="x 20 < 16 x ?")
#DeRainBow(12)

q=SangNom(1,15).UnFilter(10,10)
FieldDeinterlace(blend=false)
first=last

edges = Overlay( DEdgeMask(0,255,0,255, "4 10 4 0 0 0 -4 -10 -4", setdivisor=true, divisor=2)
\ ,DEdgeMask(0,255,0,255, "4 0 -4 10 0 -10 4 0 -4", setdivisor=true, divisor=2)
\ ,mode="lighten" ).Levels(40,1,170,0,255,false)

blue=BlankClip(length=framecount, width=width, height=height, pixel_type="YV12",fps=25,color=$7777ff)

sauber=VagueDenoiser(threshold=3,method=3,nsteps=6,chromaT=0.8,
\wiener=true, wratio=6, interlaced=false).
\RemoveDirt(tolerance=30, mthreshold=60, athreshold=300, pthreshold=10).
\UnDot.
\UnFilter(30,30)

a=ConvertToRGB32._2DClean(0,10,2,2).ConvertToYV12.
\DideeBlur(sauber,25).UnDot
b=ConvertToRGB32._2DClean(0,10,2,2).ConvertToYV12
m=subtract(a,b).YV12LUT(YExpr="x 126 > x 255 < & x 121 < x 0 > & | 255 0 ?")
overlay(sauber,mask=m)
UnFilter(40,40)

a=MotionMask
overlay(VagueDenoiser(threshold=1.2,method=3,nsteps=6,chromaT=0.8,
\wiener=true, wratio=5, interlaced=false),mask=a)
DideeBlur(sauber,2)

second=last

overlay(q,mask=edges)

sauber=VagueDenoiser(threshold=2,method=3,nsteps=6,chromaT=0.8,
\wiener=true, wratio=6, interlaced=false).
\RemoveDirt(tolerance=30, mthreshold=60, athreshold=300, pthreshold=10).
\UnDot.
\UnFilter(30,30)

a=ConvertToRGB32._2DClean(0,10,2,2).ConvertToYV12.
\DideeBlur(sauber,25).UnDot
b=ConvertToRGB32._2DClean(0,10,2,2).ConvertToYV12
m=subtract(a,b).YV12LUT(YExpr="x 130 > x 255 < & x 122 < x 0 > & | 255 0 ?")
overlay(sauber,mask=m)
UnFilter(25,25)

third=last
deen("c3d", thrY=37)
RemoveDirt(tolerance=20, mthreshold=55, athreshold=300, pthreshold=10)
#Temporalsoften(3,5,5,mode=2,scenechange=10)

#UnFilter(20,20)

#crop(32,140,-20,-120)
#LanczosResize(664,288)
return last

[krieger2005]

I will now do a comparison. I have the following Script:

avisource("D:\VTS\Çèìíÿÿ âèøíÿ 3\AVI\film4.avi",false,"YV12")

q=SangNom(1,15).UnFilter(10,10)
FieldDeinterlace(blend=false)
first=last

edges = Overlay( DEdgeMask(0,255,0,255, "4 10 4 0 0 0 -4 -10 -4",
\setdivisor=true, divisor=2)
\,DEdgeMask(0,255,0,255, "4 0 -4 10 0 -10 4 0 -4",
\setdivisor=true, divisor=2)
\,mode="lighten" ).Levels(40,1,170,0,255,false)

blue=BlankClip(length=framecount, width=width, height=height, pixel_type="YV12",fps=25,color=$7777ff)

sauber=VagueDenoiser(threshold=3,method=3,nsteps=6,chromaT=0.8,
\wiener=true, wratio=6, interlaced=false).
\RemoveDirt(tolerance=30, mthreshold=60,
\athreshold=300, pthreshold=10).
\UnDot.
\UnFilter(30,30)

a=ConvertToRGB32._2DClean(0,10,2,2).ConvertToYV12.
\DideeBlur(sauber,25).UnDot
b=ConvertToRGB32._2DClean(0,10,2,2).ConvertToYV12
m=subtract(a,b).YV12LUT(YExpr="x 126 > x 255 < & x 121 < x 0 > & | 255 0 ?")
overlay(sauber,mask=m)
UnFilter(40,40)

a=MotionMask
overlay(VagueDenoiser(threshold=1.2,method=3,nsteps=6,chromaT=0.8,
\wiener=true, wratio=5, interlaced=false),mask=a)
DideeBlur(sauber,2)

overlay(q,mask=edges)

return last


Till now (2 hour coding time) VirtualDub show me a Total time of 14:19:00:00 (14 Days and 19 Hours).
Now i will splice the script and at the end i will sum the encoding time... Maybe the next Days you will not hear anything about this comparison...

[Didée]

Well, I see no particular technical reason why your script should be so *extraordinarily* slow.

Slow the script is for sure: VagueDenoiser 2 times, RGB filtering, masking operations etc. etc. - But 14 days is a little too much, even for patient people.

My best bet is, that your machine is running out of physical memory and starts swapping. As soon as Avisynth's allocated memory gets swapped to HD, everything slows down to ridiculous speeds. That's normal.

Fire up TaskManager, and look closely how the memory managment behaves when loading the script in Vdub, and when starting to "play" the script. Especially look for the memory allocated by Vdub, and for the size & usage of the swap file. Probably you'll see something interesting!

(BTW, even on "modern" Win-XP, to my very first actions after installation belongs: a)-disable (remove) swap file b)-defragment HD c)-recreate swap file with MIN == MAX size ...)


Some other small tips:


- be paranoid, set parantheses.

In above script, this refers to undot(), ConvertToXYZ(), MotionMask(), and return(last)

Donald Graft states for ages that it MAY slow down a script when calling a parameterless filter without empty parantheses. Well, I don't know if there is any proof that this is really the case. But I decided to be paranoid and to *always* set those parantheses.
As a side-effect, this makes the script (for me at least) easier to read, especially when using a text editor with colorful syntax highlighting. ;)


- since your occupying MaskTools anyways, you should not use overlay(), but yv12layer() / MaskedMerge() / Logic() , as far as applicable.

Your line "[last.] overlay(sauber,mask=m)" then would read
"[last.] MaskedMerge(sauber, m, Y=3,U=3,V=3, useMMX=true)", for example.

Similar applies to "m=subtract(a,b).YV12LUT(... ...", which can be replaced by yv12subtract.
Since yesterday, it is even possible to do the subtraction and the LUT in one single step through yv12LUTxy(), hehe. But that's marginal in this case.


- your used method of edge mask creation is mine ;) - but yours is a stone old version. Better do it like this:

logic( Dedgemask(...),Dedgemask(..),"max") .levels(...)



Hope something useful was in there ;)


- Didée

[krieger2005]

It seems that this problem is a problem of my computer and not of avisynth... which problem this is, i don't know till now. I used a simple "FieldDeinterlace(blend=false)" for a movie and it take round about 12 hours to encode (with 3-4 fps average)

@Didée

Similar applies to "m=subtract(a,b).YV12LUT(... ...", which can be replaced by yv12subtract.
Since yesterday, it is even possible to do the subtraction and the LUT in one single step through yv12LUTxy(), hehe. But that's marginal in this case.


yv12subtruct gave me an other result as substract. And yv12LUTxy give me yet another result. Is
"subtract(a,b)" = yv12LUTxy(a,b,"a b -")
? I think no.

I tried also your ne version of Edge-Detection (with Mask-Tools 1.5.1) and it gave me different result... I wonder. But the i turn back to Mask-Tools 1.5 and the result was the same: "Dedgemask" in MaskTools 1.5.1 give the different result.

So long, i thank you very much, explicitly Didée.

[Didée]

Originally posted by krieger2005
But the i turn back to Mask-Tools 1.5 and the result was the same: "Dedgemask" in MaskTools 1.5.1 give the different result.
Yes - yesterday evening I already suspected that in v1.5.1, DEdgeMask gives a different result *if* it is called with the (obsolete) "setdivisor" parameter. But I only had a few minutes at my own PC, then. I first want to definetly confirm it, before I'll whine at Manao's shoulder ;)

When using v1.5.1, simply don't use the "setdivisor" parameter, and it should be fine.

For yv12subtract, you'll have to find the right combination of "tol=-1|0|1" and "wideRange=true|false" to get the right behaviour, heh.

Good luck with bug-hunting your machine's setup BTW.

- Didée

[Manao]

I didn't remembered how was working 'setdivisor', and I switched its behavior. Ooops :)

I corrected the behavior, and you'll find the new dll here (http://jourdan.madism.org/~manao/MaskTools-v1.5.1.zip).

I hope this time it'll work as expected.