MVTools without idx

[Delerue]

Fizick, I've been testing your alpha for 4 days, and until now no problems at all. About memory usage, take a look at this (all tests with MPC HC 1.1.839 + FFDShow 2228):

MVTools 1.11.4.3 -> MVFlowFPS:

backward_vec = source.MVAnalyse(blksize=8, overlap=2, isb = true, pel=1, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=8, overlap=2, isb = false, pel=1, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
den=FramerateDenominator(source), mask=2, idx=1)
distributor()


Memory usage 211 MB.

MVTools 2.0.9.0 -> MVFlowFPS:

super = source.MVSuper(pel=1)
backward_vec = MVAnalyse(super, blksize=8, overlap=2, isb = true, search=2)
forward_vec = MVAnalyse(super, blksize=8, overlap=2, isb = false, search=2)
source.MVFlowFps(super, backward_vec, forward_vec, num=2*FramerateNumerator(source), \
den=FramerateDenominator(source))
distributor()


Memory usage 428 MB.

I don't know if there's any memory leak, but there was a huge increase, don't you think?

About CPU optimization, I don't know if you tried something with MVAnalyse and/or MVFlowFPS, but as far as I tested, both versions look like the same.

[Fizick]

Delerue,
v2.X uses Avisynth cache for superframes, so more frames are contolled by avisynth (in addition to source). Memory size may be contolled by SetMemoryMax. Probably you have 2Gb memory, and memorymax is about 512 Mb.
i am not sure how exactly avisynth use cache memory if there is no many frames requests.
probably growing it is limited by max frame distance from current?

[Delerue]

I see. I indeed have 2 GB. There's no problem to me to have a video using almost 500 MB; I only told you about memory usage because you're in alpha stage and I thought that it would be useful somehow, hehehe. Keep going. ;)

[Fizick]

Delerue,
Of course, your report is useful!
Now I see, that you use pel=1, and i discovered, that I use non-optimal processing for pel=1 in v2.0.9 (non-needed finest clip).
It is removed in v2.0.9.1, please test it.

http://avisynth.org.ru/mvtools/mvtools-v2.0.9.1.zip

[Delerue]

Cool. I noticed a great performance improvement. Thanks. ;)

[Fizick]

and what about memory?

[Delerue]

and what about memory?

I didn't check it before you ask now. Take this: cut in half. :eek:

[Fizick]

so, this alpha 2.0.9.1 is rather mature alpha :)
Its a time to add link to it at my site :)

I consider vector format possible change.
for 2.1.X development it would be interesting (may be) to get vectors, make some manipulations with them an put to vector stream again, and recalulate.

now we have mask(kind=5) to have a vectors for processing. it is easy to implement complementary function. But these vectors are limited by value to signed char limit =127, i.e. 63 pixels max for pel=2. We need in two byte color format :)

[IanB]

A thought on improving memory usage in the Super clip.

The current implementation is to vertically pack all the various level combinations of samplings of the original image into 1 big frame.

Strike me that not all of these level sampling components are always used, so generating and storing them is a waste. So just store each sampling component in its own avs frame and multiplex the frames by frame number instead of stacking vertically.

i.e. A given super clip has say 9 level sampling components.

So you vi_of_super.num_frames *= 9; in the constructor.

In MVFrame or such instead of using PlaneSuperOffset(... Level ...) etc you resolve to pclip_of_super.GetFrame(n*9+Level, env);

In MVSuper::GetFrame you get Level=n%9; and Source_N=n/9;

A further refinement would be to also multiplex the luma and chroma planes individually as well, so 9 becomes 27. i.e. to get use (n*9+Level)*3+Plane and to resolve use Plane=n%3; Level=(n/3)%9; Source_N=n/(9*3); If you go this far just use RGB32 as you do for the vector clips.

[Fizick]

IanB,
Yes, memory usage is not perfect.

1. All levels are used in MVAnalyse (so, generation is not a waste), but in MVAnalyse only.
MVCompensate (et al) uses finest level only. It is biggest through, especially for widely used pel=2 (and pel=4 of course). We have 4*height. coarse levels use about 1*height max.

2. I certainly do not want to return to RGB32 and split a chroma! I like to have native color format for every level. It was one of my aims. I like to have possibility to see every level and its padding (i already fixed 2 bugs when i saw intermediate clip :) ). We may also apply some external filter to whole super frame (or to sublevel with a mask).

3. Anyway, I considered to split the super to finest level and coarse levels. Your suggestion is interesting. Thanks for this and others! I considered and considerer. But I am wonder, how we can multiplex frames with different width and height.

4. Where we really waste a memory, it is in MVFlowXXX, when I build full finest clip (instead separated by pel subplanes). I succesfully impemented (non-public) version without this finest clip, but modified MVFlowFps is slower by 15 percent. Reason - too many calls (for every pixel, not to every block!) to get address of specific pel plane. I may add it as an option for testing.

5. IanB, may CACHE_RANGE help? Can we get some memory improving if we explitily set cache range by MVSuper(..., cache_radius=1) if we know, that there are MVAnalyse(delta=1) only?

[mikeytown2]

An idea for #3

Throw good programing out the door and use some global variables. So instead of multiplexing we have completely separate streams. Sadly this might bring back some form of idx; but it would allow you a lot more storage space for passing info and such.

[superuser]

super=MVSuper(hpad=8, vpad=8, pel=2, sharp=2, rfilter=1)
vb=MVAnalyse(super,isb=true) # any old MVAnalyse params besides above
vf=MVAnalyse(super,isb=false)
MVDegrain1(last,mvbw=vb,mvfw=vf,super=super)

I note some speed increasing and memory usage decreasing. Please test.

About syntax and implementation.
superframe contains all source video data. So, it is possible implement clients (like MVDegrain1) without usage of source (i.e. last clip):
MVDegrain1(super,mvbw=vb,mvfw=vf)




tested above and works fine.

though had couple of points - If I pass in the source clip to MVAnalyze, it gives be invalid argument error.

source = last
super=MVSuper(hpad=8, vpad=8, pel=2, sharp=2, rfilter=1)
vb=MVAnalyse(source, super,isb=true)
vf=MVAnalyse(source, super,isb=false)


If I remove source from the MVAnalyze it works fine.

Also can clip especially vectors be passed in as mvbw=vb,mvfw=vf.

source = last
super=MVSuper(hpad=8, vpad=8, pel=2, sharp=2, rfilter=1)
vb=MVAnalyse(source, super,isb=true)
vf=MVAnalyse(source, super,isb=false)
MVDegrain1(super,mvbw=vb,mvfw=vf)


If I try this it does not work and do not think it is expected to work. If I substitute above with following it works fine:


source = last
super=MVSuper(hpad=8, vpad=8, pel=2, sharp=2, rfilter=1)
vb=MVAnalyse(super,isb=true)
vf=MVAnalyse(super,isb=false)
MVDegrain1(super, vb, vf)


....

Can MVDegrain and in general mvtools be used along with MT? Yesterday I was running MVDegrain2 on a source of about 3 hours of run length and had experienced system crash (Vista 64 bit). Not sure exactly how frames it was into it when system crashed as I was asleep than and in morning saw system crashed. I was doing 2 pass XVid encode and have xvid stats file produced during 1 pass of XVid, in which system had crashed, tell us about how many frames it had processed to know many frames the script choked on. Though I had other filters in the script, so later today will try out only with MVDegrain2 and see what happens. I have wrapped MVDegrain2 in a function and have this processed using MT. Hopefully this should not be a problem. This was a working script with new addition being MVDegrain2 using latest mvtools bundle to chain of filters. Following are filters/steps I am using in the script

- Tdeint and TDecimate
- crop
- upscale
- Masked HDRAGC
- MVDegrain2 with MT
- fft3dfilter
- LSF with MT
- SootheT and Dehalo with MT
- downsize to desired resolution
- Tweak

[MfA]

An idea for #3

Throw good programing out the door and use some global variables. So instead of multiplexing we have completely separate streams. Sadly this might bring back some form of idx; but it would allow you a lot more storage space for passing info and such.
You could just use the pointer for the videoframe together with a hash map to store arbitrary amounts of meta data for each frame.

[Fizick]

superuser,
MVAnalyse (clip super, int "blksize",....) does not use source clip.
Are you read docs about new syntax changes? :)

Can MVDegrain and in general mvtools be used along with MT?
Are you say about MT("....") ? Yes, it probably may be used. But you know about frame dividing with MT.

[superuser]

superuser,
MVAnalyse (clip super, int "blksize",....) does not use source clip.
Are you read docs about new syntax changes? :)

sorry, my bad ...

Can MVDegrain and in general mvtools be used along with MT?
Are you say about MT("....") ? Yes, it probably may be used.

Yes, this is what I am doing:



...
...
MT("mc_degrain(src)", threads=4)
...
...

function mc_degrain(clip src)
{
last = src
super=MVSuper(hpad=8, vpad=8, pel=2, sharp=2, rfilter=1)
vb=MVAnalyse(super,isb=true)
vf=MVAnalyse(super,isb=false)
return MVDegrain1(last,mvbw=vb,mvfw=vf,super=super)
}




But you know about frame dividing with MT.

no clue about it ... so can this mean using MT can result into different results? :confused:

[Fizick]

return MVDegrain1(last,mvbw=vb,mvfw=vf,super=super) ?
I am sure you use not latest 2.0.9.1 MVTools.

Yes, MT can result into different results (from single-treaded).
The search region is halved (i forget, vertically or horizontally).
But results may be quite good anyway.

[superuser]

I am sure you use not latest 2.0.9.1 MVTools.

sorry copy paste mistake :p. below r somewht the lines I used. Currently away from the system I have script on.

function mc_degrain()
{
...
...
return MVDegrain1(last,super, vb, vf)
}

I am using the latest bundle of mvtools.


Yes, MT can result into different results (from single-treaded).
The search region is halved (i forget, vertically or horizontally).
But results may be quite good anyway.

Oh ok, if frame search region is getting halved (either on vertically or horizontally), so overall is there much speed gain when using MT. At present say for search param I use exhaustive (search=3), I think. So would it be better to use in single threaded version (without MT) and may be use search as 1 or 2 (logarithmic I think would be better) - to get expeceted results of MVtools than instead of going for fraction of speed bump using MT? What I am thinking of: if MT halves the search region and MVTools based on search parameter passed would look for more data. Coz of MT data is reduced (either on vertically or horizontally), can there be chances of artifacts? I know it is a broad ranged question, and I am putting it here as it ran through my thoughts.

[Fizick]

superuser, please read MT docs at wiki.
halved is probably my wrong term.
With MT, the full frame is divided by 2 (or more) parts., and every thread has own MVTools(Analyse+Degrain) which works in single (one) part only.
So, if object is stay at same part in a time, it is OK. If object cross parts boundary, MVTools can not find it with any search param :)
Of cource, MVTools probably will find some most similar block, but it may be other object.

[superuser]

thnxs fizicks for pointing tht out, learned something new :) ... I was confusing between SetMTMode() and MT() .... may be will try out couple 1000 frames with and without MT() to check it out more.

[AVIL]

@Fizick

I have 2 clips a, b with same framesize, same framerate, same colorspace but non related contents. Both are YUY2 clips, translated in planar form with interleaved2planar.

I do motion compensation on clip a

s=a.MVSuper(planar=true)
vf1a=s.MVAnalyse(delta=1,isb=false,truemotion=true,overlap=4,search=3,searchparam=3)
ca=a.mvcompensate(s,vf1a,planar=true)


I use same vectors to mocomp clip b:

cb=b.mvcompensate(s,vf1a,planar=true,mode=1)


I expected that cb will be a (strange) mocomped version of b. Instead is an exact copy of ca (excepted first frame what is an exact copy of b).

¿Is it correct?

[Fizick]

AVIL, you should produce other supeclip from clip b, and use it in MVCompensate instead of s.

Currently in MVCompensate the source clip is used for audio only, and at bad places: blocks with bad SAD and at scenechenges (like start of clip).

It may be discussed and changed if we found a consensus (in particully with famous trick-lover Didee! :)

[AVIL]

@Fizick

Thanks. Everything is fine now. And gain in speed is noticeable too.

[IanB]

1. All levels are used in MVAnalyse (so, generation is not a waste), but in MVAnalyse only.
MVCompensate (et al) uses finest level only. It is biggest through, especially for widely used pel=2 (and pel=4 of course). We have 4*height. coarse levels use about 1*height max.Yes, I do not fully understand the code, so maybe all the components of all the levels are always needed, I must take your word on this. But they are not all needed at the same time, there is room here for sequencing, which would allow scope for some parallel processing. Also when tight for memory it is better to recalculate a frame again than to crash because a malloc fails. And it is better to recalculate just 1 components of 1 level than the entire set for all levels when available buffers are marginal. (1 humongous composite frame either fits or does not fit. With say 16 component elements any number between 0 and 15 could still fit where a single giant frame cannot.)

2. I certainly do not want to return to RGB32 and split a chroma! I like to have native color format for every level. It was one of my aims. I like to have possibility to see every level and its padding (i already fixed 2 bugs when i saw intermediate clip :) ). We may also apply some external filter to whole super frame (or to sublevel with a mask).It's only an obvious parallelisation, if they are separate they can be calculated in parallel. Far more likely is you want to apply a filter to a level or set thereof. Having the components distinct in Avisynth terms make this easy. My previous thoughts re strings and clips or even more so with the AVSValue arrays idea, offers even simpler access for this case.

3. Anyway, I considered to split the super to finest level and coarse levels. Your suggestion is interesting. Thanks for this and others! I considered and considerer. But I am wonder, how we can multiplex frames with different width and height.If a clip is private like your, you can do whatever you like. Conventional filters may choke if things are not as expected. But each PVideoFrame has it's own shape and size and these are only the same as the VideoInfo values by polite convention. Nothing says these must match always. Just thinking outside the box.

4. Where we really waste a memory, it is in MVFlowXXX, when I build full finest clip (instead separated by pel subplanes). I succesfully impemented (non-public) version without this finest clip, but modified MVFlowFps is slower by 15 percent. Reason - too many calls (for every pixel, not to every block!) to get address of specific pel plane. I may add it as an option for testing.Yes on any CPU architecture that uses cached memory you must arrange your data for good locality. Also you want addressing in inner loops to be simple, like p+=n where n={1,2,4,8,16, ...}.

5. IanB, may CACHE_RANGE help? Can we get some memory improving if we explicitly set cache range by MVSuper(..., cache_radius=1) if we know, that there are MVAnalyse(delta=1) only?No CACHE_RANGE never helps memory usage, it always uses equal or more. Sometime it helps speed by forcing cache buffers to be kept for when they are needed again, but all cache buffers must be expendable. Remember the cache only works when a given frame is requested a 2nd time! And it is only worthwhile when the set of frames that need to be kept between the 1st and 2nd access can fit in the available memory.

So I ask the question inside point 1 above again :- Are all the components, of all the levels, always needed all together every single time?

[Fizick]

So I ask the question inside point 1 above again :- Are all the components, of all the levels, always needed all together every single time?

Thanks for discussion.

In MVAnalyse we use them all in one frame processing to estimate and produce motion vectors.
MVAnalyse firstly uses coarsest level, produces coarsest vectors, then interpolates them to finer level and uses finer levels data to estimate correspondent finer vectors, and so on up to finest level.

So, the answer is: all levels, but not it single time together.

Note: Currently motion vectors of all levels is always present in output vector clip (it is used for storage).

It is other memory problem. I consider how to decrease memory usage for vector clip (not super clip only).
Usually we use top finest levels vectors only in all clients functions (besides special MVShow). Memory is 6 doubleword per block. For small block size it is rathe big value. I consider to convert vector components to short from int and remove not-used 3 components besides SAD (it was used in MVDenoise only). I already removed waste dummy motion compensation storage from vector clip, and it helps to decrese memory usage and improve performance by several procents.


Not (closely) memory related problem: I consider to add variable block sizes to MVTools like h.264 with macroblocks say 16x16 and subparts 8x8 or even 4x4 (and / or may be others 8x16 etc). All this info will be not so easy to code, store and decode :), and it is not very compatible with current MVTools design, in particular PlaneofBlocks concept, and block overlap. And it will be need in some additional parameters (type of macroblock) and storage. I will try to find the simplest solution if it exists. May be not optimal by speed but quality.

[Amefurashi]

Got a problem here: when I serve this rather simple script in VirtualDub

mpeg2source("C:\MINORITY\VIDEO_TS\MR.d2v")

super=MVSuper()
vb=MVAnalyse(super,isb=true)
vf=MVAnalyse(super,isb=false)
MVDegrain1(super, vb, vf)

I get this error message:


http://xs432.xs.to/xs432/08430/screencap1184.png


Any ideas? I'm using AviSynth 2.5.8 RC4a... :confused:

Thanks!

[Fizick]

And what is line 11 in your script with six lines? :)

[Amefurashi]

And what is line 11 in your script with six lines? :)

Sorry, I removed the "loadplugin" lines!
It's the last one ;)

[Fizick]

it is quite possible plugins conflct. so post your full script. If you say A, say B too. :)
What version MVTools do you use? may be some other older floating around.

Whats with some other video, for example ColorBars(pixel_type="YV12")

[Didée]

The script of Amerufashi does work for me, no exception.


in particully with famous trick-lover Didee! :)
Waitaminute! Generally, I'm not such a big fan of playing tricks. However: if there's a certain operation I'd like to do, which basically seems not available from the current set of tools, but a partly approximation can be achieved by using some "tricks", then ... well, so be it.

From the point of writing a script, I've no objections against the current concept of MVSuper. It reflects the old functionality completely (i.e. there's no loss of "features"), and it avoids idx collisions. Good deal so far.

But I got a small surprise when using MVTools2 in practice. Took the script "TemporalDegrain", and modified it for MVTools2 by removing all idx'es, and supplying according MVSuper frames instead.

Comparison of original TemporalDegrain (MVTools v1.10.2.1) vs. modded TemporalDegrain (MVTools v2.0.9.1)

|frames per fixed| phys. | virt.
| time interval | memory | memory
-----+----------------+--------+--------
MVT1 | 1166 (100%) | 392k | 459k (100%)
| | |
MVT2 | 947 (81%) | 378k | 397k (86%)

Memory usage decreased by 16%. Good. Speed reduced by 19%. Not so good.
Are there any optimizations in 1.10.2.1 that are not yet implemented in 2.0.9.1 ?


Note that TemporalDegrain is not an overly complex script, it's ~relatively~ easy. In essence, it boils down to

srch = source.FFT3DFilter()
vectors = srch.MVAnalyse()
source.MVDegrain(vectors).MVDegrain(vectors)

with a few more spatial operations inbetween that skeleton.

(The complete modified function can be DL'ed here (http://www.mediafire.com/file/uyiywe3fzm0/TemporalDegrain_MVT2.avs) (Mediafire). I'm rather confident that all changes related to MVTools2 are correct.)

[thetoof]

I noticed that you're still using the old hierarchical levels reduce method... I thought that since the new "smooth motion estimation" became the default, it was some important improvement that we should benefit from.

Does it mean that some scripts like TD and TGMC must be changed to use rfilter=1 or that a proper mod for MVT2 should use rfilter=0?

[Didée]

Oh. Nah, that was only an accident.

I did the initial test without specifying rfilter at all. Then, I put "rfilter=0" to see if it makes a noticeable differences in speed (it didn't). Then I uploaded the script, still having rfilter=0 in it. That's all.

[Delerue]

Fizick, is there any way to avoid mixed frames in MVFlowFPS? I mean, sometimes is very noticeable that the last frame from take A is mixed with the first frame of take B, like this:

http://img.photobucket.com/albums/v256/Delerue/Merge.jpg

In the case above, we can't see this mix with MVFlowFPS disabled.

Thanks

[Fizick]

thetoof,
rfilter=0 or =1 are simply a little different resizers like bilinearresize and reduceby2.

Delerue,
probably this mixed frame is at scenechange?

If YES: currently at scenechages (= poor estimation with big SAD) MVFlowFPS is downgraded to simple ConvertFPS function (with blending). Probably I can add some option like blend=false, with downgrading to ChangeFPS instead (you will have one or more repeated frames at scenechanges), if you prefere it.

If NOT: provide more info.

[Delerue]

Yes, the two mixed frames come from two different (and consecutive) takes/scenes. Well, I think that repeating the frame is a better idea, because blending doesn't improve the motion, instead it mix incompatible frames that sometimes can annoy people. What you think?

[Fizick]

Delerue,
I do not know what is better in general, but I will implement blend=false option to MVFlowFps (and probably to some other functions, if requested).
We (you) prefer to repeat the previous frame always or nearest to intermediate time value (i.e. next frame if time=70 percents inbetween?) Consider changing FPS from 25 to 100 Hz.

I do not remember changeFPS logic.

[Fizick]

Didée,
all optimizations of 1.10.2.1 are implemented in 2.0.9.1. Moreover, from 1.11.4 too, which use lesser memory for vector clips.

v.2.0 use avisynth cache for superframes, and it is probably the reason for some speed decreasing if memory is low. So, iMVTools2.0 speed is more dependent on memory size. Try set SetMomoryMax to bigger value if you can. I am interested in results.

Will look to script later.

[Delerue]

I don't know, but always repeat the previous frame seems to be a good idea, and the easiest to implement, no? :)

[Didée]

Hehe, I have a little story.

Adjusted SetMemoryMax up & down (my default: 216, it's a PC with 512MB RAM), even reduced frame size to half, but no major change: The MVTools2 version was constantly slower by ~16%, with only minimal changes.

Then I remembered something: boy, on this machine here, I'm still running Avisynth v2.5.6 ...

Went ahead, installed latest Avisynth v2.5.8-RC4 ...
Result: Under 2.5.8, the processing speed of the MVTools2 version was still the same. Almost no change.

But now the catastrophe: All of a sudden, the original MVTools 1.x version was ~30% slower than before!

Or, the other way round:
Compared to running under AS v2.5.8, TemporalDegrain runs 40~45% faster under Avisynth 2.5.6!

At least that's my result here ...


I'm close to losing faith, and will keep using 2.5.6 as long as possible. I'm still stuck with this poor old slow PC, but nonetheless do fiddle a lot with pretty complex scripting stuff. Such an TREMENDOUS speed loss is definetly not an option for me, no way, not at all.

@ IanB: if you're still in search for a showstopper for 2.5.8 final - here you have mine. :)

[thetoof]

Quick tests: memory usage decreased, minimum memory decreased, speed is unchanged.

Version | frames/2:30 | SetMemoryMax | Peak Mem Usage/ VM Size
---------+-------------------+--------------+--
1.11.4.3 | It just hangs | 50 | 145/162k Unsufficient
1.11.4.3 | 310 | 200 | 300/315k ( 65.08% / 63.38% )
1.11.4.3 | 311 | 1000 | 461/497k (100.00% / 103.33% )
1.11.4.3 | 310 | undefined | 461/481k (100.00% / 100.00%)

2.0.9.1 | 308 | 50 | 114/129k ( 24.73% / 26.82% )
2.0.9.1 | 310 | 200 | 267/283k ( 57.92% / 58.84% )
2.0.9.1 | 311 | 1000 | 318/333k ( 68.98% / 69.23% )
2.0.9.1 | 308 | undefined | 318/333k ( 68.98% / 69.23% )

Avisynth 2.5.8 RC4

Script v1:
setmemorymax(x)
srch=mpeg2source("Op.d2v")
ov=4
blksize=8
pel=2
b3vec = srch.MVAnalyse(isb=true, delta=3, overlap=ov, blksize=blksize, pel=pel,idx=1)
b2vec = srch.MVAnalyse(isb=true, delta=2, overlap=ov, blksize=blksize, pel=pel,idx=1)
b1vec = srch.MVAnalyse(isb=true, delta=1, overlap=ov, blksize=blksize, pel=pel,idx=1)
f1vec = srch.MVAnalyse(isb=false, delta=1, overlap=ov, blksize=blksize, pel=pel,idx=1)
f2vec = srch.MVAnalyse(isb=false, delta=2, overlap=ov, blksize=blksize, pel=pel,idx=1)
f3vec = srch.MVAnalyse(isb=false, delta=3, overlap=ov, blksize=blksize, pel=pel,idx=1)
srch.MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=1).MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=1).MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=1).MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=1)

Script v2:
setmemorymax(x)
srch=mpeg2source("Op.d2v")
ov=4
blksize=8
pel=2
srch_super = srch.MVSuper(pel=pel)
b3vec = srch_super.MVAnalyse(isb=true, delta=3, overlap=ov, blksize=blksize)
b2vec = srch_super.MVAnalyse(isb=true, delta=2, overlap=ov, blksize=blksize)
b1vec = srch_super.MVAnalyse(isb=true, delta=1, overlap=ov, blksize=blksize)
f1vec = srch_super.MVAnalyse(isb=false, delta=1, overlap=ov, blksize=blksize)
f2vec = srch_super.MVAnalyse(isb=false, delta=2, overlap=ov, blksize=blksize)
f3vec = srch_super.MVAnalyse(isb=false, delta=3, overlap=ov, blksize=blksize)
srch.MVDegrain3(srch_super,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)
MVDegrain3(srch_super,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)
MVDegrain3(srch_super,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)
MVDegrain3(srch_super,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)


The simplified syntax of the multi branch would be more than welcome... Instead of isb/delta, a simple refframes= or tradius= is much simpler.

edit: I installed 2.5.6 Oct 28 2005 and it had no influence on the results... Can someone else confirm?

[Didée]

I'll try to test my scenario on the more beefy office PC - dated too, but at least a dual core /w 1 GB RAM.

@ thetoof: your scripts do not fully reflect that scenario. Those scripts re-use the same old stuff over and over again, so they're not that demanding on the cache system like they look like at first glance...

Modified to be similar to my TemporalDegrain test:

script1:
setmemorymax(x)
source=mpeg2source("Op.d2v")
srch=source.temporalsoften(2,2,2,20,2) # just to do something temporal here
ov=4
blksize=8
b3vec = srch.MVAnalyse(isb=true, delta=3, overlap=ov, blksize=blksize,idx=1)
b2vec = srch.MVAnalyse(isb=true, delta=2, overlap=ov, blksize=blksize,idx=1)
b1vec = srch.MVAnalyse(isb=true, delta=1, overlap=ov, blksize=blksize,idx=1)
f1vec = srch.MVAnalyse(isb=false, delta=1, overlap=ov, blksize=blksize,idx=1)
f2vec = srch.MVAnalyse(isb=false, delta=2, overlap=ov, blksize=blksize,idx=1)
f3vec = srch.MVAnalyse(isb=false, delta=3, overlap=ov, blksize=blksize,idx=1)
source.MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=2)
MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=3)
MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=4)
MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=5)


script2:
setmemorymax(x)
source=mpeg2source("Op.d2v")
srch=source.temporalsoften(2,2,2,20,2) # just to do something temporal here
ov=4
blksize=8
srch_super = srch.MVSuper()
b3vec = srch_super.MVAnalyse(isb=true, delta=3, overlap=ov, blksize=blksize)
b2vec = srch_super.MVAnalyse(isb=true, delta=2, overlap=ov, blksize=blksize)
b1vec = srch_super.MVAnalyse(isb=true, delta=1, overlap=ov, blksize=blksize)
f1vec = srch_super.MVAnalyse(isb=false, delta=1, overlap=ov, blksize=blksize)
f2vec = srch_super.MVAnalyse(isb=false, delta=2, overlap=ov, blksize=blksize)
f3vec = srch_super.MVAnalyse(isb=false, delta=3, overlap=ov, blksize=blksize)

super1 = source.MVSuper()
source.MVDegrain3(super1,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)
super2 = last.MVSuper()
last.MVDegrain3(super2,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)
super3 = last.MVSuper()
last.MVDegrain3(super3,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)
super4 = last.MVSuper()
last.MVDegrain3(super4,b1vec,f1vec,b2vec,f2vec,b3vec,f3vec)

(Better don't try that with full-HD stuff...)

[thetoof]

Oh dang, you're right. I was just about to leave, so I'll run the tests when I come back to confirm your findings.

[Fizick]

Didee,
I can confirm that 2.0 is slower for this script (did not try 2.5.6).
Small more optimal setting.
Use MVSuper(...,levels=1) if you do not MVAnalyse it (second and third item in your script).

Will research.

[IanB]

Compared to running under AS v2.5.8, TemporalDegrain runs 40~45% faster under Avisynth 2.5.6!Are you comparing apples with apples here. I think you will find 2.5.6 almost totally ignores SetMemoryMax and you are running a defacto very large cache. If you set the cache to the same resultant size the speed should be the same with 2.5.8.

These are 4 items from the change list that are related. The 1st is probably what is happening here.* Fixed Cache memory oversubscription of SetMemoryMax() limit.
* Initial Memory Max value clamped to 512MB.
* Default Memory Max value restored to quarter of Free memory. Minimum 16Mb. As per 2.5.7.
* SetCacheHints(CACHE_RANGE, n) will now surrender frames to satisfy SetMemoryMax().

[Adub]

So, then by increasing SetMemoryMax() can we reach the "40-45%" speed increase on 2.5.8 that Didee had back with 2.5.6?

I myself am using 2.57, so I am curious if a specific setmemorymax call can increase my speed as easily.

[morsa]

That thing about the scene change blending is exactly the problem I suffer everytime I do framerate conversion.

I tried sometime ago to use the mvtools scenechange detection to generate a mak an replace the frames by the ones given by changeFPS, but it seems scene change detection wasn't working reliable.

So If I had this modification applied to MVtools it will make my life a lot easier.The situation right now is that after changing framerate, I need to manually check every cut and replace frames manually, because sometimes they are just blended and someothers they are really trashed.

[IanB]

@Merlin7777,

Assuming Didée's problem is actually that the cache size he is using is to small for the script he is running, then yes setting SetMemoryMax to the appropriate value for the script in question will help a lot.

The Avisynth cache is a simple LRU scheme. If the size is insufficient to hold all the buffers between the initial request for a given frame and the subsequent request for the same frame again then the cache will be absolutely useless.

[Adub]

Is there a quick and dirty way to calculate the appropriate value for a setmemorymax line? I have 4gb (well on a 32bit xp install) and if there is a way that I can use more effectively, I would like to know if there is a procedure, or a good rule of thumb that I should use when dealing with scripts like TemporalDegrain or MC_Spuds or MCTemporalDenoise.

I seem to recall an old test that someone ran ages ago using a function that Didee wrote called "Bob" or something, with took seconds per frame to render. It would generate several scripts, all with different SetMemoryMax lines. I remember that we came to specific conclusion, something along the lines of 50-75% of total ram was optimal. I just wish I could find that thread again.


Edit: AHA!!
http://forum.doom9.org/showthread.php?t=120371

It seems half of total ram is usually a good choice.

[IanB]

something along the lines of 50-75% of total ram was optimal.Hardly! Using a sledge hammer to crack wall nuts here. Any amount more than enough to satisfy a given script will work the same! Well except for the overhead involved in managing a larger amount of memory. And of course the risk of using up the entire 32bit address space, which is of course catastrophic.

[Fizick]

i can not confirm v2.5.6 speed increasing.

[IanB]

@Fizick, and you will have a devil of a time trying. The nearest you will get to seeing the issue reliably is to compare SetMemoryMax(1) and SetMemoryMax(SomethingBig) under 2.5.8, under 2.5.6 the cache does not obey SetMemoryMax correctly so you cannot force the issue without a debugger (and some hair loss ;) ).