Trying to make MDegrain3 not apply on a portion of a movie, need help.

[Chainmax]

I am backing up a series and this latest episode I'm tackling has a section that is particularly nasty noise-wise. After some experimentation with nuke-it-from-orbit filtering applied to that section, I decided to leave it be, as the loss of detail and huge amount of extra processing time was not worth it.

Since the scene is short and mostly static, leaving the brunt of the denoising out of it is a reasonable compromise. Thing is, the main denoising is done with MDegrain3, like so:

source=last
preNR=source.dfttest(sigma=1.00,tbsize=3).FFT3DFilter(sigma=9,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16)
preNR_super=preNR.MSuper(pel=2, sharp=1)
source_super=source.MSuper(pel=2, sharp=1)
backward_vec3 = MAnalyse(preNR_super,isb = true, delta = 3, overlap=4)
backward_vec2 = MAnalyse(preNR_super,isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(preNR_super,isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(preNR_super,isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(preNR_super,isb = false, delta = 2, overlap=4)
forward_vec3 = MAnalyse(preNR_super,isb = false, delta = 3, overlap=4)
source.MDegrain3(source_super,backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=350)

The problem section is frames 5535 to 5658. Having never had to make multi-line filters like this one not work on a range, I am at a loss on how to proceed without overcomplicating things. Is there an elegant-ish way to do this, or will I have to cough a mess of Trims and splices?

[cretindesalpes]

f1 = filter1 ()
f2 = filter2 ()
f1.rfs (f2, "[5535 5658]")

rfs is a modification of ReplaceFramesSimple and can be found here (http://forum.doom9.org/showthread.php?p=1644971#post1644971).
You can add more ranges in the mappings string if required.

[martin53]

The usual solution is wrap the multi-line process in a function
function filter(clip source) {
preNR=source.dfttest(sigma=1.00,tbsize=3).FFT3DFilter(sigma=9,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16)
preNR_super=preNR.MSuper(pel=2, sharp=1)
source_super=source.MSuper(pel=2, sharp=1)
backward_vec3 = MAnalyse(preNR_super,isb = true, delta = 3, overlap=4)
backward_vec2 = MAnalyse(preNR_super,isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(preNR_super,isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(preNR_super,isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(preNR_super,isb = false, delta = 2, overlap=4)
forward_vec3 = MAnalyse(preNR_super,isb = false, delta = 3, overlap=4)
source.MDegrain3(source_super,backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=350)
}

and call the function
source=last
last.trim(...,...).filter
\++ last.trim(...,...)
\++ last.trim(...,...).filter

[TurboPascal7]

That usual solution is a really bad idea for caching, performance and stability. Plus with temporal filters you will get a bit different processing on first/last frames.

A better solution would be doing it like

source=last
filtered = filter() # you can just put all the code here and use source instead of last later
filtered.trim(...,...)
\++ last.trim(...,...)
\++ filtered.trim(...,...)

But this gets unreadable pretty fast. cretindesalpes's advice works better for this kind of filtering.

[martin53]

That usual solution is a really bad idea for caching, performance and stability. Plus with temporal filters you will get a bit different processing on first/last frames.
Please indicate how the filter call will affect stability.
There is nothing to be cached beyond the trim points, so what should be the matter here. And same with performance: there are 2 instances of the filter functions, but these are only called for their smaller part of the clip. In sum, exactly the same number of frames is sent through the filter.
We are in consent that there is a difference in processing adjacent to the trim points.
But in my opinion, it is not a good, but in contrary a bad idea to include some frames of the unfiltered part in the filtering. Probably there's a scene change, so these frames have nothing to do with the frames at the other side of the fence. That's why I deliberately put the filter to the end of the graph. And Cretindesalpes' proposal suffers from the same issue.

[TurboPascal7]

Please indicate how the filter call will affect stability.
There is nothing to be cached beyond the trim points, so what should be the matter here. And same with performance: there are 2 instances of the filter functions, but these are only called for their smaller part of the clip. In sum, exactly the same number of frames is sent through the filter.
In perfect world - maybe. Avisynth, however, likes to crash when you create too many filter instances. The most reasonable explanation for this I've heard is "imagine filters allocating 1.5GB of memory (total) in constructors and avisynth's cache being limited to 1GB. After allocating 500MB of its cache, avs will happily continue allocating more memory eventually crashing thanks to the 32-bit memory limit". Exact numbers are not important - I had a short (1700 frames) script crashing every 300 frames or so with avs2yuv patched for 4GB memory.

If only there was a way for avisynth to detect that filter XXX was called on clip YYY with the same parameters before and reuse the same filter instance...

[Gavino]

The original question had only one unfiltered section, so martin53's approach is reasonable for that (particularly considering his point about scene change). However, if there are many sections, cretindesalpes's approach is prefererable to avoid excessive memory requirements.

[StainlessS]

ClipClop is another similar alternative to rfs.

[Chainmax]

I tried martin53's suggestion. I will update on how the encoding process goes. Thanks. :)

[raffriff42]

The usual solution is wrap the multi-line process in a function...

That usual solution is a really bad idea for caching, performance and stability. Plus with temporal filters you will get a bit different processing on first/last frames.

Don't want to sidetrack the thread, but do you have more information on this? I thought, reading the the Avisynth wiki (http://avisynth.nl/index.php/The_script_execution_model/The_fetching_of_frames), that any filter not in the graph for the frame currently being fetched will not be called. I suppose caching might make this more complicated. Anyway, I like using lot of small functions for code re-use and would like to know if there is really a performance hit.

[Gavino]

I thought, reading the the Avisynth wiki (http://avisynth.nl/index.php/The_script_execution_model/The_fetching_of_frames), that any filter not in the graph for the frame currently being fetched will not be called.
That's correct. However, all filters are loaded at compile-time and remain loaded (and occupying memory) until the script is unloaded, even if they are not being used for the current frame.
Anyway, I like using lot of small functions for code re-use and would like to know if there is really a performance hit.
There is no performance hit with functions (compared with using the same code outside a function).
TurboPascal7's objection was not to the function itself, but the way it was used. In martin53's script, the function was called twice (and in a larger example potentially many more times), whereas with cretindesalpes's approach (and with TurboPascal7's own script), the function would be called just once and the different frame ranges obtained from that single instance.

[raffriff42]

In martin53's script, the function was called twice (and in a larger example potentially many more times), whereas with cretindesalpes's approach (and with TurboPascal7's own script), the function would be called just once and the different frame ranges obtained from that single instance. Thank you for the answer. So, if I understand you: if a function call appears twice in a script, two instances of the function are generated, and the filter calls within each instance will have redundant caches, and this is where the inefficiency (and bugs) come in? This being true even if (and this is where I was confused) the function would only be called once for any given frame? (compile time != run time)

[Gavino]

if a function call appears twice in a script, two instances of the function are generated, and the filter calls within each instance will have redundant caches, and this is where the inefficiency (and bugs) come in?
Yes, more or less.
More accurately, each time the function is called in the script, its body is evaluated, generating a fresh section of filter graph, with new instances of the filters used. (The function itself does not exist at run-time, but the filter instances do.)

It's not just the caches, the filters themselves (depending on their requirements) may allocate memory and each instance will have its own copy. So it makes sense to create as few instances as possible when you have the choice.

[WorBry]

Great stuff; I was contemplating more or less the same thing myself, although in the context of a predefined ('wrapped', as it were) function (SMDegrain specifically).

So, just to be clear - the consensus is that when seeking applying a given function to multiple ranges in a clip, the most efficient (least resource-demanding) way is cretindesalpes's rfs approach?

I'm just trying to figure how to apply that to a defined function with variable parameters when you want to apply the function with different settings to different segments of the clip.

Taking SMDegrain, just as an example

http://doom10.org/index.php?topic=2178.0

Lets say I have a clip of 500 frames and wish to apply:

SMDegrain(tr=1) to frame ranges 50 - 100 and 250 - 300
and
SMDegrain(tr=3) to frame ranges 150 - 200 and 350 - 400

I can see how that might be done using martin53's sequential 'trim/filter' call method, but I just can't figure how you would write that using rfs. Does each of those ranges require a separate rfs/function call in the mapping string, or is possible to apply more than one range to a given rfs/function call?

[Gavino]

So, just to be clear - the consensus is that when seeking applying a given function to multiple ranges in a clip, the most efficient (least resource-demanding) way is cretindesalpes's rfs approach?
Yes, as long as the same function parameters are to be used for each range and, for temporal filters, that martin53's point (post #5) about possible processing differences at range boundaries is not a problem.

Lets say I have a clip of 500 frames and wish to apply:

SMDegrain(tr=1) to frame ranges 50 - 100 and 250 - 300
and
SMDegrain(tr=3) to frame ranges 150 - 200 and 350 - 400

I can see how that might be done using martin53's sequential 'trim/filter' call method, but I just can't figure how you would write that using rfs. Does each of those ranges require a separate rfs/function call in the mapping string, or is possible to apply more than one range to a given rfs/function call?
You can have more than one range in a call, but since it replaces frames from one clip by those from another clip, you need two replacement clips, one for each distinct SMDegrain call. In your example, you would have two rfs calls, each with two ranges.
d1 = SMDegrain(tr=1)
d3 = SMDegrain(tr=3)
rfs (d1, "[50 100] [250 300]")
rfs (d3, "[150 200] [350 400]")

[WorBry]

You can have more than one range in a call, but since it replaces frames from one clip by those from another clip, you need two replacement clips, one for each distinct SMDegrain call. In your example, you would have two rfs calls, each with two ranges.
d1 = SMDegrain(tr=1)
d3 = SMDegrain(tr=3)
rfs (d1, "[50 100] [250 300]")
rfs (d3, "[150 200] [350 400]")

Yes, exactly that. So it's that simple. Process efficiency aside, that definitely makes for greater ease and economy in the scripting - not having to account for the un-filtered segments. Brilliant. :)

[StainlessS]

ClipClop allows for up to 255 replacement clips without multiple plugin instances (and unlimited number of ranges).

see MediaFire in sig