Dogway's Filters Packs

[Dogway]

LeXXuz, try now, I updated SMDegrain with some improvements. First unprocessed planes are copied from input to mfilter clip when using mfilter obviously, this makes more sense.
Also multi (tr > 6) happened to don't be correct for some modes like LFR, mfilter or mode!="MDegrain", that is fixed now, and also with nice for loops, so less code lines.

About SceneStats that's a delicate issue due to the complexity of the filter. First I'm taking things slower now, and second I decided to wait until the issue happens to me again instead of searching for it, but it's acknowledged and will tackle with it when that happens.

DTL, do you have literature about this approach (covariance for texture/sharpness metrics)? What variables would covariance employ in this case?

[LeXXuz]

Thanks Dogway, updating now.

I don't know if that would be any simpler, but if you could just disable that error overlay that would be a big help already.

Otherwise I'll wait with these sources, no problem at all. Enough other stuff to work on from my collection.

[Dogway]

I don't think it's that easy, as I said the filter is quite complex, and even if I fix the eventual errors (at least the ones I was having) fades in/out will still need to be trimmed out. If someone comes up with a fade detector that would be welcomed.

[LeXXuz]

Originally Posted by Dogway

I don't think it's that easy, as I said the filter is quite complex, and even if I fix the eventual errors (at least the ones I was having) fades in/out will still need to be trimmed out. If someone comes up with a fade detector that would be welcomed.

Okay, no problem. And thanks for the suggestion with Expr() and ex_lut() for mfilter preview. They show way better which areas are processed.

[DTL]

"do you have literature about this approach (covariance for texture/sharpness metrics)? What variables would covariance employ in this case?"

Covariance is major part of most 'complex' metrics (more complex in compare with 'simple' SAD). SSIM and VIF widely uses covariance component. SSIM have 'structure' part based on covariance and VIF also uses covariance in output computing formula. You can look into SSIM and VIF computing functions in mvtools to see how it calculated from blocks arrays samples:

SSIM https://github.com/DTL2020/mvtools/b...MFunctions.cpp

Full SSIM is SSIM_Light * SSIM_Contrast * SSIM_Structure . SSIM_Light is sort of simple average of samples. SSIM_Contrast is some function of (variance_a, variance_b) where variance_a is fsX and variance_b is fsY at https://github.com/DTL2020/mvtools/b...ctions.cpp#L57

And SSIM_Structure is function of covariance(a,b) (fsXY) - https://github.com/DTL2020/mvtools/b...ctions.cpp#L66 . And after getting all 3 components of SSIM output is simple multiplication.

But for some specific analisys each part of SSIM may be used separately. Most interesting for textures may be SSIM_Structure calculated from covariance.

VIF is https://github.com/DTL2020/mvtools/b...FFunctions.cpp

It uses covariance of DWT - https://github.com/DTL2020/mvtools/b...ctions.cpp#L66 (DWT_a component) for VIF_A part and also some covariance for VIF_E part - https://github.com/DTL2020/mvtools/b...ctions.cpp#L98

It may be possible to convert C-computing to AVS scripting computing and you can make full or structure-only SSIM in AVS function.

The VIF DWT-based require DWT (it is not complex but more slow). The most diffrerence between SSIM and VIF for structure part is SSIM compute from direct input blocks samples covariance and VIF compute covariance from DWT-transformed blocks (may be sort of low-pass filtering or twice reducing ?).

[Dogway]

Thanks for the links, I will have a look later. If none of covariance components come from a texture analysis it would be a moot point. I have SSIM ported to a function in SimilarityMetrics, but I don't recall any texture extraction.
VIF doesn't sound familiar to me so will have a look (EDIT: ok, VIF -Visual Information Fidelity- from my metrics issue). DWT is frequency based so that might be useful.

The problem probably with blocks is that small details vanishes from the block average, unless augmentation is performed, that or dynamic sized blocks come into play like x265.

[Boulder]

For visualizing the effect of filters outside MDegrain, I've used the Subtract method like Subtract(finalclip_nonblurred, finalclip.mergechroma(finalclip_nonblurred)).levels(127*256, 1, 132*256, 0*256, 255*256).subtitle("Effect of blur in result", y=40). Here 'finalclip' is the normal output, finalclip_nonblurred the one where there is no denoising other than MDegrain and then I just want to see how luma looks like so the chroma planes are merged from the normal output.

Funny thing is that the filter gets quite a big weight even if you use something like thsad=10000 in MDegrain, it won't change the output much if you limit the effect of MDegrain itself. I was always under the impression that thsad can be used to control which parts of the frame get processed by MDegrain and which parts are skipped because the calculated thsad is too high.

[DTL]

" If none of covariance components come from a texture analysis it would be a moot point. "

Covariance in SSIM is direct math covariance between samples of compared blocks. It shows how the samples of the compared blocks changes in similar way at each sample. If one block is blurry and second is sharp (non-damaged by blur) - the covariance metric will be lower.

Covariance is
fsXY += (float)((pWorkSrc[x] - isuX) * (pWorkRef[x] - isuY));
- sum of multiplication of (blocks values - mean block value (mean DC component)). So if samples changes in same direction from mean value - the mul operation gives higher result. Also it is positive for both positive and negative difference from mean value (DC) if both differences have same sign.

So if some processing start to damage texture in compare with 'initial source' - the covariance metric become lower and it can be detected and the processing may be adjusted to keep texture more non-damaged. Also the processing may be optimized (zopti ?) to maximize covariance metric between input and output.

[LeXXuz]

Originally Posted by Boulder

Funny thing is that the filter gets quite a big weight even if you use something like thsad=10000 in MDegrain, it won't change the output much if you limit the effect of MDegrain itself. I was always under the impression that thsad can be used to control which parts of the frame get processed by MDegrain and which parts are skipped because the calculated thsad is too high.

I think without raising thSCD as well you will not do much, even with thSAD that exorbitant high, since every frame is probably considered a scene change now?

[Boulder]

Originally Posted by LeXXuz

I think without raising thSCD as well you will not do much, even with thSAD that exorbitant high, since every frame is probably considered a scene change now?

thSAD is not directly connected to scene change detection, you can use a higher thSAD value than thSCD1 and the frame will still show motion vectors with MShow (which is very useful with showsad=true). As far as I know, thSCD1 sets the threshold (per block) and thSCD2 then sets the max amount of blocks triggered by thSCD1.

[DTL]

If you set thSAD to very high value and scene detection still not skip frame - you simply lost protection from bad blends and MDegrainX will work as linear motion compensated blending engine (may be equal to MCompensate + Average()). If MVs are not perfect or there is other non-supported and not-compensated transforms left it will cause either blurring or visible several different blocks blended. thSAD adjustment in MDegrainX is only to protect user for possible artifacts of the very very simple temporal denoise method of different frames blocks weighted averaging.

In a perfect world with ideal transforms compensation no any protection required (see RIFE + Average() denoise example).

thSCD protects from more general error of blending different scenes in single frame.

[FTLOY]

Originally Posted by Dogway

Simply used:

pre=ex_Median(mode="IQMST",thres=255)
STTWM(sw=50,tw=50,aw=100,sthres=5,tthres=5)
SMDegrain(tr=2,thSAD=200,thSADC=100,contrasharp=true,prefilter=pre,str=1.2,refinemotion=true)
FrameRateConverterMIX(FrameDouble=true)

in AvsPmod, and I think without Prefetch() even as I only use it when encoding.
You might need to review your workflow, use avs+ test 9, setmemorymax to an appropiate value, and load the mt-modes.avsi. Also test with other source loaders.

I think I might have figured out what's going on...

What type (resolution) video did you test this script on ?

I am trying to use FrameConverter on 4K HDR10 content, and I think the combination is just too much for it, 'cause it seems to be ok, but slow, on lesser resolution's.

[LeXXuz]

Originally Posted by FTLOY

I think I might have figured out what's going on...

What type (resolution) video did you test this script on ?

I am trying to use FrameConverter on 4K HDR10 content, and I think the combination is just too much for it, 'cause it seems to be ok, but slow, on lesser resolution's.

Could be a memory problem. How much RAM does your system have? If you have 32GB for example, try adding

SetMemoryMax(20*1024) # max RAM usage in kilobytes
SetCacheMode(0)

at the very beginning of your script. If your machine has more RAM you can raise that even more.

I have some script combinations which have very high RAM demands and either will not start or be very very slow on higher resolutions, if I don't set MemoryMax correctly.

Also, some filters don't really like caching, especially in multithreading with a high prefetch count, so I have to set cache mode explicitely to zero.

[FTLOY]

Originally Posted by LeXXuz

Could be a memory problem. How much RAM does your system have? If you have 32GB for example, try adding

SetMemoryMax(20*1024) # max RAM usage in kilobytes
SetCacheMode(0)

at the very beginning of your script. If your machine has more RAM you can raise that even more.

I have some script combinations which have very high RAM demands and either will not start or be very very slow on higher resolutions, if I don't set MemoryMax correctly.

Also, some filters don't really like caching, especially in multithreading with a high prefetch count, so I have to set cache mode explicitely to zero.

I do only have 32Gb....

And I have this in every script:-

SetCacheMode(1)
SetMemoryMax(16384)

And the Prefetch is 16, but it's not included in the scripts.

[LeXXuz]

Originally Posted by FTLOY

I do only have 32Gb....

And I have this in every script:-

SetCacheMode(1)
SetMemoryMax(16384)

And the Prefetch is 16, but it's not included in the scripts.

Try with cache mode 0.

[Boulder]

Originally Posted by DTL

If you set thSAD to very high value and scene detection still not skip frame - you simply lost protection from bad blends and MDegrainX will work as linear motion compensated blending engine (may be equal to MCompensate + Average()). If MVs are not perfect or there is other non-supported and not-compensated transforms left it will cause either blurring or visible several different blocks blended. thSAD adjustment in MDegrainX is only to protect user for possible artifacts of the very very simple temporal denoise method of different frames blocks weighted averaging.

In a perfect world with ideal transforms compensation no any protection required (see RIFE + Average() denoise example).

thSCD protects from more general error of blending different scenes in single frame.

My case was just an example of trying to emphasize the effect. My default values for thSAD and thSCD1 are 400 and 600 respectively. thSCD1 sometimes needs to be bigger if the source is very grainy (despite heavy prefiltering).

[FTLOY]

Originally Posted by LeXXuz

Try with cache mode 0.

Isn't 0 default, so I could simply remove that call...

[LeXXuz]

I don't trust defaults.

[FTLOY]

Originally Posted by LeXXuz

I don't trust defaults.

I might as well raise the "memorymax", too...

[LeXXuz]

You probably have to. 16GB is not very much for more complex filtering of 4k content.