Introducing Zopti (ex AvisynthOptimizer)

[kedautinh12]

I think f3kgrain will make a good addgrain :D
https://github.com/kedaitinh12/AVSPlus-Scripts/blob/master/f3kgrain_v0.4MOD.avsi

[DTL]

"not sure if it respects grain distribution though"

If you can do something in math - you can get set of samples values and perform statistical analysis: if it good or bad fit natural photon noises: Poisson distribution properties - https://en.wikipedia.org/wiki/Poisson_distribution
Gaussian - https://en.wikipedia.org/wiki/Normal_distribution

It is also one thing to add to MDegrain - it currently compare current frame with ref frame. But when looking for median area value it is better to compare current frame with 'median' value. So current algoriphm skips some blocks that fits in the thSAD from median value but falls out of thSAD check if current and ref blocks are on the different sides of the 'distribution curve'.

It can be partially workarounded with multi-pass MDegrainin - 1st pass moves block samples values closer to its median and next passes can add to averaging more blocks. But it is very slow. Faster it make with statistical analysis of the currently loaded pool of blocks in tr-scope around current frame in 1-pass.

Now about trymany:

"trymany, what does it do"

It is all inside PseudoEPZSearch() in PlanesofBlocks.cpp https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/PlaneOfBlocks.cpp#L901

For each round of search at each level it is possible to be predictors:
1. Zero - its current block position with zero move. Starting predictor at each round/level.
2. Global - it is median vector of motion of the whole frame.
3. Hierarchical - it is the only interconnecting between levels predictor - spatially interpolated MV of the prevoius level (to 4 surrounding blocks).
4. 3 surrounding blocks predictors (left, top (found at current level), bottom (interpolated from prev level) + their median = 4 more predictors
5. Temporal - same block MV from previous frame
Total 3+4+1=up to 8 predictors.

And at the end - the Refine() function that actually perform the requested search with requested radius around _requested_ position.

If trymany disabled:
Check performed for each 1..8 predictors once (at its MV value) and progressively collected best (lowest) 'cost' value (also best SAD value in parallel).
After it - Refine() is called once to perform actual search around best found 'prediction' position.

If trymany enabled:
After each check of 1..8 predictors called Refine() search function to try to get better MV (cost) around each predictor.
Finally Refine() called for the best of the best found MV (at the predictors + refining stage). Its output SAD value is written to output array.

cost = SAD + penalty * SAD; (for some predictors penalty is individual setting - like zero, global. new is general penalty for other predictors and searches (refining))

This may be better for motion interpolation task but skips some best SAD values. So sometime setting all penalties to zero may give better denoising results with lower thSAD value. But if it cause to worse MVs search - it may cause error-blending distortions (?).

" why trymany=true gives worse results than trymany=false?"

Generally trymany should give best possible MVs but its actual results may depend on both MAnalyse settings and input footage.

"I half made a SAD analysis tool I need to finish it but it's a manual task."

It is possibly good to use MAnalyse as SAD calculating engine - it have at least many good speed optimizations. Just set levels to 1 and search radius to 0 and it should return SAD of the zero predictor. But current Avisynth looks like not allow to use MAnalyse output as input to other filters ? Though it can output result to binary file and it can be parsed after somehow loading in the Avisynth or by external software.

"in filmed content grain is the actual image not a transparent layer over it,"

I think of natural photon-shot noise model as of addition of noise-distributed random values to some zero-noise source patch (flat or textured). So the processing of MDegrain is real physics based - it average many equal patches with added random noise with bipolar symmetric (enough symmetric at high photon-numbers and Natural-Gaussian distribution - but it is not exactly valid for dark low-photons areas with Poisson distribution ?) distribution. So after averaging enough many patches (large enough tr-value of MDegrain) you can finally got initial noiseless patch looking.

So it is important for synthetic noise generator to have natural random values distribution. Not 'flat' in some total deviation range.

For best possible results the thSAD value must be inifinitely big so all possible max noise deviations are totally compensated. But SAD also used in block-matching processing so too high thSAD cause errors-blending at moving areas. May be it is also good to add some thSAD modulation depending on the MVs length (like increasing thSAD at the areas with low or zero MVs that may be hint of static areas).

[Dogway]

@zorr: True! how odd, truemotion arg doesn't override the truemotion settings, while at it I also tested truemotionR and temporal, and they didn't output a different GMSD metric. I also thought about the correlation matrix, there's a way to know for certain how they relate each other, this is me thinking as I haven't done any further research. You have to lock the GMSD score. Yes, you can't so the procedure is to shoot a few thousand randomized iterations and pick the first three blocks with same and best scores. That is pick 5 rows with score 95.0 +- 0.001 (ie out of 100.0), pick 7 rows with score 86.0, and 5 (or whatever matched) rows of score 80. Do the correlation matrix of each one, fit a regression for the 3 correlation points to check if the relationship is linear or not (always relative to the GMSD score of course).

About the issue I have with your newer version, the script seems to be processed until the end then in the result file I get this error message:
Error getting frame property "_PlaneGMSD": property is not set
(C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0\work\Zopti_test2s - MDegrain3 - copia_1642960023349_5_1642960023509.avs, line 102)
(C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0\work\Zopti_test2s - MDegrain3 - copia_1642960023349_5_1642960023509.avs, line 106)
And this is the script: pastebin (https://pastebin.com/qB7rmHuX)

@DTL, I have read about Poisson distribution but haven't worked with it. So you say that MDegrain is a gaussian weighted median of some sorts? My SAD analysis tool is basically a flat patch searcher, I got the patch size from Neat video, I think it was 128x128, but I will try to check with your suggestion, I didn't know MAnalyse could output SAD value (?), interesting. Decoupling thSAD from block matching would also make sense or the modulation you talk about as it makes total sense, it would make the use of prefilters less necessary.

I will try to render a few low sample shots for MDegrain tests and post here when I get some time.

[DTL]

"MDegrain is a gaussian weighted median of some sorts? "

It is simple weighted averaging tool. Currently - blocks averaging with or without overlapping. Gaussian is expected noise distribution of some natural sources. As I read somewhere in math wiki - with high enough samples (photons) number the Poisson distribution is close to Gaussian. And as natural video cameras typically works with low to medium photon numbers per sample it mean at low samples values its noise deviation should be closer to Poisson distribution and with high samples values - to Gaussian distribution.

"I didn't know MAnalyse could output SAD value (?)"

It is the natural MAnalyse output format:
x, y, SAD (all integers 32bit signed). The output file have some short 'header' and list of all blocks data in that format.

outfile

Name of file to write motion vectors data, or an empty string (nothing written). This data may be used by some external program or may be by next MVTools versions for second pass coding, etc. Produced binary file has a header (MVAnalysisData structure, see MVInterface.h source code), and the data sequence:
◾Frame number,
◾vector data (Vx, Vy, SAD) of every block,
◾next valid frame number,
◾this frame vector data,
◾and so on…

So as DX12_ME hardware do not output SAD - it was added separate SAD compute shader to calculate SAD for complete output. Before compute shader version it perform call to SearchMVs() of GroupofPlanes to calculate SAD only and fill output structure with SAD values.

The mean frame SAD you can see in MShow() output with showsad=true. Though it is rendered in output framebuf for visual reading - not returned as some variable to script for processing.
I sometime use MShow(showsad=true) to check median SAD to adjust initial thSAD in MDegrain - it should be a bit more minimum or the degraining is about disabled (weights of most ref blocks = 0).

May be it also possible to get access in script environment to the output of MAnalyse data currently passed to MDegrain (and all other clients filters) and fetch x,y,SAD data for analysis. May be ask pinterf to patch Avisynth to allow some data fetching from this 'pseudoclip' ?
Or add some 'clip-source' converter filter to mvtools to 'convert' MAnalyse output to 'filter-readable/script-readable clip' inside AVS+ environment with Eval() (?) function. Like convert to virtual clip of BlckX (number of blocks per width) width and BlckY (number of blocks per height) height clip of RGB48 format and you can get x,y,SAD triplets as R,G,B 16bit sample components. RGB96 looks not supported as pixel_fmt in current AVS+ to read full 32bit signed values - but >16bit SAD is mostly possible only for HBD and large block sizes. Typical 'good' SAD values are about thSAD values and good fit in 16bit signed.

[zorr]

It is long to type - better next day. It is night here and better to try to sleep somehow.

Yes, do get some sleep, I'm not in a hurry. :)

4:8 for YV12 not mean UV is twice more weighted.

Thanks for the detailed explanation, but my test clip in the MFlowFPS tests was converted to YV24 before it was processed with MVTools so it doesn't apply there.

I found this in your other message, is this basically the chroma scaling algorithm currently used in MVTools?


// YV12 YV16 YV24
// nLogXRatioUV 1 1 0
// nLogYRatioUV 1 0 0
// effective_chromaSADscales: (shift right chromaSAD)
// chromaSADscale=0 -> 0 1 2 // default. YV12:no change. YV24: chroma SAD is divided by 4 (shift right 2)
// =1 -> -1 0 1 // YV12: shift right -1 (=left 1, =*2) YV24: divide by 2 (shift right 1)
// =2 -> -2 -1 0 // YV12: shift right -2 (=left 2, =*4) YV24: no change
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV));
effective_chromaSADscale -= chromaSADscale; // user parameter to have larger magnitude for chroma SAD
// effective effective_chromaSADscale can be -2..2.
// when chromaSADscale is zero (default), effective_chromaSADscale is 0..2


That would mean in my test the effective_chromaSADscale was 2 so chroma really was weighted twice as high as luma.

Also it may be not correct to count U and V channels 'equal' to Y channel in this summing

Yes I agree, notice how the range of typical values in U and V channels is much narrower than the luma range, this will make the SAD of U and V effectively smaller. This may be the real reason why chroma needs to be scaled higher - at least when the chroma channels are good enough quality and their SAD values are reliable and can help finding the matching blocks.

https://i.postimg.cc/YSHpxTcC/luma-chroma-histogram.png


So current default 4:2 for YV12 still make chroma weight about 1/4 of luma or less. And increasing of scaleCSAD to 2 only make weights of SAD_Y and sum of SAD_U + SAD_V about equal.

Just wondering about this, so in order to get maximum possible chroma weight the input should be converted to YV24? Perhaps the ScaleCSAD should work so that it's applied on top of the internal autoscaling (4:2 for YV12, 1:1 for YV24 etc), then the chroma scale would behave the same no matter which input format is chosen.

There's of course the possibility to pre-process the clip so that saturation is increased, this will then give additional weight to chroma SAD values.

[DTL]

"my test clip in the MFlowFPS tests was converted to YV24 before it was processed with MVTools so it doesn't apply there."

If your YV24 is not natural RGB-full-band sourced but upconverted from typical distribution YV12 - it 'half-banded full-sized' chroma. It still have 'noise level' lower in compare with true full-band YV24. Though its SAD_UV from full-size UV may be bigger in compare with YV12 UV blocks.

"is this basically the chroma scaling algorithm currently used in MVTools?"

Yes - it is from latest 2.7.45 release. It is 'chroma weight selection in addition operation of saduv to total output SAD'. Yes - it may directly mean lumaSAD is _never_ any 'autoscaled/normalized' (with any scaleCSAD setting and any pixel format). So any addition of chroma can only increase total output SAD.

"in order to get maximum possible chroma weight the input should be converted to YV24? "

As I see with YV24 the nLogXRatioUV/nLogYRatioUV both 0. So initial
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV)); = 2
and even providing scaleCSAD = 2 will result
effective_chromaSADscale -= chromaSADscale; of only 0
and 0 result with passing SAD_U+SAD_V to output sum unchanged.
Translating eff_sclale to actual scale function is in https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/def.h#L123 . So it looks for YV24 it is impossible with current max scaleCSAD value of 2 to get any amplification to SAD_U+SAD_V over 1.
It looks the internal functions are adjusted in such way that in YV12 and in YV24 scaleCSAD=2 will result in 'equal luma + sum on chromas' balance. And default scaleCSAD=0 mean in both YV24 and YV12 it is about 1:1/4 (it marked as 4:2 as 4: (1+1) but each U and V is 1/4 in relation with 1 Y). Though UV in YV24 have 4x more samples and _may_ even in upconverted from YV12 'half-banded' state give bigger SAD value (not sure and not tested). The purpose of internal functions may be to output more or less equal SAD values from MAnalyse to MDegrain in any input format from YV24 to YV12 (no need to significantly re-adjust thSAD and other thresholds).

" the possibility to pre-process the clip so that saturation is increased, this will then give additional weight to chroma SAD values."

With high-saturated colours it will simply clip to min/max UV values and it will be close to useless. Only will help on low-saturated areas.

[zorr]

I knew your latest script version but it wasn't working correctly on my end
...
maybe you can sorta add the array trick you had in mind, ArrayAdd() is now internal

Yes I think I need to look into that. Hopefully ArrayAdd() is thread-safe. :)

What I aim to find is a real correlation between settings, one way would be to limit the dataset to the first quantile or maybe pick only the pareto front. But I'm not sure if comparing by pairs is enough, raising thSAD might imply a more dimensional behaviour where thSADC needs to be lowered, maybe same for thSADR and who knows what else (lambda, badSAD, etc), everything is on multivariate analysis.

I agree, definitely there are interactions between three or more parameters and that's what makes analyzing the results difficult.

I have thought about how to find the best values for any input clip and I think a good approach may be simply to collect lots of optimization results with different videos and then run an automatic test which tries each of them and reports the best one found.

The problem I see is that a simple AddGrainC() can't mimic real grain. You would need to use something more like GrainFactory3mod, and then encode it so DCT takes the grain to lower frequencies (DCTFlicker as Didée called it). It's an option sure. But I have reserves with synthetic grain, in filmed content grain is the actual image not a transparent layer over it, grain models the detail contours so I like to think of grain as a convolution layer.

Definitely if synthetic grain is used it should be as realistic as possible. It doesn't really matter if adding it to original clean clip takes a lot time, it can be "burned in" and then read directly as a source clip while the reference clip is the original. I think it's worth trying, it doesn't need to replicate real noise perfectly. Even if we had it perfect it would only be perfect for one clip so the algorithm will have to handle some variation in the noise profile anyway.

By the way in my random tests trymany gave better scores but was too slow so I defaulted to false.

The clip I used might be a special case as it has a very challenging, repetitive background which is bound to confuse the motion analysis.

[zorr]

"trymany, what does it do"

Thanks for that, I got the basic idea. :thanks:

This may be better for motion interpolation task but skips some best SAD values. So sometime setting all penalties to zero may give better denoising results with lower thSAD value. But if it cause to worse MVs search - it may cause error-blending distortions (?).

I think it has to do with the repetitive moving background in the test video. It's a difficult case on purpose and may need special settings compared to a typical case.

[zorr]

@zorr: True! how odd, truemotion arg doesn't override the truemotion settings

It says this on the manual page:

truemotion - This is a preset of these parameters values. It allows easy to switch default values of all "true motion" parameters at once. Set it true for true motion search (high vector coherence), set it false to search motion vectors with best SAD. Default is true since v1.4.10. In any case you can tune each parameter individually.

I take this to mean that whatever truemotion is set to, the parameter value (if set) will always take precedence.

I also thought about the correlation matrix, there's a way to know for certain how they relate each other, this is me thinking as I haven't done any further research. You have to lock the GMSD score. Yes, you can't so the procedure is to shoot a few thousand randomized iterations and pick the first three blocks with same and best scores. That is pick 5 rows with score 95.0 +- 0.001 (ie out of 100.0), pick 7 rows with score 86.0, and 5 (or whatever matched) rows of score 80. Do the correlation matrix of each one, fit a regression for the 3 correlation points to check if the relationship is linear or not (always relative to the GMSD score of course).

That sounds interesting. If you'd like to test this idea with a lot of data I can send you a result file with almost 700k results, that should be enough. :)

About the issue I have with your newer version, the script seems to be processed until the end then in the result file I get this error message:
Error getting frame property "_PlaneGMSD": property is not set

I was able to replicate the error (had to remove FluxSmooth and Luma_Rebuild though and use show=false instead of show=0 in GMSD, maybe we have different versions?). So the error happens because GMSD writes the frame props to the first clip in the argument list, and you want them in the second (the processed clip). So just switch the parameters around. But then there was another error:

RequestLinear: internal error (frame not cached)!

And that went away be commenting out RequestLinear(30). Perhaps there's a better way to handle it, because the processing took very long time at the last frame - it was probably recalculating all the frames which no longer were in the cache.

[zorr]

May be it also possible to get access in script environment to the output of MAnalyse data currently passed to MDegrain (and all other clients filters) and fetch x,y,SAD data for analysis. May be ask pinterf to patch Avisynth to allow some data fetching from this 'pseudoclip' ?
Or add some 'clip-source' converter filter to mvtools to 'convert' MAnalyse output to 'filter-readable/script-readable clip' inside AVS+ environment with Eval() (?) function. Like convert to virtual clip of BlckX (number of blocks per width) width and BlckY (number of blocks per height) height clip of RGB48 format and you can get x,y,SAD triplets as R,G,B 16bit sample components. RGB96 looks not supported as pixel_fmt in current AVS+ to read full 32bit signed values - but >16bit SAD is mostly possible only for HBD and large block sizes. Typical 'good' SAD values are about thSAD values and good fit in 16bit signed.

I was thinking the same and actually asked Pinterf (https://forum.doom9.org/showthread.php?p=1946741#post1946741) in the MVTools thread. No response though, perhaps he was too busy to even think about it. :)

This would be a great addition and would make testing new motion vector algorithm ideas much easier.

[zorr]

As I see with YV24 the nLogXRatioUV/nLogYRatioUV both 0. So initial
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV)); = 2
and even providing scaleCSAD = 2 will result
effective_chromaSADscale -= chromaSADscale; of only 0

That's not what I got. The value of chromaSADscale for YV24 is 0 when ScaleCSAD is 2 so effective_chromaSADscale -= chromaSADscale; is 2-0 = 2.

With high-saturated colours it will simply clip to min/max UV values and it will be close to useless. Only will help on low-saturated areas.

If you look at the range of U and V in the histogram you can see that it really is low-saturated, in this clip the chroma could easily be scaled by four and it still would not get clipped (and that's in addition to the ScaleCSAD multiplier 2). In my experience this is typical - chroma values are using much smaller range than luma. But of course you have to be careful and check before you do the upscaling.

But it would be better to allow larger ScaleCSAD values in MVTools at least in order to do some experiments - it could be a very simple and cheap way to get better motion vectors. Scaling the chroma by adjusting saturation can be used as a workaround to see if it really helps.

[DTL]

"=2"

eff_chroma is not final multiplier - it is bitshift control value. It looks you not look into source: https://github.com/pinterf/mvtools/blob/mvtools-pfmod/Sources/def.h#L123

// effective scale: 1 -> div 2
// 2 -> div 4 (YV24 default)
// -2 -> *4
// -1 -> *2
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);


To have sum of SAD_U + SAD_V to be multiplied by 2 you need eff_chroma control value of 1 (bitshift 1 to the left).

The comment also shows: YV24 scaleCSAD=0 is converted to eff_scale = 2 that mean (sum_of_sad_U+V) /4 . Even increasing user-control param scaleCSAD to 2 only can lower eff_scale to 0 that mean passing sum_of_sad_U+V unchanged (= multiplier 1).

[Boulder]

It says this on the manual page:

truemotion - This is a preset of these parameters values. It allows easy to switch default values of all "true motion" parameters at once. Set it true for true motion search (high vector coherence), set it false to search motion vectors with best SAD. Default is true since v1.4.10. In any case you can tune each parameter individually.

I take this to mean that whatever truemotion is set to, the parameter value (if set) will always take precedence.


Truemotion=true works like a preset in an encoder - it sets up a bunch of predefined values which you can then choose to override if necessary.

In fact, it might not be a bad idea to have a real preset functionality if something generic is found with all these tests.

[zorr]

eff_chroma is not final multiplier - it is bitshift control value. It looks you not look into source: https://github.com/pinterf/mvtools/blob/mvtools-pfmod/Sources/def.h#L123

Oh right, I did look at the code but I wasn't paying attention. Also I had a misunderstanding what the value of chromaSADscale is. It's just the ScaleCSAD parameter (defaults to 0 if not given in script).


// effective scale: 1 -> div 2
// 2 -> div 4 (YV24 default)
// -2 -> *4
// -1 -> *2
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);


To have sum of SAD_U + SAD_V to be multiplied by 2 you need eff_chroma control value of 1 (bitshift 1 to the left).

I think you mean control value of -1. :)

The comment also shows: YV24 scaleCSAD=0 is converted to eff_scale = 2 that mean (sum_of_sad_U+V) /4 . Even increasing user-control param scaleCSAD to 2 only can lower eff_scale to 0 that mean passing sum_of_sad_U+V unchanged (= multiplier 1).

Yes that's true. The algoritm only ever allows you to get 4 times the chroma weight of YV12 and in YV24 that means the chroma doesn't get any scaling as it already has 4 times the weight of YV12 chroma.

In the documentation the chroma weight of YV24 is written as 4:8 implying that chroma has double the weight compared to luma. That is justified by considering that the chroma has two channels which both contribute to the final SAD value. In the source code it's clarified: 4 : (4+4) = 4:8.

I just may have to run some chroma weighting tests to see if the results can indeed be improved by scaling the chroma even further. Or perhaps the scaling should be more fine-tuned like you suggested and the best scaling factor lies somewhere from 0.5 to 1.0 for YV24.

[zorr]

First chroma scaling results are in. I did a quick test to see if keeping all the previous optimal parameters of these tests (https://forum.doom9.org/showthread.php?p=1945684#post1945684) and only scaling the chroma using Tweak(sat=chromaScale/100.0, coring=false) would give better results. The scaling range was 50..200 -> from 0.5 to 2.0. There are only 151 different values so they can all be tested with exhaustive algorithm.

https://i.postimg.cc/zGqCGzt8/halos-ghosts-judder8b-blocksize8-2022-01-25-01-22-05-optimize-exhaustive-run-01.png

Looks like the parameters were so tightly optimized for the original chroma that any changes to it result in clearly worse score.

The next step is to try to run the optimization with more unlocked parameters to see if that allows finding a better chroma scale and better GMSD score.

[DTL]

But it would be better to allow larger ScaleCSAD values in MVTools at least in order to do some experiments - it could be a very simple and cheap way to get better motion vectors. Scaling the chroma by adjusting saturation can be used as a workaround to see if it really helps.

Here is new build with scaleCSADfine added new parameter - https://drive.google.com/file/d/1OAqXVBvrX3a6KVbY6nH5-TJQrED2sli9/view?usp=sharing . It is float muliplier with currently unlimited range. Now both scaleCSAD and scaleCSADfine works at once.

The tests shows that if adjusting scaleCSAD it is required to adjust thSAD in MDegrain at about same proportion (with >1 scaling). Such changing scaleCSAD from 0 to 2 with YV12 input require change of thSAD from 175 to 350 to keep about the same 'degrain level'. The requirement to adjust >1 parameter at once makes analysis much harder/longer. It is now require to make many encodings with different luma/chroma SAD ratio and adjust thSAD to get about the same MPEG output speed for each tested ratio and check for the quality of fine textures.

Default internal value for scaleCSADfine is 1.0 so it do not change old processing.

[DTL]

New update with many new params to optimize: https://drive.google.com/file/d/1BUP0sQ3zRinr0tVKfIumPUqLK6r5GTCF/view?usp=sharing
Based on commit - https://github.com/DTL2020/mvtools/commit/ee38893b26054dde0369ae2e142e76d1c2b0e6bc
(Windows-7 compatible build. No DX12 features.)

MDegrainN new added params:
1. adjSADzeromv (1.0 - default, no op) - possible SAD multiplier for zero-move blocks (before thSAD processing for getting block's weighting value). Float value. Recommended values: 0.9..0.4-. Possible medium values 0.75..0.5. Allow to increase degraining at static areas.

Example: setting 0.5 result thSAD for zero move blocks (static) will be thSAD*2.

2. adjSADcohmv (1.0 - default, no op) - possible SAD multiplier for blocks in coherent moving areas (before thSAD processing). Float value. Recommended values: 0.9..0.4-. Possible medium values 0.75..0.5. Allow to increase degraining at big enough coherent moving areas of much larger 1 block_size size (like camera pan movement over non-changing scene).

3. thCohMV (-1 default, no op) - threshold to detect if block's move vector is equal to surround blocks (top,left,right, down) move vectors. -1 - disables this part of processing (faster), 0 - lowest working value. Recommended values 0..4. Possible range - 0..unlimited int. Too high values will create error-blended blocks (like with too high thSAD value).

Example: setting to 4 allow SAD_of_MVs of 4 surrounding blocks with current block (x_cur, y_cur) to be up to 4.


int iabs_dc_x = SADABS(v_upper.x - x_cur) + SADABS(v_left.x - x_cur) + SADABS(v_right.x - x_cur) + SADABS(v_lower.x - x_cur);
int iabs_dc_y = SADABS(v_upper.y - y_cur) + SADABS(v_left.y - y_cur) + SADABS(v_right.y - y_cur) + SADABS(v_lower.y - y_cur);

if ((iabs_dc_x + iabs_dc_y) <= ithCohMV)
{
block_sad = (sad_t)((float)block_sad * fadjSADcohmv);
}
where int SADABS(int x) { return (x < 0) ? -x : x; }


Setting adjSADzeromv = 1.0 and adjSADcohmv = 1.0 disables new processing (faster, compatible with old scripts).

Still only C-version (no SIMD acceleration) so may be visibly slower if enabed new added processing.

Typical my processing settings now (for Full HD footage from 3chip video cam ENG/EFP type):
YV12 format:
MAnalyse(old args..., scaleSCAD=2)
MDegrainN(old args..., thSAD=250, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=4)

Using non-default new arguments assumed to allow much lower 'general' thSAD control value (about half of typical old) to keep more details on arbitrary moving objects (and skin details) while keeping noise and static areas low for better MPEG compression and looking. Possible target content type: low movement TV series (culture, nature). May visibly fail if many moving and changing its view at move time objects. For high-move footages with complex scene change recommended to disable adjSADcohmv (1.0 - default or keep thCohMV to default -1) or not set adjSADcohmv too low.

Task for auto-optimization: found sensitivity to new arguments and typical good/best values.

[zorr]

New update with many new params to optimize
...
MDegrainN new added params:
1. adjSADzeromv (1.0 - default, no op)
2. adjSADcohmv (1.0 - default, no op)
3. thCohMV (-1 default, no op)
...
Task for auto-optimization: found sensitivity to new arguments and typical good/best values.

I haven't done MDegrain tests yet, would have to come up with a good noisy / clean reference video pair. Dogway had MDegrain tests under way, perhaps he can test the new options as well.

By focusing on MFlowFPS / MBlockFPS it's possible to find good parameters for MAnalyse which will then benefit pretty much all MVTools operations including MDegrain.

Doing MDegrain properly would mean calculating optimal MAnalyse parameters for many different frame deltas, I think there may be some variation depending on the delta.

[DTL]

"there may be some variation depending on the delta."

It may be additional big field of experiments: currently only thSAD in only MDegrainN can be interpolated between 2 points - start and end in tr-length. But may be other params is also good to make interpolated. The new added adjustments for thSAD internal processing too. Currently MDegrainN have about 12 params to adjust in smooth gradation - already large enough set for manual testing the best combo.
Most hard to adjust are dependent like thSAD/adjSADzeromv/adjSADcohmv.

Also it is good to make the optimization target like 'minimum output x264-encoded filesize (with fixed set of params for x264)'. To select some balance between image quality and final encoded filesize. Typically the best image quality may not be equal to minimum encoded filesize.

[zorr]

I was able to replicate the error (had to remove FluxSmooth and Luma_Rebuild though and use show=false instead of show=0 in GMSD, maybe we have different versions?). So the error happens because GMSD writes the frame props to the first clip in the argument list, and you want them in the second (the processed clip). So just switch the parameters around. But then there was another error:

RequestLinear: internal error (frame not cached)!

And that went away be commenting out RequestLinear(30). Perhaps there's a better way to handle it, because the processing took very long time at the last frame - it was probably recalculating all the frames which no longer were in the cache.

I now have a better solution, it works reliably and is faster and makes the script simpler.

It requires a new version of Zopti which is ready to go but it's getting late here so I'll do the release tomorrow along with more detailed explanation.

[zorr]

This release introduces a better way for Avisynth scripts to write the results and fixes a couple of bugs. I recommend the upgrade especially if you're using more than one thread, there was a rare but possible bug which could result in flawed results.


supports a new way for an Avisynth script to write the result file
-first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
-then come the per frame result lines in any order, using the same format as before
-do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
-this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
-old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
-new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value

Download link updated at first post.

Here's an example of the new way to write results, based on Dogway's script from here (https://forum.doom9.org/showthread.php?p=1961928#post1961928). Only showing the part which does the writing.

resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write the number of frames to tell the optimizer how many result lines to expect
WriteFileStart(resultFile, "FrameCount()")

GMSD(src.ConvertBits(32, fulls=false, fulld=true), ConvertBits(32, fulls=false, fulld=true), show=false)

global GMSD = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
""",local=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (GMSD, time)
global delimiter = "; "
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")


This new method to write results has been completely reliable in my tests, I have been running slightly modified script by Dogway for a couple of days using 12 threads and have over 26000 results so far and no problems. The same script was very slow by reading the GMSD frameProps at the last frame causing many frames to be evaluated twice and also unreliable using the original method because the order of lines written cannot be guaranteed.

[Dogway]

Thanks for the update! Eager to test. I was a bit let down with the idea of having to run thousands of runs for the correlation test so took an hiatus.
This was my accumulated runs (random or not). Pitty that the interesting part of the convex hull didn't get sampled. I merged all the result files by hand though since they were independent.
https://i.imgur.com/0HY3EkUl.png

This was my correlation coefficients with the above sample data with the method I explained earlier. The correlation median of similar GMSD (high) score blocks. Greyed out are not reliable enough due to few data samples, as they need to be totally random otherwise you get a bias.
http://i.imgur.com/TI6Y73ch.png (https://i.imgur.com/TI6Y73c.png)

With values over +-0.6 you can satisfy linear correlation y = ax+b, you can fill that into "min:" and "max:" filter with some bias offset (ie. min: 1.3 * lambda + 100 max: 1.3 * lambda + 500)

[zorr]

Boulder, I might as well report some findings so far with your MDegrain script as I have plenty of data on it. Some changes I made to the script:


I used FFVideoSource instead of DGSource. I tried with DGSource and noticed it reports errors (popup windows) when used with a large number of threads. I will investigate this more later.
removed the preprocessing filters (ex_FluxSmoothST and ex_Luma_Rebuild) because my Avisynth version didn't support reading frameProps outside ScriptClip (I could try adding those later though).
the clip brazil-ref you shared probably already had the trimming baked into it because the clips were out of sync with the trim lines so I removed them and verified that the clips are in sync after that
I removed the optimization of truemotion and truemotionR and set them to false as their value has no effect on the result
used the new method to write results which was introduced in Zopti v1.2.2


[EDIT] Removed visualizations of bad data

I'm using dynamic iteration count with -dyniters 360 and -dynphases 10, with these setting my earlier MFlowFPS script took about 100000 iterations to finish. It's currently altering between phase 0,1 and 0,3 so it's quite early but we can already see some trends.

The -vismode heatmap is also useful when trying to determine if the search ranges are large enough (or too large).

[EDIT] Removed conclusions based on bad data.

Also looks like we have different GMSD scores for some reason, perhaps something to do with the trimming.
[EDIT] Oh boy, just noticed you have another set of trims further down which balances the first ones. So I have been calculating almost meaningless results. :o

[Dogway]

Haha, yes no issues. It happened to me as well after a few days.

Some observations, prefiltering is a MUST with such noisy sources. Maybe you can use vanilla MinBlur(2).
This is an updated recommendation from my old SMDegrain examples, you need Zs_RF_Shared.avsi which is nice for old AVS+ versions:

pre=fluxsmootht(2).MinBlur(2).Dither_Luma_Rebuild(3)

If MinBlur() is utterly slow (it uses medianblur()) use MinBlur(1).
Luma_Rebuild is also very recommended specially for movies. There are some movies like comedies that are very bright, but generally some brightneing of the low range is recommended, this also expands the luma/chroma range to PC levels. With this I think we are mostly on the same page.

Another thing is that searchRange is bound to searchAlgo, it means different things for different algos, therefore I normally run random iterations to know which range suits which algo. I had some conclusions after much testing, at least for MDegrain and this sample. In all cases, all pareto front (and most of my higher random results) was composed of the next values so they can be locked. searchRange is a bugger, I actually want to set it to 2 because higher than that smears shading but GMSD keeps scoring high for higher values.

BlkSize = 16
BlkSizeR = 8
overlapR = 4
pel = 1 # 1 for HD
sharp = 2 # optimize sharp = _n_ | 2 | sharp
lvl = 1
rfilter = 3
scaleCSAD = 2
trymany = false
truemotion = false
truemotionR = false

dct = 0 # optimize dct = _n_ | 0 | dct
dctre = 9 # optimize dctre = _n_ | 9 | dctre
searchAlgo = 1 # optimize searchAlgo = _n_ | 1 | searchAlgo
searchAlgoR = 4 # optimize searchAlgoR = _n_ | 4 | searchAlgoR
searchRange = 13 # optimize searchRange = _n_ | 13 | searchRange


These can be optimized. These are very refined ranges so expand them 25% more
thSAD = 285 # optimize thSAD = _n_ | 235..285 ; filter:x 5 % 0 == | thSAD
thSADC = 200 # optimize thSADC = _n_ | 140..250 ; filter:x 10 % 0 == | thSADC
thSADR = 280 # optimize thSADR = _n_ | 150..300 ; filter:x 10 % 0 == | thSADR
overlap = 4 # optimize overlap = _n_ | 4,8 | overlap
temporal = true # optimize temporal = _n_ | true,false | temporal


# TRUEMOTION SETTINGS
lambda = 440 # optimize lambda = _n_ | 440 | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 1320 # optimize lambdaR = _n_ | 960..2000 ; min:lambda ; max:lambda 3 * ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114 # optimize pnew = _n_ | 114 | pnew
pnewR = 136 # optimize pnewR = _n_ | 100..250 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 100 # optimize pzero = _n_ | 100 | pzero
lsad = 6000 # optimize lsad = _n_ | 1000..8000 ; filter:x 100 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79 # optimize plevel = _n_ | 79 | plevel

# lambda is not used for global predictor
pglobal = 8 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2 # optimize badrange = _n_ | 0..50 ; filter:x 2 % 0 == | badrange
badSAD = 1350 # optimize badSAD = _n_ | 1100..2200 ; filter:x 50 % 0 == | badSAD
searchRangeR = 13 # optimize searchRangeR = _n_ | 2..18 | searchRangeR
searchRangeFinest = 17# optimize searchRangeFinest = _n_ | 9..20 | searchRangeFinest
sglobal = true # optimize sglobal = _n_ | true,false | sglobal

[zorr]

Some observations, prefiltering is a MUST with such noisy sources. Maybe you can use vanilla MinBlur(2).

I assume you have already optimized the prefiltering with Zopti? :) If not, that's something it could do, either separately or (more thoroughly) combined with the optimization of other parameters.

And if you already have a good prefiltering figured out it's not a problem if it's slow. You can "bake it in" by saving the prefiltered clip as lossless video (Lagarith, Huffyuv etc) and just load the pre-filtered clip in the script you want to optimize. I installed the latest Avisynth+ and already tested this, it saves about 13 seconds per iteration in your script (running a single thread).

I just started running your original MDegrain script again, this time with the prefiltering included, no stupid trim issues and even using the original DGSource (doesn't seem to have problems with the latest AviSynth).

This is an updated recommendation from my old SMDegrain examples, you need Zs_RF_Shared.avsi which is nice for old AVS+ versions:

pre=fluxsmootht(2).MinBlur(2).Dither_Luma_Rebuild(3)

So this works better than
pre=ConvertBits(16,fulls=false).ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)?

I had some conclusions after much testing, at least for MDegrain and this sample.

Thanks, let's see if I come to the same conclusions after running about 20k iterations...

[Dogway]

hehe No I haven't run prefiltering through Zopti, one at a time. But you need some "dumb" temporal prefiltering and fluxsmooth is very good for that, minblur is the same but spatially. By Didée's words you want to soften the edges (not to mention flat areas!) otherwise "you'll end up with pretty big SADs wherever there's an edge. Means, little to nothing will happen on edges". It's been time tested.

You said you can't run my prefiltering, so I assembled an equivalent using old filters included in Zs_RF_Shared.avsi. But now that you're using new AVS+, use the "ex_" prefiltering since they are optimized. If you want to test this area I have some defaults at SMDegrain(). It's dependant on source grain, I would ditch dfttest (slow) and ex_knlmeans (slow and not as good as BM3D).

Thanks for testing, do an integrity check/run before running so nothing is mangled.

[kedautinh12]

I don't know why but in TemporalDegrain2 postFFT KNLMEANSCL faster than BM3D

[zorr]

Thanks for testing, do an integrity check/run before running so nothing is mangled.

It's looking good, very similar GMSD scores compared to your data. Also checked the clips are in sync with Interleave(), this time returning them at the end of the script and not at the start. :)

https://i.postimg.cc/qMKm38fD/mdegrain-fast.png

I will post some observations next.

[Dogway]

Thank you, very beautiful curve. Grabbing only the data samples of the inflection area can be useful for some study as it's where most of the things are going on. Not sure how to isolate those.
When I run my tests ex_MinBlur() was some kind of broken for HBD, so you might have more accurate results.

[zorr]

All right, let's see some visualizations again with correct data. I will remove the previous shots from a couple of posts up so nobody gets wrong information.

So there are still some parameters which have their best values at or very near the edge of the search range which could be extended. It's easy to see these by -vismode line.

https://i.postimg.cc/TwtTXz2S/lambda.png
Lambda is at the minimum allowed and the trend points to better results with lower lambda still.

https://i.postimg.cc/vTDb80cm/lsad.png
lsad is not at the very edge but there's a similar (although noisy) trend as with lambda.

https://i.postimg.cc/8knTXg0f/pnew.png
Looks like pnew is at the top of a hill, but might be worthwile to check if there's really a downward slope with smaller values.

https://i.postimg.cc/nLZx9FCg/pnewR.png
pnewR is quite noisy but there's a clear trend, smaller values should be tested.

https://i.postimg.cc/5NYbZKh7/pzero.png
There's no slope near pzero but there are almost as good results to the right. It might help the other parameters to find new optimal values if we allow more breathing room for pzero.

https://i.postimg.cc/wTCgHcH6/search-Range-Finest.png
searchRangeFinest is quite jumpy, we could try our luck and extend the range a bit to the left, there should be faster combinations at least.

https://i.postimg.cc/fRYZbtTY/thSAD.png
thSAD is also almost the edge, could try to extend the range upwards and get rid of the smallest values.

I will let this run continue a bit further and then restart with the extended or moved ranges.

[Dogway]

The results align well to my pareto front.
http://i.imgur.com/BU9V8muh.png (https://i.imgur.com/BU9V8mu.png)
If I'm correct the blue line tells how many results contain said value right?

These graphs are good to find a pareto front for a certain task, like MDegrain in this case with this clip.
But to better understand the correlation between settings we need a correlation study (without bias)

[zorr]

If I'm correct the blue line tells how many results contain said value right?

Yes. The axis on the right displays the scale for the number of results.

These graphs are good to find a pareto front for a certain task, like MDegrain in this case with this clip.

You don't need these graphs for the pareto front, Zopti simply lists it whenever you use the -mode evaluate. :) But the graphs certainly will help in other ways.

But to better understand the correlation between settings we need a correlation study (without bias)

Yes that could certainly be useful. Calculating correlations is not very hard, it could be built into Zopti. I think Principal Component Analysis (PCA) could also be useful.

[Dogway]

Sure, everything has to do with the covariance between the 2 sets. Have never worked with PCA though. Will start my stats studies in March, by the way bought your recommended book (physical), thanks for the suggestion!

[zorr]

by the way bought your recommended book (physical), thanks for the suggestion!

Where's my commission! (just kidding) :)

Decided to stop the run and made some adjustments to the search ranges. After 55k iterations it was still going back and forth between phases 0.0 and 0.3 (and a short visit to 0.4). Looks like there's some action in the flat top part also.

https://i.postimg.cc/zBR8Xd0p/mdegrain-fast-2.png

I will start a new one but I will do a MCompensate test before that.

MCompensate is actually the most direct way to evaluate the quality of the motion vectors. So I think the optimization of MSuper, MAnalyse and MRecalculate could be done with MCompensate and then the same parameters should work with MDegrain, MFlowFPS etc where the quality of motion vectors is harder (or slower) to evaluate. This should of course be tested so after we have arrived with a robust set of parameters for MDegrain for this source clip we could try to reproduce them using MCompensate.

[Dogway]

I explored that route and I don't recommend it. I started with the same, comparing motion vectors from a HFR clip and a motion interpolated one and that doesn't translate well into MDegrain. With MDegrain you can get away with DCT>1 with motion interpolation probably not (too slow to read pareto). Also searchRange for MDegrain is not recommended (smearing) while with motion interpolation helps a lot.

I stepped back and started with the basic MDegrain, the only thing I care is parameter correlation. Once I get that I will expand into motion interpolation. Maybe in a few days if you haven't already I will try to do some runs with randomized iterations using new version 1.2.2 and do my correlation matrix. Another run to verify the correlation matrix. This is fine if for example you want to change searchAlgo in a filter then you want to default searchRange to a different number so the filter always gives out optimal values for all clips, as I'm currently making a SceneStats() filter to output that, scene stats and make automatic filtering decisions by these stats.


EDIT: I did some correlation studies only for the thSADs given the rest is equal (using my highest scored settings) and found thSADR is directly proportional until the critical point, that is, if you use too much thSAD, thSADR is also going to go wrong and best score is inversely proportional (-1.9x+682). Since we don't know the critical point and this is also subjective given the user desired amount of denoising I think a positive correlation is fine.
thSADR = round(exp(-101./(thSAD*0.83)) * 360)
Also studied thSADC (I needed to set chroma=true, plane=4, and in GMSD V=true):
thSADC = round(thSAD * 0.755 * (0.25*exp(scaleCSAD*0.693)))

For this example my results were (GMSD score of 146.09013):
thSAD = 245
thSADR= 219
thSADC= 185
It might vary depending on other parameters but I would consider these still first class parameters where all the rest should conform to.

By the way a question, is it possible to tell sensitivity to only care for score and not score * speed?

[zorr]

I explored that route and I don't recommend it. I started with the same, comparing motion vectors from a HFR clip and a motion interpolated one

So you had a clip with high frame rate and tried to interpolate the missing frames from a lower frame rate clip? In that case what I'm suggesting is different.

MCompensate can be used to take for example frame n+1 and using motion vectors to rearrange it into frame n. If the vectors are perfect then GMSD(n, MCompensate(n+1)) will be zero. Also the parameters of MCompensate have nothing to optimize (recursion is the only one that could be experimented with but it would only benefit MCompensate so it should be at default value 0). This means the results can directly tell the quality of the motion vectors.

and that doesn't translate well into MDegrain. With MDegrain you can get away with DCT>1 with motion interpolation probably not (too slow to read pareto). Also searchRange for MDegrain is not recommended (smearing) while with motion interpolation helps a lot.

If what you're saying is true then MDegrain does not want as good motion vectors as possible for the best results, it wants something else. It would be interesting to know what exactly it is and why.

My mental image of how MDegrain works is that it creates those new frames using the same process as MCompensate and takes averages of pixels in temporal dimension (or ignores them if SAD is too bad). It also has the thSAD2, thSADC2 parameters to give the closest frames more weight in the average calculation. It should be possible to recreate the exact functionality using MCompensate and Average(). Or better because each motion vector delta could use different set of parameters for MAnalyze etc.

I stepped back and started with the basic MDegrain, the only thing I care is parameter correlation. Once I get that I will expand into motion interpolation. Maybe in a few days if you haven't already I will try to do some runs with randomized iterations using new version 1.2.2 and do my correlation matrix. Another run to verify the correlation matrix. This is fine if for example you want to change searchAlgo in a filter then you want to default searchRange to a different number so the filter always gives out optimal values for all clips, as I'm currently making a SceneStats() filter to output that, scene stats and make automatic filtering decisions by these stats.

I can help you with the correlation matrix by calculating you more data. Since the tests require purely random data perhaps I could add an option for that in Zopti. Of course since you already have set specific ranges for the parameters it's not completely random but assuming those ranges cover all sensible use cases it should work. And it would be too difficult to find any real correlations with full ranges anyway.

EDIT: I did some correlation studies only for the thSADs given the rest is equal (using my highest scored settings)
...
It might vary depending on other parameters

Just be careful not to make too strong conclusions. Some parameters like the search algorithm can have big effects on the optimal range of other parameters. But since you have the option to choose which algorithms to allow in your functions you could make correlations that work for those.

By the way a question, is it possible to tell sensitivity to only care for score and not score * speed?

I'm not sure I follow, the sensitivity only cares for the first optimized parameter, which usually is the quality. If you want to use the sensitivity calculations perhaps I should give an option to limit how many results to include, right now it's the last two generations that are used. However it would take too long to calculate the sensitivity from much larger result set because it's O(N^2) algorithm.

[DTL]

If the vectors are perfect then GMSD(n, MCompensate(n+1)) will be zero. "

It is unlikely for natural scenes. The 'vector' compensation only cover more or less good Translate transform of possible basic Scale/Rotate/Translate transforms of object. Also object (block) may change its looking because of other reasons (like simply change of incident light colour (3 components also - hue/sat/intensity)). So in current mvtools the possible compensation is very limited. In the better new versions it is possible to add more at least basic transforms to search and compensate - scale and rotate (at least 1 of 3 axises rotate). So it is not possible to expect ideal result from the still very limited tool.

In addition to 'basic' transforms may be other like Skew/Perspective

https://desktop.arcgis.com/en/arcmap/10.3/tools/coverage-toolbox/GUID-D81DC3FC-6A37-4C30-BBD3-C9B23F874ECE-web.gif

I hope in some future there will be more advanced version of mvtools with search/measure/compensate for more number of transforms. But it is very computational expensive so speed may be 100 or 10000 times slower - not very practical. It is a task for the current high speed massive multithreaded compute accelerators and future (if we will see it in current dying civilization).
Also the multi-transforms search is not simply linear sequence of several transforms search - it is multi-pass refining and even slower.

It still may be some research which of the other transforms are the most valueable with natural scenes and is worth to put developers resources to implement next in addition to current Translate-transform only in mvtools. I think it may be Rotate transform.
Also the Translate transform may be treated as special case of Rotate transform with very low angle and very large radius (distance to origin). It looks the Translate transform was most useful for MPEG coders and enough cheap in computing at old hardware so it become first and main transform to compensate. But it still not cover all natural transform cases.

So the full transform structure of block to compensate will be much more complex like:
1. Translate MV (dx,dy)
2. Rotate angle(s) (rz,(rx,ry)), +rotate origin x,y ?
3. Scale (sx,sy)
4. Skew (..x,..y)
5. Perspective (kx,ky)
6. Lighting (YUV/HSB/...)
...

" MDegrain works is that it creates those new frames using the same process as MCompensate and takes averages of pixels in temporal dimension (or ignores them if SAD is too bad). It also has the thSAD2, thSADC2 parameters to give the closest frames more weight in the average calculation. It should be possible to recreate the exact functionality using MCompensate and Average(). "

MDegrain works on per-block basis. For each block of current frame it fetches block of ref frame and shift (translates) to align with current block and check for SAD - it SAD below thSAD - it calculate non-zero averaging weight and use this block in averaging process (actually all ref blocks are used but blocks with zero weight add nothing to output result - it is to make processing a bit simple, so processing speed do not depends on number of zero-weighted blocks). But with overlap enabled there is more blocks shifted a bit (more than simple (frame_width/block_sizeH)x(frame_height/block_sizeV)). So I not sure if MCompensate may give exact result.

The real sad truth about current mvtools-mdegrain: it currently mostly tool to detect static and somehow moving blocks and sometime if moving block is mostly Translate-transformed only - it can also add it to averaging and degrain. But if moved (transformed) block is somehow other way transformed and still have SAD below thSAD - it also averaged but processing cause blurring because the transform is not good (totally/best) reversed (compensated).

[Dogway]

I did two different tests (for optical flow purposes both). One simply output the SAD mask to give a SAD score (script here https://pastebin.com/4QTLzrVy), and second took an HFR clip from Youtube, tonemaped it, downscaled it to 1080p. Then compared the original frames from the newly interpolated ones. The first test didn't translate well for MDegrain and not much either for MFlowFPS() so I think the second test is more suited.

For MFlowFPS() for example you want higher search ranges because the primary issue with it is high motion objects so you want to extend the search range. With MDegrain() I observed that anything higher than searchRange=2 smears shading a lot specially with truemotion=false, so I'm trying to find a sweet spot with truemotion=true settings. Yes you can use low frequency restoration (as seen in SMDegrain) but that will also restore DCTFlicker. With MCompensate() you still need the higher searchRange to do the motion compensation and probably DCT=1 will also be worth it there where for MDegrain it doesn't make such a visible change to be worth the performance hit, let's say MDegrain is more indulgent with the motion vectors.
So to sum it up:
Motion Interpolation clients: high fidelity vectors + high search range (blending okeish)
MDegrain: medium fidelity vectors + low search range (blending or smearing not bad)

By the way I'm refining by chunks, simply to get some correlations but I actually managed to raise the score to 146.1026

Check with this as a starting template:
thSAD = 245
thSADR = 219
thSADC = 185
BlkSize = 16
BlkSizeR = 8
overlap = 8
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = true
truemotionR = true
temporal = true

# TRUEMOTION SETTINGS
lambda = 310
lambdaR = 960
pnew = 114
pnewR = 136

pzero = 134
lsad = 6000
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79
lvl = 1

# lambda is not used for global predictor
pglobal = 8

badrange = 2
badSAD = 1350
dct = 0
dctre = 9
rfilter = 3
searchAlgo = 1
searchAlgoR = 4
searchRange = 8
searchRangeR = 11
searchRangeFinest = 12
sglobal = true


I can help you with the correlation matrix by calculating you more data. Since the tests require purely random data perhaps I could add an option for that in Zopti. Of course since you already have set specific ranges for the parameters it's not completely random but assuming those ranges cover all sensible use cases it should work. And it would be too difficult to find any real correlations with full ranges anyway.

Thanks. Currently I'm running these small tests to release new SMDegrain but later I want to do the full correlation map. It only makes sense if the score is in the highs, so all data samples below a given threshold (ie. if a known setting hampers scores in all cases) are discarded. Then I divide data samples by same score blocks so the correlation can be done at several points for more certainty, and finally do the median (I like the interquantile mean though). I'm not going to say it's a perfect solution but it's a start, after that you only know how much linearity correlation there is, but not the linear factor so I will try to do some regressions as I did above with thSAD. For me thSAD is a first class setting so thSADR and thSADC are derived from it, other first class settings include searchAlgo, searchrange and DCT. I'm trying to look for more of this kind, probably also lambda, pnew and pzero. Don't forget to lock thSADC (my bad) as GMSD score only uses Y plane.

I'm not sure I follow, the sensitivity only cares for the first optimized parameter, which usually is the quality. If you want to use the sensitivity calculations perhaps I should give an option to limit how many results to include, right now it's the last two generations that are used. However it would take too long to calculate the sensitivity from much larger result set because it's O(N^2) algorithm.

So currently it only optimizes the score? I thought it was also looking at the time to find the best score/time relation (pareto front)

[DTL]

[It looks 'pelsearch' is control radius at level=0, edited]

Also if using sub-pel processing modes it may be good to compare different interpolation algoriphms. Built-it 0,1,2 not looks as best. The current versions allow to provide external pelclip with any other upsizing algoriphm.
Also the MSuper(sharp=2) is documented as 'for sharper Wiener interpolation (6 tap, similar to Lanczos)' - but its kernel not looks like sinc-based (natural Lanczos kernel):
https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/Interpolation.cpp#L2094

pDst[i] = std::min(max_pixel_value, std::max(0, ((pSrc[i - 2]) + (-(pSrc[i - 1]) + (pSrc[i] << 2)
+ (pSrc[i + 1] << 2) - (pSrc[i + 2])) * 5 + (pSrc[i + 3]) + 16) >> 5));

So in the current development build I will try other sinc-based kernel sub-pel shifting at processing time. The users of old builds may try to provide 'pelclip' externally resized. But the best interpolation algoriphm may greatly depends on the 'conditioning' of source to process. With 'badly conditioned' sources sinc-based may give too much ringing and worse results. It is one of the sad place when going to the best quality sub-pel transformations refining/compensation. Most of other transforms compensating like Rotate is mostly based on sub-pel samples recalculating and even can not have any 'full-pel' search/compensate mode available at all. So the interaction of the sub-pel interpolation algoriphm with actual spectral distribution of the source footage will greatly change the performance of any sub-pel-based search/compensation degrainers.

[Dogway]

Thank you DTL, this might explain why fine details get warped in high motion right? I wonder what are the alternatives by looking into other optical flow solutions, some search should always be performed at finest level. In the code 'searchparam' is clamped to 'pelsearch' at finest level, maybe that's why 'pelsearch' optimizes to a high number like 17. I'm curious to know if then at finest level 'pelsearch' is no-op.

For resizing lately I'm digging blackman resize, it's like lanczos but with less ringing so it's ok to use 6 or 8 taps and get a sharp output with moderate ringing. I searched for blackmanminlobe but it's only a thing of fmtconv, couldn't find any literature about it. I have no idea how the internal Wiener kernel is supposed to look like, maybe it's worth providing a pelclip in this case.


EDIT: reading your previous post, it's interesting how the internals work, as always the simplicity of the concept stands out. One question, for many of those transformations shouldn't the block size be adaptive? Also as I see it it's practically a bunch of matrix operations, maybe this task can be accelerated by the GPU? I think with the implementation of scaling and horizontal transforms the motion estimation would greatly improve.

[DTL]

"why fine details get warped in high motion right?"

It depends on both motion speed and other reasons.

Currently if all previous levels fail to produce really best motion vector - the refining at finest level (after checking all possible predictors) is limited to 1 full-pel (distance between samples in input source). So it speed of movement > 1 sample per the time between frames it may fail to reach best position (at the last refining stage). Ofcourse it is expected the all previous stages (predictors + hierarchical refinement) will provide already best vector and no more refining is needed. But it may not every time be true.

Other reasons - the fast motion may cause also other transformations (like Skew because of rolling-shutter in some cameras and many more) that can not be compensated with current engine (using only 'solid-Translate' block compensating). And if thSAD is set too high - it will cause averaging/blending of not ideally all-transforms compensated blocks and blur of details.

Some idea to think about: Is it possible to separate SAD value increasing from non-ideal reverse-transform (transforms compensating) of blocks from natural noise SAD value increasing ? Currently engine can not determine if ref block is not perfectly match in shape/position (total geometric compensation is done) the src block and may perform failed blending. The single SAD-based decision of either put block in blending or not looks far from perfect for 'ideal denoiser'. May be other metric is possible to detect if 2 noised blocks are not good compensated so it is better to pass more noise and keep more details (skip badly compensated ref block from blending).

Currently single scalar SAD value control too much in the denoising engine and this cause fails sometime. Need to add more complex metrics/algoriphms.
In the latest testbuilds the testing of MVs length was added in algoriphm of selecting block to blending or not. Idea was: Typically static and very slow moving blocks are less distorted and simple Translate compensation can be enough (that equal to zero MV for full static block). Though if lighting changes it is also not best decision. Or other symmetrical transforms like Skew may give zero Translate MV value, or slow rotation around center, etc.

"maybe that's why 'pelsearch' optimizes to a high number like 17"

Oh - that is strange. The documentation also says: "and pelsearch is the radius parameter at finest (pel) level. " . And 'pelsearch' should be equal to 'pel' default. But at debugging I typically got searchparam =1 passed to PlaneofBlocks->SearchMVs at level=0 even if using searchparam=2 (default). Need to check it again. If pelsearch of 17 is passed to level=0 search even with pel=1 it will cause extremely great loss of speed.

Well - it looks the 'pelsearch' can really set very high search radius at level=0. But I never use it. So the first part of post https://forum.doom9.org/showthread.php?p=1963476#post1963476 looks invalid and I will delete it.

If large 'pelsearch' value really make better result - it mean the previous levels of hierarchical search significantly fails to provide good predictor for final search more or less frequently.

[zorr]

If the vectors are perfect then GMSD(n, MCompensate(n+1)) will be zero. "

It is unlikely for natural scenes.

Indeed. It's so unlikely that if the GMSD score 0.0 (perfect match) is returned it is assumed to be an error and the worst possible score is returned for that frame instead. It's not a theoretical situation either - at least some older versions of MVTools sometimes returned the reference frame instead of the MCompensated frame in rare error cases which resulted in that perfect score so I had to add this check. I haven't seen it happen in more recent versions of MVTools but it's better to be safe. So if you have wondered what this line
global GMSD = (GMSD == 0.0 ? 1.0 : GMSD)
is doing in the script, this is the reason.

The 'vector' compensation only cover more or less good Translate transform of possible basic Scale/Rotate/Translate transforms of object.
...
So it is not possible to expect ideal result from the still very limited tool.

I wasn't implying that perfect score would be expected, just that the GMSD score can tell a lot about the quality of the motion vectors. And especially it should be a good match for MDegrain.

Also the Translate transform may be treated as special case of Rotate transform with very low angle and very large radius (distance to origin).

Yes this is probably why the translate alone can be very successful. Most of the motion shown in video is so slow that curved motion paths can be quite accurately approximated with linear segments where transform is all you need.

[DTL]

" Most of the motion shown in video is so slow that curved motion paths can be quite accurately approximated with linear segments where transform is all you need. "

It may about work with very low tr values - like for sequential frames only (tr about 1). But for good degrain we need tr as large as possible and with increasing frame-distance rotation quickly cause great mismatching of ref and src blocks (SAD increases and far-tr blocks got zero weight and stops add to the output in good case or start to smooth outout in worse case). So increasing of tr value do not add to degraining any more - only waste time.
Currerntly MDegrain check current block vs all blocks in tr-scope. I calculate for block 8x8 and rotation around center - the qpel shift at the middle of edge starts with rotation angle about 3.6 degrees (arctg((1/4)/4)). It looks about 3..4 degrees is the lowest granularity for rotation angle for rotation search. For blocksize 16x16 it about 2 times lower - about 1.5 degrees.

Also rotation (around Z-axis) is not only about curved path - the curved path may be OK if block keeps it orientation relative to frame sampling grid.

It looks complex transforms quickly make increasing of tr-value useless for getting better degrain result. So best results with 'large' tr values typically got at the scenes with lots of static areas. Where the motion-search engine simply confirms the block is static or compensates the very slight residual movements of camera at tripod.

[zorr]

I did two different tests (for optical flow purposes both). One simply output the SAD mask to give a SAD score (script here https://pastebin.com/4QTLzrVy), and second took an HFR clip from Youtube, tonemaped it, downscaled it to 1080p. Then compared the original frames from the newly interpolated ones. The first test didn't translate well for MDegrain and not much either for MFlowFPS() so I think the second test is more suited.

I have also thought about using the SAD mask alone as the optimization criteria. But there's something weird about the output of MMask(kind=1). I think it should output one color brightness per block (the SAD of that block) but the output is not a low-resolution grid but a blurred one. Perhaps it's doing some kind of interpolation as well. DTL, can you take a look at it? :)

MCompensate should be a much better match for MDegrain. Neither of them do motion interpolation whereas MFlowFPS does. It's more difficult task to create good vectors for motion interpolation where the coherence of vectors is more important.

For MFlowFPS() for example you want higher search ranges because the primary issue with it is high motion objects so you want to extend the search range. With MDegrain() I observed that anything higher than searchRange=2 smears shading a lot specially with truemotion=false

The main difference is that while MDegrain can be picky and only accept high quality vectors (with low SAD value) MFlowFPS has no choice, it has to use the best motion vectors that were found. Therefore MFLowFPS needs more extreme settings to find those good enough vectors. Those hard-to-find vectors MFlowFPS needs may still be too low quality to be useful for MDegrain so it makes no sense to even try that hard, therefore lower search range works better.

Yes you can use low frequency restoration (as seen in SMDegrain) but that will also restore DCTFlicker.

Sorry I'm not familiar with the concept of low frequency restoration and DCTFlicker, can you elaborate those?

With MCompensate() you still need the higher searchRange to do the motion compensation and probably DCT=1 will also be worth it there where for MDegrain it doesn't make such a visible change to be worth the performance hit, let's say MDegrain is more indulgent with the motion vectors.

Yes agreed and I see your point why MCompensate may not be that useful for finding vectors for MDegrain. What should be done is have MCompensate also reject too high SAD blocks (with the thSAD parameter) but I'm not sure how to tell which thSAD value to use. Perhaps we could calculate optimal parameters for a few different thSAD values and SMDegrain could use those depending on which thSAD user has selected?

By the way I'm refining by chunks, simply to get some correlations but I actually managed to raise the score to 146.1026

So overlap was changed from 4 to 8, also pzero is outside the original range. That's why I like to start the optimization with all parameters unlocked and with quite large ranges... it does take longer to get results though.

Thanks. Currently I'm running these small tests to release new SMDegrain but later I want to do the full correlation map. It only makes sense if the score is in the highs, so all data samples below a given threshold (ie. if a known setting hampers scores in all cases) are discarded. Then I divide data samples by same score blocks so the correlation can be done at several points for more certainty, and finally do the median (I like the interquantile mean though). I'm not going to say it's a perfect solution but it's a start, after that you only know how much linearity correlation there is, but not the linear factor so I will try to do some regressions as I did above with thSAD. For me thSAD is a first class setting so thSADR and thSADC are derived from it, other first class settings include searchAlgo, searchrange and DCT. I'm trying to look for more of this kind, probably also lambda, pnew and pzero.

Ok so if you provide the script I can run some results and give you the output. Do you think it would help if Zopti had a setting where it only tries random settings (still making sure they pass the restrictions stated in the script, of course)? Right now you can do that by creating a large population but it's not optimal, for example the creation of new random settings is single threaded and I think the resolve phase should kick in immediately if it's known that it always takes over 100 tries to find random parameters which pass the restrictions. And I think doing the rest of your process in Zopti could also be possible, not sure how you take the median though.

Don't forget to lock thSADC (my bad) as GMSD score only uses Y plane.

Good catch, I'll disable it for the next run.

So currently it only optimizes the score? I thought it was also looking at the time to find the best score/time relation (pareto front)

Yes the pareto front is based on both quality and speed (or whichever properties you have defined in your script). However the sensitivity estimation (I think we were discussing that and not pareto) only cares about the first property, usually it's the quality. The sensitivity estimation is just used to select which parameters are mutated and you can turn that feature off if you want.

[zorr]

But for good degrain we need tr as large as possible and with increasing frame-distance rotation quickly cause great mismatching of ref and src blocks (SAD increases and far-tr blocks got zero weight and stops add to the output in good case or start to smooth outout in worse case). So increasing of tr value do not add to degraining any more - only waste time.

Hmm.. perhaps this could be tested on script level. Create several rotated and/or scaled versions of the frame and do motion interpolation between original and the rotated/scaled future/past frames. Then you'd need to select the best motion interpolated block from all of these variations based on the SAD (using MMask). Some blending would be needed in order to avoid harsh edges. And oh boy that would be slow and take tons of memory!

[Dogway]

https://i.imgur.com/iLn521l.png
This is the pruning I plan to do with the results. I only care where the action is taking place (the blue shaded area), but for correlation analysis I will only take the upper half (yellow shaded).

I will divide the yellow area in horizontal slices of same GMSD score and do the correlations.

If that area is full of say pnew=114 by refinement convergence I'm not going to take any stat from it (it's biased), so I need pnew to get there by randomness. If it happens to be all pnew=114 I would know then that it's a parameter that is first class and probably shouldn't be exposed (no quality/performance gain from it).

When all correlations and stats are done we test the results with another clip.
I will probably run the Spearman's correlation and mutual information, RDC or CoS. I read about PCA but it only describes the orthogonal vectors of the covariance ellipsoid, I'm not sure what to do with that. With high Spearman's correlations I will go one by one, plot it and fit a function.


Let me cook the script.

EDIT: The script. You might want to exclude 1 from DCT and DCTre since it will slow down things a lot, same for 3 for searchAlgo and searchAlgoR. Also starting iterations with script settings so we have a high score reference.

setmemorymax(16384/8)
DGSource("Brazil-cleanREF.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=AssumeFPS(24000,1001)
DGSource("Brazil-prefiltered.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
pre=AssumeFPS(24000,1001)
DGSource("brazil2.dgi",cl=0,ct=40,cr=0,cb=40)
AssumeFPS(24000,1001)


thSAD = 245 # optimize thSAD = _n_ | 150..300 ; filter:x 5 % 0 == | thSAD
thSADR = 219 # optimize thSADR = _n_ | 150..300 ; filter:x 5 % 0 == | thSADR
thSADC = 185
BlkSize = 16
BlkSizeR = 8
overlap = 8
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = true
truemotionR = true # optimize truemotionR = _n_ | false,true | truemotionR
temporal = true # optimize temporal = _n_ | true,false | temporal

# TRUEMOTION SETTINGS
lambda = 310 # optimize lambda = _n_ | 100..2000 ; filter:x 20 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 960 # optimize lambdaR = _n_ | 300..3000 ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114 # optimize pnew = _n_ | 20..200 ; filter:x 2 % 0 == | pnew
pnewR = 136 # optimize pnewR = _n_ | 20..200 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 134 # optimize pzero = _n_ | 0..200 ; filter:x 2 % 0 == | pzero
lsad = 6000 # optimize lsad = _n_ | 1000..8000 ; filter:x 50 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79 # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1 # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 8 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2 # optimize badrange = _n_ | 0..50 | badrange
badSAD = 1350 # optimize badSAD = _n_ | 0..3000 ; filter:x 10 % 0 == | badSAD
dct = 0 # optimize dct = _n_ | 0..10 | dct
dctre = 9 # optimize dctre = _n_ | 0..10 ; filter:x 1 != | dctre
rfilter = 3
searchAlgo = 1 # optimize searchAlgo = _n_ | 0..5 ; filter:x 3 != | searchAlgo
searchAlgoR = 4 # optimize searchAlgoR = _n_ | 0..5 ; filter:x 3 != | searchAlgoR
searchRange = 8 # optimize searchRange = _n_ | 0..20 ; filter: dct 0 > dct 5 < and x 5 < true ? | searchRange
searchRangeR = 11 # optimize searchRangeR = _n_ | 0..20 | searchRangeR
searchRangeFinest = 12# optimize searchRangeFinest = _n_ | 0..20 | searchRangeFinest
sglobal = true # optimize sglobal = _n_ | true,false | sglobal


trim(1,12)+\
trim(60,96)+\
trim(173,285-12)


C=ConvertBits(16,fulls=false)

Pre = pre #C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true

superfilt = MSuper(pre, hpad=16, vpad=16, sharp=sharp, rfilter=rfilter, pel=pel, mt=true) # all levels for MAnalyse
superR = MSuper(C, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=true, chroma=false)
superRe = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=true)
bak2 = MAnalyse(superfilt, isb=true, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 = MAnalyse(superfilt, isb=true, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 = MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 = MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe, bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
bak1 = Recalculate ? MRecalculate(superRe, bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
fwd1 = Recalculate ? MRecalculate(superRe, fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe, fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=true)



resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFileStart(resultFile, "FrameCount()")

GMSD(ConvertBits(32, fulls=false, fulld=true), src.ConvertBits(32, fulls=false, fulld=true), Y=true,U=false,V=false)

global GMSD = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
""",local=false)

global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

global delimiter = "; "
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")



Sorry I'm not familiar with the concept of low frequency restoration and DCTFlicker, can you elaborate those?

You apply a highpass on the smeared clip and add that to a lowpassed reference.
This seems good in principle but low frequency detail is not stable in the reference (grainy) clip because as per Didée's words DCT compression propagates grain detail into the low frequencies which he calls DCT flicker. A further DCTFlicker pass is necessary, you can do this as temporal denoising the lowpassed clip, use a debanding filter or a mix of both.

I made it into a function.
function ex_LFR (clip c, clip ref, int "LFR", int "UV") {

w = c.width ()
h = c.height()
LFR = Default( LFR, 300*(w/1920.)) # Default is 300 for 1080p
UV = Default( UV, 1)

LFR = max(LFR,50)
Fs = max(w,h) * 2 # Frequency sample rate is resolution * 2 (for Nyquist)
k = sqrt(log(2)/2) * LFR # Constant for -3dB
LFR = Fs / ( k * 2 * pi ) # Frequency Cutoff for Gaussian Sigma

ex_makeadddiff(c, c.ex_gaussianblur(LFR, UV=UV), ref.ex_gaussianblur(LFR,UV=UV), UV=UV) }



https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/Interpolation.cpp#L2094

pDst[i] = std::min(max_pixel_value, std::max(0, ((pSrc[i - 2]) + (-(pSrc[i - 1]) + (pSrc[i] << 2)
+ (pSrc[i + 1] << 2) - (pSrc[i + 2])) * 5 + (pSrc[i + 3]) + 16) >> 5));


Is it possible to convert that to fmtc_resample impulse coefficients to have a look?

[DTL]

In the same file it is commented as:
https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/Interpolation.cpp#L1879
// so called Wiener interpolation. (sharp, similar to Lanczos ?)
// invarint simplified, 6 taps. Weights: (1, -5, 20, 20, -5, 1)/32 - added by Fizick


Have more ideas how to keep fine low-contrast details from blur:
It may blur because 2-frames motion search engine may return noise-distorted motion vectors if we even have totally static frame. It can be checked with static frame clip + AddGrain and look at motion vectors with MShow().
The 'penalty' settings in MAnalyse may be also attempt to decrease noise-based false-motion but MAnalyse have typically only access to pair of frames and can not perform more complex motion vectors processing in time-dimension.
Only MDegrainN have access to all motion vectors in tr-scope and only at this stage we can make additional refining (post-processing) of motion vectors before usage in motion compensating and weighted-blending.
Current ideas: To make 'denoising' of motion vectors in tr-scope either using low-pass filtering or +additional threshold-based correction of too deviated motion vectors from mean (calculated from low-pass filtering of a sequence of motion vectors in the all tr-scope for current block). But it need some thinking how to control low-pass filtering from user-params without overloading user with lots of new params to adjust (one way may be to provide impulse coefficients of filter as array of params).

[Dogway]

Thanks
imp=[0.03125, 0, -0.15625, 0, 0.625, 1.0, 0.625, 0, -0.15625, 0, 0.03125]
a=fmtc_resample(1280*2,634*2, css="420", kovrspl=2, cnorm=true, center=true, kernel="impulse",impulse=imp)

Looks like a softer version of spline64. I don't know, normally one wants the pelclip of the motion search to be on the soft side, after all we do prefiltering to clean edges. But for the pelclip of the MSuper to be fed to MDegrain it could be improved with for example blackman taps=8, or should they use identical scaler?

I managed to clean some blotches from the ex_lfr() function by running it through a limiter. It's not perfect but there's an improvement, in any case I'm testing with anime which might be easier. Another option is to run ex_SmoothGrad() but maybe a little slow in the global scope.

ex_makeadddiff(c, lpc, !DCT ? lpr : ex_limitdif(lpc, lpr, thr=0.3, elast=4.0, UV=UV), UV=UV)

[DTL]

should they use identical scaler?"

They can use different sub-sample interpolation.

For MAnalyse it can be tuned to best motion vector search and for MDegrain it must be tuned for best sub-sample image data interpolation. Because with pel >1 it directly go in the output image result. But best interpolation is depend on input data spectrum. Unfortunately we still have no any one standard on interpolation processing of digital moving pictures data. I think it is typically implied to be something about sinc-based interpolation from times of hybrid analog/digital mixed systems. So for 'best' sharpness it can go up to 'pure sinc' like SincResize/SincLin2Resize. Unfortunately for small block-based interpolation it may be too truncated sinc kernel and too highly distort edges of small block with small margings (to have fast small kernel convolution) with Gibbs. So I currently set Lanczos-weighted sinc into SO=5 MAnalyse (still not finally debugged / fine tuned).

Typically user must adjust interpolation method of MDegrain to current input content to check if it not too distort it and max possible sharpness reached. If too smooth interpolation used - it will also cause decreasing sharpness on any moved blocks (down to half-sample moves with small sharp details).

Currently the interpolation is provided via 'super' clip and it can be different for MAnalyse and for final output MDegrain. Also currently SO=5 MAnalyse uses internal runtime interpolation for SAD calculation stage using about Lanczos 8 samples kernel (+-4) to decrease data traffic from memory.

[Dogway]

While testing in deep_resize() I noticed SincResize with 4 taps applies some kind of local contrast on lower frequencies so a small deconvolution had to be performed. For utmost sharpness this is fine but for speed I found blackmanresize to be visually more pleasant with 8 taps than lanczos4 which showed some ringing and aliasing.