Introducing Zopti (ex AvisynthOptimizer) [Archive] - Page 7

View Full Version : Introducing Zopti (ex AvisynthOptimizer)

Pages : 1 2 3 4 5 6 [7]

DTL

11th February 2022, 00:53

"SincResize with 4 taps applies some kind of local contrast on lower frequencies "

*In a perfect world* SincResize must have as much taps as possible. But it require *perfectly conditioned content* (+processing in the conditioning domain that is not standard defined - it may be either linear data or system-transfer transformed data). So the selection of interpolation method is greatly depends on input content - for some badly conditioned may be only Bilinear/Bicubic is good. Or some non-linear (nnedi ?).
The internal math of MSuper/MDegrain is about linear processing and for best results it may be good to feed in with linear (that is typically RGB) data, not system-transformed. Also the linear averaging operation on system-transformed data is not very good. The natural photon noise is distributed in linear domain and after conversion to non-linear transfer its distribution become distorted. So linear averaging may be less effective ?
Also the interpolation operation for sub-sample precision processing better works (less ringing and other distortions) in its target conditioning domain that may be linear (may depend on the source of current digital moving picture content).

So better workflow of MAnalyse/MDegrain may be:
1. Feed standard system-transformed YCrCb data to MAnalyse (via first MSuper filter). Motion search may better work with system-transfered data ?
2. Feed linear RGB data to MDegrain (via second MSuper filter producing second super clip).
3. Transform/pack back to system format (transfer/matrix/chroma_subsampling) when required.

Much less questions is if we process with pel=1 (not going into sub-sample precision) but it produce less quality with moving fine detailed objects (make more blurring and/or distort motion).

zorr

12th February 2022, 00:52

Let me cook the script.

EDIT: The script. You might want to exclude 1 from DCT and DCTre since it will slow down things a lot, same for 3 for searchAlgo and searchAlgoR. Also starting iterations with script settings so we have a high score reference.

Thanks, I will get started tomorrow, getting a bit late today. :) So I assume these are using the same clips as before and you are also loading the preprocessed clip?

zorr

12th February 2022, 21:51

Dogway, I did a test run with the script, just 24 results. Some observations:

Your best settings resulted in GMSD score 146.05907, not 146.1026. Not sure where the difference is coming from. It could be that the prefiltered clip is different somehow. I saved it as Lagarith from VDub using the same format as the source and DGIndexNV is not able to read that one so I left it as Lagarith.
There was already a better combination with score 146.07826.
It took 1h 34 minutes to run those 24 tests. Mainly because there were some settings that took almost an hour and a half to process. So I guess we should consider adding a timeout, I don't think anyone is patient enough to wait that long for 150 frames let alone for a whole movie... at that rate a 25 fps 90 minute movie would take 56 days. :eek:
Searchalgos 6 and 7 are pure horizontal and vertical exhaustive searches, they might be useful for some specific clips but never in general use. Those could be dropped.

Dogway

13th February 2022, 03:44

I encoded the prefiltered clip with transparent settings in x264, with --opencl (makes a slight difference). Now I realize and forgot to change "--input-range tv" to "pc", don't know if that makes a difference since it's only a tag. Also check avstp.dll is not in plugin path.
SET PROFILE= high
SET BITDEPTH= 8
SET PRESET= veryslow
SET TUNE= film
SET ME= umh
SET QP= 8

I repeated the run and got again 146.10242, a bit lower because some value rounding per my rules.
146.10242 90230 thSAD=245 thSADR=220 truemotionR=true temporal=true lambda=320 lambdaR=960 pnew=114 pnewR=136 pzero=134 lsad=6000 plevel=79 pglobal=8 badrange=2 badSAD=1350 dct=0 dctre=9 searchAlgo=1 searchAlgoR=4 searchRange=8 searchRangeR=11 searchRangeFinest=12 sglobal=true

I don't know why the difference, I guess you are using my (latest) GMSD version (https://github.com/Dogway/Avisynth-Scripts/blob/master/EX%20mods/SimilarityMetrics.avsi). In any case I upload my prefiltered clip here (https://www.mediafire.com/file/okkn37nq9kd6gzi/Brazil-prefiltered.mkv/file). If you still get some different scores I wouldn't worry since we only care about relative scores.

For the high times, did you take out DCT and DCTre=1 and searchAlgo and searchAlgo=3? Yes I forgot about those (vertical and horizontal) while expanding the ranges, I edited my script (https://forum.doom9.org/showthread.php?p=1963517#post1963517) (also with below DCT rules).
I gave a look at my charts with those taken out and probably the culprit is DCT (DCTre seems fine), anything in 0 < DCT < 5 is several times slower. My fastest one with that was 525510ms, too slow. The only way to alleviate times with those DCT is to lower searchRange below 5. In any case if you can send me the output samples, I'm also going to run tests and fill the charts so we can have a broader look even with higher DCT values.
I also realized that very long times can add a bias to the correlation because those ones will add less samples to the data set. In any case though as I said I will pick only the parallelogram of the inflection area of the pareto front, so high times will naturally be taken out.

EDIT: Got a new record (and this is using random iterations, so I'm a bit surprised):
146.10953 314810ms thSAD=220 thSADR=175 truemotionR=true temporal=false lambda=160 lambdaR=2420 pnew=64 pnewR=22 pzero=122 lsad=2650 plevel=39 pglobal=12 badrange=29 badSAD=2700 DCT=10 DCTre=7 searchAlgo=0 searchAlgoR=2 searchRange=13 searchRangeR=11 searchRangeFinest=8 sglobal=false

EDIT2: I did a random run of 230 iterations, calculated the parallelogram section and all DCT with 1, 3 or 4 were out (because of very bad quality/speed ratio). I think it is safe to assume to take those out so instead of lowering searchRange for those we can simply drop them. With this it will run fast enough (DCT=2 a bit slowish though).

From this:
dct = 0 # optimize dct = _n_ | 0..10 | dct
searchRange = 8 # optimize searchRange = _n_ | 0..20 ; filter: dct 0 > dct 5 < and x 5 < true ? | searchRange
to this:
dct = 0 # optimize dct = _n_ | 0,2,5,6,7,8,9,10 | dct
searchRange = 8 # optimize searchRange = _n_ | 0..20 ; filter: dct 2 == x 5 < true ? | searchRange

zorr

13th February 2022, 22:56

For the high times, did you take out DCT and DCTre=1 and searchAlgo and searchAlgo=3?

Yes the script version I used already had a filter to reject those.

probably the culprit is DCT (DCTre seems fine), anything in 0 < DCT < 5 is several times slower. My fastest one with that was 525510ms, too slow. The only way to alleviate times with those DCT is to lower searchRange below 5.

I run some 700 random results without timeouts, only dct=1, dctre=1 and searchAlgo 3 rejected. Here's a table of the fastest and slowest time per dct (also included a colum with the dctre value of the slowest result to see if it's always 2-4).

dct fastest slowest slowest dctre
0 33470 191010 2
1 - - -
2 373470 7703930 0
3 294230 8109110 3
4 291710 7540350 3
5 38190 270050 0
6 45160 300320 9
7 41280 390440 2
8 40100 342090 2
9 39090 248980 10
10 40250 291160 2

So yes looks like dct 2-4 (and 1 even though it wasn't tested) are much slower. But there can be pretty slow results with the other dct values as well, 390 seconds (with dct=7) is already 6,5 minutes. And if we allow that much time there are already results in dct 2-4 which run faster than that so rejecting them outright might seem unfair.

I guess the real question is can the dct 2-4 be competitive in such a restricted time limit. In the pareto front of those 700:ish results none used dct or dctre 2-4 so that indicates they might not be, but it's a bit limited amount of data for my taste.

The distribution of runtimes looks like this:

https://i.postimg.cc/PJJYP6Cy/runtime-distribution.png

35% is under 100 seconds, 62% is under 200 seconds, 68% is under 300 seconds. [EDIT these were incorrect for 200 and 300 seconds]

Another way to look at this graph is to think of the surface area as the total time to calculate results. The vast majority of time is going for those last few % with the slowest execution times.

With a time limit that problem could be solved in a fair manner, doesn't matter which dct is used. On the other hand with a time limit the ones which fail to execute under the limit are just wasted time as we will not get a result for them. But with a limit of say 200 seconds means 38% of results is wasted like this. Also in the pareto front the best (and slowest) executed in 169 seconds so looks like we should be able to get pretty good results in under 200 seconds. I think we could calculate which time limit maximizes the number of useful results...

I also realized that very long times can add a bias to the correlation because those ones will add less samples to the data set. In any case though as I said I will pick only the parallelogram of the inflection area of the pareto front, so high times will naturally be taken out.

The purpose of this exercise is not to be completely fair but to find correlations which you can use to automate some parameters. :) The range we use in the search is already "not fair". We can and should also add other limiting factors to maximize the number of useful results because nobody will use the kind of parameters that take two months to denoise a movie.

EDIT: Got a new record (and this is using random iterations, so I'm a bit surprised):
146.10953 314810ms thSAD=220 thSADR=175 truemotionR=true temporal=false lambda=160 lambdaR=2420 pnew=64 pnewR=22 pzero=122 lsad=2650 plevel=39 pglobal=12 badrange=29 badSAD=2700 DCT=10 DCTre=7 searchAlgo=0 searchAlgoR=2 searchRange=13 searchRangeR=11 searchRangeFinest=8 sglobal=false

In my MCompensate test I had to adjust the allowed parameter ranges multiple times to arrive at the optimal results. I think the same may apply here and we still don't quite know what the optimal range is.

EDIT2: I did a random run of 230 iterations, calculated the parallelogram section and all DCT with 1, 3 or 4 were out (because of very bad quality/speed ratio). I think it is safe to assume to take those out so instead of lowering searchRange for those we can simply drop them. With this it will run fast enough (DCT=2 a bit slowish though).

It's not quite that simple, I had results with dct=2 searchRange 15 faster than searchRange 2, for example. Here are the fastest results with dct=2:

145.35135 373470 thSAD=160 thSADR=230 truemotionR=true temporal=false lambda=1620 lambdaR=1020 pnew=70 pnewR=28 pzero=68 lsad=2400 plevel=25 pglobal=13 badrange=3 badSAD=2540 dct=2 dctre=0 searchAlgo=6 searchAlgoR=7 searchRange=2 searchRangeR=19 searchRangeFinest=3 sglobal=false
145.52867 428370 thSAD=220 thSADR=155 truemotionR=true temporal=false lambda=1200 lambdaR=400 pnew=120 pnewR=196 pzero=178 lsad=4650 plevel=53 pglobal=18 badrange=39 badSAD=1810 dct=2 dctre=10 searchAlgo=7 searchAlgoR=6 searchRange=15 searchRangeR=8 searchRangeFinest=1 sglobal=false
145.22943 428830 thSAD=265 thSADR=285 truemotionR=false temporal=false lambda=1920 lambdaR=2800 pnew=96 pnewR=34 pzero=84 lsad=3750 plevel=67 pglobal=7 badrange=14 badSAD=2480 dct=2 dctre=4 searchAlgo=6 searchAlgoR=6 searchRange=2 searchRangeR=8 searchRangeFinest=4 sglobal=true
145.55482 472610 thSAD=205 thSADR=155 truemotionR=true temporal=false lambda=1320 lambdaR=2840 pnew=88 pnewR=36 pzero=86 lsad=6100 plevel=75 pglobal=19 badrange=0 badSAD=220 dct=2 dctre=10 searchAlgo=6 searchAlgoR=5 searchRange=7 searchRangeR=9 searchRangeFinest=3 sglobal=true
145.43944 479630 thSAD=160 thSADR=205 truemotionR=true temporal=true lambda=1640 lambdaR=380 pnew=140 pnewR=50 pzero=84 lsad=1500 plevel=79 pglobal=19 badrange=23 badSAD=1670 dct=2 dctre=0 searchAlgo=7 searchAlgoR=2 searchRange=10 searchRangeR=8 searchRangeFinest=3 sglobal=false
145.43701 504800 thSAD=170 thSADR=195 truemotionR=false temporal=false lambda=1840 lambdaR=2060 pnew=144 pnewR=76 pzero=196 lsad=5750 plevel=37 pglobal=8 badrange=15 badSAD=2990 dct=2 dctre=10 searchAlgo=0 searchAlgoR=6 searchRange=1 searchRangeR=0 searchRangeFinest=7 sglobal=true
145.41245 525230 thSAD=260 thSADR=230 truemotionR=true temporal=false lambda=580 lambdaR=1440 pnew=100 pnewR=102 pzero=106 lsad=2550 plevel=21 pglobal=14 badrange=40 badSAD=720 dct=2 dctre=4 searchAlgo=7 searchAlgoR=7 searchRange=20 searchRangeR=4 searchRangeFinest=2 sglobal=true
145.64351 562840 thSAD=175 thSADR=150 truemotionR=true temporal=false lambda=340 lambdaR=2980 pnew=50 pnewR=182 pzero=86 lsad=3950 plevel=47 pglobal=12 badrange=36 badSAD=2720 dct=2 dctre=10 searchAlgo=4 searchAlgoR=7 searchRange=1 searchRangeR=19 searchRangeFinest=17 sglobal=false

I will look at the GMSD issue next. Didn't know that you had your own version of it.

[EDIT] I get GMSD 146.10782 with your prefiltered clip. Closer, but still a small difference. Going to try your GMSD version next.

zorr

14th February 2022, 22:38

Dogway, I switched to your latest GMSD version, the score is now 146.10785 so only a very minor change (got 146.10782 with standard GMSD). Also tried upgrading to latest Avisynth+ 3.7.2 test 3, that didn't have any effect. MVTools is already the latest version. You're using 64bit version as well, right? It's a bit mysterious but I guess we can go forward even with these slight differences.

The issue with DGSource came back, it seems to only appear when there are enough simultaneous threads using it. I run the tests using 12 threads and each script loads three sources with DGSource. Three out of 12 scripts give incorrect results because one or more of those DGSources pops an error message "Failed to create video decoder (2)" but runs anyway and the GMSD score drops to around 112. I guess I'm going to have to load at least one of the sources with FFVideoSource to avoid this issue. Or I could concatenate two of those sources as one file and split it in the script.

I thought about how to calculate the timeout which gives most results in a given time, it's some kind of infinite fraction which I don't know how to solve so I wrote a simple simulator instead. It takes 100 million samples per timeout using the distribution of execution times I collected and returns the average time per result. The accuracy is about 10-50ms between different runs. The results for 100 - 300 seconds looks like this:

https://i.postimg.cc/bJTChSjf/average-runtime-per-timeout.png

The best average runtime is about 220 seconds using a timeout of 170 seconds. I think using at least 200 would be better so we don't rule out too many high quality combinations. Perhaps the margin should be even larger, what do you think?

Using the timeout of course has a little problem that combining mine and your results is more difficult as the execution times are a bit different on my system and yours. But we can calculate the speed ratio and scale the timeout accordingly.

Do you think that using a timeout is good or do you prefer just dropping dct 2-4? I'll do whatever you think is better, the data is generated for you anyway. :)

Dogway

15th February 2022, 00:28

I don't like timeouts much because first you waste time and second how much is too much? you might be ruling out some combinations that we might need.
One solution would be to look at the frame to frame render time, so no time is wasted. Say, abort if the mean time of the 5 first render times are above x, in this case I would set it to a rather high-ish threshold like 500000ms/150

In any case I prefer to limit my times by settings so I know what's going on. If something makes the script too slow, search that up and write a rule. Aside from the settings noted above searchAlgo=5 slow things a bit, like I get 300000ms or some higher on my CPU with DCT=0, but still I think I may want to run with it. So if you know you want to try DCT=2 and searchAlgo=5, just make sure both don't coincide. I mean, this is some kind of a global scope test so we want (a bit) high times in, insanity times out.
One note; whenever possible I observed it's better (more random) to define the range as "0,1,2,4,5" than write a rule like "filter: x 3 !=". Maybe you can have a look if that's correct.

The case where DCT=2 and searchRange=15 was fastest might be due to searchRangeFinest.
Practically the main performance hoggers are DCT and searchAlgo and in second term, searchRange and searchRangeFinest (related to the previous). At least that I know of from which we are trying. I suspect something else is going on with badSAD and/or badRange. If you want tomorrow I can run a test bench for DCT=2 and see what causes high or low times for this one.

EDIT: I wanted to combine results, but if we get different scores it might not mean much. Yes, I'm on x64, latest on everything included DGDecNV (just in case). Check if you have latest ExTools and fmtconv, and the order of the GMSD() clips, it shouldn't matter but just in case, it's defined as GMSD(clip dist, clip ref), so I use GMSD(last, src) references.

Latest script version with the new rules, you can set timeout to 500000ms I doubt you will reach it often or hope so.
And one last thing, is it ok speed wise to use MDSI(), with or without downsample=true? I say because GMSD only cares about structure (edges), but searchRangeFinest affects shading which is low frequency.
setmemorymax(16384/8)
DGSource("Brazil-cleanREF.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=AssumeFPS(24000,1001)
DGSource("Brazil-prefiltered.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
pre=AssumeFPS(24000,1001)
DGSource("brazil2.dgi",cl=0,ct=40,cr=0,cb=40)
AssumeFPS(24000,1001)

thSAD = 220 # optimize thSAD = _n_ | 150..300 ; filter:x 5 % 0 == | thSAD
thSADR = 175 # optimize thSADR = _n_ | 150..300 ; filter:x 5 % 0 == | thSADR
thSADC = 185
BlkSize = 16
BlkSizeR = 8
overlap = 8
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = true
truemotionR = true # optimize truemotionR = _n_ | true,false | truemotionR
temporal = false # optimize temporal = _n_ | true,false | temporal

# TRUEMOTION SETTINGS
lambda = 160 # optimize lambda = _n_ | 100..2000 ; filter:x 20 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 2420 # optimize lambdaR = _n_ | 300..3000 ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 64 # optimize pnew = _n_ | 20..200 ; filter:x 2 % 0 == | pnew
pnewR = 22 # optimize pnewR = _n_ | 20..200 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest, and searchRangeFinest with badRange)
pzero = 122 # optimize pzero = _n_ | 0..200 ; filter:x 2 % 0 == | pzero
lsad = 2650 # optimize lsad = _n_ | 1000..8000 ; filter:x 50 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 39 # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1 # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 12 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 29 # optimize badrange = _n_ | 0..50 | badrange
badSAD = 2700 # optimize badSAD = _n_ | 0..3000 ; filter:x 10 % 0 == | badSAD
dct = 10 # optimize dct = _n_ | 0,2,5,6,7,8,9,10 | dct
dctre = 7 # optimize dctre = _n_ | 0,2,3,4,5,6,7,8,9,10 | dctre
rfilter = 3
searchAlgo = 0 # optimize searchAlgo = _n_ | 0,1,2,4,5 ; filter: dct 2 == x 5 != true ? | searchAlgo
searchAlgoR = 2 # optimize searchAlgoR = _n_ | 0,1,2,4,5 | searchAlgoR
searchRange = 13 # optimize searchRange = _n_ | 0..20 ; filter: dct 2 == x 5 < true ? | searchRange
searchRangeR = 11 # optimize searchRangeR = _n_ | 0..20 | searchRangeR
searchRangeFinest = 8# optimize searchRangeFinest = _n_ | 0..20 | searchRangeFinest
sglobal = false # optimize sglobal = _n_ | true,false | sglobal

trim(1,12)+\
trim(60,96)+\
trim(173,285-12)

C = ConvertBits(16,fulls=false)
Pre = pre #C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true

superfilt = MSuper(pre, hpad=16, vpad=16, sharp=sharp, rfilter=rfilter, pel=pel, mt=false) # all levels for MAnalyse
superR = MSuper(C, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false, chroma=false)
superRe = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
bak2 = MAnalyse(superfilt, isb=true, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 = MAnalyse(superfilt, isb=true, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 = MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 = MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe, bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
bak1 = Recalculate ? MRecalculate(superRe, bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
fwd1 = Recalculate ? MRecalculate(superRe, fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe, fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=false)

resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFileStart(resultFile, "FrameCount()")

GMSD(ConvertBits(32, fulls=false, fulld=true), src.ConvertBits(32, fulls=false, fulld=true), Y=true,U=false,V=false)

global GMSD = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
""",local=false)

global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

global delimiter = "; "
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")

DTL

15th February 2022, 12:43

Starting experiment with denoising of static-image clip I found some feature of current mvtools algoriphm - it sometime converts luma (value) noise into 'phase' noise. I.e. 2D errors of MVs cause shifting of parts of image in 2D space. Also it looks the 'MVs coherency' algorithm in MAnalyse cause random noised blocks to have same 2D error in nearby areas so large enough parts of image got phase-noise shift.

LoadPlugin("mvtools2.dll")
LoadPlugin("AddGrainC.dll")
LoadPlugin("RawSourcePlus.dll")
RawSourcePlus("out8.raw", 308,308, "Y8")
Loop(100)
ConvertToYV12()

ColorYUV(gain_y=-150,off_y=70)

frsrc=last

AddGrain(50)
#return last
padd=8

tr = 2 # Temporal radius
super1 = MSuper (pel=1, chroma=true)
multi_vec1 = MAnalyse (super1, overlap=0, multi=true, search=3, searchparam=2, delta=tr,chroma=true,mt=false)
frp=MDegrainN (super1, multi_vec1, tr, thSAD=1000, thSAD2=1000-1, thSCD1=2000, thSCD2=2000, mt=false)

super = MSuper(pel=4, mt=false)
forward_vec1 = MAnalyse(super, isb = false, search=3, searchparam=2, chroma=false, delta = 1, mt=false, levels=0)
frmsh=MShow(super,showsad=true, forward_vec1, thSCD1=2000, thSCD2=2000, scale=10)

frp=AddBorders(frp, padd,padd, padd, padd)
frsrc=AddBorders(frsrc, padd,padd, padd, padd)

difclip=Subtract(frp, frsrc)
difclip=ColorYUV(difclip, gain_y=300, off_y=-150)

StackHorizontal(difclip, frsrc, frp, frmsh)

out8.raw zoneplate file is https://drive.google.com/file/d/1gjAswPsugMI9EXrea4FQhaGwCcNZ8j1o/view?usp=sharing

Result is
https://i4.imageban.ru/out/2022/02/15/36e5a09f1cdf646140e3b533c866f647.png

at some of frames. It is visible as difference clip shows great correlation with low frequencies parts of the source (not noised original). Also the MVs errors (all MVs here are errors of search) at the low freq parts of image have great correlation though it is static image with random noise.

So do the used GMSD metric more or less sensitive to such types of distortions ? The simple metrics like PSNR will treat such phase-noise as more or less great distortion because significant parts of image is shifted.

The phase-noised image looks good denoised, but may still be not very nice to mpeg compressor.

zorr

16th February 2022, 00:32

I don't like timeouts much because first you waste time and second how much is too much? you might be ruling out some combinations that we might need.
One solution would be to look at the frame to frame render time, so no time is wasted. Say, abort if the mean time of the 5 first render times are above x, in this case I would set it to a rather high-ish threshold like 500000ms/150

Yes you need to be careful with the timeout and set it high enough. I already had an idea why the timeout wouldn't rule out important combinations, I guess I should share my data.

https://i.postimg.cc/YSTkrdNW/mdegrain2.png

This is from the random population. You can see the pareto line ends at 170 seconds. Considering you're only going to use the population "where the action is taking place" that would be results with time around 30-80 seconds. With more results the "action area" is going to move towards the top and left meaning the execution times will be even faster. Considering that perhaps a timeout of 300 seconds would surely not prune anything that's really needed.

And when it comes to wasting time, the timeout is of course meant to do the opposite and save time. Even with the timeout of 500 seconds it's going to be a huge chunk (see the professional-quality graph below). The waste is only really there if we reject results that would have been useful if we hadn't aborted.

https://i.postimg.cc/xTZ6FB9M/runtime-distribution-saved2.jpg

The idea of looking at the per frame timings is a good one, I have already investigated a similar idea. The idea was to try to predict if the total quality would be close to the best value found, based on the first frames using statistics. In the end the benefit of the prediction was so small that it wasn't worth the added complexity. This was a bit different situation though because with quality you want to be absolutely sure you don't throw away a good result, with speed there's more potential because there is a much larger spread of values. But I'd like to collect some stats on how well the first few frames can predict the total runtime before I go down that path. Preliminary data indicates the per frame execution times vary a lot so it may not be a sure thing.

In any case I prefer to limit my times by settings so I know what's going on. If something makes the script too slow, search that up and write a rule. Aside from the settings noted above searchAlgo=5 slow things a bit, like I get 300000ms or some higher on my CPU with DCT=0, but still I think I may want to run with it. So if you know you want to try DCT=2 and searchAlgo=5, just make sure both don't coincide. I mean, this is some kind of a global scope test so we want (a bit) high times in, insanity times out.

I like that too because then we'll learn more about the performance characteristics of MVTools. I'm just afraid that the rules to control the runtime may become really complex, and there's also the risk of pruning combinations that would have good enough performance.

One note; whenever possible I observed it's better (more random) to define the range as "0,1,2,4,5" than write a rule like "filter: x 3 !=". Maybe you can have a look if that's correct.

That's correct. When the parameter's value is checked after mutation and it's not valid according to the filter then the closest valid value is chosen instead. This introduces bias. Why a random valid value is not used is that it would no longer respect the original mutation amount. We could just try the mutation again and hope to hit a valid value but that would be very inefficient.

I suspect something else is going on with badSAD and/or badRange.

It's definitely a factor, but only on those frames where the motion vectors are difficult to find and SAD is considered "bad" on one or more blocks. The search is done only to those bad blocks so it may be skipped completely or in extreme cases calculated for all blocks. So the performance depends not only on the chosen settings but also on the contents of the frame (and previous, next frames).

EDIT: I wanted to combine results, but if we get different scores it might not mean much. Yes, I'm on x64, latest on everything included DGDecNV (just in case). Check if you have latest ExTools and fmtconv, and the order of the GMSD() clips, it shouldn't matter but just in case, it's defined as GMSD(clip dist, clip ref), so I use GMSD(last, src) references.

I'll try updating ExTools, that one is a bit older than your latest.

In the mean time I started another optimization with revised ranges based on the first test (below the changes). This is still using overlap = 4 so perhaps even better results can be found with overlap = 8.

# parameter sharp no longer optimized (only had one possible value)
# thSAD range extended from 230..290 to 230..330
# thSADR range extended from 150..300 to 120..300
# lambda range moved from 400..900 to 100..600 (offset -300)
# lambdaR range moved from 960..2000 to 760..1800 (offset -200), maximum changed from "lambda 3 *" to "lambda 6 *"
# pnew range moved from 100..200 to 40..140 (offset -60)
# pnewR range extended from 100..250 to 0..250
# pzero range moved from 28..100 to 58..130 (offset 30)
# lsad range changed from 1000..8000 to 100..6000
# badrange range extended from 0..50 to 0..70
# badSAD range extended from 1100..2200 to 1100..2400
# dctre options changed from 0,2,3,6,7,9 to 0,6,7,9
# optimization of thSADC disabled as GMSD doesn't evaluate the chroma by default

The best result found so far is 146.21552.

https://i.postimg.cc/Twq2YZR4/mdegrain-fast2.png

Latest script version with the new rules, you can set timeout to 500000ms I doubt you will reach it often or hope so.

Based on the latest results you might want to adjust the ranges of pnewR and lsad to include smaller values, the best result has pnewR=2 and lsad=200. The other parameters have the best value inside your ranges.

And one last thing, is it ok speed wise to use MDSI(), with or without downsample=true? I say because GMSD only cares about structure (edges), but searchRangeFinest affects shading which is low frequency.

Not sure, I haven't done performance comparisons between GMSD and MDSI. But I say if MDSI better represents the quality it should be used.

I did a couple of graphs to show the vast difference in speed of dct 2-4 versus others. These are from the same random population. First all the paretos grouped by dct:

https://i.postimg.cc/s22DDnBj/groupby-dct-all.png

dct 2-4 are in their own group far away from the global pareto. This also tells us that a timeout of 500 seconds would actually cut out the best results of dct 2-4. Here you can also see that the filter:x 1 != has caused dct 0 and dct 2 have more samples than the others.

https://i.postimg.cc/VknXWk4r/groupby-dct-5-10.png

Only dct 5-10 here, they are all grouped quite close together. The top quality results are from dct 6 and 9.

https://i.postimg.cc/s23GP9vq/groupby-dct-0.png

Only dct 0 in this one, it dominates the pareto of faster results.

I'm thinking dct 1-4 might not have any representation in the "action" area and therefore we might as well leave them out. Or do another run where they are the only dct values allowed.

Dogway

16th February 2022, 01:51

zorr

17th February 2022, 01:44

It depends on the (final) inflection curve width, I need enough samples in the horizontal dimension. The slowest "low" pareto time must match the fastest "high" pareto (high and low of the inflection curve), this width segment has also to be applied on the "high" pareto to extract the correct parallelogram of the inflection curve.

All right, I trust that you know what you are doing. :)

there's no convex hull.

Just a clarification, you probably know this but the pareto front is not a convex hull even though it's drawn in a similar manner. Actually the pareto boundary is not a straight line between the pareto front points, it should be drawn as two orthogonal lines. I draw it as a straight line to make it easier for the eyes. And naturally convex hull covers all sides which is not the case here. The pareto front is formed from the results which are non-dominated (no other result is at least as good in quality/speed and better in either quality or speed).

Check my samples spreadsheet.

Tried to but it requires access rights, I sent the request.

GMSD by default uses "downsample=true", so it is safe to assume MDSI works well also with "downsample=true", and it will be faster. I updated the metrics script yesterday for using downsample when clip is HD. You'd need to adjust timeout if you are going to use it.

MDSI might be better but perhaps we should try to evaluate which one better represents the quality in this case. We could do a medium length run with both and check which one found the better looking result. In my earlier similarity metric tests I concluded that GMSD is probably better for most cases but it may depend. One fun test would be to optimize for both GMSD and MDSI and visually evaluate the pareto front. That might be even more fair test than two different runs.

I tried your latest ExTools but it didn't change the GMSD score (and it shouldn't have because it's not used when the preprocessed clip is just loaded from the disk). Also updated MaskTools2, same story. I thought I found the culprit when I noticed I have IQA_downsample.avsi which has a different implementation of _IQA_downsample() but that wasn't it either.

Since there is still small variation between our scores I think we should do a small validation test, say 10 random results. I can send you the log of my results and you can repeat the same tests using -mode validate.

A couple of notes about your latest script: I wonder why truemotionR is optimized, perhaps because MRecalculate doesn't have lsad and pnew which are controlled by truemotion? Anyway I tested it and changing it true/false didn't have any effect on the score.

I noticed the plevel has wrong range, the valid values are 0..2.

The ranges look good except I would extend the range of pnewR to zero and lsad to 50 (can't be zero because that's illegal value and must be divisible by 50). This is based on my latest run, for example here's pnewR.

https://i.postimg.cc/520fQsxy/line-pnew-R.png

You mentioned earlier that GMSD favored something that caused smearing. Do you have an example (script + which frame) to demonstrate that? If it looks like I think it does then there might be a way to boost it so that GMSD considers it a more severe artifact.

Boulder

17th February 2022, 05:36

In my regular "downscale - upscale" method, GMSD favours a slightly softer result compared to MDSI. I often use it for more grainy sources where the resulting average bitrate would be very high.

Dogway

17th February 2022, 11:32

Yes, I checked MDSI and it's pretty much also edge based too. GMSD fared better for this cases in some tests I did.

Send me the test scripts so I can do a -mode validate but I won't be able to check until the night.
I fixed already your range observations, I think I referenced an old documentation for plevels.

zorr

17th February 2022, 22:53

Send me the test scripts so I can do a -mode validate but I won't be able to check until the night.

I generated the test cases using your latest script with my modifications. I made a package with the script and the result log. You can download the zip from here (https://drive.google.com/file/d/1szzudGoyxO2N-5pMhQOfPD3m7P9TFzws/view?usp=sharing). Also included is the latest Zopti jar for version 1.2.3 WIP since the validate mode in previous versions only validates the pareto front and here we want to validate all results. Just replace the jar file in Zopti bin folder.

You can run the validation with command
zopti -mode validate -log "path/to/file/mdegrain_validate run-01.log" -threads 10 -count 1 -types avg

Adjust file location and number of threads. Also change the first line of the mdegrain_validate run-01.log file to point to correct .avs file location.

There are 10 results, including some with dct=2. One of them takes about 45 minutes to run even with searchRange=4.

The validation writes two files, in this case they have the same information because count is 1. Usually validation is done with multiple measurements and the other file contains all individual results and the other one contains the average/min/max/median of those multiple results.

If the quality is different than what is found in the log Zopti writes a warning line to console output and in the end the number of warnings. I expect 10 warnings but the interesting thing will be how different the results are.

Dogway

17th February 2022, 23:36

Ok, went ahead to check and didn't read your example call haha, it's taking a while (single threaded) but so far 5 validations and in all there's a score difference of 0.005, always lower. I don't know what to think about it... You are running latest GMSD, AVS+, DGSource, MVTools2, fmtconv, and latest ExTools.

Do you want me to re run them with your call or is it not necessary?

zorr

18th February 2022, 00:22

Ok, went ahead to check and didn't read your example call haha, it's taking a while (single threaded) but so far 5 validations and in all there's a score difference of 0.005, always lower.

That's interesting, also helpful if the difference is almost a constant. I wonder if it's because you're running Intel and I have AMD, after all floating point math is not exact in different architectures. The delta is quite large though so it must be some cumulative effect.

I don't know what to think about it... You are running latest GMSD, AVS+, DGSource, MVTools2, fmtconv, and latest ExTools.

GMSD Similarity Metrics Pack v2.6 (16-02-2022)
AVS+ AviSynth+ 3.7.2 (r3620, master, x86_64)
DGSource 239.0.0.0
MVTools2 2.7.45.0 (20210608)
fmtconv r28
ExTools ExTools v7.5 (12-02-2022)

Do you want me to re run them with your call or is it not necessary?

The result will be the same so no need to run again but get a couple more results if you have the time so that we can confirm the quality delta.

Dogway

18th February 2022, 00:35

Yep, all latest except AVS+ version, I use r3622, but I was also thinking on float point precision from the beginning, didn't know you had an AMD, did a search on AVX512 in Expr docs but it's not used anywhere so we are on par with that (AVX2 top here).
The difference is not exactly 0.005, it fluctuates a tiny bit but never above the third digit. Probably I would consider it safe to sum 0.005 to my scores.

Validation results so far:
145.82434 85382 thSAD=265 thSADR=245 temporal=true lambda=1740 lambdaR=760 pnew=174 pnewR=24 pzero=100 lsad=6100 plevel=1 pglobal=0 badrange=16 badSAD=2300 dct=8 dctre=0 searchAlgo=2 searchAlgoR=4 searchRange=8 searchRangeR=11 searchRangeFinest=6 sglobal=false
146.01328 85692 thSAD=220 thSADR=270 temporal=true lambda=160 lambdaR=980 pnew=74 pnewR=48 pzero=100 lsad=250 plevel=2 pglobal=4 badrange=50 badSAD=80 dct=6 dctre=6 searchAlgo=0 searchAlgoR=1 searchRange=8 searchRangeR=0 searchRangeFinest=2 sglobal=false
146.00676 97915 thSAD=220 thSADR=265 temporal=false lambda=900 lambdaR=700 pnew=98 pnewR=18 pzero=188 lsad=1700 plevel=0 pglobal=20 badrange=33 badSAD=1290 dct=10 dctre=9 searchAlgo=4 searchAlgoR=2 searchRange=10 searchRangeR=1 searchRangeFinest=0 sglobal=false
145.94038 109009 thSAD=255 thSADR=190 temporal=false lambda=400 lambdaR=960 pnew=100 pnewR=78 pzero=42 lsad=6800 plevel=0 pglobal=10 badrange=26 badSAD=2380 dct=10 dctre=6 searchAlgo=1 searchAlgoR=0 searchRange=1 searchRangeR=11 searchRangeFinest=20 sglobal=true
145.94707 187095 thSAD=195 thSADR=185 temporal=false lambda=1080 lambdaR=2800 pnew=68 pnewR=78 pzero=134 lsad=100 plevel=2 pglobal=7 badrange=10 badSAD=110 dct=7 dctre=10 searchAlgo=0 searchAlgoR=1 searchRange=15 searchRangeR=16 searchRangeFinest=17 sglobal=false
145.62733 209594 thSAD=155 thSADR=235 temporal=false lambda=1960 lambdaR=1620 pnew=108 pnewR=68 pzero=98 lsad=2100 plevel=0 pglobal=2 badrange=33 badSAD=930 dct=8 dctre=4 searchAlgo=2 searchAlgoR=5 searchRange=13 searchRangeR=18 searchRangeFinest=2 sglobal=true
145.80855 212743 thSAD=215 thSADR=195 temporal=false lambda=860 lambdaR=1420 pnew=146 pnewR=12 pzero=172 lsad=3000 plevel=2 pglobal=0 badrange=46 badSAD=2650 dct=10 dctre=4 searchAlgo=5 searchAlgoR=2 searchRange=20 searchRangeR=15 searchRangeFinest=12 sglobal=true
145.86792 501016 thSAD=225 thSADR=300 temporal=true lambda=520 lambdaR=360 pnew=192 pnewR=156 pzero=162 lsad=6600 plevel=1 pglobal=14 badrange=20 badSAD=170 dct=2 dctre=7 searchAlgo=0 searchAlgoR=0 searchRange=4 searchRangeR=13 searchRangeFinest=4 sglobal=true

Yours for comparison:
145.82846 84450 thSAD=265 thSADR=245 temporal=true lambda=1740 lambdaR=760 pnew=174 pnewR=24 pzero=100 lsad=6100 plevel=1 pglobal=0 badrange=16 badSAD=2300 dct=8 dctre=0 searchAlgo=2 searchAlgoR=4 searchRange=8 searchRangeR=11 searchRangeFinest=6 sglobal=false
146.01889 85180 thSAD=220 thSADR=270 temporal=true lambda=160 lambdaR=980 pnew=74 pnewR=48 pzero=100 lsad=250 plevel=2 pglobal=4 badrange=50 badSAD=80 dct=6 dctre=6 searchAlgo=0 searchAlgoR=1 searchRange=8 searchRangeR=0 searchRangeFinest=2 sglobal=false
146.01184 98740 thSAD=220 thSADR=265 temporal=false lambda=900 lambdaR=700 pnew=98 pnewR=18 pzero=188 lsad=1700 plevel=0 pglobal=20 badrange=33 badSAD=1290 dct=10 dctre=9 searchAlgo=4 searchAlgoR=2 searchRange=10 searchRangeR=1 searchRangeFinest=0 sglobal=false
145.94518 109910 thSAD=255 thSADR=190 temporal=false lambda=400 lambdaR=960 pnew=100 pnewR=78 pzero=42 lsad=6800 plevel=0 pglobal=10 badrange=26 badSAD=2380 dct=10 dctre=6 searchAlgo=1 searchAlgoR=0 searchRange=1 searchRangeR=11 searchRangeFinest=20 sglobal=true
145.95306 178040 thSAD=195 thSADR=185 temporal=false lambda=1080 lambdaR=2800 pnew=68 pnewR=78 pzero=134 lsad=100 plevel=2 pglobal=7 badrange=10 badSAD=110 dct=7 dctre=10 searchAlgo=0 searchAlgoR=1 searchRange=15 searchRangeR=16 searchRangeFinest=17 sglobal=false
145.63268 197430 thSAD=155 thSADR=235 temporal=false lambda=1960 lambdaR=1620 pnew=108 pnewR=68 pzero=98 lsad=2100 plevel=0 pglobal=2 badrange=33 badSAD=930 dct=8 dctre=4 searchAlgo=2 searchAlgoR=5 searchRange=13 searchRangeR=18 searchRangeFinest=2 sglobal=true
145.8131 203980 thSAD=215 thSADR=195 temporal=false lambda=860 lambdaR=1420 pnew=146 pnewR=12 pzero=172 lsad=3000 plevel=2 pglobal=0 badrange=46 badSAD=2650 dct=10 dctre=4 searchAlgo=5 searchAlgoR=2 searchRange=20 searchRangeR=15 searchRangeFinest=12 sglobal=true
145.87228 907400 thSAD=225 thSADR=300 temporal=true lambda=520 lambdaR=360 pnew=192 pnewR=156 pzero=162 lsad=6600 plevel=1 pglobal=14 badrange=20 badSAD=170 dct=2 dctre=7 searchAlgo=0 searchAlgoR=0 searchRange=4 searchRangeR=13 searchRangeFinest=4 sglobal=true
145.73233 934690 thSAD=265 thSADR=255 temporal=false lambda=1100 lambdaR=1360 pnew=36 pnewR=170 pzero=10 lsad=2050 plevel=2 pglobal=4 badrange=35 badSAD=320 dct=2 dctre=2 searchAlgo=4 searchAlgoR=1 searchRange=4 searchRangeR=10 searchRangeFinest=10 sglobal=false
146.09076 2676410 thSAD=250 thSADR=290 temporal=true lambda=480 lambdaR=2740 pnew=64 pnewR=24 pzero=40 lsad=1700 plevel=0 pglobal=2 badrange=17 badSAD=530 dct=2 dctre=9 searchAlgo=1 searchAlgoR=2 searchRange=4 searchRangeR=10 searchRangeFinest=19 sglobal=false

By the way your last one took a lot of time, dct=2 and searchRange=4, so the culprit could be searchRangeFinest. In any case for me it's fine to drop DCT from 2 to 4 if you want, or limit searchRangeFinest but those didn't get good scores anyway.

I'm going to let it optimize overnight with the last refined script version (sans DCT 2 ~ 4), and check the results tomorrow. If you can share your highest score parameters so far (146.21552) so I can include it in the following runs for pareto reference.

EDIT: My results, took about 8h so I guess I can do 1000 iterations per night. One question though, do you know how I can continue with this "optimization" without creating a new one?

https://i.imgur.com/awrSxonl.png

The difference in time can roughly be mapped like this:
time = time/100.
time > 178.04 ? 170.306*pow(time,0.25045)-436.243 : \
1.0849*time-7.839
time = time*100

zorr

18th February 2022, 22:23

Yep, all latest except AVS+ version, I use r3622

Where did you get that one? I'm using the latest 3.7.2 test 3 version pinterf shared in AviSynth+ thread Vol.2, the Github only has 3.7.1 versions.

By the way your last one took a lot of time, dct=2 and searchRange=4, so the culprit could be searchRangeFinest.

I did about 2140 results using the latest script, including dct=2 but limiting the seachRange and searchAlgo. The script runtime graph now looks like this:

https://i.postimg.cc/sDcXbYSy/runtime-distribution-dct2.png

Not surprisingly the slowest results have all dct=2. The fastest dct=2 was 498 seconds so the lump at 500 is where dct=2 results start. The limits are mostly restricting the time to stay below 1000 seconds but some of them are higher still. Here's a zip (https://drive.google.com/file/d/1hrKFyTsFevCRm8gQIA_RvpTGHaGO2X_i/view?usp=sharing) of the dct=2 results sorted by time if you want to try to figure out what makes them slow.

In any case for me it's fine to drop DCT from 2 to 4 if you want, or limit searchRangeFinest but those didn't get good scores anyway.

Yes I think that's a reasonable option right now. We can come back to DCT 1-4 later. I know they have potential because in my MFlowFPS tests DCT=1 was the top quality result.

If you can share your highest score parameters so far (146.21552) so I can include it in the following runs for pareto reference.

146.21703 95570 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dctre=6 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true

One question though, do you know how I can continue with this "optimization" without creating a new one?

Not possible at the moment, I will make a Zopti version which supports -continue with the mutation algorithm, at least with the initial random generation. But the risk of getting duplicates is pretty low so you can just start a new one and concatenate the result files manually later.

The difference in time can roughly be mapped like this:
time = time/100.
time > 178.04 ? 170.306*pow(time,0.25045)-436.243 : \
1.0849*time-7.839
time = time*100

I'd like to get more data for this curve fitting function. We can do it the other way, if you send me your 1000 results I can run the validation on them.

Dogway

18th February 2022, 23:02

Where did you get that one? I'm using the latest 3.7.2 test 3 version pinterf shared in AviSynth+ thread Vol.2, the Github only has 3.7.1 versions.

From here (https://forum.doom9.org/showthread.php?p=1963492#post1963492)I think.

Yes I think that's a reasonable option right now. We can come back to DCT 1-4 later. I know they have potential because in my MFlowFPS tests DCT=1 was the top quality result.

I also think they have potential but not as much for MDegrain client as for MFlowFPS.

146.21703 95570 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dctre=6 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true

Do you have DCT and searchAlgo values for this?

I'd like to get more data for this curve fitting function. We can do it the other way, if you send me your 1000 results I can run the validation on them.

OK, they uploaded here (https://www.mediafire.com/file/v1c0rskswsoez7o/Zopti_test2s_-_MDegrain6_-_Random_%25282022-02-18_00-34-31%2529_optimize_mutation_pop-500_mutcount-60-1_mutamount-0.5-0.01_iters-500_run-01.log/file).
The fitting is pretty basic given I only had like 8 samples and 1 of them was an outlier. But for short to medium times it seemed to follow a linear function.

zorr

19th February 2022, 00:31

From here (https://forum.doom9.org/showthread.php?p=1963492#post1963492)I think.

Looks the same link I used. What does Version() output?

146.21703 95570 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dctre=6 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true

Do you have DCT and searchAlgo values for this?

dct = 0
searchAlgo = 1

These may also be relevant

BlkSize = 16
BlkSizeR = 8
overlap = 4
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = false
truemotionR = false

OK, they uploaded here (https://www.mediafire.com/file/v1c0rskswsoez7o/Zopti_test2s_-_MDegrain6_-_Random_%25282022-02-18_00-34-31%2529_optimize_mutation_pop-500_mutcount-60-1_mutamount-0.5-0.01_iters-500_run-01.log/file).

Thanks, I'll do the validation after I have run about 10k results without dct 1-4, started already.

The fitting is pretty basic given I only had like 8 samples and 1 of them was an outlier. But for short to medium times it seemed to follow a linear function.

I will do a scatter plot and try to fit some functions.

I have thought about adding a conversion mode to Zopti, you could for example convert the result file into CSV or JSON format for easier importing to other software. Also sorting by GMSD or time could be useful. Do you need anything like that?

Dogway

19th February 2022, 11:39

Yes, I actually have a better global picture of what's going on when I add to Excel and color code the values. Then I can simply sort by columns and check the relationships at the same time for different parameters (like sectioned by DCT...), prune those that don't need, etc. Just like in the Google spreadsheets.
CSV output would be very useful because I always have to remove "param_name=". I have a regex for that but better if it's built internally.
As for sorting it could also be useful, currently I can sort in Notepad2 with Alt+O, but this only takes into account the first column (GMSD).

This is my version()
https://i.imgur.com/uAoTfdO.png

zorr

19th February 2022, 22:44

One question though, do you know how I can continue with this "optimization" without creating a new one?

The option -continue now works with mutation algorithm for the initial random generation population. Get the latest jar here (https://drive.google.com/file/d/1M_dIRPR2yfHVps_9DG_gy_shlLn2F5DW/view?usp=sharing).

I have about 5.5k results now.

https://i.postimg.cc/tCfCPMZR/mdegrain-latest-fast-dct-5k.png

Dogway

19th February 2022, 23:38

Thanks! I will leave it tonight using the -continue flag.

The curve is shaping up nicely, is there any reason you use different values for population and iterations?

Also if you can share your top results with overlap=8, as we are testing with that one locked. At the time I did some study on overlap and found that in average it was slower and with worse scores, but we can go back to it after the correlations, same with DCT 1~4.

By the way, I think that rather than trying to map GMSD and time from your logs to mine or the other way around, it would be better to section an area (the 'action' area) and validate those.

zorr

20th February 2022, 23:12

The curve is shaping up nicely, is there any reason you use different values for population and iterations?

The mutation algorithm works by creating an offspring for every member of the population and then selecting the best 50% which brings the population back to original size. That is also done in the first generation so N*2 solutions will be generated, where N is the population size. And that's the number of iterations. I guess it also works if iterations is smaller than N*2 but I wanted to be precise. So in a nutshell if you want X random results you set the iterations to X and population size to X/2 in the mutation algorithm.

Also if you can share your top results with overlap=8, as we are testing with that one locked.

My best overlap=8 result currently is from the ongoing random run, 146.19359. The earlier larger runs were using overlap=4.

At the time I did some study on overlap and found that in average it was slower and with worse scores, but we can go back to it after the correlations, same with DCT 1~4.

Overlap=8 was slower and worse, or overlap=4? Anyway I think overlap is one of those hard to predict parameters, the best option may depend on the source material.

By the way, I think that rather than trying to map GMSD and time from your logs to mine or the other way around, it would be better to section an area (the 'action' area) and validate those.

Hmm yes, of course that means after the area is sectioned and validated the section could be elsewhere and not anymore in the 'action' area. Also if the area is not large enough it might introduce selection bias. But I get your point, it's a good idea as long as the section is selected carefully. I guess there's no need to do the speed validation run of your 1000 results?

I have finished 10k results and continued, now over 11k. I think 100k would be a nice target number, takes at least two weeks though...

There are already some trends visible, especially if you add the average score per parameter value (the green line). The average might actually be a better indicator when using purely random samples.

https://i.postimg.cc/pXbKp8cS/mdegrain-random-th-SAD.png

Dogway

21st February 2022, 01:00

This is my highest with overlap=8
146.20343 235920 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dct=0 dctre=6 searchAlgo=1 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true
Just changed overlap for your highest score.

The sectioned 'action' area is going to be shifted to the right to the same amount as highest time minus lowest time of the 'action' area. So all slices have the same horizontal dimension (and enough room) and minimizes bias.
The 'validation' is only to take my samples instead of merging them manually while trying to map time and scores. You don't save time but do save randomness.

zorr

5th March 2022, 21:23

Zopti v1.2.3 is ready. This one has a new conversion feature to help importing Zopti data into other software. Also improvements to -vismode line and some other minor things. Details below:

new mode: convert
-converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
-writes a new file and doesn't remove the original
-does not overwrite an existing file (error message given in such situation)
-supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
-sort direction can be specified by adding a + or - after the field name, for example "-sort time-" sorts by time, descending
-ascending direction (+) is the default if no + or - is given
-multiple sort criterias supported, for example "-sort dct GMSD-" sorts by dct (ascending) and the rows with the same dct value by GMSD (descending)
-output format: CSV
-specified with -format csv or by not giving format option
-target file name is the same as the original .log file but file ending changes from .log to .csv
-csv file includes title row with the names of result values and optimized parameters
-can be opened directly (for example) in Excel
-uses the default decimal format of the OS with floating point values
-output format: zopti
-specified with -format zopti
-keeps the original Zopti format and header rows, only sorts the results when option -sort is given
-filename is "converted " + original file name
-example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
-example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
improvements to visualization mode: line
-now supports option -types which controls which aggregate lines are displayed in the chart
-valid values are "best", "worst", "average", "median", "count" and "samples"
-best = best result per parameter value
-worst = worst result per parameter value
-average = average result per parameter value
-median = median result per parameter value
-count = number of samples per parameter value
-samples = display all results as small dots
-separate multiple values with space, for example "-types best average"
-default value is "best count" which will display the best value per parameter value and the number of samples per parameter value
-a legend is shown if more than one type is selected
display of option -groupby improved, pareto fronts can no longer be cut off the screen
option -continue also supported with mutation algorithm
-but only for the generation of the initial random population
-useful if you only want to test independent random samples with a large population
validation mode behavior change: it now validates all the results in the log file and not just the pareto front

Download link updated at the first post.

Ceppo

6th March 2022, 11:14

In love with the project, waiting for the documentation!

zorr

9th March 2022, 01:33

Dogway, you can now download 100k random mdegrain results (https://drive.google.com/file/d/11dvn95o5j7e5F8WArh77fO_d4PJXttbR/view?usp=sharing). :) The best result was surprisingly good 146.22812.

https://i.postimg.cc/fbrPntM8/mdegrain-100k.png

I'm working on a new sample histogram visualization, here's an example of parameter thSAD:

https://i.postimg.cc/FzQDN08V/mdegrain-100k-samplehist.png

Dogway

9th March 2022, 09:16

Thanks! Nice convex graph!

Here are the isolated action areas (https://www.mediafire.com/file/icjrwezwarsm5yc/mdegrain_100k_-_ACTION_AREAS.zip/file) if you also want to run some tests. And here the Excel chart (https://www.mediafire.com/file/5ox04pviij5h5tg/ZORR_100K.xlsx/file).
Initially I was only going to work with the action area between convex inflection and convex top, but that yielded only 60 samples, so I also include the expanded action area (blue shaded (https://forum.doom9.org/showthread.php?p=1963517#post1963517)) between convex bottom and convex top, this results in 683 samples which is better to work with. The GMSD of the bottom sample is 145.967 so I guess it's fine enough for the correlations to still be meaningful.

As anchor points I used:
146.08434 34410 TOP
146.16876 47800 AREA1 BOTTOM
145.9666 28280 AREA2 BOTTOM

zorr

16th March 2022, 01:16

Dogway, I took your action area 2 samples (683 results) and decided to do a correlation test of my own. I was curious how much the results change if we crop the video by 10%, 20%, 30% etc. The cropping is done by taking the same amount on each edge, so for example crop by 10% takes 5% from left, right, top and bottom edges. Also important to set downsample=true parameter to GMSD, otherwise it will automatically be false when cropping drops the resolution to SD (happens at 50% or more cropping). I calculate the correlation between original and cropped results and also the average time. The cropping was done within the script so there's a small performance hit from the cropping itself which could be avoided.

Here's the results:

https://i.postimg.cc/mDg2ksQy/crop-correlation.png

crop correlation average GMSD average time time %
no crop 1,0000 146,0399465 48027,01 100,00 %
10 % 0,9935 145,9842366 40753,73 84,86 %
20 % 0,9849 145,9095277 33480,41 69,71 %
30 % 0,9751 145,8502522 26815,10 55,83 %
40 % 0,9479 145,8150252 21069,68 43,87 %
50 % 0,8983 145,8019399 16655,15 34,68 %
60 % 0,8704 145,7848522 14720,74 30,65 %
70 % 0,8553 145,7376085 14241,59 29,65 %

Above 0,7 correlation is already considered strong so these are all pretty good but I would be more comfortable with something like above 0,97 correlation which would already give a nice 44% performance improvement using 30% crop, or perhaps even 40% with 0,95 correlation and more than twice as fast.

Here's what crops 10%, 20%, 30% and 70% look like in a scatter plot:

https://i.postimg.cc/NfDtyVjN/crop-correlation-scatter.png

You can download an Excel with all the calculations here (https://docs.google.com/spreadsheets/d/1KJw2PwpLt11Nv5r1XwTJPaVb4h-RwMmR/edit?usp=sharing&ouid=106586920922614226357&rtpof=true&sd=true).

Dogway

16th March 2022, 21:30

Cropping to save time right? 10% already achieves a nice perf up, 30% looks like the limit. This can come very handy when optimizing for MFlow. I actually cropped my interpolation sample to the "motion area", to maximize the worst scenario case.

Currently I'm finishing SceneChange and SceneStats filter and after that I'm free to dedicate the needed time to work on the correlations, I plan to implement, as I fear it's not already in any Python package, the CoS (https://www.researchgate.net/publication/318889643_A_Copula_Statistic_for_Measuring_Nonlinear_Dependence_with_Application_to_Feature_Selection_in_Machine_Learning) (Copula Statistic) correlation which seems to be the state-of-the-art algo for multivariate analysis.

zorr

17th March 2022, 00:02

Yes, cropping could be used to save time. I'm also going to investigate how to select the most important frames, again saving time. I have a theory that the frames with the largest variance in GMSD should be used. Zopti could first determine the variance and then select the frames based on that.

Dogway

22nd March 2022, 09:52

That's certainly true. But it just happens that MFlowInter and MFlowFps have an unfortunate looking failure case, which the MCompensate doesn't have. So even with (actually, especially with) bad vectors MCompensate will look better. Of course you can't use MCompensate to generate an inbetween frame, it can only recreate a complete frame using other frames.

But if we first create the inbetween frame with MFlowInter / MFlowFps and then use MCompensate to reconstruct that frame using two nearby frames something magical happens...

https://i.postimg.cc/wvDbnFMP/mflowinter-mcompensate.gif

Here we see the original frame (orig), a reconstructed frame from MFlowInter (inter) and finally the MCompensated frame (final).

Interesting, I would like to know more about it, how you mix MFlowFPS output with MCompensate, is that with a badvectors mask?

I have been a bit busy with latest repo scripts but I'm almost done, I think I can start with the correlations later this week.
I run some visualizations in -vismode and found it very useful (didn't know how to write the calls). Analyzing only the sectioned area I found the following to be meaningful.
searchAlgo searchRange (optimal)
0 4
1 5
2 3
4 5
5 4

searchAlgoR searchRangeR (optimal)
0 4
1 1
2 6
4 6
5 6

searchAlgo 4 for qual/perf
searchAlgo 2 for qual
searchAlgoR 0 for qual/perf
searchAlgoR 4 for qual
searchRange 3 for qual
searchRangeR0 for qual

dct 0 for qual
dctre 4~5 for qual

pglobal 4 for qual/perf
pglobal 6 for qual
# although 10~13 achieved the best next results

plevel 0 for qual
sglobal true for qual

temporal true for qual/perf
temporal false for qual
pnew 20~36 for qual (and prolly qual/perf)

lambda 100-287 for qual/perf
lambda 664-851 for qual
badSAD 1798-2095 for qual and qual/perf

thSAD 250-290 for qual and qual/perf (in this clip)
lsad 50-844 is the most constant best

These are loose but safe observations given the results.
For future tests we can lock plevel, sglobal, and probably temporal, dct and dctre.
Others like pzero, pnew, searchRangeFinest, etc need to wait for the correlations.

One Zopti option I couldn't make to work was -mapres 200x200, it always opens at max screen height but only for -vismode. I use jdk-16.0.1 on Win7.

Also something looked iffy with -vismode line, checking on 'dct':
zopti -mode evaluate -autorefresh false -vismode line -param dct -mapres 200x200 -log "results.log"
https://i.imgur.com/lf5CNOIl.png

It shows values 0, 1, 2, 3, 4, 5 and 6, but we skipped dct=1 and also included up to 9.

-------------------

On another note when running an optimization I get this error after population count runs out:
Parameter sensitivity estimation with 256 result combinations
java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
at avisynthoptimizer.Parameter.updateAverages(Parameter.java:954)
at avisynthoptimizer.Parameter.updateSensitivityEstimation(Parameter.java:972)
at avisynthoptimizer.SensitivityEstimation.estimateParameterSensitivity(SensitivityEstimation.java:102)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:315)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:38)
at org.uma.jmetal.util.evaluator.impl.MultithreadedSolutionListEvaluator.lambda$evaluate$1(MultithreadedSolutionListEvaluator.java:36)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(UnknownSource)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

I use the last avs template we came to use with the next batch script:
SET RUNS= 1
SET ALG= spea2
SET POPULATION= 8
SET INITIAL= script
SET ITERS= dyn
SET MUTAMOUNT= 0.3 0.01
SET MUTCOUNT= 60% 1
SET CROSSPROB= 0.1
SET CROSSDIST= 20
SET SENSITIVITY= TRUE
SET DYNPHASES= 1
SET DYNITERS= 8
SET ERRORS= STOP
SET PRIORITY=
SET THREADS= 8
SET TIMEOUT= 1440
SET CONTINUE=

java -jar Zopti.jar "C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.2\Tests\Zopti_test2s - MDegrain7 - Random.avs" ^
-alg %ALG% -initial %INITIAL% -sensitivity %SENSITIVITY% -iters %ITERS% -dyniters %DYNITERS% -dynphases %DYNPHASES% -pop %POPULATION% -runs %RUNS% ^
-mutcount %MUTCOUNT% -mutamount %MUTAMOUNT% -timeout %TIMEOUT% -threads %THREADS%

I tested with several options, halving threads, doubling population, etc. With both Zopti 1.2.2 and 1.2.3.

zorr

24th March 2022, 00:36

Interesting, I would like to know more about it, how you mix MFlowFPS output with MCompensate, is that with a badvectors mask?

First you create two new frames with MCompensate, one from the previous original frame (Mprev) and one from the next original frame (Mnext). The target frame for both is the new interpolated frame (inter) in between them created by MFlowFPS. Then you create a pixel mask where luma is 255 where difference of Mprev and inter is smaller than the difference of Mnext and inter. You could use this mask directly to take pixels from Mprev and Mnext but it looks better if the mask is processed a little bit. I have tried downscaling and upscaling back to original size with point resize so that bigger blocks are selected and also blurring so that the edges are not harsh. This is something that Zopti could optimize but I never got very far with that experiment.

I'll post my old script in the next post (too large to fit here).

One Zopti option I couldn't make to work was -mapres 200x200, it always opens at max screen height but only for -vismode. I use jdk-16.0.1 on Win7.

-mapres is meant for -vismode heatmap only and it sets the maximum resolution of the heat map, not in pixels but in how many rows and columns it has. If you want to see the visualizations in specific size you can use the -shot parameter, for example -shot 200x200 creates a screenshot with that size (or approximately that, it also depends on the natural dimensions of the chart).

Also something looked iffy with -vismode line, checking on 'dct':
...
It shows values 0, 1, 2, 3, 4, 5 and 6, but we skipped dct=1 and also included up to 9.

Yes that's a bug I'm aware of. It's showing the internal mapping of the values into continuous range. I have tried to display the original labels but there seems to be a bug with the XChart library and it's not working the way it's supposed to.

On another note when running an optimization I get this error after population count runs out:

Thanks for the report, I will try to reproduce it.

zorr

24th March 2022, 00:38

Here's an old script I used to test the MCompensate concept. It's using Corrector plugin and I'm not sure if there's a 64 bit version, you can easily replace it with Expr (the code is commented so you know what it's supposed to do).
The code is old and ugly, didn't spend time cleaning it up. :)

TEST_FRAMES = 10
MIDDLE_FRAME = 50

# source clip
AVISource("d:\process2\1 deinterlaced.avi")

# original framerate
FPS_NUM = 50
FPS_DEN = 1
AssumeFPS(FPS_NUM, FPS_DEN)

#return last

# needed for some parameter combinations
#ConvertToYV24()
ConvertToYV16() # RemoveGrain only works in planar colorspaces

orig = last

weight = 1.0/3.0
avg = Average(trim(orig,2,0), weight, trim(orig,1,0), weight, orig, weight)
#return avg

# preprocessing - blur and/or degrain
#searchClip = orig
#searchClip = last.Blur(1.58).Blur(1.58)
searchClip = RemoveGrain(orig, 16)
searchClip = RemoveGrain(searchClip, 23)

super_pad = 16 #- optimize super_pad = _n_ | 0,4,6,8,12,16,18,24,32,48 | super_pad
super_pel = 4 #- optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 #- optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 4 #- optimize super_rfilter = _n_ | 0..4 | super_rfilter
super_search = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, searchClip)
super_render = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, orig, levels=1)

blockSize = 8 #- optimize blockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 4 #- optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 1 #- optimize searchRange = _n_ | 1..30 | searchRange
searchRangeFinest = 12 #- optimize searchRangeFinest = _n_ | 1..60 | searchRangeFinest
lambda = 830*(blockSize*blockSize)/(8*8) #- optimize lambda = _n_ | 0..20000 ; filter:x 10 % 0 == | lambda
lsad=7752 #- optimize lsad=_n_ | 8..20000 ; filter:x 8 + 10 % 0 == | LSAD
pnew=0 #- optimize pnew=_n_ | 0..256 | pnew
plevel=2 #- optimize plevel=_n_ | 0..2 | plevel
overlap=0 #- optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=0 #- optimize overlapv=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlapv
divide=0 #- optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true #- optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 5836 #- optimize badSAD = _n_ | 4..10000 ; filter:x 4 + 10 % 0 == | badSAD
badRange = 28 #- optimize badRange = _n_ | 4..100 | badRange
badAlgo = 0 #- optimize badAlgo = _n_ | 0,1 | badAlgo
meander = false #- optimize meander = _n_ | false,true | meander
temporal = true #- optimize temporal = _n_ | false,true | temporal
trymany = true #- optimize trymany = _n_ | false,true | trymany
dct = 5 #- optimize dct = _n_ | 0,1,2,3,4,5,6,7,8,9,10 D | dct
scaleCSAD = 0 #- optimize scaleCSAD = _n_ | -2..2 | scaleCSAD

delta = 1
useChroma = true
badRange = (badAlgo == 0 ? badRange : -badRange)
bv = MAnalyse(super_search, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
fv = MAnalyse(super_search, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)

threshold = 16320 # max for thSCD1
maskScale = 1 #- optimize maskScale = _n_ | 1..600 | maskScale
mask_fps = 2 #- optimize mask_fps = _n_ | 0..2 | mask_fps
inter = orig.MFlowFPS(super_render, bv, fv, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold, thSCD2=100)

### mcompensate pass
interpolated = mcompensate_pass(inter.ConvertToYV16(), orig)
### mcompensate pass

#return Interleave(inter.selectOdd().ConvertToYUY2(), interpolated) # compare result from MFlowFPS and MCompensate
#return Interleave(orig.ConvertToYUY2(), interpolated) # MFlowFPS + MCompensate final clip

fps_only = interpolated #inter.SelectOdd()

# second pass
super_search3 = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only)
super_render3 = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only, levels=1)
bv3 = MAnalyse(super_search3, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
fv3 = MAnalyse(super_search3, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
inter3 = fps_only.MFlowFPS(super_render3, bv3, fv3, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold, thSCD2=100)

### second mcompensate pass
interpolated2 = mcompensate_pass(inter3.ConvertToYV16(), orig)
### second mcompensate pass

#return Interleave(trim(orig,1,0), interpolated2.ConvertToYV16()) # compare original and twice interpolated frames

fps_only2 = interpolated2 # inter2.SelectOdd()

delimiter = "; "

inter_yv12 = fps_only2.ConvertToYV12()
orig_yv12 = orig.ConvertToYV12()
avg_yv12 = avg.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
avg_yv12 = avg_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

global total = 0.0
global ssim_avg = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim_avg = SSIM_FRAME(inter_yv12, avg_yv12)
ssim = (ssim > ssim_avg) ? ssim : 0.0
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig_yv12, inter_yv12, avg_yv12, delta, frame_count")

# NOTE: AvsTimer call should be before the WriteFile call
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
resultFile = "perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

function mcompensate_pass(clip input, clip orig) {

searchClip2 = RemoveGrain(input, 19) #- optimize RemoveGrain(input, _n_) | 11,12,19,20 D | mrmgrain
searchClip2 = RemoveGrain(searchClip2, 0) #- optimize RemoveGrain(searchClip2, _n_) | 0..24 D | mrmgrain2

msuper_pad = 16 #- optimize msuper_pad = _n_ | 6,8,12,16,18 | msuper_pad
msuper_pel = 4 #- optimize msuper_pel = _n_ | 2,4 | msuper_pel
msuper_sharp = 2 #- optimize msuper_sharp = _n_ | 0..2 | msuper_sharp
msuper_rfilter = 0 #- optimize msuper_rfilter = _n_ | 0..4 | msuper_rfilter
super_search2 = MSuper(hpad=msuper_pad, vpad=msuper_pad, pel=msuper_pel, sharp=msuper_sharp, rfilter=msuper_rfilter, searchClip2)
super_render2 = MSuper(hpad=msuper_pad, vpad=msuper_pad, pel=msuper_pel, sharp=msuper_sharp, rfilter=msuper_rfilter, input, levels=1)

mblockSize = 8 #- optimize mblockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:mdivide 0 > 8 2 ? ; filter:moverlap moverlapv max 2 * x <= | mblockSize
msearchAlgo = 6 #- optimize msearchAlgo = _n_ | 0..7 D | msearchAlgo
msearchRange = 1 #- optimize msearchRange = _n_ | 1..10 | msearchRange
msearchRangeFinest = 1 #- optimize msearchRangeFinest = _n_ | 1..60 | msearchRangeFinest
mlambda = 3290*(mblockSize*mblockSize)/(8*8) #- optimize mlambda = _n_ | 0..40000 ; filter:x 10 % 0 == | mlambda
mlsad=10042 #- optimize mlsad=_n_ | 8..20000 ; filter:x 8 + 10 % 0 == | mLSAD
mpnew=20 #- optimize mpnew=_n_ | 0..256 | mpnew
mplevel=2 #- optimize mplevel=_n_ | 1..2 | mplevel
moverlap=4 #- optimize moverlap=_n_ | 4 | moverlap
moverlapv=4 #- optimize moverlapv=_n_ | 4 | moverlapv
mdivide=0 #- optimize mdivide=_n_ | 0..2 ; max:mblockSize 8 >= 2 0 ? moverlap 4 % 0 == 2 0 ? min | mdivide
mglobalMotion = false #- optimize mglobalMotion = _n_ | false,true | mglobalMotion
mbadSAD = 656 # optimize mbadSAD = _n_ | 4..20000 ; filter:x 4 - 500 % 0 == | mbadSAD
mbadRange = 4*10 # optimize mbadRange = _n_*10 | 1..10 | mbadRange
mbadAlgo = 1 #- optimize mbadAlgo = _n_ | 0,1 | mbadAlgo
mmeander = true #- optimize mmeander = _n_ | false,true | mmeander
mtemporal = true #- optimize mtemporal = _n_ | false,true | mtemporal
mtrymany = false #- optimize mtrymany = _n_ | false,true | mtrymany
mdct = 5 #- optimize mdct = _n_ | 0,2,5,6,9,10 D | mdct
mscaleCSAD = -2 #- optimize mscaleCSAD = _n_ | -2..2 | mscaleCSAD

delta = 1
useChroma = true
mbadRange = (mbadAlgo == 0 ? mbadRange : -mbadRange)
bv2 = MAnalyse(super_search2, isb = true, blksize=mblockSize, search=msearchAlgo, searchparam=msearchRange, pelsearch=msearchRangeFinest, chroma=useChroma, delta=delta, lambda=mlambda, lsad=mlsad, pnew=mpnew, plevel=mplevel, global=mglobalMotion, overlap=moverlap, overlapv=moverlapv, scaleCSAD=mscaleCSAD, divide=mdivide, badSAD=mbadSAD, badrange=mbadRange, meander=mmeander, temporal=mtemporal, trymany=mtrymany, dct=mdct)
fv2 = MAnalyse(super_search2, isb = false, blksize=mblockSize, search=msearchAlgo, searchparam=msearchRange, pelsearch=msearchRangeFinest, chroma=useChroma, delta=delta, lambda=mlambda, lsad=mlsad, pnew=mpnew, plevel=mplevel, global=mglobalMotion, overlap=moverlap, overlapv=moverlapv, scaleCSAD=mscaleCSAD, divide=mdivide, badSAD=mbadSAD, badrange=mbadRange, meander=mmeander, temporal=mtemporal, trymany=mtrymany, dct=mdct)

threshold = 16320 # max for thSCD1
f1 = MCompensate(input, super_render2, fv2, thSCD1=threshold, thSAD=threshold)
b1 = MCompensate(input, super_render2, bv2, thSCD1=threshold, thSAD=threshold)

#return StackHorizontal(inter, f1, b1)
#return Interleave(inter.SelectOdd().Subtitle("inter"), f1.SelectOdd().Subtitle("f1"), b1.SelectOdd().Subtitle("b1"))
#return inter.SelectOdd()

inter = input.SelectOdd()
f1 = f1.SelectOdd()
b1 = b1.SelectOdd()

# Corrector needs YUY2
f1c = f1.ConvertToYUY2()
b1c = b1.ConvertToYUY2()
interc = inter.ConvertToYUY2()
origc = orig.ConvertToYUY2()

# take pixel from the frame (b2 or f2) whose value is closest to inter (processing Y U and V individually)
comb = Corrector(interc, interc, b1c, f1c, b1c, f1c, mode=1, th=255)

#f1 = f1.ConvertToYV16()
#b1 = b1.ConvertToYV16()
comb_yv16 = comb.ConvertToYV16()

# create masks where value is 255 when combined pixel has the same value as source
mask_f1 = mt_lutxy(f1, comb_yv16, yexpr=mt_polish("((x==y)?255:0)"), U=-128, V=-128)
mask_b1 = mt_lutxy(b1, comb_yv16, yexpr=mt_polish("((x==y)?255:0)"), U=-128, V=-128)

# downscale masks, essentially generate averages for how many times f1 and b1 had correct colors
downScale = 0 #- optimize downScale = _n_ | 0,2,4,8,16,32 | downScale
scaled_f1 = downscale(mask_f1, orig, downScale)
scaled_b1 = downscale(mask_b1, orig, downScale)

# create new mask with value 255 where scaled_f1 mask has larger value and value 0 otherwise
mask_combined = mt_lutxy(scaled_f1, scaled_b1, yexpr=mt_polish("((x>y)?255:0)"), U=-128, V=-128)

# refine mask: remove single pixels where all 8 neighbours are opposite color
# phase 1: calculate average of 9x9 neighbourhood
mask_combined_1 = mask_combined.ConvertToYV16().mt_edge(mode="1 1 1 1 1 1 1 1 1 9", thY1=0, thY2=255, Y=3, U=-128, V=-128)
# phase 2: change pixel color to opposite if average is in certain range
mask_filled = mt_lutxy(mask_combined, mask_combined_1, yexpr=mt_polish("((x==255)?((y<30)?0:255):((y>225)?255:0))"), U=-128, V=-128)

# scale mask back to original size
mask_combined = mask_combined.PointResize(orig.Width, orig.Height)
mask_filled = mask_filled.PointResize(orig.Width, orig.Height)

#return Interleave(mask_combined, mask_filled)
#return mask_combined

mask_final = mask_filled

#mask_f1 = mask_f1.ConvertToYUY2()
#mask_b1 = mask_b1.ConvertToYUY2()
mask_final = mask_final.ConvertToYUY2()

blurAmount = 155*10 #- optimize blurAmount = _n_*10 | 101..200 | blurAmount
# mask_final = blurrr(mask_final, blurAmount)
mask_final = mask_final.GaussianBlur(varY=blurAmount)

#return mask_final

# create final result by combining b1 and f1 with the mask
interpolated = Overlay(b1, f1, mask=mask_final, output="YUY2")

#final = Interleave(origc.Subtitle("orig"), f1c.Subtitle("f1"), b1c.Subtitle("b1"), comb.SubTitle("comb"), mask_f1.Subtitle("mask f1"), mask_b1.Subtitle("mask b1"),mask_final.Subtitle("final mask"), interc.Subtitle("inter"), mask_combined.Subtitle("final"))
final = Interleave(origc.Subtitle("orig"), interc.Subtitle("inter"), interpolated.Subtitle("final"))
#return mask_final
#return interc
return interpolated
#return final

}

function downscale(clip c, clip orig, int scale, int "currScale") {
String SSS = """
currScale = Default(currScale, 2)
targetWidth = orig.Width/currScale
targetWidth = (targetWidth % 2 == 1) ? targetWidth+1 : targetWidth
c = c.BilinearResize(targetWidth, orig.Height/currScale)
return (currScale==scale) ? c : downscale(c, orig, scale, currScale*2)
"""
return scale >= 2 ? Eval(SSS) : c
}

function blurrr(clip c, int times) {
String SSS = """
c = c.Blur(1.58)
return times==1 ? c : blurrr(c, times-1)
"""
return times>0 ? Eval(SSS) : c
}

Dogway

27th March 2022, 10:37

Thanks for the script, I will try to clean it up and add into comparisons for the interpolation tests.

I ran a test to confirm my above assertions that we can lock plevel, pnewR, temporal, sglobal, dct, and dctre. I took all my best scores from pareto of your 100K test and also some internal tests and simply swapped those values, then ran a validation. In all cases but one (it preferred dctre=7) I got same or better score with times same or better in average, the "hyphen" depicts a new pareto front.

INPUT
146.1719 56870 thSAD=290 thSADR=275 temporal=TRUE lambda=680 lambdaR=1420 pnew=38 pnewR=0 pzero=22 lsad=3900 plevel=0 pglobal=13 badrange=43 badSAD=1860 dct=0 dctre=5 searchAlgo=2 searchAlgoR=5 searchRange=3 searchRangeR=6 searchRangeFinest=0 sglobal=TRUE
146.13617 41180 thSAD=285 thSADR=195 temporal=TRUE lambda=260 lambdaR=1820 pnew=20 pnewR=0 pzero=64 lsad=7500 plevel=0 pglobal=12 badrange=49 badSAD=2230 dct=0 dctre=5 searchAlgo=4 searchAlgoR=4 searchRange=2 searchRangeR=5 searchRangeFinest=19 sglobal=TRUE
146.1149 37630 thSAD=300 thSADR=250 temporal=TRUE lambda=420 lambdaR=2360 pnew=22 pnewR=0 pzero=86 lsad=2500 plevel=0 pglobal=5 badrange=44 badSAD=1470 dct=0 dctre=5 searchAlgo=0 searchAlgoR=0 searchRange=1 searchRangeR=2 searchRangeFinest=6 sglobal=TRUE
146.08434 34410 thSAD=215 thSADR=295 temporal=TRUE lambda=200 lambdaR=1800 pnew=48 pnewR=0 pzero=40 lsad=1250 plevel=0 pglobal=5 badrange=26 badSAD=1920 dct=0 dctre=5 searchAlgo=4 searchAlgoR=0 searchRange=3 searchRangeR=1 searchRangeFinest=6 sglobal=TRUE
146.00838 29060 thSAD=285 thSADR=205 temporal=TRUE lambda=240 lambdaR=1340 pnew=52 pnewR=0 pzero=192 lsad=7850 plevel=0 pglobal=19 badrange=20 badSAD=1740 dct=0 dctre=5 searchAlgo=0 searchAlgoR=1 searchRange=0 searchRangeR=0 searchRangeFinest=9 sglobal=TRUE
146.15271 376230 thSAD=260 thSADR=210 temporal=TRUE lambda=480 lambdaR=1660 pnew=86 pnewR=0 pzero=42 lsad=1850 plevel=0 pglobal=12 badrange=49 badSAD=1950 dct=0 dctre=5 searchAlgo=2 searchAlgoR=1 searchRange=16 searchRangeR=13 searchRangeFinest=0 sglobal=TRUE
146.19511 107890 thSAD=245 thSADR=220 temporal=TRUE lambda=240 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=15 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
146.20343 235920 thSAD=245 thSADR=220 temporal=TRUE lambda=250 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=17 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=1 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
-146.19547 68920 thSAD=245 thSADR=220 temporal=TRUE lambda=260 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=13 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE

OUTPUT
-146.18489 60372 thSAD=290 thSADR=275 temporal=TRUE lambda=680 lambdaR=1420 pnew=38 pnewR=0 pzero=22 lsad=3900 plevel=0 pglobal=13 badrange=43 badSAD=1860 dct=0 dctre=5 searchAlgo=2 searchAlgoR=5 searchRange=3 searchRangeR=6 searchRangeFinest=0 sglobal=TRUE
146.17502 49104 thSAD=285 thSADR=195 temporal=TRUE lambda=260 lambdaR=1820 pnew=20 pnewR=0 pzero=64 lsad=7500 plevel=0 pglobal=12 badrange=49 badSAD=2230 dct=0 dctre=5 searchAlgo=4 searchAlgoR=4 searchRange=2 searchRangeR=5 searchRangeFinest=19 sglobal=TRUE
-146.17696 37867 thSAD=300 thSADR=250 temporal=TRUE lambda=420 lambdaR=2360 pnew=22 pnewR=0 pzero=86 lsad=2500 plevel=0 pglobal=5 badrange=44 badSAD=1470 dct=0 dctre=5 searchAlgo=0 searchAlgoR=0 searchRange=1 searchRangeR=2 searchRangeFinest=6 sglobal=TRUE
145.85498 39661 thSAD=215 thSADR=295 temporal=TRUE lambda=200 lambdaR=1800 pnew=48 pnewR=0 pzero=40 lsad=1250 plevel=0 pglobal=5 badrange=26 badSAD=1920 dct=0 dctre=5 searchAlgo=4 searchAlgoR=0 searchRange=3 searchRangeR=1 searchRangeFinest=6 sglobal=TRUE
-146.12015 31993 thSAD=285 thSADR=205 temporal=TRUE lambda=240 lambdaR=1340 pnew=52 pnewR=0 pzero=192 lsad=7850 plevel=0 pglobal=19 badrange=20 badSAD=1740 dct=0 dctre=5 searchAlgo=0 searchAlgoR=1 searchRange=0 searchRangeR=0 searchRangeFinest=9 sglobal=TRUE
146.16965 136785 thSAD=260 thSADR=210 temporal=TRUE lambda=480 lambdaR=1660 pnew=86 pnewR=0 pzero=42 lsad=1850 plevel=0 pglobal=12 badrange=49 badSAD=1950 dct=0 dctre=5 searchAlgo=2 searchAlgoR=1 searchRange=16 searchRangeR=13 searchRangeFinest=0 sglobal=TRUE
146.19472 68707 thSAD=245 thSADR=220 temporal=TRUE lambda=240 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=15 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
-146.197 115782 thSAD=245 thSADR=220 temporal=TRUE lambda=250 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=17 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=1 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
146.19452 69217 thSAD=245 thSADR=220 temporal=TRUE lambda=260 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=13 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE

This is only an insight for optimizations, for correlations I need all the values to draw a bigger picture, will start tomorrow.

For some reason the error I reported "fixed" after I changed the variable ERROR (https://forum.doom9.org/showthread.php?p=1966188#post1966188)from STOP to CONTINUE, despite not being called in the Zopti line.

zorr

29th March 2022, 22:37

Thanks for the script, I will try to clean it up and add into comparisons for the interpolation tests.

This is something I also want to do... whenever I get the time. :) I've been thinking about a kind of MVToolsHQ library with focus on quality and using every trick in the book, optimized by Zopti.

I ran a test to confirm my above assertions that we can lock plevel, pnewR, temporal, sglobal, dct, and dctre. I took all my best scores from pareto of your 100K test and also some internal tests and simply swapped those values, then ran a validation. In all cases but one (it preferred dctre=7) I got same or better score with times same or better in average, the "hyphen" depicts a new pareto front.

That's good to know, eliminating some parameters from the optimization always helps.

Meanwhile I've got some really promising results on the selection of frames. Looks like we can throw away 60% of the frames and still have over 0.99 correlation.

Dogway

2nd April 2022, 11:11

This is something I also want to do... whenever I get the time. :) I've been thinking about a kind of MVToolsHQ library with focus on quality and using every trick in the book, optimized by Zopti.

Yep, so far our tests have been quite focused, truemotion=true, refinemotion=true. But I think this is fine in order to win small battles first. For example I noticed that for copulas I shouldn't mix continuous with discreet distributions so I also need to lock those args that are boolean or with a narrow range like DCT.

I also found that the CoS algo uses a gaussian copula, I don't know if it's "ok" to change that distribution type but I feel more confident using a separate algo, currently looking on T-copula as our samples (slices in the section area) are rather few.

In any case I'm going to lock those and a few more args and run a test so I can actually use all the samples (I presume most will be confined around the original 'action' area).

There are also some/many args that depend on motion/vector length. This is a double-edged sword, default for high motion or for low motion? You can't have both so in my opinion for MDegrain it's better to default for low motion as some of my visual tests have shown.

Ceppo

3rd April 2022, 14:53

@zorr
Does your creation allow to choose the best trade off between speed/quality or it's just about quality?

EDIT:
I need to speed up CQTGMC (https://forum.doom9.org/showthread.php?p=1963693#post1963693) while losing as less precision as possible.

EDIT:
I was thinking to speed up the process you could try my CDuplicate + CReplaceDuplicate(drop=true) to remove duplicate or frames very similar to each other so that only frames with a strong differences are checked.

zorr

4th April 2022, 22:01

@zorr
Does your creation allow to choose the best trade off between speed/quality or it's just about quality?

It finds the "pareto front" of results, which contains all the best combinations in speed/quality. For example in this image of 100k results the red dots represent the different tradeoffs between speed and quality. The fastest result is the leftmost dot and the one with highest quality is the topmost. The other red dots are other non-dominated combinations which are "best" in their own way. They're basically the best quality you can get using this much time.

https://i.postimg.cc/1XNx45Zh/mdegrain-100k-edit.png

EDIT:
I need to speed up CQTGMC (https://forum.doom9.org/showthread.php?p=1963693#post1963693) while losing as less precision as possible.

Optimizing QTGMC (or a variation of it) directly is probably too hard, as Zopti needs some kind of reference frames to determine the quality. In theory you could make a reference by having a progressive video which you then interlace and throw away half the fields and then try to reconstruct the original frames with QTGMC, but since the algorithm also does noise removal the similarity metric is affected negatively by that.

What you can do however is to improve MVTools which QTGMC is mostly based on, if you can find better settings for MVTools it will also improve the quality and/or speed of QTGMC.

EDIT:
I was thinking to speed up the process you could try my CDuplicate + CReplaceDuplicate(drop=true) to remove duplicate or frames very similar to each other so that only frames with a strong differences are checked.

That might work but we can't remove frames freely, in order to do proper motion compensation MVTools needs two previous and two next frames also. I'm studying how to remove the insignificant frames by trying all combinations of them and comparing the result to the original calculation which uses all frames. Of course the same limitation on which frames can be removed applies there as well.

zorr

4th April 2022, 22:31

Yep, so far our tests have been quite focused, truemotion=true, refinemotion=true.

Truemotion is just a preset for variables lambda, lsad, plevel, pnew and global. We're optimizing all of those individually so the preset value has no effect.

What's refinemotion?

But I think this is fine in order to win small battles first. For example I noticed that for copulas I shouldn't mix continuous with discreet distributions so I also need to lock those args that are boolean or with a narrow range like DCT.

Definitely you have to start from the small battles as even they can be overwhelming at first. You could also consider all distributions discrete since Zopti doesn't support anything else. All the real-valued parameters need to be discretized.

But some of the parameters are also non-continuous, like the dct you mentioned. There's no expectation that changing dct one bigger will change the results in certain direction as it's simply switching to a different algorithm. Those might cause problems with any statistical methods. I haven't read about copulas before so I don't know how those behave though.

currently looking on T-copula as our samples (slices in the section area) are rather few.

Well nothing's stopping you from using more samples, there are 100k of them after all. :) I do get that you want to focus on the higher quality results but even there you could widen the range and get more statistically robust results using more samples. Or do several tests using different amount of samples.

In any case I'm going to lock those and a few more args and run a test so I can actually use all the samples (I presume most will be confined around the original 'action' area).

Ok, let me know when you find a good range, I can provide more results.

There are also some/many args that depend on motion/vector length. This is a double-edged sword, default for high motion or for low motion? You can't have both so in my opinion for MDegrain it's better to default for low motion as some of my visual tests have shown.

Yes for noise removal to be effective relatively low motion is required (and noticing the noise in high motion frames is much more difficult anyway).

I stumbled upon this new correlation algorithm called "distance correlation (https://en.wikipedia.org/wiki/Distance_correlation)" which can reveal not just linear dependencies but pretty much any kind of relationship. That could also be useful metric.

Dogway

22nd April 2022, 09:45

I had a break to do some Zopti and finally did a random run with the locked params, all in all only optimizing the next:
lambda,lambdaR,pnew,pnewR,pzero,lsad,pglobal,badSAD,badrange

Very interesting results:
https://i.imgur.com/fbX0Hml.png

That diagonal blob happened by chance showing a strong relationship between the params so it looks like they are all mostly correlated in some way which makes it perfect to construct qual/perf trade-off presets.

I don't know what kind of distributions they follow, actually looking at "-vismode line" it doesn't show any pattern whatsoever even in the original sectioned AREA so I will do it "by hand" with a scatter plot matrix. 1K is not much so I will go up to around 3K.

First correlate obvious params; lambda with lambdaR, pnew to pnewR, badSAD to badrange, and all the penalties. lsad might be also related to lambda. Prune outliers and finally correlate all of them.

After this I will run again a clean 3K random run with a synthetic clip by adding grain with cretindesalpes' new plugin and reassess the results.

At the end I will add in the variables, everything should be automatized by 4 variables; thSAD, tr, clip/scene motion and qual/perf preset.
There are settings that depend on other things but they should correlate with the above 4, for example badrange is in pixel units, therefore depends on input_size*pel, same for searchRange and searchRangeFinest. lambda depends on blksize, such that by default is 1000*blksize*blksizeV/64, with 1000 being substituted with my correlation function which depends on thSAD and probably lsad.

I'm creating now a global clip stats parse filter from the output of SceneStats() log file, so one can know how much motion has the whole clip (in average) so people can set up better MVTools settings. A per-scene is also possible but some people might not want to run MVTools in a runtime environment.
A SAD analysis tool is also in the works, would like to do with expressions since MVTools is so slow.

EDIT: I was testing the new additions on -vismode line and I find them great, specially the average mode. Typically I like median more as it's more robust but it creates spikes. Do you think IQM could be added? it mixes the best of both worlds. Simply create a ranking order, prune the the first and last 25% of the samples and calculate the average of the remaining. There are softer versions though like the Trimean or Winsorized mean.

zorr

23rd April 2022, 21:30

I had a break to do some Zopti and finally did a random run with the locked params
...
everything should be automatized by 4 variables; thSAD, tr, clip/scene motion and qual/perf preset

Looks interesting and hopefully you'll find those correlations.

After this I will run again a clean 3K random run with a synthetic clip by adding grain with cretindesalpes' new plugin and reassess the results.

Yes that's a good idea also, looks like the grain is very realistic and we can have a "perfect" reference clip.

I'm creating now a global clip stats parse filter from the output of SceneStats() log file, so one can know how much motion has the whole clip (in average) so people can set up better MVTools settings.

Would be interesting to figure out the best settings automatically based on the scene stats. That would involve a lot of testing though.

EDIT: I was testing the new additions on -vismode line and I find them great, specially the average mode. Typically I like median more as it's more robust but it creates spikes. Do you think IQM could be added?

Sure, IQM sounds very simple. There are some subtle rules when the number of results is not divisible by 4 but those are not a problem.

anton_foy

23rd April 2022, 23:21

Yes that's a good idea also, looks like the grain is very realistic and we can have a "perfect" reference clip.
Just made a simulation of a high ISO uhd clip with and without noise if someone is interested. I recorded a clean plate with very high ISO in low light to isolate as noise and then applied it onto a clean noise free clip and matched the noise to another real noisy clip.

DTL

24th April 2022, 07:07

For low self-noise silicon-imaging new cameras most of noise is photon shot. It have poisson distribution on low levels and gauss at medium and high. Film grain may have different distribution. So for degrain/denoise work we need to use different noise simulators. For film intermediate image capturing and direct electronic low self-noise image capturing cameras. The film is some mix of both photon shot noise with film grain.

Dogway

24th April 2022, 09:36

I finished the 3K run, I'm going to plot the scatter plot matrix. I tried showing the optimize animation but all I got was rendering png's to the animation folder. Is there a way to show the animation in the scatter plot window?
I used:
zopti -mode evaluate -autorefresh true -animation 1

I also noticed the CSV output is rounding score to 3 decimals

@DTL: I haven't read the implementation papers but most content that is to denoise use to be bluray films, specially non-digital, so most of the grain would be a mix of shot noise (low light), negative stock grain, and positive print film stock grain. Each stock has its own characteristics but probably targeting a 5207+2383/2393 would work for films in the last 30 years. Grain is better (more filmic) implemented in log space by the way.

EDIT: Here the 3K plot with traced limits
https://i.imgur.com/9l7k6x4m.png (https://i.imgur.com/9l7k6x4.png)

EDIT: Here the scatter plot matrix of the first 6 patches out of 18. Maybe I need more samples.
https://i.imgur.com/qYtSZIAh.png (https://i.imgur.com/qYtSZIA.png)

EDIT: By the way, I took the liberty to optimize bicubic's b and c coefficients and for upscaling it always optimized to Catmull-Rom, although I like Precise better (adds some acutance). For downscaling it optimized to -0.83,0.14 for halving, and -1.00,0.16 for 1080p to 720p. My reference was a tuned SSIM_downsample().