Introducing Zopti (ex AvisynthOptimizer) [Archive] - Page 2

View Full Version : Introducing Zopti (ex AvisynthOptimizer)

Pages : 1 [2] 3 4 5 6 7

zorr

1st October 2018, 21:18

Version 0.9.3-beta (https://drive.google.com/open?id=1qwrfRAij12rz1fXcf-mERSSuKw-7397f) released. The only change is that avsr was upgraded to latest version 0.1.7.

zorr

2nd October 2018, 23:59

It's time to take a closer look at how to adjust the optimization process. Let's run the optimizer using the same script and settings used in the last tutorial:

optimizer <path_to_your_script> -iters 100

The program displays:

Arguments
iters = 100

Running optimization for script d:/optimizer/test/flower/denoise.avs
Using these settings:
ARGUMENT DESCRIPTION VALUE
-runs runs 5
-alg algorithm spea2
-pop population 8
-iters iterations 100
-mutamount mutation amount 0.3 0.01
-mutcount mutation count 60% 1
-crossprob crossover probability 0.1
-crossdist crossover distribution 20
-sensitivity sensitivity estimation true
-dynphases dynamic phases N/A
-dyniters iterations per phase N/A

You can stop the optimization after this text is displayed.

The "Arguments" section lists the arguments and their values as they were understood by the optimizer.

The next section is a handy cheat sheet on what arguments are available and their current values. The first column ARGUMENT tells the argument name you can use to specify the setting. The DESCRIPTION column contains a short description of what the argument does. And finally the VALUE is the current value used for the argument. Most of these are using the default values, we only specified the -iters argument. If you run the optimizer in another mode (like "evaluate") the listed arguments are specific to that mode.

I spent quite a while figuring out good default values so they should work reasonably well, but I have only tested them on a few different optimization tasks so they might not be good for every case. It takes a lot of effort to test these settings because to determine if one value is better than another one should run the optimization task many times with each value in order to gain enough statistical significance. I mostly used 20 runs per parameter value.

Let's take a look at the arguments one by one.

-runs specifies the number of optimization runs. a "run" is one complete optimization cycle which itself is specified with the -iters argument. I talked about the need for multiple runs earlier but I will repeat the points here: Since the optimization process is depending on random numbers the outcome is not always the same and there can be large differences in the final result. If you only run the optimization once you cannot be really sure whether the results are good or bad. Another useful aspect of multiple runs is that the variance of the best result can tell us about how easy or hard this optimization task is. Large variance means difficult task. And if the task is difficult we can try to increase the iterations. I don't have a good answer on how many runs are enough. If you can only run N iterations should you run for example three runs with N/3 iterations or eight runs with N/8 iterations? More iterations is better but more runs is also better, to a point.

-alg specifies the metaheuristic algorithm used in the optimization. Currently there are three options: "nsga-ii", "spea2", "mutation" and "exhaustive". NSGA-II and SPEA2 are very good and well known algorithms. I got slightly better results with spea2 so it's the default. If you're interested in how these algorithms work you should check out the free ebook Essentials of Metaheuristics (https://cs.gmu.edu/~sean/book/metaheuristics/). The third option "mutation" is a very simple algorithm I wrote which only uses mutation. It can find a reasonably good result faster than the other algorithms but it will lose with large iteration counts. Finally we have the "exhaustive" option, it simply tries all the possible (and valid) parameter combinations. It can be useful if you only have a few parameters and can limit the number of values per parameter so that the number of combinations doesn't get too high. I have tried some other metaheuristic algorithms like CMA-ES, BFGS (Broyden–Fletcher–Goldfarb–Shanno) and SMPSO (particle swarm algorithm) but I didn't get as good results with them. The SMPSO is still waiting for a more thorough examination, it is promising. I should also note that the algorithms I'm using are not the basic variations, I have changed the way the mutations work and got better results that way.

-pop specifies the population size which is a term often used with genetic algorithms. It's basically how many individual results are kept in memory during the optimization. The genetic algorithms (like NSGA-II and SPEA2) work by doing crossovers between two individuals and then mutating (randomizing) the results slightly. The crossover operation takes some values from one individual and some from the other. The new individuals are rated and finally the best ones are selected as the new "generation". The default population size of 8 seems very small and maybe you're wondering why it should be small at all, after all it's not a problem to keep thousands or even millions of results in memory. Yes, in theory you should get better results with a larger population size but it does have a drawback: it makes the progress slower. If the population size is much larger than the size of the pareto front that means many less than optimal results are kept around and are used in the crossovers. Combining two bad results might create a very good individual but it's more likely to happen when combining two good results. But if you are going to run with a large iteration count then perhaps increasing the population size will also help. A larger population may also be needed with a difficult optimization task. If you want a reasonably good result fast use the "mutation" algorithm with a population size of 1.

-iters specifies the number of iterations. One iteration means one execution of the script we're trying to optimize. You can give the iteration count as a number (for example 1000) but there are other indirect ways. You can give a time limit in days, hours and minutes. For example 5h30m would run 5 hours and 30 minutes. 1d12h would be one day and 12 hours. You can use spaces if you put quotes around the value, for example "2h 45m". Using the time limit can be useful if you have a specific deadline for the results, or if you want to try what the optimizer can find during the night while you sleep. Just remember that the time limit applies to a single run, so if you start an optimization with 3 runs and 1h iterations it will take a total of 3 hours. During the optimization the maximum iteration count is still displayed on each result line but it is only an estimation.

20 / 345 : 4.772059 20ms sigma=349 blockTemporal=1 blockSize=50 overlap=15
21 / 347 : 4.815294 20ms sigma=477 blockTemporal=2 blockSize=64 overlap=15
22 / 349 : 4.875304 20ms sigma=591 blockTemporal=2 blockSize=30 overlap=6
23 / 350 : 4.880693 110ms sigma=800 blockTemporal=5 blockSize=61 overlap=7
24 / 343 : 4.909643 150ms sigma=800 blockTemporal=5 blockSize=61 overlap=21

There also also two keywords that trigger a special dynamic iteration mode: "dyn" and "dynbk". dyn stands for "dynamic" and means the iteration count depends on how the optimization is progressing. There are two additional arguments that define how the dynamic iteration is behaving: -dynphases and -dyniters. -dynphases defines how many distinct "phases" the algorithm is using (default is 10 but it's displayed as "N/A" in the example since it does not apply to the chosen iteration method). The phase goes from 0.0 to 1.0 during the iteration and affects how large and common mutations are. In the beginning (phase 0.0) the mutations are large and applied to many parameters. In the end (phase 1.0) they are small and applied to few parameters (in general, you can also change that). If dynphases has a value of 10 the phase range is divided into 10 steps and the phases used are thus 0.0, 0.1, 0.2, ..., 0.9 and 1.0. The dynamic iteration stays at the current phase step as long as it's still making progress. The "not making progress" is triggered when there has not been new pareto front results in the last -dyniters iterations. In that case the algorithm moves to the next phase step and resets the counter. The default value of -dyniters is also 10. "dynbk" is much like "dyn" but it can also move backwards (hence the name, "dynamic backtracking") to the previous phase step. That happens whenever a new pareto front result is found. I have not done a comprehensive study on which of these algorithms gives better results. The dynamic iteration counts are useful when you are not limited by a specific deadline and just want to find the best result. When you're running a dynamic iteration the phase step changes are displayed in the console, for example:

6 iterations remaining is this generation
No improvement in 10 iterations - moving to phase 1/10

Also instead of displaying the maximum iteration number the current phase is displayed (there's no reliable way to estimate maximum iteration count):

43 / 0,10 : 4.470519 20ms sigma=474 blockTemporal=-1 blockSize=22 overlap=0
44 / 0,10 : 4.90395 60ms sigma=665 blockTemporal=3 blockSize=41 overlap=10
45 / 0,10 : 4.757739 10ms sigma=490 blockTemporal=3 blockSize=32 overlap=0

-mutamount specifies the mutation amount. The amount is proportional to the allowed value range given in the script for each parameter you're optimizing. For example if you have a parameter with a value range 0..100 that would mean the parameter has 101 valid values. This number is multiplied by the mutation amount to get the largest possible change the mutation is allowed to make. So with mutation amount 0.2 the mutations would vary from -20.2 to 20.2. You can give two values for the mutation amount, the first is used in the beginning (phase 0.0) and the last in the end (phase 1.0) and linearly interpolated in between. If you only give one value that is used in all phases.

-mutcount specifies the mutation count. Whenever mutation is applied the first step is deciding how many parameters will be mutated and this argument defines just that. Like with -mutamount you can give a different value for the beginning and end phases. What's more the count can be given as a percentage of the number of optimized parameters in the script. So if you have 20 parameters to optimize and specify -mutcount 50% the algorithm will mutate 10 parameters. You can mix both presentations, for example the default -mutcount is "60% 1" which means mutating 60% of the parameters in the beginning and one in the end.

-crossprob specifies the probability of the crossover operation. If the probability is 1.0 the operation is applied to every new individual, if it's 0.0 it is never applied. In my tests I have found that this crossover argument is not that critical for a successful optimization.

-crossdist specifies the "distribution index" of the "simulated binary crossover" which is the crossover method used in NSGA-II and SPEA2. To be honest I don't fully understand what it does. I haven't investigated what value would be optimal for this argument.

-sensitivity specifies whether the sensitivity estimation algorithm is used. This algorithm is trying to determine how "sensitive" each parameter is, that is how much changing the parameter's value will affect the result. The sensitivity is then used by scaling the applied mutation amounts. The results are usually better when sensitivity estimation is on. If you want to switch it off set the value as "false".

Now you know how to change the optimization process. The default values are good most of the time but feel free to try different things. The most important arguments are probably -iters (and -dynphases and -dyniters if dynamic iteration is used), -runs, and -pop, followed by -mutamount and -mutcount. If you find good settings for a specific script please let me know.

In the next episode we will focus on the visualization of the results. :cool:

zorr

3rd October 2018, 21:50

Version 0.9.4-beta (https://drive.google.com/open?id=1imbzhqHIuGPEHq1qfzUIGe6mTmvhOTxA) released. Some excessive logging removed in timed iteration mode.

Seedmanc

9th October 2018, 10:13

Ok, so I gave it a try, here are the impressions.
First off, the Avstimer failed to load on Win10/Avisynth+MT, when loading via "loadplugin" it errored with "platform returned code 126: module not found", the way it reacts when I try to load a non-existant dll. I tried replacing avisynth.dll with the one from non-plus version, but it only changed the wording of the error.
Fortunately my main OS is Win7 with Avisynth 2.6MT installed where it worked, however Avstimer always returned time of 9999999ms. I suppose that made the optimization task much less efficient, because where I expected it to take tens of minutes for 10 720p frames in 3 runs, it took 15 minutes for the first run, 2.5 hours on second and 1.5 hours on third. I ran it with algo "mutation" and 100 iters. My system is Core i5 2550k OC'd to 4.3GHz, 16Gb RAM

Here's a script I used, modified from what you offered in the other thread:

TEST_FRAMES = 10
MIDDLE_FRAME = 600

# original framerate
FPS_NUM = 30
FPS_DEN = 1

# source clip
Asrc=FFmpegSource2("f:\Hibikin - Watashtachi wa Zutto... Deshou (AVS test video 60fps 720p 10bit CRF0).mkv" ).assumefps(60)
Asrc=Asrc.trim(0,60*30-9)+Asrc.trim(60*40+60*60+9,0) # this is usually the part I worked with when manually adjusting the parameters before
asrc.selecteven
AssumeFPS(FPS_NUM, FPS_DEN)

#return last

# needed for some parameter combinations
ConvertToYV24()

orig = last

super_pel = 2 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, orig )

blockSize = 32 # optimize blockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:divide 0 > 8 2 ? ; filter:overlap 2 * x <= | blockSize
searchAlgo = 3 # optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 4 # optimize searchRange = _n_ | 1..30 | searchRange
searchRangeFinest = 4 # optimize searchRangeFinest = _n_ | 1..60 | searchRangeFinest
lambda = 16000 # optimize lambda = _n_ | 0..20000 | lambda
lsad=400 # optimize lsad=_n_ | 8..20000 | lsad
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=0 # optimize plevel=_n_ | 0..2 | plevel
overlap=16 # optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
divide=2 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true # optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 2000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 4..50 | badRange
meander = true # optimize meander = _n_ | false,true | meander
temporal = false # optimize temporal = _n_ | false,true | temporal
trymany = false # optimize trymany = _n_ | false,true | trymany
dct = 0 # optimize dct = _n_ | 0,2,3,4,5,6,7,8,9,10 D | dct

delta = 1
useChroma = true
bv = MAnalyse(super_render, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
fv = MAnalyse(super_render, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)

threshold = 10000
maskScale = 100 # optimize maskScale = _n_ | 1..300 | maskScale
mask_fps = 2 # optimize mask_fps = _n_ | 0..2 | mask_fps
inter = orig.MFlowFPS(super_render, bv, fv, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )

# return this to look at the clip with doubled framerate
#return inter

fps_only = inter.SelectOdd()

# second pass
super_render2 = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only )
bv2 = MAnalyse(super_render2, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)
fv2 = MAnalyse(super_render2, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)
inter2 = fps_only.MFlowFPS(super_render2, bv2, fv2, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )
fps_only2 = inter2.SelectOdd()

delimiter = "; "

inter_yv12 = fps_only2.ConvertToYV12()
orig_yv12 = orig.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

global total = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig_yv12, inter_yv12, delta, frame_count")

# NOTE: AvsTimer call should be before the WriteFile call
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
resultFile = "f:\avsoptim\results\perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="f:\avsoptim\results\perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

A few notes:
1) I dropped the RemoveGrain call since my sources are CGI and clean enough already.
2) The FrameEvaluate you use doesn't seem to be the native one, as it complained that it doesn't have the argument "args". It worked when I installed GScript, you might want to add that to the list of dependencies.
3) Shouldn't the optimizer params description for DCT include the D flag, since, much like the searchAlgo param, it is "non-linear" and we can't make assumptions about the value and effect? I put the flag there.

The results were confusing.

Run 1 best: 9.534311 9999999 super_pel=4 super_sharp=1 super_rfilter=2 blockSize=8 searchAlgo=4 searchRange=4 searchRangeFinest=8 lambda=0 lsad=461 pnew=255 plevel=1 overlap=0 divide=0 globalMotion=false badSAD=1424 badRange=50 meander=true temporal=true trymany=false dct=8 maskScale=123 mask_fps=2
Run 2 best: 9.535164 9999999 super_pel=4 super_sharp=0 super_rfilter=0 blockSize=12 searchAlgo=4 searchRange=28 searchRangeFinest=60 lambda=1855 lsad=7420 pnew=136 plevel=0 overlap=6 divide=0 globalMotion=true badSAD=6286 badRange=14 meander=false temporal=true trymany=true dct=6 maskScale=177 mask_fps=2
Run 3 best: 9.553077 9999999 super_pel=4 super_sharp=0 super_rfilter=3 blockSize=12 searchAlgo=4 searchRange=1 searchRangeFinest=34 lambda=18163 lsad=8 pnew=9 plevel=2 overlap=6 divide=0 globalMotion=true badSAD=9869 badRange=10 meander=true temporal=false trymany=true dct=5 maskScale=23 mask_fps=2

I can see some tendencies when analyzing the logs manually, sorting by the SSIM so there's that at least. Visually, however, I can't say it looks better than what the hand-picked parameters provide, but that's to be expected from a first attempt with a low iter count.
I think the large search space (22 parameters) might have affected it too. I intend to run this again overnight with whatever iter count it manages to do in time, but first I need to have the Avstimer fixed.
Maybe I should leave the truemotion tuning to later and first try to see what more common set of parameters this tool can generate to compare with the manually tuned ones. Then once I have the good params fixed, I can have it experiment with truemotion.

Another question I wanted to ask, is there a way to estimate the amount of time/iters/population required for a certain amount of tunable script parameters involved? I don't really understand much the math involved, but I have a feeling that either the population size or the iters count should scale up with the parameter count somehow. Seeing how the first run took 15 minutes while the second one 2.5 hours it means not every run will even attempt to cover all parameters involved, since obviously the first one omitted the slowest stuff like DCT 4. But then you said Mutation algo can work with population 1.

I assume, if we were using brute-force instead of metaheuristics, then for every new parameter introduced (or a value added to a list of possible values for existing parameters) the total combination amount would double. Is there a way to roughly estimate the effect of parameter addition here?

vcmohan

9th October 2018, 12:39

I tried to understand this thread but could not. However broadly I find that various parameters for a VHS conversion are attempted so as to optimize the FPS and may be other parameters results of which are not quantizable. I also have seen mention of use of an example frame or frames to arrive at the desired quality.

In oil exploration where a large number of parameters which can vary independantly over some ranges are used to estimate reserves of oil present. Often Monte Carlo simulation is used.

In recent times AI is being used extensively for solving various problems. ANN s can use thousands of parameters to arrive at a solution which can mimic as close as possible an example.

May be tried if applicable. If my suggestion is absolutely off track please ignore the post.

Groucho2004

9th October 2018, 14:42

First off, the Avstimer failed to load on Win10/Avisynth+MT, when loading via "loadplugin" it errored with "platform returned code 126: module not found"
The avstimer 32 bit plugin that zorr provides is linked against MSVCR71.DLL which is a runtime DLL that is necessary for dynamically linked VC 7.1 binaries. I can only assume that zorr took the original project file and did not modify it for newer versions of VC.

Seedmanc

9th October 2018, 16:38

The avstimer 32 bit plugin that zorr provides is linked against MSVCR71.DLL which is a runtime DLL that is necessary for dynamically linked VC 7.1 binaries. I can only assume that zorr took the original project file and did not modify it for newer versions of VC.

Thanks, that fixed it. The problem with the incorrect time being reported still remains though.

zorr

9th October 2018, 22:25

The avstimer 32 bit plugin that zorr provides is linked against MSVCR71.DLL which is a runtime DLL that is necessary for dynamically linked VC 7.1 binaries. I can only assume that zorr took the original project file and did not modify it for newer versions of VC.

That's correct. I don't know much about building Visual Studio projects and the myriad of dll versions. I only have 32bit Avisynth installed so my testing is limited to that. Should the AVSTimer be modified to use some other dll version? :confused:

StainlessS

9th October 2018, 22:40

For anybody that needs, here VS CPP v7.0 and v7.1 dll's,

http://www.mediafire.com/file/1220u8g1drmubi8/MSVCR71_70_Runtimes.rar/file#

EDIT: Put in system32, or SysWow64 if 64 bit.

EDIT: MSVCR70.DLL (vs 2002) and MSVCR71.DLL (vs 2003).

Groucho2004

9th October 2018, 23:43

There's no reason whatsoever to link a current binary against these ancient 7.x DLLs which cause nothing but grief.

zorr, post your current project, I'll have a look.

I posted makefiles with the modified code which have the correct compiler and linker settings which you can use to build the DLLs from the command line.

zorr

9th October 2018, 23:47

Ok, so I gave it a try, here are the impressions.

Thanks, much appreciated. :)

however Avstimer always returned time of 9999999ms.

That may not be the fault of Avstimer. The optimizer sets this value as the timing result when the SSIM is invalid (that is, zero). It's done that way in order to keep the invalid results out of pareto front (an invalid result might be faster than any other result and stay in the pareto front due to being the fastest). You could check what Avstimer really returns by running the script manually (in VirtualDub or Avsr for example) and see what's inside the perFrameResults.txt. Paste it here as well, maybe there are more clues as to what is wrong.

I suppose that made the optimization task much less efficient, because where I expected it to take tens of minutes for 10 720p frames in 3 runs, it took 15 minutes for the first run, 2.5 hours on second and 1.5 hours on third. I ran it with algo "mutation" and 100 iters. My system is Core i5 2550k OC'd to 4.3GHz, 16Gb RAM

That's a large variation in runtime but not entirely unexpected. Even without dct=1 there are some very slow parameter combinations and if the optimizer stumbles upon one of those it might keep trying the slow ones if they happen to give good results. So most likely at least some results in the pareto front were using those slow combinations and mutating them resulted in mostly slow combinations. This could be verified looking at the log files or visualizing them.

Here's a script I used, modified from what you offered in the other thread:

I tried it and after swapping the source and result files found it worked just fine.

A few notes:
1) I dropped the RemoveGrain call since my sources are CGI and clean enough already.

Good call, removing extraneous parameters will help the optimizer.

2) The FrameEvaluate you use doesn't seem to be the native one, as it complained that it doesn't have the argument "args". It worked when I installed GScript, you might want to add that to the list of dependencies.

Sorry about that, I have ScriptClip installed and used its syntax there and forgot to remove the "args". It's not actually necessary and works just fine without it.

3) Shouldn't the optimizer params description for DCT include the D flag, since, much like the searchAlgo param, it is "non-linear" and we can't make assumptions about the value and effect? I put the flag there.

That's true, I added the dct parameter later and didn't really think it through. Yes, it should have the D flag.

The results were confusing.

I can see some tendencies when analyzing the logs manually, sorting by the SSIM so there's that at least. Visually, however, I can't say it looks better than what the hand-picked parameters provide, but that's to be expected from a first attempt with a low iter count.

Yes, maybe that denoising example gave a little too rosy picture of the process. That truly is a toy problem compared to MVTools and 100 iterations will not get you very far here. I am currently running a test with 5000 iterations and 5 runs.

I think the large search space (22 parameters) might have affected it too.

Yes, there's a thing called the curse of dimensionality (https://en.wikipedia.org/wiki/Curse_of_dimensionality). Searching in a high dimensional space is much more difficult than doing it in a low dimensional space. The script I gave is perhaps too brutal, my goal is to make an easier one with smaller search space once I have found the smaller subspace where the good results can be found.

Maybe I should leave the truemotion tuning to later and first try to see what more common set of parameters this tool can generate to compare with the manually tuned ones. Then once I have the good params fixed, I can have it experiment with truemotion.

That's a very reasonable idea. You can go even further and just search around some settings you know are good.

Another question I wanted to ask, is there a way to estimate the amount of time/iters/population required for a certain amount of tunable script parameters involved? I don't really understand much the math involved, but I have a feeling that either the population size or the iters count should scale up with the parameter count somehow.

I would love to have that kind of estimation but alas I don't know how to calculate it. But it totally makes sense that increasing the iters / population is needed for larger parameter count. The current default values were determined using a script with 22 parameters so they should work quite well with your script.

Seeing how the first run took 15 minutes while the second one 2.5 hours it means not every run will even attempt to cover all parameters involved, since obviously the first one omitted the slowest stuff like DCT 4. But then you said Mutation algo can work with population 1.

I believe the dct=4 has been tried but for some reason was not better than what was in the pareto front at the time. That could be simply because at the same time some other parameter was also mutated and that one brought the quality down more than dct=4 brought it up. Search the logs for dct=4 to be sure.

I assume, if we were using brute-force instead of metaheuristics, then for every new parameter introduced (or a value added to a list of possible values for existing parameters) the total combination amount would double. Is there a way to roughly estimate the effect of parameter addition here?

We can calculate the number of possible combinations by multiplying the number of allowed values of each parameter. For example with

super_pel = 2 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter

there are 2*3*5 = 30 combinations. Doing the math for your script we get 2*3*5*9*8*30*60*20001*19993*257*3*17*3*2*9997*47*2*2*2*10*300*3 = 4 136 285 698 342 782 746 022 144 000 000. :eek: Not every one of those combinations is valid though. :)

zorr

10th October 2018, 00:06

Groucho2004

10th October 2018, 00:14

Uploaded it here (https://drive.google.com/open?id=1qfID-V-531JGCx7sMtMbUkWjQ_eHh05Z). Thanks!The avstimer.dll you have in your "Release" directory is not the same as the one you posted before. It also has the correct VC2015/2017 dependencies.

Edit: Checked the time stamp on the first one you posted - It's Kassandro's DLL from 2005. :rolleyes:

zorr

10th October 2018, 00:29

The avstimer.dll you have in your "Release" directory is not the same as the one you posted before. It also has the correct VC2015/2017 dependencies.

Edit: Checked the time stamp on the first one you posted - It's Kassandro's DLL from 2005. :rolleyes:

I guess I only added the missing x64 version. Is that one using the correct dependencies as well?

zorr

10th October 2018, 00:33

Plugins package (https://drive.google.com/open?id=1iKH05RPHMywkRtka4b4o--LrES8hmAbO) with SSIM and AvsTimer has been updated, hopefully with a more functional 32bit AvsTimer. :)

Groucho2004

10th October 2018, 00:47

Plugins package (https://drive.google.com/open?id=1iKH05RPHMywkRtka4b4o--LrES8hmAbO) with SSIM and AvsTimer has been updated, hopefully with a more functional 32bit AvsTimer. :)
All good now.

StainlessS

10th October 2018, 02:43

There's no reason whatsoever to link
True, so long as there is source, otherwise, ...

Unfortunately, not everyone is willing to go to the bother of compiling Ye Olde stuff, even if source does exist.

Above link has been there for several years I think, not altogether a bad idea to have access to such when required. (EDIT: not needed in this case when
there is someone as affable as Ye Olde G2K4 :) )

Groucho2004

10th October 2018, 08:06

There's no reason whatsoever to link
True, so long as there is source, otherwise, ...Well, the word "link" usually implies that the source is present... :)

Seedmanc

10th October 2018, 18:44

The new avstimer reports time correctly, thanks.

Some other ideas to add, previously it was mentioned that some parameter combinations might have no effect on time or SSIM, but what if we knew for sure what those combinations are, it would be nice to be able to mark them as such so the optimizer could skip them. For example, in MSuper the sharp parameter is only for pel>1, it won't raise an error otherwise but it'll be a waste of time.
Regarding the ConvertToYV24, can you give an example what kind of parameter combinations would be unsuitable for YV12 so this becomes necessary? It slows down the processing considerably, so I'd like to get rid of that.

Another thing, the badRange parameter of MAnalyze says that we need to use positive values for UMH search and negative for exhaustive. Unfortunately it does not disclose why, however it doesn't raise an error anyway. But the FPS for negative values is almost a half of what's achieved for positive ones (when you set badSAD to 0 so the wide search is always invoked, not a real-life scenario, but might happen a lot during the search). In SVP a similar parameter is described as using an adaptive radius when negative to save time, claiming it takes about 2/3 of time for a similar result, but it seems we have the opposite here. Anyway, how would we describe it in the settings to use negative values when searchAlgo is 3?

Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there.

To comment on your post about the runs vs iters, you said, I don't have a good answer on how many runs are enough. If you can only run N iterations should you run for example three runs with N/3 iterations or eight runs with N/8 iterations? More iterations is better but more runs is also better, to a point.. But judging from the example where you ran a script for 100, 200 500 and 2000 iterations, the key was to see how the results stabilize among multiple runs, converging to a single set of parameters and resulting SSIMs/times. Hence, multiple runs are only used to check if the process has stabilized enough, but multiple iterations are the requirement for that stabilization. In best case all you'd need is 2 runs and a large amount of iters to see if the results became close to each other. I use 3 since it's the lowest amount from which you can already calculate both the average and the median.
If I'm correct about that all we need to figure out is how to scale population count over the increasing search space. What configuration is more likely to try the largest subset of the search parameters, the high-iter low-pop or the vice-versa?

zorr

11th October 2018, 00:24

Some other ideas to add, previously it was mentioned that some parameter combinations might have no effect on time or SSIM, but what if we knew for sure what those combinations are, it would be nice to be able to mark them as such so the optimizer could skip them.

Using the dependency definitions (min / max / filter) we can guarantee that the optimizer will not try certain combinations. It doesn't really matter what the purpose is (in some cases to avoid completely invalid combinations, in some cases to avoid doing useless work).

For example, in MSuper the sharp parameter is only for pel>1, it won't raise an error otherwise but it'll be a waste of time.

You could define it like this:

super_pel = 2 # optimize super_pel = _n_ | 1,2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 ; max:super_pel 1 == 0 2 ? | super_sharp

Translation: if super_pel is 1 then then the maximum value of super_sharp is 0, otherwise it is 2. But note that in your script super_pel only has values 2 and 4 so in this case all the values of super_sharp are valid. My experience so far has been that the best results pretty much always use super_pel=4, may not be worth your time to try the other values unless you're also optimizing for speed.

https://i.postimg.cc/WzrmWXSw/groupby_super_pel.png

Regarding the ConvertToYV24, can you give an example what kind of parameter combinations would be unsuitable for YV12 so this becomes necessary? It slows down the processing considerably, so I'd like to get rid of that.

The error message you get is "MAnalyse: wrong overlap for the colorspace subsampling for divide mode". Looking at the MVTools source I was able to gather that it's triggered when overlap is not divisible by xRatioUV (or overlapv is not divisible by yRatioUV). And the xRatioUV is 2 for YUY2 and (1 << vi.GetPlaneWidthSubsampling(PLANAR_U)) for other formats. Assuming you convert to YV12 instead (which is needed for the SSIM anyway) then you can avoid the error by making the overlap values divisible by 4. Like this (filter dependency is no longer needed so I removed that):

overlap=0 # optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlap
overlapv=0 # optimize overlapv=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlapv

You could also avoid the speed penalty by preprocessing the video to YV24 and use that instead. It might still be slower than processing YV12 but at least you don't pay the price of conversion every time. And also consider the cropping idea, that can give you a significant speed increase.

Another thing, the badRange parameter of MAnalyze says that we need to use positive values for UMH search and negative for exhaustive. Unfortunately it does not disclose why, however it doesn't raise an error anyway.

I think this means that using negative values selects the exhaustive algorithm and using positive values selects the UMH algorithm for this wide search, independently on what is used in the first search. Btw I didn't even realize you can use negative values, I guess I now have to test those too. :)

Anyway, how would we describe it in the settings to use negative values when searchAlgo is 3?

You could use the min and max dependencies. Or the filter, you can do pretty much anything you could think of with that one. :)

Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there.

I don't know more than you about this, if Pinterf is around maybe he could clarify. But it strikes me odd that the result would be different when temporal is disabled and MT is on/off. That could be a bug.

But judging from the example where you ran a script for 100, 200 500 and 2000 iterations, the key was to see how the results stabilize among multiple runs, converging to a single set of parameters and resulting SSIMs/times. Hence, multiple runs are only used to check if the process has stabilized enough, but multiple iterations are the requirement for that stabilization.

I was just showing off. :D It's rare that you can make the results converge to the exact same values. When that does happen though it's a strong indicator that we've found the optimal result. In a more realistic scenario the iteration counts are always insufficient to make the results converge in that way. There may be some parameters with the same values (like super_pel=4) but many others are different, some wildly.

There is another reason to do multiple runs. The beginning of the search usually "locks" the search into a certain corner of the search space and it may never get out of that within the iteration count. So it could be that you get significantly better result in one out of say, 10 runs. If you only ever do 3 runs maybe you'll never get to that lucky corner. Here's a recent example:

Run 1 best: 9.790725 2130 rmgrain=12 super_rfilter=1 blockSize=8 searchAlgo=3 searchRange=1
searchRangeFinest=7 lambda=2213 LSAD=2744 plevel=2 overlap=0 globalMotion=true badSAD=9559
badRange=34 meander=false temporal=false trymany=false dct=1 maskScale=2
Run 2 best: 9.787563 1210 rmgrain=19 super_rfilter=1 blockSize=8 searchAlgo=1 searchRange=1
searchRangeFinest=3 lambda=3931 LSAD=19316 plevel=2 overlap=0 globalMotion=false badSAD=2770
badRange=13 meander=true temporal=true trymany=false dct=1 maskScale=2
Run 3 best: 9.790011 1510 rmgrain=19 super_rfilter=1 blockSize=8 searchAlgo=3 searchRange=1
searchRangeFinest=4 lambda=2629 LSAD=1684 plevel=2 overlap=0 globalMotion=true badSAD=9831
badRange=10 meander=true temporal=false trymany=false dct=1 maskScale=2
Run 4 best: 9.789404 1190 rmgrain=19 super_rfilter=0 blockSize=8 searchAlgo=1 searchRange=1
searchRangeFinest=2 lambda=2142 LSAD=2679 plevel=2 overlap=0 globalMotion=true badSAD=9558
badRange=45 meander=true temporal=false trymany=false dct=1 maskScale=2
Run 5 best: 9.788453 1220 rmgrain=8 super_rfilter=0 blockSize=8 searchAlgo=1 searchRange=1
searchRangeFinest=3 lambda=3432 LSAD=6410 plevel=2 overlap=0 globalMotion=true badSAD=4531
badRange=4 meander=true temporal=false trymany=false dct=1 maskScale=1

There are three results below 9.79 and two above it. If you're unlucky you'll get those below 9.79 results in your three runs. These results were run with 5000 iterations, the differences are larger with a smaller iteration count.

In best case all you'd need is 2 runs and a large amount of iters to see if the results became close to each other. I use 3 since it's the lowest amount from which you can already calculate both the average and the median.

Maybe you're right. It's a tough call because we're talking about probabilities here. So if you do *this*, then *that* happens with a certain probability, but not always. One could do a large number of runs and then using that data calculate what are the odds for a certain result using N runs. But I think it also depends on the particular script and how difficult it is to optimize.

If I'm correct about that all we need to figure out is how to scale population count over the increasing search space. What configuration is more likely to try the largest subset of the search parameters, the high-iter low-pop or the vice-versa?

Strictly speaking there is no difference, the number of iterations defines how large the searched subset is (as the optimizer never tries duplicates within one run). But I guess you're asking how to get the widest possible subset. For that question the high population should do better in terms of how wide the search is but it's going to do less mutations of the best result and therefore might end up with a worse result than a smaller population. The "mutation" algorithm with population 1 is the most narrow search possible, it simply keeps the best result and mutates it until one of the mutations is better. If you just want to make the search wider, you can do that also by cranking up the mutation amount and count.

Seedmanc

12th October 2018, 18:25

Two problems, one is that after I modified the script to disable YV24 conversion, limit the sharp parameter and introduce the negative badRange (didn't work with -50..50 values so I added a boolean flag according to which I do or do not multiply the range by -1), the script started crashing, complaining that "MflowFps can't work in reentrant multithreading" even though I didn't touch anything MT-related. While I do have MT versions of Avisynth 2.6 and Avisynth+ installed I do not call setmtmode or prefetch in either but it crashes in both randomly, I'm lucky if I manage to get one run finished.

I used to get this error before AvsOptimizer sometimes too, usually when switching between multiple heavy scripts in Avsp editor, but it was a rare occurrence. Before I edited the script I managed to run it overnight totalling in over a thousand iterations without a problem. If it didn't work right away I'd consider that launching two upsampling operations in succession like we do here might've been the culprit.

Here's my script as of now:

TEST_FRAMES = 10
MIDDLE_FRAME = 600

# original framerate
FPS_NUM = 30
FPS_DEN = 1

# source clip
Asrc=FFmpegSource2("c:\Hibikin - Watashtachi wa Zutto... Deshou (AVS test video 60fps 720p 10bit CRF0).mkv",threads=1 ).assumefps(60)
Asrc=Asrc.trim(0,60*30-9)+Asrc.trim(60*40+60*60+9,0)
asrc.selecteven
AssumeFPS(FPS_NUM, FPS_DEN)

#return last

# needed for some parameter combinations
#ConvertToYV24()

orig = last

mt=true #happens without this parameter as well since mt is true by default, just introduced it for later use

super_pel = 2 # optimize super_pel = _n_ | 1,2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 ; max:super_pel 1 == 0 2 ? | super_sharp
super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, orig, mt=mt )

blockSize = 12 # optimize blockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:divide 0 > 8 2 ? ; filter:overlap 2 * x <= | blockSize
searchAlgo = 3 # optimize searchAlgo = _n_ | 0..5 D | searchAlgo
searchRange = 4 # optimize searchRange = _n_ | 1..30 | searchRange
searchRangeFinest = 4 # optimize searchRangeFinest = _n_ | 1..40 | searchRangeFinest
lambda = 16000 # optimize lambda = _n_ | 0..20000 | lambda
lsad=400 # optimize lsad=_n_ | 8..20000 | lsad
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=0 # optimize plevel=_n_ | 0..2 | plevel
overlap=4 # optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlap
divide=2 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true # optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 2000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 2..50 | badRange
negBadRange = false # optimize negBadRange = _n_ | false,true | negBadRange
meander = true # optimize meander = _n_ | false,true | meander
temporal = false # optimize temporal = _n_ | false,true | temporal
trymany = false # optimize trymany = _n_ | false,true | trymany
dct = 0 # optimize dct = _n_ | 0,2,3,4,5,6,7,8,9,10 D | dct

delta = 1
useChroma = true

resultBadRange = (negBadRange ? -1 : 1)*badRange
bv = MAnalyse(super_render, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct, mt=mt)
fv = MAnalyse(super_render, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct, mt=mt)

threshold = 10000
maskScale = 100 # optimize maskScale = _n_ | 1..300 | maskScale
mask_fps = 2 # optimize mask_fps = _n_ | 0..2 | mask_fps
inter = orig.MFlowFPS(super_render, bv, fv, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )

# return this to look at the clip with doubled framerate
#return inter

fps_only = inter.SelectOdd()

# second pass
super_render2 = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only )
bv2 = MAnalyse(super_render2, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, mt=mt)
fv2 = MAnalyse(super_render2, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, mt=mt)
inter2 = fps_only.MFlowFPS(super_render2, bv2, fv2, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )
fps_only2 = inter2.SelectOdd()

delimiter = "; "

inter_yv12 = fps_only2#.ConvertToYV12()
orig_yv12 = orig#.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

global total = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig_yv12, inter_yv12, delta, frame_count")

# NOTE: AvsTimer call should be before the WriteFile call
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
resultFile = "c:\avsoptim\results\perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="c:\avsoptim\results\perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

The second thing is that limiting iterations by time is rather loose, apparently because it does not include the time spent on generating populations, so where I set it to 3 runs with 3 hours it went for 3:05, 3:20 and over 4 hours on last run. Not very convenient when you expect it to be done in the morning.

Also, for some reason sensitivity estimation is always marked as N/A in the log, even if I try to pass -sensitivity true. It would be interesting to see which parameters are more important.

Groucho2004

12th October 2018, 21:14

zorr

12th October 2018, 23:36

Two problems, one is that after I modified the script ... the script started crashing, complaining that "MflowFps can't work in reentrant multithreading" even though I didn't touch anything MT-related. While I do have MT versions of Avisynth 2.6 and Avisynth+ installed I do not call setmtmode or prefetch in either but it crashes in both randomly, I'm lucky if I manage to get one run finished.

I used to have a similar problem when I ran with a high resolution source, it was because the script was running out of memory. With

SetMemoryMax(2048)

the problem went away. I tried your script again and didn't have any problems, but I'm using a low resolution source.

Here's my script as of now:

I like the way you implemented the positive / negative badRange. It's pretty much the way I would have done it, separating the algorithm into its own parameter.

The second thing is that limiting iterations by time is rather loose, apparently because it does not include the time spent on generating populations

Yes, the mutation algorithm had such a flaw. It should be fixed now in the latest 0.9.5-beta (https://drive.google.com/open?id=1lxlm95gcBausN3YNBGOKNIAZ9kr5Jwqx).

Also, for some reason sensitivity estimation is always marked as N/A in the log, even if I try to pass -sensitivity true. It would be interesting to see which parameters are more important.

The mutation algo is not quite as fully featured as the others, it's missing the sensitivity estimation at the moment. I will add it in the next version. It's probably not going to happen during this weekend though, I'm moving tomorrow and about to start renovating the new house...

Seedmanc

13th October 2018, 00:12

Groucho2004, here are the logs: https://pastebin.com/trPLFs2w (primary setup with avs2.6), https://pastebin.com/TjdMquUr (secondary setup with avs+). For some reason it complains about the absence of fft3w lib in the latter, even though mvtools dct modes work fine.

zorr, thanks for the memory suggestion and the time fix, I'll try to run it overnight now with those. I wonder if it would be possible to use the 3Gb or 4Gb limits reliably since I'm on 64bit OS. SEt says (https://forum.doom9.org/showthread.php?p=1312666#post1312666) it requires the patching of the runner app (avsr in our case) for support, I assume it applies regardless of MT usage, does avsr have that?

Groucho2004

13th October 2018, 02:12

StainlessS

13th October 2018, 02:56

W10

[CPP 2.5 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth+\plugins+\avstimer.dll [2018-10-07] # 2.5 Plugin in Plugins+ folder
C:\Program Files (x86)\AviSynth+\plugins+\ffms2.dll [2013-05-21] # 2.5 Plugin in Plugins+ folder

[CPP 2.6 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth+\plugins\avstimer.dll [2018-09-11] # see above. (earlier plug is v2.6, later 2.5 ?????? )
C:\Program Files (x86)\AviSynth+\plugins+\mvtools2.dll [2.5.11.22] # Plugins/+ mixup with below
C:\Program Files (x86)\AviSynth+\plugins\mvtools27.dll [2.7.31.0] # ditto

EDIT: Presumably, the only plugz that should be in plugins+ should be those that use avs+ colorspaces, or functionality.
EDIT: The 'hazy' part is standard v2.6 plug with standard VERSION 6 header, versus avs+ VERSION 6 Header,
ie should avs+ VERSION 6 header but without avs+ colorspace or functionality be in + or standard plugins (?).

Groucho2004

13th October 2018, 03:21

I wonder if it would be possible to use the 3Gb or 4Gb limits reliably since I'm on 64bit OS. SEt says (https://forum.doom9.org/showthread.php?p=1312666#post1312666) it requires the patching of the runner app (avsr in our case) for support, I assume it applies regardless of MT usage, does avsr have that?avsr is LAA (Large Address Aware). Patching is only necessary when the application is not built with that linker switch (VirtualDub, for example).

Besides, since there is a 64 bit version of the Optimizer, why not use that?

Lastly, why would you use SEt's ancient Avisynth MT? AVS+ is quite stable, has a 64 bit version, uses less memory and, from my experience, has better MT support.

Groucho2004

13th October 2018, 12:18

EDIT: Presumably, the only plugz that should be in plugins+ should be those that use avs+ colorspaces, or functionality.
EDIT: The 'hazy' part is standard v2.6 plug with standard VERSION 6 header, versus avs+ VERSION 6 Header,
ie should avs+ VERSION 6 header but without avs+ colorspace or functionality be in + or standard plugins (?).
I too think that in order to earn the hallowed privilege to reside in the "+" directory, plugins should support AVS+ colour spaces and/or register their own MT modes.

There's no reason to use any other header for 2.6 plugs than the one that comes with AVS+ which is fully backward compatible with classic AVS.

Seedmanc

14th October 2018, 12:53

So, after much suffering (because nothing helped against the crashes, not even running it on 64bit avs+) I figured out to take a look at the intermediate avs scripts generated by the optimizer to find what has brought it to its knees. https://pastebin.com/bgUELWu6 this one crashes 100% of the time, can you tell what the problem is?
I'll save you time, it's the combination of pel 4, blocksize 8 with overlap 4, divide > 0 with large search radius and (surprise!) removal of ConvertToYV24. What does it have to do with MT? Well I don't know, ask the mvtools maker about their error reporting style.
Further testing revealed that for pel 4 it is enough to have search radius of 4 to cause error, with pel 2 it takes around 12 and I couldn't reproduce it for pel 1. While I admit that having a search radius larger than block size seems strange it doesn't cause an error for YV24 or divide 0 / pel 1. What's more weird is that unlike the chroma subsampling violation it does not necessarily raise an error right away, sometimes it happens in the middle or at the end of the script, sometimes it's not the MT error but some random access violation. Moreso, setting chroma to false doesn't help (but converting to YV24 still does even then). Nothing in the readme has prepared me for this.
Worst thing I don't even know how to report it, the thread has been abandoned for months.

I guess more filters/minmaxes are in order, but with the current notation it's hard to figure out how to write them. Set divide to 0 when blocksize is 8 and overlap 4 and pel > 1.

In a way, Optimizer can be used as an automated plugin testing tool, since it tries out so many parameter combinations and reveals all kinds of bugs and readme inconsistencies.
Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.

StainlessS

14th October 2018, 16:36

Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.

Can I also suggest that all non-beta versions be linked in first post only and not in temporal posts.

Seems that the [EDIT: forum] spell checker works again, been missing for some time (or maybe its because I'm on W7_64 at the moment, instead of XP32/64).
[EDIT: Or maybe spell check is via current version of Firefox for W7, and no longer supported for last for XP FireFox v52.xx ESR]

zorr

14th October 2018, 23:24

So, after much suffering (because nothing helped against the crashes
...
it's the combination of pel 4, blocksize 8 with overlap 4, divide > 0 with large search radius and (surprise!) removal of ConvertToYV24.
...
Further testing revealed that for pel 4 it is enough to have search radius of 4 to cause error, with pel 2 it takes around 12 and I couldn't reproduce it for pel 1.

I was able to reproduce this even with my low resolution source. It needs to be (or converted to) YV12 though, even YUY2 works. With my slightly older MVTools it also crashes with super_pel=2, but not 1.

While I admit that having a search radius larger than block size seems strange

I don't think the block size affects the search radius. The block size tells how large blocks the algorithm is trying to track, while the search radius tells how far the block can move between frames. I guess the practical upper limit for the search radius is the width of the frame (or height, if that one is larger). [EDIT] Actually it's the diagonal of the frame, Sqrt(width*width + height*height). And here we're assuming the radius is defined in pixels, which it is not in every search algorithm.

What's more weird is that unlike the chroma subsampling violation it does not necessarily raise an error right away, sometimes it happens in the middle or at the end of the script, sometimes it's not the MT error but some random access violation.

I found a MVTools2 bug earlier which was only triggered every 10th or 20th run of the script, reported here (https://github.com/pinterf/mvtools/issues/14).

I don't even know how to report it, the thread has been abandoned for months.

I guess pinterf is very busy but he did respond to my bug reports swiftly, I think the deciding factor there was that he was able to reproduce the problem. So in this case with almost 100% error rate I think he can find the issue pretty soon.

I guess more filters/minmaxes are in order, but with the current notation it's hard to figure out how to write them. Set divide to 0 when blocksize is 8 and overlap 4 and pel > 1.

Certainly possible but I would do that only as last resort, let's give Pinterf a chance to do a fix first. With the details you figured out your error report would be very good. Getting rid of the reverse polish notation is on my todo list.

In a way, Optimizer can be used as an automated plugin testing tool, since it tries out so many parameter combinations and reveals all kinds of bugs and readme inconsistencies.

I agree, I have found two bugs from MVTools using the optimizer, both were fixed. Perhaps I should also add a mode to the optimizer where you give it a failing script and it tries to figure out all the parameter combinations that trigger the error.

Another fun (or not) idea that is already possible would be to find the slowest possible parameter combinations... perhaps combined with the worst quality too! :)

Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.

Thanks for the suggestion, I have updated the first post.

zorr

17th October 2018, 23:19

AvisynthOptimizer v0.9.6-beta (https://drive.google.com/open?id=1DrjDag1P7rOD9evkL-1ALf0-Zvf52adu) released. This version has improvements to mutation algorithm. It now supports sensitivity estimation, has colored console output that is more like the other algorithms and has the correct number of script evaluations when using a fixed iteration count.

Seedmanc, do you mind if I create an MVTools2 bug report about the issue you encountered? Or would you like to do it yourself?

Seedmanc

18th October 2018, 09:18

zorr, yes, I would rather have you do it, please go ahead.

Does the new version still require full path to the avs script?

On another note, in the FRC thread you mentioned this:
Oh and one more thing, I came up with a way to limit the ugly artifacts you often get with MFlowFPS when good motion vectors are not found. Basically you reconstruct the frame created with MFlowFPS using MCompensate and the original frames. You just need to find good parameters for that, which you can do with the optimizer. I will give an example of that later.
I'm not sure I understand this. if you can't get good vectors for MFlow then you don't have good vectors for any other tool anyway.

pinterf

18th October 2018, 12:52

Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there.
My MvTools2 fork was originated from 2.6.0.5 which had internal multithreading through avstp.dll. Internal multithreading served well when used in non-MT capable avisynth versions or scripts which could not be run in MT for some reason. So the parameter mt applies for internal multithreading.

The reason why "temporal" is not multithreading friendly is that it requires linear frame access: there is only a single internal buffer that holds the vectors from previous frame. Previous vectors are used _only_ if frame order is linear from MAnalyze point of view: the current frame number = previously analyzed frame number + 1. So the word "multithreading" here refers to the Avisynth-level multithreading schema.

MAnalyze automatically reports to work in MT_MULTI_INSTANCE mode under Avisynth+. Perhaps MAnalyze could adaptively report MT_SERIALIZED when temporal=true is set.

zorr

18th October 2018, 23:34

zorr, yes, I would rather have you do it, please go ahead.

Ok, I will do some more investigation and then create the bug report.

Does the new version still require full path to the avs script?

Yes, unfortunately. But now that you mentioned it I will take a look at how hard it would be to fix.

I'm not sure I understand this. if you can't get good vectors for MFlow then you don't have good vectors for any other tool anyway.

That's certainly true. But it just happens that MFlowInter and MFlowFps have an unfortunate looking failure case, which the MCompensate doesn't have. So even with (actually, especially with) bad vectors MCompensate will look better. Of course you can't use MCompensate to generate an inbetween frame, it can only recreate a complete frame using other frames.

But if we first create the inbetween frame with MFlowInter / MFlowFps and then use MCompensate to reconstruct that frame using two nearby frames something magical happens...

https://i.postimg.cc/wvDbnFMP/mflowinter-mcompensate.gif

Here we see the original frame (orig), a reconstructed frame from MFlowInter (inter) and finally the MCompensated frame (final).

zorr

18th October 2018, 23:43

The reason why "temporal" is not multithreading friendly is that it requires linear frame access: there is only a single internal buffer that holds the vectors from previous frame. Previous vectors are used _only_ if frame order is linear from MAnalyze point of view: the current frame number = previously analyzed frame number + 1.

Does this mean that previous vectors are sometimes used depending on the scheduling of cores, or does this condition never happen in practice when multithreading is enabled?

pinterf

19th October 2018, 15:50

It depends on scheduling.
Btw, new mvtools2 released, fixing an issue in MAnalyze which could cause artifacts also in MFlow*** at larger blocksizes/lambda.

zorr

19th October 2018, 23:29

It depends on scheduling.
Btw, new mvtools2 released, fixing an issue in MAnalyze which could cause artifacts also in MFlow*** at larger blocksizes/lambda.

Thanks! I used the latest version to test the crash issue Seedmanc found and it's still there. I created a bug report (https://github.com/pinterf/mvtools/issues/17).

zorr

22nd October 2018, 22:44

AvisynthOptimizer version 0.9.7-beta (https://drive.google.com/open?id=13gen5EKFj6XJDbuz7mnmEIaKGTl9OX3t) released.

The source file path (or any path) doesn't need to be an absolute file path anymore. This is implemented by calling SetWorkingDir() in the beginning of the script to set the working directory as the original script's directory.

Seedmanc, was this the issue you asked about, I may have misunderstood because you said "full path to the avs script"...

Thanks to Pinterf the crash issue is fixed in the latest MVTools2 version (https://github.com/pinterf/mvtools/releases/download/2.7.33/mvtools-2.7.33-with-depans.7z). I have run some 43 000 tests and found no crashes.

Seedmanc

29th October 2018, 20:44

By re-enabling MT and trimming the required 10-frame clip into a separate video I've managed to speed up things from around 150 iters per hour to a whole thousand, and I can finally see it converging to a more or less singular set of parameters overnight.

However, there are still problems. It seems the nature of SSIM makes preference of sharp lines to textures and fills, which makes the results very biased sometimes. For example, it always prefers pel=4 over pel=2, despite it being said in the readme that it's not necessarily better, especially considering the time penalty. Apparently pel=2 gives somewhat aliased edges, hardly noticeable by eye, but too important for the metric. Another problem is that it also prefers sharp=0 to sharp=2, even when it's clearly visible that the former looks considerably blurrier than the original video. Perhaps it is again due to the extra attention to lines, especially with the double upsampling method used here, the halos around edges become extra prominent. Though it's not just about SSIM, when comparing PSNR or VQM (the latter uses DCTs for comparison), using the MSU VQMT software, it was noticeable how the graphs align in parallel to each other, as if sharp=2 incurred a constant penalty in the metrics value, independent of the scene complexity. Another reason might be that I'm testing it on a 2D animation (or rather, 3D CGI which is cellshaded to look 2D), which means lots of very sharp edges with flat fills around them. In this situation, a mere half-pixel shift of an edge makes a lot more difference (relatively) than dozens of pixels away if the background is the same color. I can't be bothered to test if the problem is as strong on real footages, though.
What I tried to do, however is to obscure the influence of sharpness-related options by downscaling the video just before passing it to SSIM. 1/2 was not enough, but 1/4 by each side did the trick - sharpness or aliasing no longer affected the metrics, resulting in pel=2 and sharp=2 getting about the same share in the results pool as other values. When calculating total SSIM over entire video and comparing visually, it looks like the sharp option does not really affect the efficiency of frame interpolation in any way, meanwhile pel=2 actually looked somewhat better and got a better SSIM than pel=4. Not very significant on its own, but considering the speed difference with pel=4, important.

Among other troublesome parameters, there are also divide and overlap. The double upsample method used here causes the SSIM to always be higher for overlap=0 and no divide. Meanwhile overlap pretty much universally gives better SSIM and visuals when comparing directly to original frames, and divide sometimes looks better as well. I couldn't find a solution here, downscaling didn't help, nor I can explain what might be throwing SSIM off in that case. Really, how can SSIM of this (overlap 0) (https://i.imgur.com/FVXRt76.jpg) be higher than of this (overlap half) (https://i.imgur.com/8dlqb0K.jpg)? Ok, I need to clarify the images here, I split the video by half in duration and stack up vertically so I only have to go through 5 frames manually instead of 10 when comparing. Then, on the left half is the video after double upsampling and the SSIM for it compared to the original frames, while on the right it's after single upsampling (how it should be) and compared to discarded frames (original video is 60 so I can drop half and still get a reasonable source framerate). As you can see, SSIM on the left is inversely proportional to the actual video quality as opposed to SSIM on the right. I suppose I'll have to fix overlap to half the blocksize in the script itself, but I'm disappointed it hates divide so much.

A few more notes, the Divide parameter should be marked with the D flag, since divide=2 isn't really any "more divided" than 1, just different modes. I also added padding parameter for MSuper and the new parameter scaleCSAD, added in 2.7, which seems to improve quality when set to positive value (and the optimizer indeed chooses the maximum value for it). However neither DCT, nor searchalgo or padding converge to any particular values even after 3000 iterations and unlocking dct=1 (I don't think I saw it choosing 1 at all). I'm going to try modifying the script so that it compares to the original discarded frames to get rid of the mistakes introduced by the double upsampling and see if it's gonna be better.
Here's a Google Spreadsheets link (https://docs.google.com/spreadsheets/d/1E3Da_djkw41ZJf0V9zkACtKE812_aHny1cFb_EstLXM/edit?usp=sharing) where I tried to analyze (for the lack of a better way) results from several 3run*3000iter trials with downscaling and without, comparing the distribution of divide, sharp and pel parameters. I couldn't quite figure out how to make use of the visualizer's groupby method, so I had to come up with my own.

zorr, I mean the need to provide full path to .avs when calling the optimizer even if they're in the same directory.
I'd like to request a way to only generate scripts for the best results of every run instead of the entire pareto front. Usually when AvsOptim finishes I end up manually comparing the run results (with the image setup of above) among themselves and with the handpicked best results from previous runs. As of now it requires a lot of manual parameter editing to match the run results reported by Evaluate mode.
Also, I wonder if it would be possible to manually provide one of the generated population members so that a new run could start with one handpicked best parameter set among others and perhaps try to improve on top of that. For example, the optimizer would use the values assigned to var in the script (before the # optimize part) as one of the population members. Does that even make sense?

zorr

30th October 2018, 02:04

However, there are still problems. It seems the nature of SSIM makes preference of sharp lines to textures and fills, which makes the results very biased sometimes.

That could be the case. Our eyes are also very sensitive to sharp lines (much more than to textures) so perhaps that is just the way SSIM is supposed to work.

For example, it always prefers pel=4 over pel=2, despite it being said in the readme that it's not necessarily better, especially considering the time penalty. Apparently pel=2 gives somewhat aliased edges, hardly noticeable by eye, but too important for the metric.

I can confirm that, pel=4 is pretty much always the chosen one for the best result. But the pareto front does contain pel=2 because it gives a significant speed increase.

Another problem is that it also prefers sharp=0 to sharp=2, even when it's clearly visible that the former looks considerably blurrier than the original video. Perhaps it is again due to the extra attention to lines, especially with the double upsampling method used here, the halos around edges become extra prominent.

In my tests sharp=2 is usually the better one, but then again I haven't done a lot testing with animations. I did test a clip of "Frozen" where sharp=2 was again the winner.

Though it's not just about SSIM, when comparing PSNR or VQM (the latter uses DCTs for comparison), using the MSU VQMT software, it was noticeable how the graphs align in parallel to each other, as if sharp=2 incurred a constant penalty in the metrics value, independent of the scene complexity. Another reason might be that I'm testing it on a 2D animation (or rather, 3D CGI which is cellshaded to look 2D), which means lots of very sharp edges with flat fills around them. In this situation, a mere half-pixel shift of an edge makes a lot more difference (relatively) than dozens of pixels away if the background is the same color.

That sounds peculiar, perhaps something is not quite right with the script. Have you tried converting the video into an lossless avi format and using that as the source? In my experience the other source filters are not as reliable.

What I tried to do, however is to obscure the influence of sharpness-related options by downscaling the video just before passing it to SSIM. 1/2 was not enough, but 1/4 by each side did the trick - sharpness or aliasing no longer affected the metrics, resulting in pel=2 and sharp=2 getting about the same share in the results pool as other values.

I am not sure if that is the best way to handle this kind of situation though. Basically the optimizer is now blind to these parameters and cannot help you find the optimal values for those. You could perhaps have the same effect by disabling the optimization for these parameters and in the end manually try if the different values for pel and sharp have an effect on the final result.

There's also a SSIM variation called Multiscale SSIM (http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf) (MS-SSIM) that actually calculates the SSIM using different scales. I think that would be a good improvement on the quality measurement. It might be possible to implement MS-SSIM using just an avisynth function.

When calculating total SSIM over entire video and comparing visually, it looks like the sharp option does not really affect the efficiency of frame interpolation in any way, meanwhile pel=2 actually looked somewhat better and got a better SSIM than pel=4. Not very significant on its own, but considering the speed difference with pel=4, important.

Can you clarify this? So was the SSIM for pel=4 better in the 10 frame part (earlier you said it prefers pel=4) but pel=2 was better when calculating for the whole clip? If you meant that then yes, this could happen because the short 10 frame segment may not be a good representation of the whole clip. The optimizer tends to "overoptimize" in a way, it only cares about this short segment it is given to work with and because it also optimizes for speed it tends to fine-tune the arguments (for example the search ranges) so that they just barely work. Perhaps it would be better to take the 10 frames from different parts of the script, even though it will increase the processing time because the algorithm uses at least 4 frames around the constructed frame. Also it might be a good idea to "loosen" the found parameters, for example using slightly larger search ranges for the whole clip.

Among other troublesome parameters, there are also divide and overlap. The double upsample method used here causes the SSIM to always be higher for overlap=0 and no divide. Meanwhile overlap pretty much universally gives better SSIM and visuals when comparing directly to original frames, and divide sometimes looks better as well.

My experience is different, optimal overlap is usually not zero (at least in both directions) and divide=2 was the optimal result in my latest run.

I couldn't find a solution here, downscaling didn't help, nor I can explain what might be throwing SSIM off in that case. Really, how can SSIM of this (overlap 0) (https://i.imgur.com/FVXRt76.jpg) be higher than of this (overlap half) (https://i.imgur.com/8dlqb0K.jpg)?

Is the SSIM value for the both frames? In that case the bottom left picture might explain it, in my eye it looks a lot more garbled and deserves lower SSIM.

As you can see, SSIM on the left is inversely proportional to the actual video quality as opposed to SSIM on the right. I suppose I'll have to fix overlap to half the blocksize in the script itself, but I'm disappointed it hates divide so much.

As fancy as the optimizer is, it's just a tool with flaws. It may not find the optimal values when given too much freedom, so giving it stricter limits will probably help. Or give it more iteration counts, those always help. :) Actually I have some pretty encouraging results from my latest optimizer run with 50 000 iterations, I have *most* of the parameters now converged.

A few more notes, the Divide parameter should be marked with the D flag, since divide=2 isn't really any "more divided" than 1, just different modes.

Good point, I will try that too.

I also added padding parameter for MSuper and the new parameter scaleCSAD, added in 2.7, which seems to improve quality when set to positive value (and the optimizer indeed chooses the maximum value for it).

Oh, I just noticed I have been reading old documents of MVTools, didn't even know about this parameter! :eek:

However neither DCT, nor searchalgo or padding converge to any particular values even after 3000 iterations

Converging is not necessary though. If the parameter doesn't change the result then it really doesn't matter which value is used. But 3000 iterations is just probably not enough for them to converge (but I can tell you that 50 000 iterations will make searchalgo converge, in my case to value 6).

unlocking dct=1 (I don't think I saw it choosing 1 at all).

That was brave, unlocking dct=1. ;) But it's definitely there when I looked at your Google spreadsheet, it's even the best result of full run 1.

I'm going to try modifying the script so that it compares to the original discarded frames to get rid of the mistakes introduced by the double upsampling and see if it's gonna be better.

A good idea. You can just remove the second pass and change which clip is used inside FrameEvaluate. Just double check that you're comparing correct frames.

May I ask you why you're doing this frame doubling to a video with 60fps rate?

Here's a Google Spreadsheets link (https://docs.google.com/spreadsheets/d/1E3Da_djkw41ZJf0V9zkACtKE812_aHny1cFb_EstLXM/edit?usp=sharing) where I tried to analyze (for the lack of a better way) results from several 3run*3000iter trials with downscaling and without, comparing the distribution of divide, sharp and pel parameters. I couldn't quite figure out how to make use of the visualizer's groupby method, so I had to come up with my own.

Thanks for that. It's high time I continue the documentation in this thread, especially the evaluation part. But that will have to wait for a little bit longer. I will hopefully have some time tomorrow to run some analysis with your data. Could you also share the script (or scripts) you used to create the logs, what will help especially with the groupby -functionality.

But just a quickie here, you could for example run

optimizer -mode evaluate -log "../scripts/script*.log" -groupby super_pel -vismode series

You should now see two lines which represent the pareto fronts with values pel=2 and pel=4.

zorr, I mean the need to provide full path to .avs when calling the optimizer even if they're in the same directory.

OK, I will have to investigate that. I have used a relative path without any issues.

I'd like to request a way to only generate scripts for the best results of every run instead of the entire pareto front. Usually when AvsOptim finishes I end up manually comparing the run results (with the image setup of above) among themselves and with the handpicked best results from previous runs. As of now it requires a lot of manual parameter editing to match the run results reported by Evaluate mode.

Very well, AvisynthOptimizer v0.9.8-beta (https://drive.google.com/open?id=1KXj2r9n3P1KBDwelYIZTUrOeGQFPw5Z5) is ready for download. The scripts-parameter now takes values "none", "pareto", "best", "bestofrun" in addition to "true" and "false" (these last two are equal to "pareto" and "none"). In your case you want the "bestofrun" option. The "best" will only create a script of the best pareto front result.

Also, I wonder if it would be possible to manually provide one of the generated population members so that a new run could start with one handpicked best parameter set among others and perhaps try to improve on top of that. For example, the optimizer would use the values assigned to var in the script (before the # optimize part) as one of the population members. Does that even make sense?

It does make sense algorithm-wise, I will have to see how to manage that with the libraries I'm using. Stay tuned...

zorr

30th October 2018, 22:04

Already mentioned in the post above but let's make it official:
AvisynthOptimizer v0.9.8-beta (https://drive.google.com/open?id=1KXj2r9n3P1KBDwelYIZTUrOeGQFPw5Z5) released.

This version adds new modes to -scripts parameter:

"none": do not write any script files (same as "false")
"pareto": write scripts from the pareto front (same as "true")
"best": write a script from the best pareto front result
"bestofrun": write scripts from the best result of each run (thanks Seedmanc for the suggestion)

There are also some improvements to groupby -functionality:
-multiple scripts with different values for a parameter can be analyzed
-maxgroups also works with parameter values given as a list (previously had to be a value range)

Seedmanc

31st October 2018, 21:38

I am not sure if that is the best way to handle this kind of situation though. Basically the optimizer is now blind to these parameters and cannot help you find the optimal values for those.
It's not really blind, I'm just removing the distractions. Super_sharp and super_pel affected sharpness and edge aliasing much more than the actual motion interpolation quality (as in vectors), therefore biasing the SSIM. At least for me it is more important to measure how well the vectors are determined when optimizing for MFlowFps, rather than how sharp the picture is. Therefore I'm removing the constant from the equation to better see the effect those parameters have (or not).
Can you clarify this?
No, I mean that I compared the SSIM effect of super_sharp and super_pel after downscaling 4x for entire video, using the MSU VQMT (http://www.compression.ru/video/quality_measure/vqmt_download.html) tool that can plot the graph for various metrics and calculate average over entire video. The downscaling removed the bias for line sharpness and allowed me to evaluate the real influence of those parameters on the vector quality. As it turned out, super_sharp didn't really had any significant effect, regardless of video length, but pel=2 seems to be somewhat better. Here's how the graph looked without downscaling for super_sharp: image (https://i.imgur.com/SNB64IH.png). That's the parallel alignment I was talking about, it's not due to some frame shift if that's what you meant.

Is the SSIM value for the both frames? In that case the bottom left picture might explain it, in my eye it looks a lot more garbled and deserves lower SSIM.
It does deserve a lower SSIM but as the image shows, for the double upsample method it gets a higher SSIM instead - 0.9564 instead of 0.9555. Single-upsampling does no such error, for it SSIM correlates with the actual visual quality more often. Again, left half of the image is what your double-upsampling produces, and the SSIM on left is calculated for it against the original frames. On the right half is the result of single-upsampling after discarding half of the frames and the SSIM comparing to the discarded frames. The middle number is the ratio between the two, doesn't really mean anything.

May I ask you why you're doing this frame doubling to a video with 60fps rate?
60 fps is the framerate of my test clip, I use it a lot since with that FPS I can drop half the frames and still end up with 30fps to upsample from while being able to compare to the discarded frames without having to do double-upsampling. But actually I can go higher that 60 too, since my monitor can show 120+. For now though I want to learn doing 30 to 60 at least.
https://pastebin.com/uWa1msZ1 - here's an average script I was using when making Google sheets.

So it looks like we figured what options differ the most between working with real footages and animation. That would be super_sharp and divide at least, since we get very different results for them. Possibly overlap and DCT too. Also it always sets maskScale to 1, even though with the default 100 it looks a little better.

zorr

31st October 2018, 23:31

It's not really blind, I'm just removing the distractions. Super_sharp and super_pel affected sharpness and edge aliasing much more than the actual motion interpolation quality (as in vectors), therefore biasing the SSIM. At least for me it is more important to measure how well the vectors are determined when optimizing for MFlowFps, rather than how sharp the picture is. Therefore I'm removing the constant from the equation to better see the effect those parameters have (or not).

Ah, I see it now. I bit surprising though that pel=2 would create better vectors. It just occurred to me that maybe it has larger effective search range, assuming that the search ranges are not scaled by pel. That certainly could explain the better vectors.

I'm going to have to do a similar experiment downscaling the result before SSIM comparison and comparing that to non-downscaled SSIM. Perhaps it's always better to downscale.

I compared the SSIM effect of super_sharp and super_pel after downscaling 4x for entire video, using the MSU VQMT (http://www.compression.ru/video/quality_measure/vqmt_download.html) tool that can plot the graph for various metrics and calculate average over entire video.

I have MSU VQMT downloaded but haven't tried it yet. I guess at least some of that functionality could be replaced with an Avisynth script and/or the optimizer with some tweaks. Are you using it primarily for comparing total SSIM of two clips and/or also to see how they change visually?

It does deserve a lower SSIM but as the image shows, for the double upsample method it gets a higher SSIM instead - 0.9564 instead of 0.9555.

Oh, but I meant that the 0.9564 frame was better. At least I can understand why SSIM would rate it such - it doesn't have a lot of higher frequency "noise" like the other picture.

Single-upsampling does no such error, for it SSIM correlates with the actual visual quality more often.

In that case (and because you do have a 60fps source) you should definitely only do a single upsampling. Do you need help with the script implementing that?

https://pastebin.com/uWa1msZ1 - here's an average script I was using when making Google sheets.

Ok thanks. I noticed you used range 0..4 for scaleCSAD but in the latest docs the valid values are -2, -1, 0, 1 and 2. Still your best results used 4 so it has to work... [EDIT] Just noticed you subtract 2 from the value, so it's just like in the docs. Nevermind. And I guess there's a bug with negative values?

Also I want to ask if the results on one tab are from a single run or from multiple runs. There are over 10000 results in full1 and full2 so I'm guessing they consist of multiple runs.

Full2 doesn't have the scaleCSAD parameter, was that your first script variation?

It would help the analyzing if you posted the original log files.

So it looks like we figured what options differ the most between working with real footages and animation. That would be super_sharp and divide at least, since we get very different results for them. Possibly overlap and DCT too. Also it always sets maskScale to 1, even though with the default 100 it looks a little better.

Also the blockSize, though I think the optimal one for that may differ for every video. I was wondering about your maskScale results too, in my tests the maskScale is very close to the maximum (actually I had to set the maximum higher for my latest run, it's one of those params without any official maximum value).

[EDIT] Forgot to ask, which algorithm are you using when running the optimization? If it's still "mutation" I recommend you try the default "SPEA2" because with thousands of iterations it gets better results.

Seedmanc

1st November 2018, 08:22

Are you using it primarily for comparing total SSIM of two clips and/or also to see how they change visually?
Yes, that's how I use it mostly, before AvsOptim I was using it a lot to manually analyze the effect of each parameter. It has a lot of various metrics available.
Oh, but I meant that the 0.9564 frame was better
I can't really say how it's better, isn't the quality of framerate upsampling determined by how correct the vectors are? And by correctness I mean how well they converge to a middle point in time between the two existing frames. In 0.956 case you can see that they don't really converge, instead of a single finger you see 3, meanwhile on 0.9555 it's more or less one. This is my primary measurement of visual quality, to see how well the fast-moving objects are treated.
I can figure out the script for single upsampling, thanks.
About negative values, when I was introducing negative badRange somehow the notion of -50..50 didn't work, so I wasn't trying it anymore (well it would actually be a wrong way for that particular parameter). Try and see if negative ranges work for you with scaleCSAD.

As I mentioned, each tab consists of log values from 3 runs of 3000 iterations together. The leftmost column has all the log values, I only removed the header and sorted by SSIM. Yes, Full2 was one of the earliest.
I've switched to SPEA2 around time when MVtools bugs were discussed, so I'm using that now.

zorr

1st November 2018, 21:40

I can't really say how it's better, isn't the quality of framerate upsampling determined by how correct the vectors are? And by correctness I mean how well they converge to a middle point in time between the two existing frames.

That's one way to look at it. But I know some people prefer the less correct (technically speaking) blending when the vectors are so bad that there are distracting artifacts. SSIM naturally doesn't know anything about the vectors, it's simply trying to figure out how similar two pictures are. But I totally understand your take on it and it's a valid point of view.

If one does the MCompensate correction after MFlowFPS the most correct vectors should produce better image quality even according to SSIM. But to do that requires that both processes are optimized at the same time roughly doubling the number of optimized parameters. Or perhaps it could be done in turns, first optimizing MFlowFPS only, then optimizing MCompensate and then optimizing MFlowFPS again. Sorry if this doesn't make much sense, I need to make a thread about that technique...

About negative values, when I was introducing negative badRange somehow the notion of -50..50 didn't work

Ok I will investigate.

As I mentioned, each tab consists of log values from 3 runs of 3000 iterations together.

There must be something else there as well, full1 has 10688 results and full2 has 14624 results. Perhaps you ran them with the time limit?

Seedmanc

2nd November 2018, 21:37

I need to make a thread about that technique...
I'll be looking forward to it. I tried it briefly now and it looks promising, it removes the artifacts like a charm. Though it also erases the moving objects out of existence too sometimes and it can't recreate the missing vectors in high-motion areas, it still looks cool. I usually average the backward and forward compensation together to smooth it out a bit.

Perhaps you ran them with the time limit?
Yes, sorry, that's what I meant. I was doing 3 runs per 3 hours on average.

ChaosKing

3rd November 2018, 14:34

What am I doing wrong? I tried different names but I always get this error msg:

Found following optimizable parameters:
# optimize tr = _n_ | 1..4 | tr
found 1 parameters to optimize

Running SPEA2
java.lang.Exception: Could not update parameter value for [tr = _n_]
at avisynthoptimizer.Parameter.getLine(Parameter.java:617)
....

TEST_FRAMES = 10 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

ffms2("E:\cut.mkv").ConvertBits(8)
source = last
last=source.AddGrain(80, 0, 0, seed=2)
tr =1 # optimize tr = _n_ | 1..4 | tr
denoised = TemporalDegrain2(degrainTR=tr)
last = denoised

# calculate SSIM value for each test frame
global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(source, denoised)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")

# measure runtime, plugin writes the value to global avstimer variable
# NOTE: AvsTimer should be called before WriteFile
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "D:\AvisynthRepository\AvisynthOptimizer-0.9.8-beta\perFrame.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="D:\AvisynthRepository\AvisynthOptimizer-0.9.8-beta\perFrame.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

# NOTE: must return last or FrameEvaluate will not run
return last

#Prefetch(0)

Seedmanc

3rd November 2018, 19:21

ChaosKing, spaces matter. In the template (after # optimize) it says tr = _n_, but in the parameter assignment (before # optimize) you're missing space after =.

ChaosKing

3rd November 2018, 20:06

Ahhhh didn't know what, thx, works now.