Introducing Zopti (ex AvisynthOptimizer)

[zorr]

Last year I dug up some old VHS videos and decided to finally digitize them. The tapes are old and I wanted to preserve them in the best quality I could afford. I soon found AviSynth and started to investigate how to do restoration and deinterlacing (for playback on computer).

This lead me to QTGMC-plugin and I was impressed by its capabilities but I did find some strange artifacts when I used the best quality settings. These artifacts were not present at lower quality settings. Being a perfectionist I naturally searched manually which of the 92 parameters were responsible for the artifacts and found them (more about that later).

There is quite a bit of noise in the old VHS tapes so naturally I investigated noise removal plugins. The temporal denoising made possible by MVTools is very important and I wanted to find out which settings are best for my VHS tapes. Reading these and other video restoration forums I found that there really isn't any one set of settings that would work for every type of video. So finding good settings would involve a lot of manual testing. Using MVTools to do motion compensation involves about 60 parameters so there's a lot of things to adjust... this task seemed hopeless.

So I began to wonder if there is anything that could automate this kind of process. Turns out that there is, and I made a tool that can help doing it. And now I'm about to share it with you.

An AviSynth script is first "augmented" by adding instructions on which parameters should be optimized. The instructions are written inside comments so the script will continue to work normally. The script also needs to measure the quality of the processed video using current parameter values or measure the running time (or preferably both the quality and runtime). The script needs to write these results to a specific file.

Now you might be wondering how on earth can the script measure the quality. I can only think of one way: to compare the frames to reference frames and measure the similarity. The closer the similary value, the better the quality. You could measure similarity in a very simplistic way using LumaDifference and ChromaUDifference+ChromaVDifference, but the best similary metric AviSynth currently has is SSIM (https://en.wikipedia.org/wiki/Structural_similarity). SSIM is not the best similarity metric anymore but it's still widely used in the image processing community. The SSIM plugin needs to have a function which returns the similarity value to the script. The only version that currently does that (as far as I know) is v0.25.1 by mitsubishi (https://forum.doom9.org/showthread.php?p=1089303#post1089303).

Ok, but where do we get the reference frames? In case of MVTools we can use the original frames as reference frames and the script will try to reconstruct them using motion compensation (but it's not allowed to use the reference frame in the reconstruction). We can do something similar if we want to use MVTools to double the framerate: we first create the double-rate video, then remove the original frames from it, then double the framerate again and finally compare these to the original frames. This idea is not limited to MVTools, you could for example do color grading using some other software and then try to recreate the same result using AviSynth. I'm sure the smart people here will find use cases I couldn't even dream about. :D

Most people don't only care about the quality, the script's run time is also important. I found a couple of different plugins to measure the runtime, but only one of them (AvsTimer) fit the bill and even then I had to make some modifications to it.

So if the purpose is to find the best settings, what do we consider the "best" when both quality and time are involved? For example we can have settings A with quality 99 (larger is better), time 2300ms (larger is worse) and settings B with quality 95 and time 200ms. Which one of these is better? We can use the concept of pareto domination to answer this question. When one solution is at least as good as the other solution in every objective (in this case the objectives are quality and speed) and better in at least one objective, it dominates the other solution. In this example, neither dominates the other. But if we have settings C with quality 99 and time 200ms it dominates both A and B. In the end we want to know all the nondominated solutions, which is called the pareto front. So there's going to be a list of parameters with increasing quality and runtime. You can have more than two objectives if you want, the same pareto concept works. There's a good and free ebook called "Essentials of Metaheuristics (https://cs.gmu.edu/~sean/book/metaheuristics/)" which describes the pareto concept and much more.

AvisynthOptimizer reads the augmented script, verifies it and starts running a metaheuristic optimization algorithm. There are currently three algorithm choices: NSGA-II, SPEA2 and mutation. The first two are some of the best metaheuristic algorithms available and also described in the ebook mentioned above. The mutation is my own simplistic algorithm (but it can be useful if you're in a hurry since it's the most efficient). All the metaheuristic algorithms are working in a similar manner generating different solutions and testing them. The optimizer does these tests by creating AviSynth scripts with certain parameter values and running them. The script then writes quality/time results into a file which the optimizer then reads. The metaheuristic then decides which parameters to try next based on the results. This continues until some end criteria is met.

There are three different ending criterias: number of iterations, time limit and "dynamic". Number of iterations is just that, the algorithm runs the script specific number of times. Setting a time limit can be pretty useful if you know how much time you can use for the optimization. You could for example let it run overnight for 8 hours and see the results in the morning. Dynamic variation is stopping only when it doesn't make any progress anymore. Making progress is defined by "no more pareto front members in last x iterations". This can be useful if you want to find the best possible results regardless of how long it takes.

During the optimization all tested parameters and their results are written to a log file. This log file can be "evaluated" during and after the optimization process. The evaluation basically means finding the pareto front and showing it. You can also create AviSynth scripts from the pareto front in order to test them yourself. It's also possible to visualize the results of the log file in a two dimensional scatter chart. This chart highlights the pareto front and shows all the other results too. The chart can also be "autorefreshing": it loads the log file every few seconds and updates the visuals, which is a fun way to track how the optimization is progressing. Here's a gif what it looks like (obviously sped up):

https://s22.postimg.cc/lgnp0apsh/animation.gif (https://postimg.cc/image/q2jt8nbbh/)

The visualization has a few other bells and whistles but one I'd like to highlight here is the group by functionality: you can group the results by certain parameter's values and show a separate pareto front for each value. I think a picture is in order here, this what grouping by MVTools' blocksize looks like:

https://s22.postimg.cc/x5rooblwx/groupby-blocksize.png (https://postimg.cc/image/hkad4d9yl/)

Measuring the script's runtime is not very accurate, ie it has some variation. All the other processes running at the same time are using CPU cycles and messing with the cache so you should try to minimize other activity on the computer. In order to get more accurate results you can run a validation on the finished results log file. In validation the idea is to run the pareto front results multiple times and calculate the average, median, minimum or maximum of these multiple measurements (you can decide which one(s)). After the pareto front is measured validation calculates the "secondary pareto front" by removing the results in the original pareto front and finding a pareto front of the remaining results. The verification is run on the secondary pareto front also because it's possible that due to inaccurate runtimes the real pareto front will contain results from the secondary pareto front. If the secondary pareto front did contain real pareto front members the validation takes the third pareto front and validates it as well. And so on until the current pareto front didn't contain any new pareto front members.

So how good is the optimizer? Let's take a look at one example. A while back there was a thread (https://forum.doom9.org/showthread.php?t=175508) about best motion interpolation filters. There's a test video with a girl waving her hand. The best options currently are John Meyer's jm_fps script and FrameRateConverter. Here's a comparison gif with those two and AvisynthOptimizer. FramerateConverter was run with preset="slow".

https://s22.postimg.cc/x71mh6xv5/ezgif-2-85352ef164.gif (https://postimg.cc/image/8qjgmpx4d/)

Now obviously I'm showing a bit of a cherry-picked example here. The optimizer was instructed to search the best parameters for this short 10 frame sequence. I have run most of my optimization runs using only 10 frames because otherwise the optimization takes too long. Ideally the optimizer would automatically select the frames from a longer video, I have some ideas on how to implement that but as of now the user has to make the selection. Also this part of the video is not the most challenging part, I decided to try something relatively easy first. After all the optimizer cannot do miracles (that feature is not finished yet ;)).

At this point I envision that AvisynthOptimizer would be an useful tool for plugin authors so they can test plugin parameters and try to search for the optimal ones. At some later point AvisynthOptimizer could be useful for normal users, when combined with a script with limited search space so that the search will not take excessively long time.

All right, that's all for now. I will add more detailed explanations and the download links tomorrow. I haven't actually finished the documentation yet, but still wanted to get this thing out there before my vacation is over. :)

[EDIT] Here are the download links:
Download the zip (version 1.2.3) here (https://drive.google.com/file/d/1Ox_HfyvpV1bbL2_j-jx9hGOOYQAunXQp/view?usp=sharing) and unzip to a folder of your liking. You will also need my modded AvsTimer and the SSIM plugin with the SSIM_FRAME-function which returns the SSIM value to the script. I have a package which contains both of those here (https://drive.google.com/open?id=1iKH05RPHMywkRtka4b4o--LrES8hmAbO).

More documentation can be found from these posts (if you're not interested in reading the whole thread):
Augmented Script (part 1/2) (https://forum.doom9.org/showthread.php?p=1851085#post1851085)
Augmented Script (part 2/2) (https://forum.doom9.org/showthread.php?p=1851086#post1851086)
Hands-on tutorial (part 1/2) (https://forum.doom9.org/showthread.php?p=1851670#post1851670)
Hands-on tutorial (part 2/2) (https://forum.doom9.org/showthread.php?p=1851672#post1851672)
Optimizer arguments (https://forum.doom9.org/showthread.php?p=1853726#post1853726)

[wonkey_monkey]

Nice!

[Seedmanc]

Sounds cool, but wouldn't it make more sense measuring quality by halving the source framerate and comparing upsampled FPS to the original frames instead of double-upsampling and comparing to the already imperfect recreated frames? Settings that worked well for a lower source framerate will surely work even better for a higher one.

Besides I don't get why everyone's using FRC as the golden measure, all it does is blending frames like ConvertFps but slow.

[zorr]

Sounds cool, but wouldn't it make more sense measuring quality by halving the source framerate and comparing upsampled FPS to the original frames instead of double-upsampling and comparing to the already imperfect recreated frames? Settings that worked well for a lower source framerate will surely work even better for a higher one.

The interpolation task is much more difficult when done using half framerate, so it could mean no good settings are found. On the other hand
you can try to offset the difficulty by selecting a moderately easy part of the video for optimization. I think this idea is worth investigating. I will run some tests later.

Besides I don't get why everyone's using FRC as the golden measure, all it does is blending frames like ConvertFps but slow.

I'm not an expert on this subject matter, just following what has been recommended by more knowledgeable people here. I did try InterFrame (http://www.spirton.com/uploads/InterFrame/InterFrame2.html) also but it gave very poor results. Do you know Avisynth plugins which do better frame interpolation than FRC?

[StainlessS]

The interpolation task is much more difficult when done using half framerate
I'm sure that someone might have some suitable true 50/60FPS stuff to donate, [AssumeFPS(60.0) to make numbers better if 59.94fps].

I did start on some half arsed similar project but gave up (well on back burner somwhere), was intending to use from script and using RT_Stats
RT_Array and/or RT_DBase, maybe DBase for arg names, ranges etc, and array for results.

Your effort looks way more organized than my totally unplanned (lets see what happens) effort.

Good look and keep us all posted, your cherry pickin's look good to me :)

EDIT: I guess your thread explains all of the 'torture' you put me through in this thread:- http://forum.doom9.org/showthread.php?t=175373

EDIT: Also take note that Source Framerate is also an indirect arg, and it would be good to also know how it
affects the results.

[zorr]

I'm sure that someone might have some suitable true 50/60FPS stuff to donate, [AssumeFPS(60.0) to make numbers better if 59.94fps].

The way I see it is that every video is unique and might need different parameters for optimal results. So doing this half-framerate test on such video will tell us one thing: what are good parametes for this one video. But with more test cases hopefully general patterns will emerge and the optimizer would only need to figure out which of these cases it is dealing with.

I did start on some half arsed similar project but gave up (well on back burner somwhere), was intending to use from script and using RT_Stats
RT_Array and/or RT_DBase, maybe DBase for arg names, ranges etc, and array for results.

Your effort looks way more organized than my totally unplanned (lets see what happens) effort.

It wasn't that organized from the start, just a crazy idea that evolved gradually into what it is today. Hats off to you for trying to do it in pure Avisynth, I didn't even think of such possibility. It was fun to figure out all the parts that were needed. How do I make Avisynth script communicate the results to another program? How to run Avisynth scripts programmatically? And so on...

Good look and keep us all posted, your cherry pickin's look good to me :)

Thanks, I have some more cherry-picked results to share but I have to finish this tutorial thread first. :)

EDIT: I guess your thread explains all of the 'torture' you put me through in this thread:- http://forum.doom9.org/showthread.php?t=175373

That would be correct. That was me trying to understand how to get the best possible quality out of MVTools.

EDIT: Also take note that Source Framerate is also an indirect arg, and it would be good to also know how it
affects the results.
Sorry, which function has this argument?

[StainlessS]

Sorry, which function has this argument?

Any function that takes a video clip with some non obligatory framerate.
(ie the results will be different depending upon input framerate, higher i/p framerate almost certainly better o/p).
EDIT: Assuming eg o/p to double rate.
EDIT: Also, assuming that I'm correct in thinking that vector length is limited to signed BYTE size, so is
also affected by input frame size (and also pel setting) [perhaps for 16 bit colorspace is limited to 16 bit vector, dont know].

[zorr]

Let's take a look at what an "augmented" script looks like. This is a complete script that contains everything AvisynthOptimizer needs. The script is using MFlowInter to reconstruct every frame of the video using the neighbour frames and then compares them to the original frames.

TEST_FRAMES = 10 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

AVISource("d:\process2\1 deinterlaced.avi")
orig = last

# you could add preprocessing here to help MSuper - not used here
searchClip = orig

super_pel = 4 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2 # optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, searchClip)
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, last, levels=1)

blockSize = 8 # optimize blockSize = _n_ | 6,8,12,16,24,32 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 5 # optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 2 # optimize searchRange = _n_ | 1..10 | searchRange
searchRangeFinest = 2 # optimize searchRangeFinest = _n_ | 1..10 | searchRangeFinest
lambda = 1000*(blockSize*blockSize)/(8*8) # optimize lambda = _n_ | 0..20000 | lambda
lsad=1200 # optimize lsad=_n_ | 8..20000 | LSAD
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=1 # optimize plevel=_n_ | 0..2 | plevel
overlap=2 # optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=2 # optimize overlapv=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / | overlapv
divide=0 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true # optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 10000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 4..50 | badRange
meander = true # optimize meander = _n_ | false,true | meander
temporal = false # optimize temporal = _n_ | false,true | temporal
trymany = false # optimize trymany = _n_ | false,true | trymany

# smallest delta is 1 but you can make the task more challenging by using larger delta (larger deltas often used in temporal denoising)
delta = 1

useChroma = true
bv = MAnalyse(super_search, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badSAD=badSAD, \
badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)
fv = MAnalyse(super_search, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badSAD=badSAD, \
badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)

# NOTE: we disable scene change detection by setting thSCD1 very high
blockChangeThreshold = 10000
maskScale = 70 # optimize maskScale = _n_ | 1..300 | maskScale
inter = last.MFlowInter(super_render, bv, fv, time=50, ml=maskScale, thSCD1=blockChangeThreshold, thSCD2=100, blend=false)

# SSIM needs YV12 colospace
inter_yv12 = inter.ConvertToYV12()
orig_yv12 = orig.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

# cut out the part used in quality / speed evaluation
inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

# calculate SSIM value for each test frame
global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")

# measure runtime, plugin writes the value to global avstimer variable
# NOTE: AvsTimer should be called before WriteFile
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "D:\optimizer\perFrame.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="D:\optimizer\perFrame.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

# return original and reconstructed frame side by side for comparison
#return StackHorizontal(orig_yv12, inter_yv12)

# NOTE: must return last or FrameEvaluate will not run
return last

That's not the most simple example I could think of but it's a good example because it uses some of the more advanced features of the optimizer.

TEST_FRAMES = 10 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number
We begin by declaring a couple of variables which help choosing the part which gets optimized. TEST_FRAMES tells how many frames long the test sequence is. MIDDLE_FRAME tells the frame number in the middle of the test sequence. So you can find an interesting part of the video and make note of the frame number and then set it as the middle frame number. These are by no means necessary in the script, they're just there to help.

super_pel = 4 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2 # optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, searchClip)
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, last, levels=1)
Skipping a couple of lines we find the first AvisynthOptimizer related code. Here we define a couple of variables (in practice constants) that are later used in MSuper calls. The comment lines which start with "# optimize" are read by AvisynthOptimizer. (NOTE: "#optimize" will not work). The comments consist of three sections separated by "|" character.

The first section (after the word optimize) tells which part of code needs to be manipulated by the optimizer. The part with "_n_" is replaced with different values, the rest is there just give enough context on where this "_n_" -part is located. Whitespace matters here, so "# optimize super_pel=_n_" would not work when the code says "super_pel = 4".

The second section tells the valid values the optimizer should try for this parameter. You can define it as a range, for example "1..5" which would mean values from 1 to 5. Or you can define the values as a list separated by commas, for example "1,2,3". Booleans are supported (usually given as "false,true"). Strings are supported as well. Floats however are not, so if the parameter is a floating point number define it something like this:

param = 100/1000.0 # optimize param = _n_/1000.0 | 0..1000 | paramName
This would give you floating point values between 0.0 and 1.0 with 0.001 intervals. You should consider what to use as the divider. A large divider means larger search space which makes the search more difficult. Too small divider could mean you miss the optimal value. One good strategy could be using a coarse interval first and do another run with finer interval after you've figured out an approximate optimal value. Don't forget the ".0" in the divider, otherwise you get integer division and no floating point numbers!

The last section is the name of the parameter. I have named some of the parameters differently than their corresponding parameter name in MVTools just to make it a little easier to remember what they do. These names are used in the log files where tested parameter values are reported. Also you can refer to other parameters by using these names (more about that in a little while).

searchAlgo = 5 # optimize searchAlgo = _n_ | 0..7 D | searchAlgo
This is another basic variable / optimizer definition but it has this "D" after the range. The D is short for "discrete" and means the optimizer should not make any assumptions that values closer to each other are more similar. Usually this is the case, for example when changing search range from 8 to 9 we expect smaller change in the result than if we change it from 8 to 20. This particular parameter controls the search algorithm and we have no idea whether algorithms 1 and 2 are more similar than, say, algorithms 1 and 3.

overlap=2 # optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
What on earth is this? :eek: Calm down, there is an explanation. :) So the middle part where valid values are described can itself be divided into sections separated by ";"-character. The first part is just a normal value list: "0,2,4,6,8,10,12,14,16". The second and third parts are there because of an important feature: conflict resolvation (not sure if that's a word). So MVTools is kinda picky on what kind of parameter values you give to it. If it doesn't like the combination it will give an error message instead of doing something useful. For example overlap cannot be larger than half the blocksize. The optimizer doesn't know this (unless we tell it) and would happily try thousands of combinations where overlap is larger than blocksize/2. One way out of this problem is to wrap the script in try..catch and write out zero as quality when this happens. That would work but the optimizer would waste a lot of time trying invalid parameter combinations.

The better solution is to tell the kind of dependencies the parameters have between them. In this case we define a max dependency: "max:blockSize 2 /". The part after "max:" is a function written in reverse polish notation. For all the non-reverse-polish people this means blockSize/2. Now the optimizer knows that the maximum value of overlap is blockSize/2. Here we are referring to another parameter "blockSize" by its name.

The final part and the most difficult one by far is the filter dependency. It's used when minimum and maximum dependencies are not enough. The idea of the filter is that the optimizer will do a test for the current parameter values during the runtime. The current value of the parameter is put inside the formula replacing the "x". Then the formula is evaluated and if it returns true the parameter value is accepted as valid. The formula in reverse polish form "x divide 0 > 4 2 ? % 0 ==", which in infix form is "x % ((divide > 0) ? 4 : 2) == 0". "divide" is not division operator but another parameter name (perhaps not the best name choice here). In plain english this states that if (divide > 0) then overlap should be divisible by 4, otherwise it should be divisible by 2.

blockSize = 8 # optimize blockSize = _n_ | 6,8,12,16,24,32 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
I will highlight this line as well because here in the filter definition we have "max", which is the maximum function. The full list of supported operators is +, -, *, /, %, <, >, <=, >=, ==, !=, ?, min, max, and, or.

Whenever you define a dependency with another parameter, you should do it for both parameters involved. So for example because overlap has a max dependency on blockSize, the blockSize should have min or filter dependency on overlap. The reason for this is to avoid bias in the search. If there is a conflict between overlap and blockSize, the optimizer will try to resolve it by changing either overlap's value or blockSize's value (and it does this in a fair way so that both get changed as often). If the dependency is defined only in one of the parameters, it will always change that parameter's value (it's not smart enough to figure out the valid values for the other one). This would introduce bias in the search and possibly ruin the chances of finding the optimal results.

That's all there is to defining the parameters to optimize. The reverse polish notation is something I would like to change since it's not the most user friendly format. The infix parsers I looked at were unfortunately not able to deal with ternary operators. But if you just want to optimize MVTools the hard part is already done and you can reuse my parameter definitions. :)

# calculate SSIM value for each test frame
global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")
This is the part where we calculate the quality using SSIM. The current frame's quality is stored at "ssim"-variable and the code also calculates the total into "ssim_total". Sometimes MVTools returns the original frame when called with bad parameter values and SSIM returns the maximum similarity 1.0. Those frames have to be rated zero instead.

# measure runtime, plugin writes the value to global avstimer variable
# NOTE: AvsTimer should be called before WriteFile
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")
This part measures the runtime of every frame. The AvsTimer plugin needed some modifications for this purpose. The original returned frames per second as integer value, I changed it to return the passed time in milliseconds. Moreover, the timer was reporting the results into Windows debug log only, my version writes the result into a global variable called avstimer (I should add a parameter where you can define the parameter name).

Part 2 below...

[zorr]

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "D:\optimizer\perFrame.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="D:\optimizer\perFrame.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

The current frame number, per frame ssim and runtime values are written to a file, separated by ";". The comment starting "# output" is another important input for the optimizer. Here we define the output values of the script in a special format. The first output is defined with out1="...", the second with out2="..." and so on. The format is "<name>: <goal>(<type>) [<unit>]". <name> is the name of the output, this is used in the visualization. <goal> is either MIN or MAX (or a lowercase variation) and tells whether we want to minimize or maximize this value. <type> can be float, int or time. Time is a float but it will be rounded to 10 milliseconds (this is the default and can be changed). The reason for the rounding is that the time measurement itself is not very accurate and a very small difference in the timing is not very important anyway. For example if we have to scripts with the same quality but the other is one millisecond faster that probably isn't a reason to consider it better (and we can't be sure which one is really faster). <unit> is an optional name for the unit. It will be used in the visualization if it's defined. All of this can also be defined in another, more verbose format. For example we could define out1="ssim: MAX(float)" with "name1=ssim, goal1=MAX, type1=float". In this case I didn't use quotes, they are not necessary if the values don't contain space characters. The rounding can be defined with roundx="...", for example "round2=10".

The "file=" -definition is the only compulsory output parameter, the others have default values. The optimizer will change the file name where the results are written for every new script that it creates. For that reason it needs to know what part of the text needs to be replaced with a new file name. We just need to repeat the same file name which is used as the resultFile's value.

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

When the script is finished it writes a line starting with "stop". This is needed to tell the optimizer that the script is finished processing and the result is ready for consumption. The optimizer is polling the file every 0.1 seconds until this line is found. I was hoping to use a more sane technique but as of now this is how it works. The polling interval is not too extreme so it doesn't use much CPU or I/O but also frequent enough so that we can get the results rapidly. Maybe the polling interval should be customizable. The code above also writes the total ssim value but does NOT write a total runtime. If the runtime were written also the optimizer would read the result using these values from the last line. If the last line doesn't contain enough values to read (in this case two would be needed, one for each output) it reads all the lines and sums the values there. In some cases the script might calculate the final result using a different logic than simple addition, it can then write the result as the last line. In that case the individual lines are not even needed so the first WriteFile() could be removed.

# return original and reconstructed frame side by side for comparison
#return StackHorizontal(orig_yv12, inter_yv12)

# NOTE: must return last or FrameEvaluate will not run
return last

The script must return "last" in order to work properly (FrameEvaluate is not called otherwise). If you're debugging the script and returning for example a comparison frame with original and reconstructed frame just remember to change it back to "last" when you're done.

The results-file will look like this:

0; 0.754317; 164.351318
1; 0.859464; 92.895966
2; 0.805696; 67.377174
3; 0.744211; 64.517632
4; 0.684871; 60.627941
5; 0.821181; 63.919125
6; 0.919067; 57.346405
7; 0.842346; 57.677219
8; 0.833121; 65.297905
9; 0.787272; 59.674370
stop 8.051547

All right, that's enough for today. Tomorrow I will cover how to run the optimizer and then add the download package. :cool:

[StainlessS]

Sorry Zorr, I had not properly read your intro, (skim read far too much), and I now understand that you are trying to optimize for the current clip,
and not a generally best settings set, for any ol' clip. The double current, chuck originals away and do it again, is a bit of lateral thinking that
skipped over this head, and (assuming it works, and I guess it must), then is quite inspired.

I can see me reading this thread a good few times from start to finish, and have also downed the meta-wottsit PDF thing to have a read of that too.

Will delete this post so as not to interfere in your postings flow, after you post the next intriguing episode.

EDIT:
(assuming it works, and I guess it must)
If it does, then you have indeed found good settings and so when using for real [ie without "double current, chuck originals away and do it again"], should be great quality.

[zorr]

I now understand that you are trying to optimize for the current clip,
and not a generally best settings set, for any ol' clip.

Optimizing for the current clip is the easier option. You just run the optimization and you should have good settings for that one clip (or at least for one small part of that one clip...). And this is already useful because different clips do need different settings to be optimal.

But in the long run I also want to see if the default settings of MVTools and other filters can be made better. That however takes a lot more work, the settings will have to be tested on many kinds of videos. Maybe we can find settings that work better for most videos than the current default settings. Maybe the current defaults are already optimal in that regard. I think it's worth trying now that we have a tool that can help.

Will delete this post so as not to interfere in your postings flow, after you post the next intriguing episode.

No need to delete, this is all relevant conversation.

If it does, then you have indeed found good settings and so when using for real [ie without "double current, chuck originals away and do it again"], should be great quality.

Yes, in this case the optimization task is more difficult than the way the script is used for real, which should make the real case better quality.

[zorr]

[EDIT] Tutorial updated to match the features of the latest version of AvisynthOptimizer (no longer using VirtualDub to run scripts, better error handling, colored text)

It's finally time for you to get your greedy hands on AvisynthOptimizer! Download the zip here (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) and unzip to a folder of your liking. You will also need my modded AvsTimer and the SSIM plugin with the SSIM_FRAME-function which returns the SSIM value to the script. I have a package which contains both of those here (https://drive.google.com/open?id=1iKH05RPHMywkRtka4b4o--LrES8hmAbO). Let's first see what is inside the AvisynthOptimizer package:

lib (folder)
AviSynthOptimizer.jar
avsr.exe
avsr64.exe
optimizer.bat
optimizer.ini
versionHistory.txt
The optimizer is written in Java so you need to have Java Runtime. If it's not installed yet, go ahead and grab it from here (http://www.oracle.com/technetwork/java/javase/jre8-downloads-2133155.html).

AvisynthOptimizer.jar contains the main code. There are some external jar dependencies in the lib-folder.

avsr.exe and avsr64.exe this is Groucho2004's avsr (https://forum.doom9.org/showthread.php?t=173259) utility which is used to run the scripts (thanks Groucho2004!)

optimizer.bat is a batch file you can use to run the optimizer.

optimizer.ini contains some configuration settings. In it's initial state it looks like this:

architecture=
log=

The first line tells the default Avisynth architecture (x86 or x64) used to run the scripts. The optimizer will ask you which one to use on the first time it's started (unless you're using 32bit Windows, in which case it doesn't need to ask) and write the value here. You can also override the default value using the -arch parameter.

The second line "log=" doesn't have a value either. The optimizer will update here the latest log file it has written to. It's useful when evaluating logs (we don't have to specify the log file, it defaults to the file found in this .ini file).

versionHistory.txt what could this be? yes, it's the version history.

Let's move on to our first optimizing task. When I was trying to come up what a suitable first demonstration I found this thread (https://forum.doom9.org/showthread.php?t=114582) where Fizick was comparing the quality of denoisers using SSIM. One of the first filters he tried was FFT3DFilter. The script is short and simple and shouldn't be too difficult to optimize. Let's find out if we can beat Fizick's best settings! :D

The challenge introduced in that thread is the following: introduce noise to a video and then try to remove it with the denoiser, finally compare the denoised frames to the original frames with SSIM. The goal is to get as good similarity as possible. The video used can be downloaded here (http://trace.eas.asu.edu/yuv/flower/flower_cif.7z). It's in raw 4:2:0 YUV format and can we read with RawSource (https://github.com/chikuzen/RawSource_2.6x/releases).

Let's first make a script that has the SSIM comparison and just for a good measure the timing measurement as well. If I was doing a serious optimization I would measure more than 5 frames but let's keep the runtime fast in this example. Note that we're adding the noise using a constant seed value so that the noise is exactly the same every time we run the script, making sure that there isn't an unknown variable messing with our quality measurements. The script is using the best FFT3DFilter settings found in the thread. Change the Rawsource path and resultFile path if needed. The example script has an absolute path for the source file but it's not necessary.

TEST_FRAMES = 5 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

RawSource("D:\optimizer\test\flower\flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)

denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(source, denoised)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last
Save the script as "denoise.avs" and run it in VirtualDub (or whatever you normally use to run the Avisynth scripts) by double-clicking the file. In VirtualDub press play once and then look for the perFrameResults.txt file. It should contain something like this:

0; 0.987766; 12.808190
1; 0.987759; 6.340095
2; 0.987965; 6.358214
3; 0.987979; 5.819328
4; 0.987961; 5.781337
stop 4.939430
The quality values (0.987...) should be the same as above, your runtimes will be faster or slower than mine. The last line with the sum of SSIM values should be the same as well.

Ok, so 4.939430 is the result we're trying to beat. Now let's change the script a bit to make it ready for the optimizer.

Replace this part

denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

with this:

sigma = 400/100.0 # optimize sigma = _n_/100.0 | 100..800 | sigma
bt = 4 # optimize bt = _n_ | -1..5 | blockTemporal
blockSize = 32 # optimize blockSize = _n_ | 2..64 | blockSize
overlap = 16 # optimize overlap = _n_ | 0..32 | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

When you start optimizing you have to guess what the suitable ranges are for each parameter. Since we already know at least one very good set of parameters we could set the range around those values. But let's not make it too easy for the optimizer and instead give it relatively large ranges. Besides, maybe the best result isn't near the values Fizick found, you never know.

We're going to try values 1.0 - 8.0 for the sigma, values from -1 to 5 for the bt (the full range allowed), values 2 - 64 for the bw and bh and 0 - 32 for the ow and oh. The last two are the overlap in x and y direction, they can use the same value because we have no reason to believe they should be different (at least in this case where the noise is uniform in all directions). The same applies for the block's size. In general it's a good idea to try to keep the number of parameters to optimize small, it makes the optimization job easier.

Now we can start the optimizer. Open a command line window in the directory you installed it to and write

optimizer <path_to_your_script>

The only thing you need to give is the path to the script you want to optimize. Mine was "../test/flower/denoise.avs".

Now since this is the first time you're running the optimizer it will ask the preferred Avisynth architecture:

Which Avisynth architecture are you (mostly) using?
NOTE: this default setting can be overridden with -arch argument
1 - 32bit (x86)
2 - 64bit (x64)

Just answer with 1 or 2 <ENTER>. The optimizer should start running and displaying the tested parameter combinations. However, it soon stops and displays an error message:

Error in script execution:

FFT3DFilter: Must not be 2*ow > bw
<script path and line number>

If you look at the last line you can see that the script is located at <optimizer_install_path>/work. That's the work folder where the optimizer is creating the scripts into. If there's an error the latest script will still be there so you can try to debug it. Also if you abort the optimization (with CTRL+C and answering Y when asked if you want to terminate the batch job) the optimizer doesn't have a chance to clean up properly so there may be some leftover script files in the work folder. It's safe to delete all of them if you're not currently running an optimization.

Let's figure out what the error message is trying to say. Looks like there is a dependency between the parameters ow and bw. We should add those to our script and try again. This dependency is almost the same we already saw in the MVTools script, namely that overlap cannot be larger than half the blockSize. The dependencies are thus:

blockSize = 32 # optimize blockSize = _n_ | 2..64 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 0..32 ; max:blockSize 2 / | overlap

Remember that we should define the dependency for both parameters involved in order not to introduce bias. Bias is bad! After you've replaced those lines run the optimizer again. This time it should be happy and keep on going.

Mutating 2 params by 28,5 %
RESOLVED: blockSize 5 -> 14
RESOLVED: blockSize 10 -> 36
RESOLVED: overlap 10 -> 6
RESOLVED: overlap 4 -> 3
* 105 / 2000 : 4.946868 50ms sigma=514 blockTemporal=5 blockSize=11 overlap=5
106 / 2000 : 4.9306154 90ms sigma=717 blockTemporal=5 blockSize=26 overlap=13
107 / 2000 : 4.911076 30ms sigma=726 blockTemporal=4 blockSize=5 overlap=2
108 / 2000 : 4.932592 60ms sigma=732 blockTemporal=5 blockSize=14 overlap=7
+ 109 / 2000 : 4.8711557 10ms sigma=752 blockTemporal=4 blockSize=16 overlap=0
110 / 2000 : 4.933645 70ms sigma=625 blockTemporal=5 blockSize=36 overlap=18
111 / 2000 : 4.937391 60ms sigma=670 blockTemporal=5 blockSize=12 overlap=6
112 / 2000 : 4.918705 30ms sigma=752 blockTemporal=4 blockSize=7 overlap=3
Parameter sensitivity estimation with 256 result combinations
-> sigma 1,573 blockTemporal 1,178 blockSize 0,600 overlap 0,720

This is a short sample from my run. Every once in a while the optimizer tells what mutation settings it is currently using. The mutations are large in the beginning and gradually become smaller towards the end. This is not particularly relevant information, just a small sanity check.

Next we have four RESOLVED lines, they tell us how the conflicts were resolved in the current generation. If you don't see any resolve lines that just means that there weren't any conflicts. The important thing here is that both blockSize and overlap get their values changed. If you have a parameter with dependencies defined but never see it on the resolved line, then something could be wrong about the definitions.

Then we have a list of results. The first number is the current iteration, always increasing by one with every result. The second number is the total number of iterations in this run. The default is 2000. The next two numbers are the SSIM value and runtime which the script calculated. The rest of the line spells out the parameters used with this result. Some of the result lines have different colors and symbols (+, *) in front of them. If you're lucky you might even see the "e" symbol. The * symbol (red line) means this is the best result found so far. Now this needs a bit more explanation... didn't we already conclude that there is not one best result but a pareto front? Yes, but here the best result is determined by sorting the results by the *first* value output from the script (in this case the SSIM value). So whenever we find the best SSIM so far the * is displayed in front of the result. The + symbol (yellow line) signifies that we have found a new pareto front member. And if you see "e", it means this result is exactly as good as some other result already in the pareto front ("e" is short for "equal"). The optimizer never tries the same parameter combination twice so this should be pretty rare, but can happen because sometimes certain parameter don't have much (or any) effect on the result. Whenever you see * or + you know the algorithm is making progress.

The last two lines say some gibberish about parameter sensitivity. This is my invention where the algorithm is trying to determine (based on recent results) how sensitive the parameters are. A sensitive parameter is one that causes a large change in the result when its value changes. The sensitivity value shown is large for sensitive parameters and small for non-sensitive parameters. The reason for this sensitivity business is that the mutations we're making are scaled by the sensitivity in order to avoid too large or too small changes for the parameters. In my testing the optimizer gives better results with the sensitivity estimation than without it so it defaults to being enabled.

2000 iterations is a bit long for this optimization task, so let's stop the optimizer (CTRL+C, Y) and make it run a much lighter task. This time we give it the number of iterations as an argument:

optimizer <path_to_your_script> -iters 100

Now the optimizer will finish much faster, on my machine it only takes about 80 seconds. But wait a minute... it didn't finish after 100 iterations but started another run! Why is it doing that? Here's the deal: randomness is a large factor in the metaheuristic algorithms. This means that the results of an optimization run will be different every time, and the differences can be large. So if you only run the optimization once and have bad luck you get bad results. This is why you should try at least a couple of runs. The default is 5 runs. Let's wait until the optimizer has finished them all. Except that we still have something fun to try...

[zorr]

While the optimizer is running we can run the visualization. Open another command line window in the optimizer's folder and run:

optimizer -mode evaluate

A window opens with a chart of the latest optimization run. I took a shot while the third run was under way:

https://s22.postimg.cc/hld1mrmvl/viz03.png

The grey dots are all the results of the latest run. The red dots which are connected by a black line are the latest run's pareto front. The black larger dots, connected by a grey line, are the global pareto front, meaning the best results from all the runs. These descriptions are also visible in the info box in the bottom right corner. The info box also shows the latest run number and the number of iterations finished.

Over the chart window there's a long text, that's the name of the latest log file where these results are stored. The name is long because it contains all the relevant parameters of the metaheuristic algorithm. It also contains the script name we're optimizing and the timestamp when the optimization started. All this information could be useful later when you have hundreds of log files.

Notice how all the results are clustered on 10ms marks on the X axis, that's because the runtimes are rounded. The chart doesn't show all the results but instead focuses on the best ones. To be precise, the chart shows the best X% of the results. The focused percentage can be defined by a parameter, the default % is 20. This is implemented because the top results are usually the interesting ones. The default percentage can be changed with the argument -top.

Close the visualization window from the X and let's try an automatically updating chart. Start it by typing

optimizer -mode evaluate -autorefresh true

The chart opens like before but now you should see that the visuals are changing every few seconds. You can also see grey rectangles around some of the results. Those rectangles mean these results are among the last N results (default N is 10). If the optimizer starts a new run the autorefreshing chart starts showing it and the run number changes in the info box.

All right, that's enough about visualization for now. In the next phase we will look at the finished results. Go grab some coffee (or something else if you don't like coffee, like me) and come back in less than 8 minutes. Or even sooner if your machine is faster than mine.

...

Now that the optimizer has finished we can try to analyze the results. Start the evaluation once more (autorefresh not needed, there's nothing to refresh)

optimizer -mode evaluate

You can marvel the wonderful chart for a while if you like but we're going to focus on the command line window where we started the visualization from. It displays (among other things) the best result (again ranked by the first output's value) of every run. If there's a lot of variation in those results and their parameter values that means our search was not thorough enough. In that case you might get better results by increasing the iteration count. Here's what I got with 100 iterations:

Run 1 best: 4.940206 70 sigma=704 blockTemporal=5 blockSize=17 overlap=7
Run 2 best: 4.947239 50 sigma=551 blockTemporal=5 blockSize=13 overlap=6
Run 3 best: 4.935884 50 sigma=771 blockTemporal=5 blockSize=19 overlap=5
Run 4 best: 4.941647 90 sigma=607 blockTemporal=5 blockSize=23 overlap=10
Run 5 best: 4.928436 20 sigma=725 blockTemporal=3 blockSize=20 overlap=7

We can see that there's some variation in the SSIM values. And hey, 3 out of 5 are better than our goal, yippee! :D The parameter values are not that close to each other either. To see if more iterations would help I tried 200 iterations: (you don't need to run these)

Run 1 best: 4.94857 60 sigma=486 blockTemporal=5 blockSize=16 overlap=8
Run 2 best: 4.947047 60 sigma=543 blockTemporal=5 blockSize=16 overlap=8
Run 3 best: 4.931532 160 sigma=474 blockTemporal=5 blockSize=43 overlap=21
Run 4 best: 4.942368 30 sigma=464 blockTemporal=3 blockSize=12 overlap=6
Run 5 best: 4.944955 30 sigma=462 blockTemporal=4 blockSize=12 overlap=6

Hmm... not much of an improvement. Let's try 500 iterations:

Run 1 best: 4.9489594 60 sigma=480 blockTemporal=5 blockSize=14 overlap=7
Run 2 best: 4.948967 60 sigma=475 blockTemporal=5 blockSize=14 overlap=7
Run 3 best: 4.948965 60 sigma=485 blockTemporal=5 blockSize=14 overlap=7
Run 4 best: 4.948967 60 sigma=475 blockTemporal=5 blockSize=14 overlap=7
Run 5 best: 4.948962 60 sigma=477 blockTemporal=5 blockSize=14 overlap=7

Wow, look at that! All parameters but sigma are equal in all runs, and even sigmas are varying very little. Looks like we've found the optimal settings, or at least something very close to it. Just for kicks this is the result with 2000 iterations:

Run 1 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 2 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 3 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 4 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 5 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7

The evaluation also shows the best and worst result of the series and the pareto front of the whole series (the global pareto front). These are the best results from all of the runs. The pareto front is sorted by the first output's value, best value first. Here's my pareto front from the 100 iterations series:

Pareto front:
4.947239 50 sigma=551 blockTemporal=5 blockSize=13 overlap=6
4.942367 40 sigma=566 blockTemporal=5 blockSize=16 overlap=6
4.9421406 30 sigma=551 blockTemporal=4 blockSize=14 overlap=6
4.93908 20 sigma=560 blockTemporal=3 blockSize=14 overlap=6
4.917379 10 sigma=790 blockTemporal=3 blockSize=15 overlap=2

As you can see the first result has the best SSIM and the worst runtime, going down the list the SSIM gets worse and the runtime gets better. This is a pretty fast script to run, even the longest time was only 50 milliseconds so in this case we can forget about the runtime and choose the result with the best SSIM value. However, some other plugins can have wildly different runtimes depending on the used parameter values (MVTools is among those). In that case you should consider if the extra quality is worth the increased runtime.

One final step in this tutorial: let's create the scripts from the pareto front.

optimizer -mode evaluate -scripts true

After you've run that you should find Avisynth scripts with names like "denoise_PARETO_01_4.947239_50.avs". The name contains the script's name, the word PARETO, a number which stands for the order in the pareto front (01 -> best, 02 -> second best and so on) and the result values (SSIM and runtime). You can then run those scripts and compare the quality with your very own eyeballs.

So far we have always evaluated the latest optimization run. If you need to go back to earlier ones, give the name of the log file(s) with the -log parameter. You can use the "*" wildcard in the name, for example if you want to analyze all logs starting with "denoise" you could run

optimizer -mode evaluate -log "../test/flower/denoise*.log"

Note that quotes are needed when using the wildcards.

I will give you one more option to play with: the -vismode (short for "visualize mode") which can take the values none, single, series and seriespareto.
In the next episode we will take a closer look at the optimizer and how to customize the optimization process besides the iteration count.

[Groucho2004]

process.vfc - this is a VirtualDub script that runs the Avisynth script using "Null video pass" meaning it doesn't write the resulting video anywhere. runavs.bat is passing this file to VirtualDub.There are much easier ways to run a script.
For example: "avsmeter script.avs -o". :cool:

[StainlessS]

JFYI, (anybodies info),
On WXP32 (last Firefox for XP32), I was getting a 'Page Is not redirecting properley' type message, but Downloaded OK on W10 (current Firefox).

EDIT: Both links at top of post #12.

[zorr]

There are much easier ways to run a script.
For example: "avsmeter script.avs -o". :cool:

D'oh! :o To my defense, I did try a few other options for running avs scripts (like AVS2AVI) but found that they weren't as reliable as VirtualDub. Back then I wanted to save the video too so I didn't investigate AVSMeter.

Luckily (or rather, by design :cool:) it's very easy to use AVSMeter instead of VirtualDub, you only need to change the runavs.bat to something like this:

title %2
"D:\optimizer\bin\tools\AvsMeter\AvsMeter" %1 -o
exit

However, I ran into some problems when I tried it. Running the denoise.avs gave this error:

D:\optimizer\bin\tools\AvsMeter>avsmeter ../../../test/flower/denoise.avs -o

AVSMeter 2.8.5 (x86) - Copyright (c) 2012-2018, Groucho2004
AviSynth 2.60, build:Mar 31 2015 [16:38:54] (2.6.0.6)

Number of frames: 5
Length (hh:mm:ss.ms): 00:00:00.200
Frame width: 352
Frame height: 288
Framerate: 25.000 (25/1)
Colorspace: i420


Exception 0xC0000094 [STATUS_INTEGER_DIVIDE_BY_ZERO]
Module: D:\optimizer\bin\tools\AvsMeter\AVSMeter.exe
Address: 0x00324CD3

Without the -o there was no exception but this report
Script runtime is too short for meaningful measurements
and the perFrameResults.txt was not generated.

I was able to make it work by adding more test frames to the script, it worked with TEST_FRAMES = 50.

Another issue is that AVSMeter takes quite a long time (several seconds) displaying "Query Avisynth info..." which then gets replaced with the AviSynth version information. Would it be possible to skip this part and start running the script right away? I didn't find a switch to suppress that.

[zorr]

JFYI, (anybodies info),
On WXP32 (last Firefox for XP32), I was getting a 'Page Is not redirecting properley' type message, but Downloaded OK on W10 (current Firefox).

Thanks for the report. I wasn't able to reproduce this but then again I'm using W10 too. Is anyone else having problems with the downloads? Would it help if I linked to the folder containing the files instead of linking directly to the files? The files are at Google Drive.

[EDIT] I replaced the download links, I probably used incorrect ones earlier.

[Groucho2004]

However, I ran into some problems when I tried it.Try "avsr (https://forum.doom9.org/showthread.php?t=173259)" instead.

[zorr]

Try "avsr (https://forum.doom9.org/showthread.php?t=173259)" instead.

That's perfect, thank you!

I decided to run some benchmarks which one is faster and also do some tests with the poll frequency. Apparently avsr is starting up faster than VirtualDub.

When running a validation on the first pareto front of denoise.avs (2000 iterations) it's finished with VirtualDub in 54.1 seconds but with avsr it's 0.6 seconds faster (measured average of 5 runs). If I change the poll frequency from 100ms to 50ms the difference is even larger, about 1.8 seconds. I tried intervals 25ms and 10ms too, there's still improvement but probably not enough to justify the cost. I decided to change the poll frequency to 50ms. With that change and using avsr the validation runs about 5% faster.

Do you mind if I add avsr to the AvisynthOptimizer package and use it as the default script runner? I will of course document that this is software made by you.

It would be nice if I could determine automatically which Avisynth platform (x86 or x64) the user is running, then I could skip one manual step of the installation.

[Groucho2004]

Apparently avsr is starting up faster than VirtualDub.VDub has a much bigger overhead so that's to be expected.

Do you mind if I add avsr to the AvisynthOptimizer package and use it as the default script runner?No, I don't mind.

It would be nice if I could determine automatically which Avisynth platform (x86 or x64) the user is running, then I could skip one manual step of the installation.Where do you want to determine this? In the batch file? In your software?

[zorr]

VDub has a much bigger overhead so that's to be expected.

Yes I was expecting that too. But data is always better than expectations. :)

No, I don't mind.

Cool, I will add it in the next version.

Where do you want to determine this? In the batch file? In your software?

In my software so that I could select automatically which version of avsr is going to end up in the runavs.bat. Otherwise user has to go and edit the runpath in the optimizer.ini.

An alternative would be to make a little setup program where the user could select the correct version from the two alternatives and .ini would be created based on that.

[Groucho2004]

It would be nice if I could determine automatically which Avisynth platform (x86 or x64) the user is running, then I could skip one manual step of the installation.You also have to consider that a user can have both 32 and 64 bit Avisynth installed. So, the choice whether to test this with 32 or 64 bit Avisynth is up to the user.

[zorr]

You also have to consider that a user can have both 32 and 64 bit Avisynth installed. So, the choice whether to test this with 32 or 64 bit Avisynth is up to the user.

That's a good point. I'm thinking the .ini defines the default but it can be overridden with a command line argument.

I also want to ask your opinion on the .bat files. It's not necessary to have them (batch can be run even without a file) but I thought it would make hacking the optimizer a bit easier (you can change what you want to run the scripts with). Removing them would make sense now that avsr does the job very well and is included. But should I still leave them in order to let others decide the script runner?

[wonkey_monkey]

How about an argument in the script that tells it whether to use x64 or x86?

Some scripts might always require one or the other.

[zorr]

How about an argument in the script that tells it whether to use x64 or x86?

Some scripts might always require one or the other.

That's a good point too. So it would go like this:

Use the platform defined in the .ini
EXCEPT if the platform is defined in the script, then use that instead
EXCEPT if the platform has been given as a command line argument, then use that

[StainlessS]

And what if no ini file (or script or command arg, use native-ness).
.

[zorr]

And what if no ini file (or script or command arg, use native-ness).
.

Great idea, except that I have no idea how to find "native-ness". :)
Besides whatever automatic method I can come up with it's still going to go wrong when people have both platforms installed. So I think I'm just going to ask the platform the first time the optimizer is run.

[Groucho2004]

So I think I'm just going to ask the platform the first time the optimizer is run.Can you elaborate on "ask the platform"? I'm not sure what you mean by that.

[wonkey_monkey]

Ask the user what platform to use.

[Groucho2004]

Ask the user what platform to use.When I hear/read "platform" I naturally associate OS platform. In this context it's obviously about the "Avisynth platform" which sounds a bit odd but that's probably just me.

[zorr]

When I hear/read "platform" I naturally associate OS platform. In this context it's obviously about the "Avisynth platform" which sounds a bit odd but that's probably just me.

Yes the idea is to ask the user which "Avisynth platform" should be the default. I'm open to suggestions on what to call that, I'm not native english speaker. :) Perhaps "version" would flow better but then again people might confuse that with version numbers. Architecture?

Perhaps I should present the question as "Are you using (mostly) 32bit or 64bit Avisynth?". But I still need a good name for the argument, right now it's -platform.

[Boulder]

This is an interesting concept, it could be quite useful in testing parameters for downscaling 4K to 1080p or 1080p to 720p (based on the fact that they will be upscaled to 4K upon playback). So, any chance this could be made Vapoursynth-compatible as well?

[ChaosKing]

It would be nice if the download links could be on the first page.
I think adapting it to VS could be achieved with the SSIM() function from muvsfunc. https://github.com/WolframRhodium/muvsfunc/blob/c223e9d8099d19b24d743e1450f67deb116ebf69/muvsfunc.py#L3371

But I don't know for what the avstimer.dll is good for...

[Groucho2004]

Yes the idea is to ask the user which "Avisynth platform" should be the default. I'm open to suggestions on what to call that, I'm not native english speaker. :) Perhaps "version" would flow better but then again people might confuse that with version numbers. Architecture?

Perhaps I should present the question as "Are you using (mostly) 32bit or 64bit Avisynth?". But I still need a good name for the argument, right now it's -platform.
First you should determine which Avisynth versions are installed.

If only 32 bit Avisynth is installed, run the 32 bit process, if only 64 bit Avisynth is installed, run the 64 bit process.
If both are installed, ask the user something like "With which Avisynth version do you want to run the test?" and then present the two choices.

[zorr]

This is an interesting concept, it could be quite useful in testing parameters for downscaling 4K to 1080p or 1080p to 720p (based on the fact that they will be upscaled to 4K upon playback).

That certainly could be done. The only difficulty in this would be that Avisynth would have to simulate the upscaling of the playback device (or is it upscaled with Avisynth during the playback?).

So, any chance this could be made Vapoursynth-compatible as well?

I haven't looked at Vapoursynth so I can't tell for sure. But basically what's needed for the optimizer to work is:
1) a way to run a script programmatically from the optimizer
2) a way to specify which values the optimizer is changing in the script
3) the script has to calculate a result: quality / runtime / any other interesting value
4) the script has to write the result into a specific file

I'm very much interested in making this happen but hopefully someone who knows Vapoursynth can help me figure out the steps above.

[zorr]

It would be nice if the download links could be on the first page.

Thanks for the suggestion, I added the download links there.

I think adapting it to VS could be achieved with the SSIM() function from muvsfunc. https://github.com/WolframRhodium/muvsfunc/blob/c223e9d8099d19b24d743e1450f67deb116ebf69/muvsfunc.py#L3371

SSIM or any other means to measure quality will work.

But I don't know for what the avstimer.dll is good for...

AvsTimer is for measuring the runtime of the script. It's not absolutely necessary if you don't care how fast or slow the script is.

[zorr]

First you should determine which Avisynth versions are installed.

Is there an easy way to do that? I guess I could scan the registry for installed software. Can I determine the platform from the software's name?

[EDIT]

32bit Avisynth can be found at
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\AviSynth
when installed on a 64bit system.

Is the 64bit version at
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Avisynth
?

What about if Windows is 32bit? I guess there can only be 32bit Avisynth but it will be at the same spot in the registry as 64bit Avisynth on a 64bit Windows.

[wonkey_monkey]

"Architecture" may be a better word than platform.

[Groucho2004]

Is there an easy way to do that? I guess I could scan the registry for installed software. Can I determine the platform from the software's name?
Checking the registry is probably the best way. Here are the keys:
32 bit AVS on 32 bit Windows:
HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth]
@="<avisynth install path>"

32 bit AVS on 64 bit Windows:
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Avisynth]
@="<avisynth install path>"

64 bit AVS on 64 bit Windows:
[HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth]
@="<avisynth install path>"

[StainlessS]

q) Is the registry always automatically cleared, upon uninstall ? (I doubt it)

[Groucho2004]

q) Is the registry always automatically cleared, upon uninstall ? (I doubt it)I'm pretty sure that this is the case for all current installers (AVS 2.6, AVS+). The Universal Installer also cleans up properly after uninstall.

If the registry entry is just an orphan and avisynth.dll is not present, the application loading Avisynth will (well, should) throw an appropriate error.

[StainlessS]

OK, was just going by the fact that many other apps leave their keys insitu on uninstall.

[Boulder]

That certainly could be done. The only difficulty in this would be that Avisynth would have to simulate the upscaling of the playback device (or is it upscaled with Avisynth during the playback?). It would be simplest to upscale with Avisynth during playback, maybe just use a common LanczosResize or Spline to make sure the upscale remains as sharp as possible without introducing other artifacts.


I haven't looked at Vapoursynth so I can't tell for sure. But basically what's needed for the optimizer to work is:
1) a way to run a script programmatically from the optimizer
2) a way to specify which values the optimizer is changing in the script
3) the script has to calculate a result: quality / runtime / any other interesting value
4) the script has to write the result into a specific file

I'm very much interested in making this happen but hopefully someone who knows Vapoursynth can help me figure out the steps above.Myrsloik, the author of Vapoursynth, is very knowledgeable so I'll post in the Vapoursynth thread to point here. At least point 1 is there out of the box, vspipe can be used to pipe the output of the script and it's commonly used with encoders. The others should also be available as VS is Python-based and thus quite flexible in itself.

[zorr]

I wanted to get rid of the polling so instead I tried waiting for the avsr process to finish until I start reading the output file. That works but is about 10% slower (when running a validation with a fast script) than the polling method.

Another thing I'm working on is improved error handling. The latest avsr will output the script error messages into standard error stream which I can read from the optimizer. However that same 10% penalty comes when I read avsr's output streams fully before I start checking the output file. That makes sense since reading the streams until nothing else is coming is pretty much the same thing as waiting for the process to finish. So what I'm going to try next is reading the streams while I'm polling the output file. In that way I can hopefully enable the improved error handling and keep the faster execution speed.

Apologies for the slow progress. I recently bought a house and now I have to plan the moving and renovations...

[Myrsloik]

That certainly could be done. The only difficulty in this would be that Avisynth would have to simulate the upscaling of the playback device (or is it upscaled with Avisynth during the playback?).



I haven't looked at Vapoursynth so I can't tell for sure. But basically what's needed for the optimizer to work is:
1) a way to run a script programmatically from the optimizer
2) a way to specify which values the optimizer is changing in the script
3) the script has to calculate a result: quality / runtime / any other interesting value
4) the script has to write the result into a specific file

I'm very much interested in making this happen but hopefully someone who knows Vapoursynth can help me figure out the steps above.

1. Use vspipe if all you need is to calculate a frame statistic
2. vspipe has --arg key=value which can set variables inside the script
3. per frame or for the whole thing? not sure what you need here
4. depends on the answer in 3, I plan to add the possibility to dump all frame properties as json in a future vspipe release

[zorr]

1. Use vspipe if all you need is to calculate a frame statistic
Ok, I'll take a look at vspipe. Is there a simple vapoursynth script example I could use to test it with?

2. vspipe has --arg key=value which can set variables inside the script
That sounds much more elegant than the way it's done with Avisynth. But I forgot to mention that usually you also need to define dependencies between variables (for example this value cannot be larger than that other value). Those definitions would still have to be inside the script. In Avisynth the definitions are inside comment blocks on the same line as the variable.

3. per frame or for the whole thing? not sure what you need here
Per frame is not necessary, just a nice feature (the per frame values are summed by the optimizer). But you can do the summing easily in the script and output a single final result. Most common operation is to calculate a SSIM similarity metric and/or the frame's runtime for each frame and sum those.

4. depends on the answer in 3, I plan to add the possibility to dump all frame properties as json in a future vspipe release
It doesn't need to happen within vspipe, the script could write the file. At least that's how it works with Avisynth.

[zorr]

Checking the registry is probably the best way. Here are the keys...
I considered this but reading the Windows registry from Java probably isn't worth the hassle (it requires external libraries). I think I will settle for a simple OS check - on a 32bit windows Avisynth is 32bit too. On a 64bit Windows I will ask the Avisynth architecture on the first run (or if information no longer present in .ini file). And the default value can be overridden with -arch parameter (thanks davidhorman for the suggestion).

[zorr]

New version 0.9.2-beta is released. I have changed the download links to point to the new version.

This version no longer uses VirtualDub to run the scripts, instead it's using Groucho2004's excellent avsr utility which is included.

The error handling is improved and any script errors will be displayed on the console window.

I will have to change the tutorial to reflect these changes. Should I just edit the original messages or post a completely new version of the tutorial? If I change the original it will be easier for people who start reading from the beginning but it will be difficult to understand the discussion that follows.

[StainlessS]

Just update original posts, and post advisory that is updated.
It is your thread to do with as you please, within reason.

[zorr]

I have updated the Hands-on tutorial (https://forum.doom9.org/showpost.php?p=1851670&postcount=12) to match the features of the latest AvisynthOptimizer version.

[zorr]

Version 0.9.3-beta (https://drive.google.com/open?id=1qwrfRAij12rz1fXcf-mERSSuKw-7397f) released. The only change is that avsr was upgraded to latest version 0.1.7.

[zorr]

It's time to take a closer look at how to adjust the optimization process. Let's run the optimizer using the same script and settings used in the last tutorial:

optimizer <path_to_your_script> -iters 100

The program displays:

Arguments
iters = 100

Running optimization for script d:/optimizer/test/flower/denoise.avs
Using these settings:
ARGUMENT DESCRIPTION VALUE
-runs runs 5
-alg algorithm spea2
-pop population 8
-iters iterations 100
-mutamount mutation amount 0.3 0.01
-mutcount mutation count 60% 1
-crossprob crossover probability 0.1
-crossdist crossover distribution 20
-sensitivity sensitivity estimation true
-dynphases dynamic phases N/A
-dyniters iterations per phase N/A

You can stop the optimization after this text is displayed.

The "Arguments" section lists the arguments and their values as they were understood by the optimizer.

The next section is a handy cheat sheet on what arguments are available and their current values. The first column ARGUMENT tells the argument name you can use to specify the setting. The DESCRIPTION column contains a short description of what the argument does. And finally the VALUE is the current value used for the argument. Most of these are using the default values, we only specified the -iters argument. If you run the optimizer in another mode (like "evaluate") the listed arguments are specific to that mode.

I spent quite a while figuring out good default values so they should work reasonably well, but I have only tested them on a few different optimization tasks so they might not be good for every case. It takes a lot of effort to test these settings because to determine if one value is better than another one should run the optimization task many times with each value in order to gain enough statistical significance. I mostly used 20 runs per parameter value.

Let's take a look at the arguments one by one.

-runs specifies the number of optimization runs. a "run" is one complete optimization cycle which itself is specified with the -iters argument. I talked about the need for multiple runs earlier but I will repeat the points here: Since the optimization process is depending on random numbers the outcome is not always the same and there can be large differences in the final result. If you only run the optimization once you cannot be really sure whether the results are good or bad. Another useful aspect of multiple runs is that the variance of the best result can tell us about how easy or hard this optimization task is. Large variance means difficult task. And if the task is difficult we can try to increase the iterations. I don't have a good answer on how many runs are enough. If you can only run N iterations should you run for example three runs with N/3 iterations or eight runs with N/8 iterations? More iterations is better but more runs is also better, to a point.

-alg specifies the metaheuristic algorithm used in the optimization. Currently there are three options: "nsga-ii", "spea2", "mutation" and "exhaustive". NSGA-II and SPEA2 are very good and well known algorithms. I got slightly better results with spea2 so it's the default. If you're interested in how these algorithms work you should check out the free ebook Essentials of Metaheuristics (https://cs.gmu.edu/~sean/book/metaheuristics/). The third option "mutation" is a very simple algorithm I wrote which only uses mutation. It can find a reasonably good result faster than the other algorithms but it will lose with large iteration counts. Finally we have the "exhaustive" option, it simply tries all the possible (and valid) parameter combinations. It can be useful if you only have a few parameters and can limit the number of values per parameter so that the number of combinations doesn't get too high. I have tried some other metaheuristic algorithms like CMA-ES, BFGS (Broyden–Fletcher–Goldfarb–Shanno) and SMPSO (particle swarm algorithm) but I didn't get as good results with them. The SMPSO is still waiting for a more thorough examination, it is promising. I should also note that the algorithms I'm using are not the basic variations, I have changed the way the mutations work and got better results that way.

-pop specifies the population size which is a term often used with genetic algorithms. It's basically how many individual results are kept in memory during the optimization. The genetic algorithms (like NSGA-II and SPEA2) work by doing crossovers between two individuals and then mutating (randomizing) the results slightly. The crossover operation takes some values from one individual and some from the other. The new individuals are rated and finally the best ones are selected as the new "generation". The default population size of 8 seems very small and maybe you're wondering why it should be small at all, after all it's not a problem to keep thousands or even millions of results in memory. Yes, in theory you should get better results with a larger population size but it does have a drawback: it makes the progress slower. If the population size is much larger than the size of the pareto front that means many less than optimal results are kept around and are used in the crossovers. Combining two bad results might create a very good individual but it's more likely to happen when combining two good results. But if you are going to run with a large iteration count then perhaps increasing the population size will also help. A larger population may also be needed with a difficult optimization task. If you want a reasonably good result fast use the "mutation" algorithm with a population size of 1.

-iters specifies the number of iterations. One iteration means one execution of the script we're trying to optimize. You can give the iteration count as a number (for example 1000) but there are other indirect ways. You can give a time limit in days, hours and minutes. For example 5h30m would run 5 hours and 30 minutes. 1d12h would be one day and 12 hours. You can use spaces if you put quotes around the value, for example "2h 45m". Using the time limit can be useful if you have a specific deadline for the results, or if you want to try what the optimizer can find during the night while you sleep. Just remember that the time limit applies to a single run, so if you start an optimization with 3 runs and 1h iterations it will take a total of 3 hours. During the optimization the maximum iteration count is still displayed on each result line but it is only an estimation.

20 / 345 : 4.772059 20ms sigma=349 blockTemporal=1 blockSize=50 overlap=15
21 / 347 : 4.815294 20ms sigma=477 blockTemporal=2 blockSize=64 overlap=15
22 / 349 : 4.875304 20ms sigma=591 blockTemporal=2 blockSize=30 overlap=6
23 / 350 : 4.880693 110ms sigma=800 blockTemporal=5 blockSize=61 overlap=7
24 / 343 : 4.909643 150ms sigma=800 blockTemporal=5 blockSize=61 overlap=21

There also also two keywords that trigger a special dynamic iteration mode: "dyn" and "dynbk". dyn stands for "dynamic" and means the iteration count depends on how the optimization is progressing. There are two additional arguments that define how the dynamic iteration is behaving: -dynphases and -dyniters. -dynphases defines how many distinct "phases" the algorithm is using (default is 10 but it's displayed as "N/A" in the example since it does not apply to the chosen iteration method). The phase goes from 0.0 to 1.0 during the iteration and affects how large and common mutations are. In the beginning (phase 0.0) the mutations are large and applied to many parameters. In the end (phase 1.0) they are small and applied to few parameters (in general, you can also change that). If dynphases has a value of 10 the phase range is divided into 10 steps and the phases used are thus 0.0, 0.1, 0.2, ..., 0.9 and 1.0. The dynamic iteration stays at the current phase step as long as it's still making progress. The "not making progress" is triggered when there has not been new pareto front results in the last -dyniters iterations. In that case the algorithm moves to the next phase step and resets the counter. The default value of -dyniters is also 10. "dynbk" is much like "dyn" but it can also move backwards (hence the name, "dynamic backtracking") to the previous phase step. That happens whenever a new pareto front result is found. I have not done a comprehensive study on which of these algorithms gives better results. The dynamic iteration counts are useful when you are not limited by a specific deadline and just want to find the best result. When you're running a dynamic iteration the phase step changes are displayed in the console, for example:

6 iterations remaining is this generation
No improvement in 10 iterations - moving to phase 1/10

Also instead of displaying the maximum iteration number the current phase is displayed (there's no reliable way to estimate maximum iteration count):

43 / 0,10 : 4.470519 20ms sigma=474 blockTemporal=-1 blockSize=22 overlap=0
44 / 0,10 : 4.90395 60ms sigma=665 blockTemporal=3 blockSize=41 overlap=10
45 / 0,10 : 4.757739 10ms sigma=490 blockTemporal=3 blockSize=32 overlap=0

-mutamount specifies the mutation amount. The amount is proportional to the allowed value range given in the script for each parameter you're optimizing. For example if you have a parameter with a value range 0..100 that would mean the parameter has 101 valid values. This number is multiplied by the mutation amount to get the largest possible change the mutation is allowed to make. So with mutation amount 0.2 the mutations would vary from -20.2 to 20.2. You can give two values for the mutation amount, the first is used in the beginning (phase 0.0) and the last in the end (phase 1.0) and linearly interpolated in between. If you only give one value that is used in all phases.

-mutcount specifies the mutation count. Whenever mutation is applied the first step is deciding how many parameters will be mutated and this argument defines just that. Like with -mutamount you can give a different value for the beginning and end phases. What's more the count can be given as a percentage of the number of optimized parameters in the script. So if you have 20 parameters to optimize and specify -mutcount 50% the algorithm will mutate 10 parameters. You can mix both presentations, for example the default -mutcount is "60% 1" which means mutating 60% of the parameters in the beginning and one in the end.

-crossprob specifies the probability of the crossover operation. If the probability is 1.0 the operation is applied to every new individual, if it's 0.0 it is never applied. In my tests I have found that this crossover argument is not that critical for a successful optimization.

-crossdist specifies the "distribution index" of the "simulated binary crossover" which is the crossover method used in NSGA-II and SPEA2. To be honest I don't fully understand what it does. I haven't investigated what value would be optimal for this argument.

-sensitivity specifies whether the sensitivity estimation algorithm is used. This algorithm is trying to determine how "sensitive" each parameter is, that is how much changing the parameter's value will affect the result. The sensitivity is then used by scaling the applied mutation amounts. The results are usually better when sensitivity estimation is on. If you want to switch it off set the value as "false".

Now you know how to change the optimization process. The default values are good most of the time but feel free to try different things. The most important arguments are probably -iters (and -dynphases and -dyniters if dynamic iteration is used), -runs, and -pop, followed by -mutamount and -mutcount. If you find good settings for a specific script please let me know.

In the next episode we will focus on the visualization of the results. :cool:

[zorr]

Version 0.9.4-beta (https://drive.google.com/open?id=1imbzhqHIuGPEHq1qfzUIGe6mTmvhOTxA) released. Some excessive logging removed in timed iteration mode.

[Seedmanc]

Ok, so I gave it a try, here are the impressions.
First off, the Avstimer failed to load on Win10/Avisynth+MT, when loading via "loadplugin" it errored with "platform returned code 126: module not found", the way it reacts when I try to load a non-existant dll. I tried replacing avisynth.dll with the one from non-plus version, but it only changed the wording of the error.
Fortunately my main OS is Win7 with Avisynth 2.6MT installed where it worked, however Avstimer always returned time of 9999999ms. I suppose that made the optimization task much less efficient, because where I expected it to take tens of minutes for 10 720p frames in 3 runs, it took 15 minutes for the first run, 2.5 hours on second and 1.5 hours on third. I ran it with algo "mutation" and 100 iters. My system is Core i5 2550k OC'd to 4.3GHz, 16Gb RAM

Here's a script I used, modified from what you offered in the other thread:

TEST_FRAMES = 10
MIDDLE_FRAME = 600

# original framerate
FPS_NUM = 30
FPS_DEN = 1

# source clip
Asrc=FFmpegSource2("f:\Hibikin - Watashtachi wa Zutto... Deshou (AVS test video 60fps 720p 10bit CRF0).mkv" ).assumefps(60)
Asrc=Asrc.trim(0,60*30-9)+Asrc.trim(60*40+60*60+9,0) # this is usually the part I worked with when manually adjusting the parameters before
asrc.selecteven
AssumeFPS(FPS_NUM, FPS_DEN)

#return last

# needed for some parameter combinations
ConvertToYV24()

orig = last


super_pel = 2 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, orig )

blockSize = 32 # optimize blockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:divide 0 > 8 2 ? ; filter:overlap 2 * x <= | blockSize
searchAlgo = 3 # optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 4 # optimize searchRange = _n_ | 1..30 | searchRange
searchRangeFinest = 4 # optimize searchRangeFinest = _n_ | 1..60 | searchRangeFinest
lambda = 16000 # optimize lambda = _n_ | 0..20000 | lambda
lsad=400 # optimize lsad=_n_ | 8..20000 | lsad
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=0 # optimize plevel=_n_ | 0..2 | plevel
overlap=16 # optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
divide=2 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true # optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 2000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 4..50 | badRange
meander = true # optimize meander = _n_ | false,true | meander
temporal = false # optimize temporal = _n_ | false,true | temporal
trymany = false # optimize trymany = _n_ | false,true | trymany
dct = 0 # optimize dct = _n_ | 0,2,3,4,5,6,7,8,9,10 D | dct

delta = 1
useChroma = true
bv = MAnalyse(super_render, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
fv = MAnalyse(super_render, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)



threshold = 10000
maskScale = 100 # optimize maskScale = _n_ | 1..300 | maskScale
mask_fps = 2 # optimize mask_fps = _n_ | 0..2 | mask_fps
inter = orig.MFlowFPS(super_render, bv, fv, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )

# return this to look at the clip with doubled framerate
#return inter

fps_only = inter.SelectOdd()

# second pass
super_render2 = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only )
bv2 = MAnalyse(super_render2, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)
fv2 = MAnalyse(super_render2, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)
inter2 = fps_only.MFlowFPS(super_render2, bv2, fv2, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )
fps_only2 = inter2.SelectOdd()


delimiter = "; "

inter_yv12 = fps_only2.ConvertToYV12()
orig_yv12 = orig.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

global total = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig_yv12, inter_yv12, delta, frame_count")

# NOTE: AvsTimer call should be before the WriteFile call
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
resultFile = "f:\avsoptim\results\perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="f:\avsoptim\results\perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

A few notes:
1) I dropped the RemoveGrain call since my sources are CGI and clean enough already.
2) The FrameEvaluate you use doesn't seem to be the native one, as it complained that it doesn't have the argument "args". It worked when I installed GScript, you might want to add that to the list of dependencies.
3) Shouldn't the optimizer params description for DCT include the D flag, since, much like the searchAlgo param, it is "non-linear" and we can't make assumptions about the value and effect? I put the flag there.

The results were confusing.

Run 1 best: 9.534311 9999999 super_pel=4 super_sharp=1 super_rfilter=2 blockSize=8 searchAlgo=4 searchRange=4 searchRangeFinest=8 lambda=0 lsad=461 pnew=255 plevel=1 overlap=0 divide=0 globalMotion=false badSAD=1424 badRange=50 meander=true temporal=true trymany=false dct=8 maskScale=123 mask_fps=2
Run 2 best: 9.535164 9999999 super_pel=4 super_sharp=0 super_rfilter=0 blockSize=12 searchAlgo=4 searchRange=28 searchRangeFinest=60 lambda=1855 lsad=7420 pnew=136 plevel=0 overlap=6 divide=0 globalMotion=true badSAD=6286 badRange=14 meander=false temporal=true trymany=true dct=6 maskScale=177 mask_fps=2
Run 3 best: 9.553077 9999999 super_pel=4 super_sharp=0 super_rfilter=3 blockSize=12 searchAlgo=4 searchRange=1 searchRangeFinest=34 lambda=18163 lsad=8 pnew=9 plevel=2 overlap=6 divide=0 globalMotion=true badSAD=9869 badRange=10 meander=true temporal=false trymany=true dct=5 maskScale=23 mask_fps=2

I can see some tendencies when analyzing the logs manually, sorting by the SSIM so there's that at least. Visually, however, I can't say it looks better than what the hand-picked parameters provide, but that's to be expected from a first attempt with a low iter count.
I think the large search space (22 parameters) might have affected it too. I intend to run this again overnight with whatever iter count it manages to do in time, but first I need to have the Avstimer fixed.
Maybe I should leave the truemotion tuning to later and first try to see what more common set of parameters this tool can generate to compare with the manually tuned ones. Then once I have the good params fixed, I can have it experiment with truemotion.

Another question I wanted to ask, is there a way to estimate the amount of time/iters/population required for a certain amount of tunable script parameters involved? I don't really understand much the math involved, but I have a feeling that either the population size or the iters count should scale up with the parameter count somehow. Seeing how the first run took 15 minutes while the second one 2.5 hours it means not every run will even attempt to cover all parameters involved, since obviously the first one omitted the slowest stuff like DCT 4. But then you said Mutation algo can work with population 1.

I assume, if we were using brute-force instead of metaheuristics, then for every new parameter introduced (or a value added to a list of possible values for existing parameters) the total combination amount would double. Is there a way to roughly estimate the effect of parameter addition here?

[vcmohan]

I tried to understand this thread but could not. However broadly I find that various parameters for a VHS conversion are attempted so as to optimize the FPS and may be other parameters results of which are not quantizable. I also have seen mention of use of an example frame or frames to arrive at the desired quality.

In oil exploration where a large number of parameters which can vary independantly over some ranges are used to estimate reserves of oil present. Often Monte Carlo simulation is used.

In recent times AI is being used extensively for solving various problems. ANN s can use thousands of parameters to arrive at a solution which can mimic as close as possible an example.

May be tried if applicable. If my suggestion is absolutely off track please ignore the post.

[Groucho2004]

First off, the Avstimer failed to load on Win10/Avisynth+MT, when loading via "loadplugin" it errored with "platform returned code 126: module not found"
The avstimer 32 bit plugin that zorr provides is linked against MSVCR71.DLL which is a runtime DLL that is necessary for dynamically linked VC 7.1 binaries. I can only assume that zorr took the original project file and did not modify it for newer versions of VC.

[Seedmanc]

The avstimer 32 bit plugin that zorr provides is linked against MSVCR71.DLL which is a runtime DLL that is necessary for dynamically linked VC 7.1 binaries. I can only assume that zorr took the original project file and did not modify it for newer versions of VC.

Thanks, that fixed it. The problem with the incorrect time being reported still remains though.

[zorr]

The avstimer 32 bit plugin that zorr provides is linked against MSVCR71.DLL which is a runtime DLL that is necessary for dynamically linked VC 7.1 binaries. I can only assume that zorr took the original project file and did not modify it for newer versions of VC.

That's correct. I don't know much about building Visual Studio projects and the myriad of dll versions. I only have 32bit Avisynth installed so my testing is limited to that. Should the AVSTimer be modified to use some other dll version? :confused:

[StainlessS]

For anybody that needs, here VS CPP v7.0 and v7.1 dll's,

http://www.mediafire.com/file/1220u8g1drmubi8/MSVCR71_70_Runtimes.rar/file#

EDIT: Put in system32, or SysWow64 if 64 bit.

EDIT: MSVCR70.DLL (vs 2002) and MSVCR71.DLL (vs 2003).

[Groucho2004]

There's no reason whatsoever to link a current binary against these ancient 7.x DLLs which cause nothing but grief.

zorr, post your current project, I'll have a look.

I posted makefiles with the modified code which have the correct compiler and linker settings which you can use to build the DLLs from the command line.

[zorr]

Ok, so I gave it a try, here are the impressions.

Thanks, much appreciated. :)

however Avstimer always returned time of 9999999ms.

That may not be the fault of Avstimer. The optimizer sets this value as the timing result when the SSIM is invalid (that is, zero). It's done that way in order to keep the invalid results out of pareto front (an invalid result might be faster than any other result and stay in the pareto front due to being the fastest). You could check what Avstimer really returns by running the script manually (in VirtualDub or Avsr for example) and see what's inside the perFrameResults.txt. Paste it here as well, maybe there are more clues as to what is wrong.

I suppose that made the optimization task much less efficient, because where I expected it to take tens of minutes for 10 720p frames in 3 runs, it took 15 minutes for the first run, 2.5 hours on second and 1.5 hours on third. I ran it with algo "mutation" and 100 iters. My system is Core i5 2550k OC'd to 4.3GHz, 16Gb RAM

That's a large variation in runtime but not entirely unexpected. Even without dct=1 there are some very slow parameter combinations and if the optimizer stumbles upon one of those it might keep trying the slow ones if they happen to give good results. So most likely at least some results in the pareto front were using those slow combinations and mutating them resulted in mostly slow combinations. This could be verified looking at the log files or visualizing them.

Here's a script I used, modified from what you offered in the other thread:

I tried it and after swapping the source and result files found it worked just fine.

A few notes:
1) I dropped the RemoveGrain call since my sources are CGI and clean enough already.

Good call, removing extraneous parameters will help the optimizer.

2) The FrameEvaluate you use doesn't seem to be the native one, as it complained that it doesn't have the argument "args". It worked when I installed GScript, you might want to add that to the list of dependencies.

Sorry about that, I have ScriptClip installed and used its syntax there and forgot to remove the "args". It's not actually necessary and works just fine without it.

3) Shouldn't the optimizer params description for DCT include the D flag, since, much like the searchAlgo param, it is "non-linear" and we can't make assumptions about the value and effect? I put the flag there.

That's true, I added the dct parameter later and didn't really think it through. Yes, it should have the D flag.

The results were confusing.

I can see some tendencies when analyzing the logs manually, sorting by the SSIM so there's that at least. Visually, however, I can't say it looks better than what the hand-picked parameters provide, but that's to be expected from a first attempt with a low iter count.

Yes, maybe that denoising example gave a little too rosy picture of the process. That truly is a toy problem compared to MVTools and 100 iterations will not get you very far here. I am currently running a test with 5000 iterations and 5 runs.

I think the large search space (22 parameters) might have affected it too.

Yes, there's a thing called the curse of dimensionality (https://en.wikipedia.org/wiki/Curse_of_dimensionality). Searching in a high dimensional space is much more difficult than doing it in a low dimensional space. The script I gave is perhaps too brutal, my goal is to make an easier one with smaller search space once I have found the smaller subspace where the good results can be found.

Maybe I should leave the truemotion tuning to later and first try to see what more common set of parameters this tool can generate to compare with the manually tuned ones. Then once I have the good params fixed, I can have it experiment with truemotion.

That's a very reasonable idea. You can go even further and just search around some settings you know are good.

Another question I wanted to ask, is there a way to estimate the amount of time/iters/population required for a certain amount of tunable script parameters involved? I don't really understand much the math involved, but I have a feeling that either the population size or the iters count should scale up with the parameter count somehow.

I would love to have that kind of estimation but alas I don't know how to calculate it. But it totally makes sense that increasing the iters / population is needed for larger parameter count. The current default values were determined using a script with 22 parameters so they should work quite well with your script.

Seeing how the first run took 15 minutes while the second one 2.5 hours it means not every run will even attempt to cover all parameters involved, since obviously the first one omitted the slowest stuff like DCT 4. But then you said Mutation algo can work with population 1.

I believe the dct=4 has been tried but for some reason was not better than what was in the pareto front at the time. That could be simply because at the same time some other parameter was also mutated and that one brought the quality down more than dct=4 brought it up. Search the logs for dct=4 to be sure.

I assume, if we were using brute-force instead of metaheuristics, then for every new parameter introduced (or a value added to a list of possible values for existing parameters) the total combination amount would double. Is there a way to roughly estimate the effect of parameter addition here?

We can calculate the number of possible combinations by multiplying the number of allowed values of each parameter. For example with

super_pel = 2 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter

there are 2*3*5 = 30 combinations. Doing the math for your script we get 2*3*5*9*8*30*60*20001*19993*257*3*17*3*2*9997*47*2*2*2*10*300*3 = 4 136 285 698 342 782 746 022 144 000 000. :eek: Not every one of those combinations is valid though. :)

[zorr]

There's no reason whatsoever to link a current binary against these ancient 7.x DLLs which cause nothing but grief.

zorr, post your current project, I'll have a look.

I posted makefiles with the modified code which have the correct compiler and linker settings which you can use to build the DLLs from the command line.

Uploaded it here (https://drive.google.com/open?id=1qfID-V-531JGCx7sMtMbUkWjQ_eHh05Z). Thanks!

[Groucho2004]

Uploaded it here (https://drive.google.com/open?id=1qfID-V-531JGCx7sMtMbUkWjQ_eHh05Z). Thanks!The avstimer.dll you have in your "Release" directory is not the same as the one you posted before. It also has the correct VC2015/2017 dependencies.

Edit: Checked the time stamp on the first one you posted - It's Kassandro's DLL from 2005. :rolleyes:

[zorr]

The avstimer.dll you have in your "Release" directory is not the same as the one you posted before. It also has the correct VC2015/2017 dependencies.

Edit: Checked the time stamp on the first one you posted - It's Kassandro's DLL from 2005. :rolleyes:

I guess I only added the missing x64 version. Is that one using the correct dependencies as well?

[zorr]

Plugins package (https://drive.google.com/open?id=1iKH05RPHMywkRtka4b4o--LrES8hmAbO) with SSIM and AvsTimer has been updated, hopefully with a more functional 32bit AvsTimer. :)

[Groucho2004]

Plugins package (https://drive.google.com/open?id=1iKH05RPHMywkRtka4b4o--LrES8hmAbO) with SSIM and AvsTimer has been updated, hopefully with a more functional 32bit AvsTimer. :)
All good now.

[StainlessS]

There's no reason whatsoever to link
True, so long as there is source, otherwise, ...

Unfortunately, not everyone is willing to go to the bother of compiling Ye Olde stuff, even if source does exist.

Above link has been there for several years I think, not altogether a bad idea to have access to such when required. (EDIT: not needed in this case when
there is someone as affable as Ye Olde G2K4 :) )

[Groucho2004]

There's no reason whatsoever to link
True, so long as there is source, otherwise, ...Well, the word "link" usually implies that the source is present... :)

[Seedmanc]

The new avstimer reports time correctly, thanks.

Some other ideas to add, previously it was mentioned that some parameter combinations might have no effect on time or SSIM, but what if we knew for sure what those combinations are, it would be nice to be able to mark them as such so the optimizer could skip them. For example, in MSuper the sharp parameter is only for pel>1, it won't raise an error otherwise but it'll be a waste of time.
Regarding the ConvertToYV24, can you give an example what kind of parameter combinations would be unsuitable for YV12 so this becomes necessary? It slows down the processing considerably, so I'd like to get rid of that.

Another thing, the badRange parameter of MAnalyze says that we need to use positive values for UMH search and negative for exhaustive. Unfortunately it does not disclose why, however it doesn't raise an error anyway. But the FPS for negative values is almost a half of what's achieved for positive ones (when you set badSAD to 0 so the wide search is always invoked, not a real-life scenario, but might happen a lot during the search). In SVP a similar parameter is described as using an adaptive radius when negative to save time, claiming it takes about 2/3 of time for a similar result, but it seems we have the opposite here. Anyway, how would we describe it in the settings to use negative values when searchAlgo is 3?

Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there.


To comment on your post about the runs vs iters, you said, I don't have a good answer on how many runs are enough. If you can only run N iterations should you run for example three runs with N/3 iterations or eight runs with N/8 iterations? More iterations is better but more runs is also better, to a point.. But judging from the example where you ran a script for 100, 200 500 and 2000 iterations, the key was to see how the results stabilize among multiple runs, converging to a single set of parameters and resulting SSIMs/times. Hence, multiple runs are only used to check if the process has stabilized enough, but multiple iterations are the requirement for that stabilization. In best case all you'd need is 2 runs and a large amount of iters to see if the results became close to each other. I use 3 since it's the lowest amount from which you can already calculate both the average and the median.
If I'm correct about that all we need to figure out is how to scale population count over the increasing search space. What configuration is more likely to try the largest subset of the search parameters, the high-iter low-pop or the vice-versa?

[zorr]

Some other ideas to add, previously it was mentioned that some parameter combinations might have no effect on time or SSIM, but what if we knew for sure what those combinations are, it would be nice to be able to mark them as such so the optimizer could skip them.

Using the dependency definitions (min / max / filter) we can guarantee that the optimizer will not try certain combinations. It doesn't really matter what the purpose is (in some cases to avoid completely invalid combinations, in some cases to avoid doing useless work).

For example, in MSuper the sharp parameter is only for pel>1, it won't raise an error otherwise but it'll be a waste of time.

You could define it like this:

super_pel = 2 # optimize super_pel = _n_ | 1,2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 ; max:super_pel 1 == 0 2 ? | super_sharp

Translation: if super_pel is 1 then then the maximum value of super_sharp is 0, otherwise it is 2. But note that in your script super_pel only has values 2 and 4 so in this case all the values of super_sharp are valid. My experience so far has been that the best results pretty much always use super_pel=4, may not be worth your time to try the other values unless you're also optimizing for speed.

https://i.postimg.cc/WzrmWXSw/groupby_super_pel.png

Regarding the ConvertToYV24, can you give an example what kind of parameter combinations would be unsuitable for YV12 so this becomes necessary? It slows down the processing considerably, so I'd like to get rid of that.

The error message you get is "MAnalyse: wrong overlap for the colorspace subsampling for divide mode". Looking at the MVTools source I was able to gather that it's triggered when overlap is not divisible by xRatioUV (or overlapv is not divisible by yRatioUV). And the xRatioUV is 2 for YUY2 and (1 << vi.GetPlaneWidthSubsampling(PLANAR_U)) for other formats. Assuming you convert to YV12 instead (which is needed for the SSIM anyway) then you can avoid the error by making the overlap values divisible by 4. Like this (filter dependency is no longer needed so I removed that):

overlap=0 # optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlap
overlapv=0 # optimize overlapv=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlapv


You could also avoid the speed penalty by preprocessing the video to YV24 and use that instead. It might still be slower than processing YV12 but at least you don't pay the price of conversion every time. And also consider the cropping idea, that can give you a significant speed increase.

Another thing, the badRange parameter of MAnalyze says that we need to use positive values for UMH search and negative for exhaustive. Unfortunately it does not disclose why, however it doesn't raise an error anyway.

I think this means that using negative values selects the exhaustive algorithm and using positive values selects the UMH algorithm for this wide search, independently on what is used in the first search. Btw I didn't even realize you can use negative values, I guess I now have to test those too. :)

Anyway, how would we describe it in the settings to use negative values when searchAlgo is 3?

You could use the min and max dependencies. Or the filter, you can do pretty much anything you could think of with that one. :)

Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there.

I don't know more than you about this, if Pinterf is around maybe he could clarify. But it strikes me odd that the result would be different when temporal is disabled and MT is on/off. That could be a bug.

But judging from the example where you ran a script for 100, 200 500 and 2000 iterations, the key was to see how the results stabilize among multiple runs, converging to a single set of parameters and resulting SSIMs/times. Hence, multiple runs are only used to check if the process has stabilized enough, but multiple iterations are the requirement for that stabilization.

I was just showing off. :D It's rare that you can make the results converge to the exact same values. When that does happen though it's a strong indicator that we've found the optimal result. In a more realistic scenario the iteration counts are always insufficient to make the results converge in that way. There may be some parameters with the same values (like super_pel=4) but many others are different, some wildly.

There is another reason to do multiple runs. The beginning of the search usually "locks" the search into a certain corner of the search space and it may never get out of that within the iteration count. So it could be that you get significantly better result in one out of say, 10 runs. If you only ever do 3 runs maybe you'll never get to that lucky corner. Here's a recent example:

Run 1 best: 9.790725 2130 rmgrain=12 super_rfilter=1 blockSize=8 searchAlgo=3 searchRange=1
searchRangeFinest=7 lambda=2213 LSAD=2744 plevel=2 overlap=0 globalMotion=true badSAD=9559
badRange=34 meander=false temporal=false trymany=false dct=1 maskScale=2
Run 2 best: 9.787563 1210 rmgrain=19 super_rfilter=1 blockSize=8 searchAlgo=1 searchRange=1
searchRangeFinest=3 lambda=3931 LSAD=19316 plevel=2 overlap=0 globalMotion=false badSAD=2770
badRange=13 meander=true temporal=true trymany=false dct=1 maskScale=2
Run 3 best: 9.790011 1510 rmgrain=19 super_rfilter=1 blockSize=8 searchAlgo=3 searchRange=1
searchRangeFinest=4 lambda=2629 LSAD=1684 plevel=2 overlap=0 globalMotion=true badSAD=9831
badRange=10 meander=true temporal=false trymany=false dct=1 maskScale=2
Run 4 best: 9.789404 1190 rmgrain=19 super_rfilter=0 blockSize=8 searchAlgo=1 searchRange=1
searchRangeFinest=2 lambda=2142 LSAD=2679 plevel=2 overlap=0 globalMotion=true badSAD=9558
badRange=45 meander=true temporal=false trymany=false dct=1 maskScale=2
Run 5 best: 9.788453 1220 rmgrain=8 super_rfilter=0 blockSize=8 searchAlgo=1 searchRange=1
searchRangeFinest=3 lambda=3432 LSAD=6410 plevel=2 overlap=0 globalMotion=true badSAD=4531
badRange=4 meander=true temporal=false trymany=false dct=1 maskScale=1


There are three results below 9.79 and two above it. If you're unlucky you'll get those below 9.79 results in your three runs. These results were run with 5000 iterations, the differences are larger with a smaller iteration count.

In best case all you'd need is 2 runs and a large amount of iters to see if the results became close to each other. I use 3 since it's the lowest amount from which you can already calculate both the average and the median.

Maybe you're right. It's a tough call because we're talking about probabilities here. So if you do *this*, then *that* happens with a certain probability, but not always. One could do a large number of runs and then using that data calculate what are the odds for a certain result using N runs. But I think it also depends on the particular script and how difficult it is to optimize.

If I'm correct about that all we need to figure out is how to scale population count over the increasing search space. What configuration is more likely to try the largest subset of the search parameters, the high-iter low-pop or the vice-versa?

Strictly speaking there is no difference, the number of iterations defines how large the searched subset is (as the optimizer never tries duplicates within one run). But I guess you're asking how to get the widest possible subset. For that question the high population should do better in terms of how wide the search is but it's going to do less mutations of the best result and therefore might end up with a worse result than a smaller population. The "mutation" algorithm with population 1 is the most narrow search possible, it simply keeps the best result and mutates it until one of the mutations is better. If you just want to make the search wider, you can do that also by cranking up the mutation amount and count.

[Seedmanc]

Two problems, one is that after I modified the script to disable YV24 conversion, limit the sharp parameter and introduce the negative badRange (didn't work with -50..50 values so I added a boolean flag according to which I do or do not multiply the range by -1), the script started crashing, complaining that "MflowFps can't work in reentrant multithreading" even though I didn't touch anything MT-related. While I do have MT versions of Avisynth 2.6 and Avisynth+ installed I do not call setmtmode or prefetch in either but it crashes in both randomly, I'm lucky if I manage to get one run finished.

I used to get this error before AvsOptimizer sometimes too, usually when switching between multiple heavy scripts in Avsp editor, but it was a rare occurrence. Before I edited the script I managed to run it overnight totalling in over a thousand iterations without a problem. If it didn't work right away I'd consider that launching two upsampling operations in succession like we do here might've been the culprit.

Here's my script as of now:


TEST_FRAMES = 10
MIDDLE_FRAME = 600

# original framerate
FPS_NUM = 30
FPS_DEN = 1

# source clip
Asrc=FFmpegSource2("c:\Hibikin - Watashtachi wa Zutto... Deshou (AVS test video 60fps 720p 10bit CRF0).mkv",threads=1 ).assumefps(60)
Asrc=Asrc.trim(0,60*30-9)+Asrc.trim(60*40+60*60+9,0)
asrc.selecteven
AssumeFPS(FPS_NUM, FPS_DEN)

#return last

# needed for some parameter combinations
#ConvertToYV24()

orig = last

mt=true #happens without this parameter as well since mt is true by default, just introduced it for later use


super_pel = 2 # optimize super_pel = _n_ | 1,2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 ; max:super_pel 1 == 0 2 ? | super_sharp
super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, orig, mt=mt )

blockSize = 12 # optimize blockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:divide 0 > 8 2 ? ; filter:overlap 2 * x <= | blockSize
searchAlgo = 3 # optimize searchAlgo = _n_ | 0..5 D | searchAlgo
searchRange = 4 # optimize searchRange = _n_ | 1..30 | searchRange
searchRangeFinest = 4 # optimize searchRangeFinest = _n_ | 1..40 | searchRangeFinest
lambda = 16000 # optimize lambda = _n_ | 0..20000 | lambda
lsad=400 # optimize lsad=_n_ | 8..20000 | lsad
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=0 # optimize plevel=_n_ | 0..2 | plevel
overlap=4 # optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlap
divide=2 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true # optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 2000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 2..50 | badRange
negBadRange = false # optimize negBadRange = _n_ | false,true | negBadRange
meander = true # optimize meander = _n_ | false,true | meander
temporal = false # optimize temporal = _n_ | false,true | temporal
trymany = false # optimize trymany = _n_ | false,true | trymany
dct = 0 # optimize dct = _n_ | 0,2,3,4,5,6,7,8,9,10 D | dct

delta = 1
useChroma = true

resultBadRange = (negBadRange ? -1 : 1)*badRange
bv = MAnalyse(super_render, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct, mt=mt)
fv = MAnalyse(super_render, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap ,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct, mt=mt)



threshold = 10000
maskScale = 100 # optimize maskScale = _n_ | 1..300 | maskScale
mask_fps = 2 # optimize mask_fps = _n_ | 0..2 | mask_fps
inter = orig.MFlowFPS(super_render, bv, fv, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )

# return this to look at the clip with doubled framerate
#return inter

fps_only = inter.SelectOdd()

# second pass
super_render2 = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only )
bv2 = MAnalyse(super_render2, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, mt=mt)
fv2 = MAnalyse(super_render2, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest,
\ chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap,
\ divide=divide, badSAD=badSAD, badrange=resultBadRange, meander=meander, temporal=temporal, trymany=trymany, mt=mt)
inter2 = fps_only.MFlowFPS(super_render2, bv2, fv2, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold )
fps_only2 = inter2.SelectOdd()


delimiter = "; "

inter_yv12 = fps_only2#.ConvertToYV12()
orig_yv12 = orig#.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

global total = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig_yv12, inter_yv12, delta, frame_count")

# NOTE: AvsTimer call should be before the WriteFile call
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
resultFile = "c:\avsoptim\results\perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="c:\avsoptim\results\perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

The second thing is that limiting iterations by time is rather loose, apparently because it does not include the time spent on generating populations, so where I set it to 3 runs with 3 hours it went for 3:05, 3:20 and over 4 hours on last run. Not very convenient when you expect it to be done in the morning.

Also, for some reason sensitivity estimation is always marked as N/A in the log, even if I try to pass -sensitivity true. It would be interesting to see which parameters are more important.

[Groucho2004]

Two problems, one is that after I modified the script to disable YV24 conversion, limit the sharp parameter and introduce the negative badRange (didn't work with -50..50 values so I added a boolean flag according to which I do or do not multiply the range by -1), the script started crashing, complaining that "MflowFps can't work in reentrant multithreading" even though I didn't touch anything MT-related. While I do have MT versions of Avisynth 2.6 and Avisynth+ installed I do not call setmtmode or prefetch in either but it crashes in both randomly, I'm lucky if I manage to get one run finished.
Run "avsmeter avsinfo -log" and post the log file. Use pastebin or similar to post the log.

[zorr]

Two problems, one is that after I modified the script ... the script started crashing, complaining that "MflowFps can't work in reentrant multithreading" even though I didn't touch anything MT-related. While I do have MT versions of Avisynth 2.6 and Avisynth+ installed I do not call setmtmode or prefetch in either but it crashes in both randomly, I'm lucky if I manage to get one run finished.

I used to have a similar problem when I ran with a high resolution source, it was because the script was running out of memory. With

SetMemoryMax(2048)

the problem went away. I tried your script again and didn't have any problems, but I'm using a low resolution source.

Here's my script as of now:

I like the way you implemented the positive / negative badRange. It's pretty much the way I would have done it, separating the algorithm into its own parameter.

The second thing is that limiting iterations by time is rather loose, apparently because it does not include the time spent on generating populations

Yes, the mutation algorithm had such a flaw. It should be fixed now in the latest 0.9.5-beta (https://drive.google.com/open?id=1lxlm95gcBausN3YNBGOKNIAZ9kr5Jwqx).

Also, for some reason sensitivity estimation is always marked as N/A in the log, even if I try to pass -sensitivity true. It would be interesting to see which parameters are more important.

The mutation algo is not quite as fully featured as the others, it's missing the sensitivity estimation at the moment. I will add it in the next version. It's probably not going to happen during this weekend though, I'm moving tomorrow and about to start renovating the new house...

[Seedmanc]

Groucho2004, here are the logs: https://pastebin.com/trPLFs2w (primary setup with avs2.6), https://pastebin.com/TjdMquUr (secondary setup with avs+). For some reason it complains about the absence of fft3w lib in the latter, even though mvtools dct modes work fine.

zorr, thanks for the memory suggestion and the time fix, I'll try to run it overnight now with those. I wonder if it would be possible to use the 3Gb or 4Gb limits reliably since I'm on 64bit OS. SEt says (https://forum.doom9.org/showthread.php?p=1312666#post1312666) it requires the patching of the runner app (avsr in our case) for support, I assume it applies regardless of MT usage, does avsr have that?

[Groucho2004]

Groucho2004, here are the logs: https://pastebin.com/trPLFs2w (primary setup with avs2.6), https://pastebin.com/TjdMquUr (secondary setup with avs+). For some reason it complains about the absence of fft3w lib in the latter, even though mvtools dct modes work fine.
Remove avstp.dll from your plugins directory and see if it still crashes.

As for fft3w - Just ignore it for now.

[StainlessS]

W10


[CPP 2.5 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth+\plugins+\avstimer.dll [2018-10-07] # 2.5 Plugin in Plugins+ folder
C:\Program Files (x86)\AviSynth+\plugins+\ffms2.dll [2013-05-21] # 2.5 Plugin in Plugins+ folder

[CPP 2.6 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth+\plugins\avstimer.dll [2018-09-11] # see above. (earlier plug is v2.6, later 2.5 ?????? )
C:\Program Files (x86)\AviSynth+\plugins+\mvtools2.dll [2.5.11.22] # Plugins/+ mixup with below
C:\Program Files (x86)\AviSynth+\plugins\mvtools27.dll [2.7.31.0] # ditto


EDIT: Presumably, the only plugz that should be in plugins+ should be those that use avs+ colorspaces, or functionality.
EDIT: The 'hazy' part is standard v2.6 plug with standard VERSION 6 header, versus avs+ VERSION 6 Header,
ie should avs+ VERSION 6 header but without avs+ colorspace or functionality be in + or standard plugins (?).

[Groucho2004]

I wonder if it would be possible to use the 3Gb or 4Gb limits reliably since I'm on 64bit OS. SEt says (https://forum.doom9.org/showthread.php?p=1312666#post1312666) it requires the patching of the runner app (avsr in our case) for support, I assume it applies regardless of MT usage, does avsr have that?avsr is LAA (Large Address Aware). Patching is only necessary when the application is not built with that linker switch (VirtualDub, for example).

Besides, since there is a 64 bit version of the Optimizer, why not use that?

Lastly, why would you use SEt's ancient Avisynth MT? AVS+ is quite stable, has a 64 bit version, uses less memory and, from my experience, has better MT support.

[Groucho2004]

EDIT: Presumably, the only plugz that should be in plugins+ should be those that use avs+ colorspaces, or functionality.
EDIT: The 'hazy' part is standard v2.6 plug with standard VERSION 6 header, versus avs+ VERSION 6 Header,
ie should avs+ VERSION 6 header but without avs+ colorspace or functionality be in + or standard plugins (?).
I too think that in order to earn the hallowed privilege to reside in the "+" directory, plugins should support AVS+ colour spaces and/or register their own MT modes.

There's no reason to use any other header for 2.6 plugs than the one that comes with AVS+ which is fully backward compatible with classic AVS.

[Seedmanc]

So, after much suffering (because nothing helped against the crashes, not even running it on 64bit avs+) I figured out to take a look at the intermediate avs scripts generated by the optimizer to find what has brought it to its knees. https://pastebin.com/bgUELWu6 this one crashes 100% of the time, can you tell what the problem is?
I'll save you time, it's the combination of pel 4, blocksize 8 with overlap 4, divide > 0 with large search radius and (surprise!) removal of ConvertToYV24. What does it have to do with MT? Well I don't know, ask the mvtools maker about their error reporting style.
Further testing revealed that for pel 4 it is enough to have search radius of 4 to cause error, with pel 2 it takes around 12 and I couldn't reproduce it for pel 1. While I admit that having a search radius larger than block size seems strange it doesn't cause an error for YV24 or divide 0 / pel 1. What's more weird is that unlike the chroma subsampling violation it does not necessarily raise an error right away, sometimes it happens in the middle or at the end of the script, sometimes it's not the MT error but some random access violation. Moreso, setting chroma to false doesn't help (but converting to YV24 still does even then). Nothing in the readme has prepared me for this.
Worst thing I don't even know how to report it, the thread has been abandoned for months.

I guess more filters/minmaxes are in order, but with the current notation it's hard to figure out how to write them. Set divide to 0 when blocksize is 8 and overlap 4 and pel > 1.


In a way, Optimizer can be used as an automated plugin testing tool, since it tries out so many parameter combinations and reveals all kinds of bugs and readme inconsistencies.
Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.

[StainlessS]

Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.

Can I also suggest that all non-beta versions be linked in first post only and not in temporal posts.

Seems that the [EDIT: forum] spell checker works again, been missing for some time (or maybe its because I'm on W7_64 at the moment, instead of XP32/64).
[EDIT: Or maybe spell check is via current version of Firefox for W7, and no longer supported for last for XP FireFox v52.xx ESR]

[zorr]

So, after much suffering (because nothing helped against the crashes
...
it's the combination of pel 4, blocksize 8 with overlap 4, divide > 0 with large search radius and (surprise!) removal of ConvertToYV24.
...
Further testing revealed that for pel 4 it is enough to have search radius of 4 to cause error, with pel 2 it takes around 12 and I couldn't reproduce it for pel 1.


I was able to reproduce this even with my low resolution source. It needs to be (or converted to) YV12 though, even YUY2 works. With my slightly older MVTools it also crashes with super_pel=2, but not 1.

While I admit that having a search radius larger than block size seems strange

I don't think the block size affects the search radius. The block size tells how large blocks the algorithm is trying to track, while the search radius tells how far the block can move between frames. I guess the practical upper limit for the search radius is the width of the frame (or height, if that one is larger). [EDIT] Actually it's the diagonal of the frame, Sqrt(width*width + height*height). And here we're assuming the radius is defined in pixels, which it is not in every search algorithm.

What's more weird is that unlike the chroma subsampling violation it does not necessarily raise an error right away, sometimes it happens in the middle or at the end of the script, sometimes it's not the MT error but some random access violation.

I found a MVTools2 bug earlier which was only triggered every 10th or 20th run of the script, reported here (https://github.com/pinterf/mvtools/issues/14).

I don't even know how to report it, the thread has been abandoned for months.

I guess pinterf is very busy but he did respond to my bug reports swiftly, I think the deciding factor there was that he was able to reproduce the problem. So in this case with almost 100% error rate I think he can find the issue pretty soon.

I guess more filters/minmaxes are in order, but with the current notation it's hard to figure out how to write them. Set divide to 0 when blocksize is 8 and overlap 4 and pel > 1.

Certainly possible but I would do that only as last resort, let's give Pinterf a chance to do a fix first. With the details you figured out your error report would be very good. Getting rid of the reverse polish notation is on my todo list.

In a way, Optimizer can be used as an automated plugin testing tool, since it tries out so many parameter combinations and reveals all kinds of bugs and readme inconsistencies.

I agree, I have found two bugs from MVTools using the optimizer, both were fixed. Perhaps I should also add a mode to the optimizer where you give it a failing script and it tries to figure out all the parameter combinations that trigger the error.

Another fun (or not) idea that is already possible would be to find the slowest possible parameter combinations... perhaps combined with the worst quality too! :)

Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.

Thanks for the suggestion, I have updated the first post.

[zorr]

AvisynthOptimizer v0.9.6-beta (https://drive.google.com/open?id=1DrjDag1P7rOD9evkL-1ALf0-Zvf52adu) released. This version has improvements to mutation algorithm. It now supports sensitivity estimation, has colored console output that is more like the other algorithms and has the correct number of script evaluations when using a fixed iteration count.

Seedmanc, do you mind if I create an MVTools2 bug report about the issue you encountered? Or would you like to do it yourself?

[Seedmanc]

zorr, yes, I would rather have you do it, please go ahead.

Does the new version still require full path to the avs script?

On another note, in the FRC thread you mentioned this:
Oh and one more thing, I came up with a way to limit the ugly artifacts you often get with MFlowFPS when good motion vectors are not found. Basically you reconstruct the frame created with MFlowFPS using MCompensate and the original frames. You just need to find good parameters for that, which you can do with the optimizer. I will give an example of that later.
I'm not sure I understand this. if you can't get good vectors for MFlow then you don't have good vectors for any other tool anyway.

[pinterf]

Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there.
My MvTools2 fork was originated from 2.6.0.5 which had internal multithreading through avstp.dll. Internal multithreading served well when used in non-MT capable avisynth versions or scripts which could not be run in MT for some reason. So the parameter mt applies for internal multithreading.

The reason why "temporal" is not multithreading friendly is that it requires linear frame access: there is only a single internal buffer that holds the vectors from previous frame. Previous vectors are used _only_ if frame order is linear from MAnalyze point of view: the current frame number = previously analyzed frame number + 1. So the word "multithreading" here refers to the Avisynth-level multithreading schema.

MAnalyze automatically reports to work in MT_MULTI_INSTANCE mode under Avisynth+. Perhaps MAnalyze could adaptively report MT_SERIALIZED when temporal=true is set.

[zorr]

zorr, yes, I would rather have you do it, please go ahead.

Ok, I will do some more investigation and then create the bug report.

Does the new version still require full path to the avs script?

Yes, unfortunately. But now that you mentioned it I will take a look at how hard it would be to fix.


I'm not sure I understand this. if you can't get good vectors for MFlow then you don't have good vectors for any other tool anyway.

That's certainly true. But it just happens that MFlowInter and MFlowFps have an unfortunate looking failure case, which the MCompensate doesn't have. So even with (actually, especially with) bad vectors MCompensate will look better. Of course you can't use MCompensate to generate an inbetween frame, it can only recreate a complete frame using other frames.

But if we first create the inbetween frame with MFlowInter / MFlowFps and then use MCompensate to reconstruct that frame using two nearby frames something magical happens...

https://i.postimg.cc/wvDbnFMP/mflowinter-mcompensate.gif

Here we see the original frame (orig), a reconstructed frame from MFlowInter (inter) and finally the MCompensated frame (final).

[zorr]

The reason why "temporal" is not multithreading friendly is that it requires linear frame access: there is only a single internal buffer that holds the vectors from previous frame. Previous vectors are used _only_ if frame order is linear from MAnalyze point of view: the current frame number = previously analyzed frame number + 1.

Does this mean that previous vectors are sometimes used depending on the scheduling of cores, or does this condition never happen in practice when multithreading is enabled?

[pinterf]

It depends on scheduling.
Btw, new mvtools2 released, fixing an issue in MAnalyze which could cause artifacts also in MFlow*** at larger blocksizes/lambda.

[zorr]

It depends on scheduling.
Btw, new mvtools2 released, fixing an issue in MAnalyze which could cause artifacts also in MFlow*** at larger blocksizes/lambda.

Thanks! I used the latest version to test the crash issue Seedmanc found and it's still there. I created a bug report (https://github.com/pinterf/mvtools/issues/17).

[zorr]

AvisynthOptimizer version 0.9.7-beta (https://drive.google.com/open?id=13gen5EKFj6XJDbuz7mnmEIaKGTl9OX3t) released.

The source file path (or any path) doesn't need to be an absolute file path anymore. This is implemented by calling SetWorkingDir() in the beginning of the script to set the working directory as the original script's directory.

Seedmanc, was this the issue you asked about, I may have misunderstood because you said "full path to the avs script"...

Thanks to Pinterf the crash issue is fixed in the latest MVTools2 version (https://github.com/pinterf/mvtools/releases/download/2.7.33/mvtools-2.7.33-with-depans.7z). I have run some 43 000 tests and found no crashes.

[Seedmanc]

By re-enabling MT and trimming the required 10-frame clip into a separate video I've managed to speed up things from around 150 iters per hour to a whole thousand, and I can finally see it converging to a more or less singular set of parameters overnight.

However, there are still problems. It seems the nature of SSIM makes preference of sharp lines to textures and fills, which makes the results very biased sometimes. For example, it always prefers pel=4 over pel=2, despite it being said in the readme that it's not necessarily better, especially considering the time penalty. Apparently pel=2 gives somewhat aliased edges, hardly noticeable by eye, but too important for the metric. Another problem is that it also prefers sharp=0 to sharp=2, even when it's clearly visible that the former looks considerably blurrier than the original video. Perhaps it is again due to the extra attention to lines, especially with the double upsampling method used here, the halos around edges become extra prominent. Though it's not just about SSIM, when comparing PSNR or VQM (the latter uses DCTs for comparison), using the MSU VQMT software, it was noticeable how the graphs align in parallel to each other, as if sharp=2 incurred a constant penalty in the metrics value, independent of the scene complexity. Another reason might be that I'm testing it on a 2D animation (or rather, 3D CGI which is cellshaded to look 2D), which means lots of very sharp edges with flat fills around them. In this situation, a mere half-pixel shift of an edge makes a lot more difference (relatively) than dozens of pixels away if the background is the same color. I can't be bothered to test if the problem is as strong on real footages, though.
What I tried to do, however is to obscure the influence of sharpness-related options by downscaling the video just before passing it to SSIM. 1/2 was not enough, but 1/4 by each side did the trick - sharpness or aliasing no longer affected the metrics, resulting in pel=2 and sharp=2 getting about the same share in the results pool as other values. When calculating total SSIM over entire video and comparing visually, it looks like the sharp option does not really affect the efficiency of frame interpolation in any way, meanwhile pel=2 actually looked somewhat better and got a better SSIM than pel=4. Not very significant on its own, but considering the speed difference with pel=4, important.

Among other troublesome parameters, there are also divide and overlap. The double upsample method used here causes the SSIM to always be higher for overlap=0 and no divide. Meanwhile overlap pretty much universally gives better SSIM and visuals when comparing directly to original frames, and divide sometimes looks better as well. I couldn't find a solution here, downscaling didn't help, nor I can explain what might be throwing SSIM off in that case. Really, how can SSIM of this (overlap 0) (https://i.imgur.com/FVXRt76.jpg) be higher than of this (overlap half) (https://i.imgur.com/8dlqb0K.jpg)? Ok, I need to clarify the images here, I split the video by half in duration and stack up vertically so I only have to go through 5 frames manually instead of 10 when comparing. Then, on the left half is the video after double upsampling and the SSIM for it compared to the original frames, while on the right it's after single upsampling (how it should be) and compared to discarded frames (original video is 60 so I can drop half and still get a reasonable source framerate). As you can see, SSIM on the left is inversely proportional to the actual video quality as opposed to SSIM on the right. I suppose I'll have to fix overlap to half the blocksize in the script itself, but I'm disappointed it hates divide so much.

A few more notes, the Divide parameter should be marked with the D flag, since divide=2 isn't really any "more divided" than 1, just different modes. I also added padding parameter for MSuper and the new parameter scaleCSAD, added in 2.7, which seems to improve quality when set to positive value (and the optimizer indeed chooses the maximum value for it). However neither DCT, nor searchalgo or padding converge to any particular values even after 3000 iterations and unlocking dct=1 (I don't think I saw it choosing 1 at all). I'm going to try modifying the script so that it compares to the original discarded frames to get rid of the mistakes introduced by the double upsampling and see if it's gonna be better.
Here's a Google Spreadsheets link (https://docs.google.com/spreadsheets/d/1E3Da_djkw41ZJf0V9zkACtKE812_aHny1cFb_EstLXM/edit?usp=sharing) where I tried to analyze (for the lack of a better way) results from several 3run*3000iter trials with downscaling and without, comparing the distribution of divide, sharp and pel parameters. I couldn't quite figure out how to make use of the visualizer's groupby method, so I had to come up with my own.


zorr, I mean the need to provide full path to .avs when calling the optimizer even if they're in the same directory.
I'd like to request a way to only generate scripts for the best results of every run instead of the entire pareto front. Usually when AvsOptim finishes I end up manually comparing the run results (with the image setup of above) among themselves and with the handpicked best results from previous runs. As of now it requires a lot of manual parameter editing to match the run results reported by Evaluate mode.
Also, I wonder if it would be possible to manually provide one of the generated population members so that a new run could start with one handpicked best parameter set among others and perhaps try to improve on top of that. For example, the optimizer would use the values assigned to var in the script (before the # optimize part) as one of the population members. Does that even make sense?

[zorr]

However, there are still problems. It seems the nature of SSIM makes preference of sharp lines to textures and fills, which makes the results very biased sometimes.

That could be the case. Our eyes are also very sensitive to sharp lines (much more than to textures) so perhaps that is just the way SSIM is supposed to work.

For example, it always prefers pel=4 over pel=2, despite it being said in the readme that it's not necessarily better, especially considering the time penalty. Apparently pel=2 gives somewhat aliased edges, hardly noticeable by eye, but too important for the metric.

I can confirm that, pel=4 is pretty much always the chosen one for the best result. But the pareto front does contain pel=2 because it gives a significant speed increase.

Another problem is that it also prefers sharp=0 to sharp=2, even when it's clearly visible that the former looks considerably blurrier than the original video. Perhaps it is again due to the extra attention to lines, especially with the double upsampling method used here, the halos around edges become extra prominent.

In my tests sharp=2 is usually the better one, but then again I haven't done a lot testing with animations. I did test a clip of "Frozen" where sharp=2 was again the winner.

Though it's not just about SSIM, when comparing PSNR or VQM (the latter uses DCTs for comparison), using the MSU VQMT software, it was noticeable how the graphs align in parallel to each other, as if sharp=2 incurred a constant penalty in the metrics value, independent of the scene complexity. Another reason might be that I'm testing it on a 2D animation (or rather, 3D CGI which is cellshaded to look 2D), which means lots of very sharp edges with flat fills around them. In this situation, a mere half-pixel shift of an edge makes a lot more difference (relatively) than dozens of pixels away if the background is the same color.

That sounds peculiar, perhaps something is not quite right with the script. Have you tried converting the video into an lossless avi format and using that as the source? In my experience the other source filters are not as reliable.

What I tried to do, however is to obscure the influence of sharpness-related options by downscaling the video just before passing it to SSIM. 1/2 was not enough, but 1/4 by each side did the trick - sharpness or aliasing no longer affected the metrics, resulting in pel=2 and sharp=2 getting about the same share in the results pool as other values.

I am not sure if that is the best way to handle this kind of situation though. Basically the optimizer is now blind to these parameters and cannot help you find the optimal values for those. You could perhaps have the same effect by disabling the optimization for these parameters and in the end manually try if the different values for pel and sharp have an effect on the final result.

There's also a SSIM variation called Multiscale SSIM (http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf) (MS-SSIM) that actually calculates the SSIM using different scales. I think that would be a good improvement on the quality measurement. It might be possible to implement MS-SSIM using just an avisynth function.

When calculating total SSIM over entire video and comparing visually, it looks like the sharp option does not really affect the efficiency of frame interpolation in any way, meanwhile pel=2 actually looked somewhat better and got a better SSIM than pel=4. Not very significant on its own, but considering the speed difference with pel=4, important.

Can you clarify this? So was the SSIM for pel=4 better in the 10 frame part (earlier you said it prefers pel=4) but pel=2 was better when calculating for the whole clip? If you meant that then yes, this could happen because the short 10 frame segment may not be a good representation of the whole clip. The optimizer tends to "overoptimize" in a way, it only cares about this short segment it is given to work with and because it also optimizes for speed it tends to fine-tune the arguments (for example the search ranges) so that they just barely work. Perhaps it would be better to take the 10 frames from different parts of the script, even though it will increase the processing time because the algorithm uses at least 4 frames around the constructed frame. Also it might be a good idea to "loosen" the found parameters, for example using slightly larger search ranges for the whole clip.

Among other troublesome parameters, there are also divide and overlap. The double upsample method used here causes the SSIM to always be higher for overlap=0 and no divide. Meanwhile overlap pretty much universally gives better SSIM and visuals when comparing directly to original frames, and divide sometimes looks better as well.

My experience is different, optimal overlap is usually not zero (at least in both directions) and divide=2 was the optimal result in my latest run.

I couldn't find a solution here, downscaling didn't help, nor I can explain what might be throwing SSIM off in that case. Really, how can SSIM of this (overlap 0) (https://i.imgur.com/FVXRt76.jpg) be higher than of this (overlap half) (https://i.imgur.com/8dlqb0K.jpg)?

Is the SSIM value for the both frames? In that case the bottom left picture might explain it, in my eye it looks a lot more garbled and deserves lower SSIM.

As you can see, SSIM on the left is inversely proportional to the actual video quality as opposed to SSIM on the right. I suppose I'll have to fix overlap to half the blocksize in the script itself, but I'm disappointed it hates divide so much.

As fancy as the optimizer is, it's just a tool with flaws. It may not find the optimal values when given too much freedom, so giving it stricter limits will probably help. Or give it more iteration counts, those always help. :) Actually I have some pretty encouraging results from my latest optimizer run with 50 000 iterations, I have *most* of the parameters now converged.

A few more notes, the Divide parameter should be marked with the D flag, since divide=2 isn't really any "more divided" than 1, just different modes.

Good point, I will try that too.

I also added padding parameter for MSuper and the new parameter scaleCSAD, added in 2.7, which seems to improve quality when set to positive value (and the optimizer indeed chooses the maximum value for it).

Oh, I just noticed I have been reading old documents of MVTools, didn't even know about this parameter! :eek:

However neither DCT, nor searchalgo or padding converge to any particular values even after 3000 iterations

Converging is not necessary though. If the parameter doesn't change the result then it really doesn't matter which value is used. But 3000 iterations is just probably not enough for them to converge (but I can tell you that 50 000 iterations will make searchalgo converge, in my case to value 6).

unlocking dct=1 (I don't think I saw it choosing 1 at all).

That was brave, unlocking dct=1. ;) But it's definitely there when I looked at your Google spreadsheet, it's even the best result of full run 1.

I'm going to try modifying the script so that it compares to the original discarded frames to get rid of the mistakes introduced by the double upsampling and see if it's gonna be better.

A good idea. You can just remove the second pass and change which clip is used inside FrameEvaluate. Just double check that you're comparing correct frames.

May I ask you why you're doing this frame doubling to a video with 60fps rate?

Here's a Google Spreadsheets link (https://docs.google.com/spreadsheets/d/1E3Da_djkw41ZJf0V9zkACtKE812_aHny1cFb_EstLXM/edit?usp=sharing) where I tried to analyze (for the lack of a better way) results from several 3run*3000iter trials with downscaling and without, comparing the distribution of divide, sharp and pel parameters. I couldn't quite figure out how to make use of the visualizer's groupby method, so I had to come up with my own.

Thanks for that. It's high time I continue the documentation in this thread, especially the evaluation part. But that will have to wait for a little bit longer. I will hopefully have some time tomorrow to run some analysis with your data. Could you also share the script (or scripts) you used to create the logs, what will help especially with the groupby -functionality.

But just a quickie here, you could for example run

optimizer -mode evaluate -log "../scripts/script*.log" -groupby super_pel -vismode series

You should now see two lines which represent the pareto fronts with values pel=2 and pel=4.

zorr, I mean the need to provide full path to .avs when calling the optimizer even if they're in the same directory.

OK, I will have to investigate that. I have used a relative path without any issues.

I'd like to request a way to only generate scripts for the best results of every run instead of the entire pareto front. Usually when AvsOptim finishes I end up manually comparing the run results (with the image setup of above) among themselves and with the handpicked best results from previous runs. As of now it requires a lot of manual parameter editing to match the run results reported by Evaluate mode.

Very well, AvisynthOptimizer v0.9.8-beta (https://drive.google.com/open?id=1KXj2r9n3P1KBDwelYIZTUrOeGQFPw5Z5) is ready for download. The scripts-parameter now takes values "none", "pareto", "best", "bestofrun" in addition to "true" and "false" (these last two are equal to "pareto" and "none"). In your case you want the "bestofrun" option. The "best" will only create a script of the best pareto front result.

Also, I wonder if it would be possible to manually provide one of the generated population members so that a new run could start with one handpicked best parameter set among others and perhaps try to improve on top of that. For example, the optimizer would use the values assigned to var in the script (before the # optimize part) as one of the population members. Does that even make sense?

It does make sense algorithm-wise, I will have to see how to manage that with the libraries I'm using. Stay tuned...

[zorr]

Already mentioned in the post above but let's make it official:
AvisynthOptimizer v0.9.8-beta (https://drive.google.com/open?id=1KXj2r9n3P1KBDwelYIZTUrOeGQFPw5Z5) released.

This version adds new modes to -scripts parameter:

"none": do not write any script files (same as "false")
"pareto": write scripts from the pareto front (same as "true")
"best": write a script from the best pareto front result
"bestofrun": write scripts from the best result of each run (thanks Seedmanc for the suggestion)


There are also some improvements to groupby -functionality:
-multiple scripts with different values for a parameter can be analyzed
-maxgroups also works with parameter values given as a list (previously had to be a value range)

[Seedmanc]

I am not sure if that is the best way to handle this kind of situation though. Basically the optimizer is now blind to these parameters and cannot help you find the optimal values for those.
It's not really blind, I'm just removing the distractions. Super_sharp and super_pel affected sharpness and edge aliasing much more than the actual motion interpolation quality (as in vectors), therefore biasing the SSIM. At least for me it is more important to measure how well the vectors are determined when optimizing for MFlowFps, rather than how sharp the picture is. Therefore I'm removing the constant from the equation to better see the effect those parameters have (or not).
Can you clarify this?
No, I mean that I compared the SSIM effect of super_sharp and super_pel after downscaling 4x for entire video, using the MSU VQMT (http://www.compression.ru/video/quality_measure/vqmt_download.html) tool that can plot the graph for various metrics and calculate average over entire video. The downscaling removed the bias for line sharpness and allowed me to evaluate the real influence of those parameters on the vector quality. As it turned out, super_sharp didn't really had any significant effect, regardless of video length, but pel=2 seems to be somewhat better. Here's how the graph looked without downscaling for super_sharp: image (https://i.imgur.com/SNB64IH.png). That's the parallel alignment I was talking about, it's not due to some frame shift if that's what you meant.

Is the SSIM value for the both frames? In that case the bottom left picture might explain it, in my eye it looks a lot more garbled and deserves lower SSIM.
It does deserve a lower SSIM but as the image shows, for the double upsample method it gets a higher SSIM instead - 0.9564 instead of 0.9555. Single-upsampling does no such error, for it SSIM correlates with the actual visual quality more often. Again, left half of the image is what your double-upsampling produces, and the SSIM on left is calculated for it against the original frames. On the right half is the result of single-upsampling after discarding half of the frames and the SSIM comparing to the discarded frames. The middle number is the ratio between the two, doesn't really mean anything.

May I ask you why you're doing this frame doubling to a video with 60fps rate?
60 fps is the framerate of my test clip, I use it a lot since with that FPS I can drop half the frames and still end up with 30fps to upsample from while being able to compare to the discarded frames without having to do double-upsampling. But actually I can go higher that 60 too, since my monitor can show 120+. For now though I want to learn doing 30 to 60 at least.
https://pastebin.com/uWa1msZ1 - here's an average script I was using when making Google sheets.

So it looks like we figured what options differ the most between working with real footages and animation. That would be super_sharp and divide at least, since we get very different results for them. Possibly overlap and DCT too. Also it always sets maskScale to 1, even though with the default 100 it looks a little better.

[zorr]

It's not really blind, I'm just removing the distractions. Super_sharp and super_pel affected sharpness and edge aliasing much more than the actual motion interpolation quality (as in vectors), therefore biasing the SSIM. At least for me it is more important to measure how well the vectors are determined when optimizing for MFlowFps, rather than how sharp the picture is. Therefore I'm removing the constant from the equation to better see the effect those parameters have (or not).

Ah, I see it now. I bit surprising though that pel=2 would create better vectors. It just occurred to me that maybe it has larger effective search range, assuming that the search ranges are not scaled by pel. That certainly could explain the better vectors.

I'm going to have to do a similar experiment downscaling the result before SSIM comparison and comparing that to non-downscaled SSIM. Perhaps it's always better to downscale.

I compared the SSIM effect of super_sharp and super_pel after downscaling 4x for entire video, using the MSU VQMT (http://www.compression.ru/video/quality_measure/vqmt_download.html) tool that can plot the graph for various metrics and calculate average over entire video.

I have MSU VQMT downloaded but haven't tried it yet. I guess at least some of that functionality could be replaced with an Avisynth script and/or the optimizer with some tweaks. Are you using it primarily for comparing total SSIM of two clips and/or also to see how they change visually?

It does deserve a lower SSIM but as the image shows, for the double upsample method it gets a higher SSIM instead - 0.9564 instead of 0.9555.

Oh, but I meant that the 0.9564 frame was better. At least I can understand why SSIM would rate it such - it doesn't have a lot of higher frequency "noise" like the other picture.

Single-upsampling does no such error, for it SSIM correlates with the actual visual quality more often.

In that case (and because you do have a 60fps source) you should definitely only do a single upsampling. Do you need help with the script implementing that?

https://pastebin.com/uWa1msZ1 - here's an average script I was using when making Google sheets.

Ok thanks. I noticed you used range 0..4 for scaleCSAD but in the latest docs the valid values are -2, -1, 0, 1 and 2. Still your best results used 4 so it has to work... [EDIT] Just noticed you subtract 2 from the value, so it's just like in the docs. Nevermind. And I guess there's a bug with negative values?

Also I want to ask if the results on one tab are from a single run or from multiple runs. There are over 10000 results in full1 and full2 so I'm guessing they consist of multiple runs.

Full2 doesn't have the scaleCSAD parameter, was that your first script variation?

It would help the analyzing if you posted the original log files.

So it looks like we figured what options differ the most between working with real footages and animation. That would be super_sharp and divide at least, since we get very different results for them. Possibly overlap and DCT too. Also it always sets maskScale to 1, even though with the default 100 it looks a little better.

Also the blockSize, though I think the optimal one for that may differ for every video. I was wondering about your maskScale results too, in my tests the maskScale is very close to the maximum (actually I had to set the maximum higher for my latest run, it's one of those params without any official maximum value).

[EDIT] Forgot to ask, which algorithm are you using when running the optimization? If it's still "mutation" I recommend you try the default "SPEA2" because with thousands of iterations it gets better results.

[Seedmanc]

Are you using it primarily for comparing total SSIM of two clips and/or also to see how they change visually?
Yes, that's how I use it mostly, before AvsOptim I was using it a lot to manually analyze the effect of each parameter. It has a lot of various metrics available.
Oh, but I meant that the 0.9564 frame was better
I can't really say how it's better, isn't the quality of framerate upsampling determined by how correct the vectors are? And by correctness I mean how well they converge to a middle point in time between the two existing frames. In 0.956 case you can see that they don't really converge, instead of a single finger you see 3, meanwhile on 0.9555 it's more or less one. This is my primary measurement of visual quality, to see how well the fast-moving objects are treated.
I can figure out the script for single upsampling, thanks.
About negative values, when I was introducing negative badRange somehow the notion of -50..50 didn't work, so I wasn't trying it anymore (well it would actually be a wrong way for that particular parameter). Try and see if negative ranges work for you with scaleCSAD.

As I mentioned, each tab consists of log values from 3 runs of 3000 iterations together. The leftmost column has all the log values, I only removed the header and sorted by SSIM. Yes, Full2 was one of the earliest.
I've switched to SPEA2 around time when MVtools bugs were discussed, so I'm using that now.

[zorr]

I can't really say how it's better, isn't the quality of framerate upsampling determined by how correct the vectors are? And by correctness I mean how well they converge to a middle point in time between the two existing frames.

That's one way to look at it. But I know some people prefer the less correct (technically speaking) blending when the vectors are so bad that there are distracting artifacts. SSIM naturally doesn't know anything about the vectors, it's simply trying to figure out how similar two pictures are. But I totally understand your take on it and it's a valid point of view.

If one does the MCompensate correction after MFlowFPS the most correct vectors should produce better image quality even according to SSIM. But to do that requires that both processes are optimized at the same time roughly doubling the number of optimized parameters. Or perhaps it could be done in turns, first optimizing MFlowFPS only, then optimizing MCompensate and then optimizing MFlowFPS again. Sorry if this doesn't make much sense, I need to make a thread about that technique...

About negative values, when I was introducing negative badRange somehow the notion of -50..50 didn't work

Ok I will investigate.

As I mentioned, each tab consists of log values from 3 runs of 3000 iterations together.

There must be something else there as well, full1 has 10688 results and full2 has 14624 results. Perhaps you ran them with the time limit?

[Seedmanc]

I need to make a thread about that technique...
I'll be looking forward to it. I tried it briefly now and it looks promising, it removes the artifacts like a charm. Though it also erases the moving objects out of existence too sometimes and it can't recreate the missing vectors in high-motion areas, it still looks cool. I usually average the backward and forward compensation together to smooth it out a bit.

Perhaps you ran them with the time limit?
Yes, sorry, that's what I meant. I was doing 3 runs per 3 hours on average.

[ChaosKing]

What am I doing wrong? I tried different names but I always get this error msg:

Found following optimizable parameters:
# optimize tr = _n_ | 1..4 | tr
found 1 parameters to optimize

Running SPEA2
java.lang.Exception: Could not update parameter value for [tr = _n_]
at avisynthoptimizer.Parameter.getLine(Parameter.java:617)
....



TEST_FRAMES = 10 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

ffms2("E:\cut.mkv").ConvertBits(8)
source = last
last=source.AddGrain(80, 0, 0, seed=2)
tr =1 # optimize tr = _n_ | 1..4 | tr
denoised = TemporalDegrain2(degrainTR=tr)
last = denoised

# calculate SSIM value for each test frame
global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(source, denoised)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")

# measure runtime, plugin writes the value to global avstimer variable
# NOTE: AvsTimer should be called before WriteFile
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "D:\AvisynthRepository\AvisynthOptimizer-0.9.8-beta\perFrame.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="D:\AvisynthRepository\AvisynthOptimizer-0.9.8-beta\perFrame.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)


# NOTE: must return last or FrameEvaluate will not run
return last

#Prefetch(0)

[Seedmanc]

ChaosKing, spaces matter. In the template (after # optimize) it says tr = _n_, but in the parameter assignment (before # optimize) you're missing space after =.

[ChaosKing]

Ahhhh didn't know what, thx, works now.

[Boulder]

I can't get it to work, I have the same problem.

source=DGSource("test.dgi",fulldepth=true)
b = -0.6 # optimize b = _n_ | -1..1 | b
c = 0.3 # optimize c = _n_ | -1..1 | c
filtered=BicubicResize(source, 1920, 800, b=b, c=c).Lanczos4Resize(3840, 1600)

I'm testing optimizing the downsizing parameters as mentioned earlier in this thread.

EDIT: changing to "b = 0.6 # optimize b = _n_ | -1..1 | b" got past the error.

[zorr]

I can't get it to work, I have the same problem.

source=DGSource("test.dgi",fulldepth=true)
b = -0.6 # optimize b = _n_ | -1..1 | b
c = 0.3 # optimize c = _n_ | -1..1 | c
filtered=BicubicResize(source, 1920, 800, b=b, c=c).Lanczos4Resize(3840, 1600)

I'm testing optimizing the downsizing parameters as mentioned earlier in this thread.

EDIT: changing to "b = 0.6 # optimize b = _n_ | -1..1 | b" got past the error.

I have located the bug, the current version cannot read negative number values. Will be fixed in the next version.

Also note that floats are not supported, so with range -1..1 the optimizer will only try values -1, 0 and 1. You can get around this limitation by using something like this:

c = 30/100.0 # optimize c = _n_ | -100..100 | c

which would result in range -1.0 .. 1.0 with steps of 0.01.

[Boulder]

I got it to run with this script, but the iterations all have the same SSIM value. If I open the script in VirtualDub2 and run it, the perFrameResults.txt is there but it is not produced when the script is run with optimizer.bat. I'm using the latest Avisynth+ build with your special plugins in x86 mode. If I run in x64 mode, I get the error message "java.lang.NumberFormatException: For input string: "I don't know what 'ssim' means." even though the plugins are in place and I have both environments of Avisynth+ installed.

source=DGSource("test.dgi",fulldepth=true)

b = 33/100 # optimize b = _n_/100 | -100..100 | b
c = 33/100 # optimize c = _n_/100 | -100..100 | c
denoised=BicubicResize(source, 1920, 800, b=b, c=c).Lanczos4Resize(3840, 1600)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(source, denoised)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

[zorr]

I got it to run with this script, but the iterations all have the same SSIM value.
b = 33/100 # optimize b = _n_/100 | -100..100 | b

When you divide by 100 you must use a floating point number as the divisor (100.0), otherwise it's integer division and most results will then end up as 0 which explains the same SSIM value.

If I open the script in VirtualDub2 and run it, the perFrameResults.txt is there but it is not produced when the script is run with optimizer.bat.

That is normal, the optimizer changes the output file name and location, the files are at <optimizer_install_dir>/work.

If I run in x64 mode, I get the error message "java.lang.NumberFormatException: For input string: "I don't know what 'ssim' means." even though the plugins are in place and I have both environments of Avisynth+ installed.

I haven't tested x64 version myself so this could be a bug. Did you set the x64 mode with "-arch x64"?

[Boulder]

When you divide by 100 you must use a floating point number as the divisor (100.0), otherwise it's integer division and most results will then end up as 0 which explains the same SSIM value.
Thanks, I think this should be mentioned this in the tutorial post. Float works only in the first parameter, not in the range call. In the range call you get "java.lang.NumberFormatException: For input string: "-100.0".

I haven't tested x64 version myself so this could be a bug. Did you set the x64 mode with "-arch x64"?It was already in the ini file as per my first test with a clean ini.

[zorr]

AvisynthOptimizer v0.9.9-beta (https://drive.google.com/open?id=1S92SJDbTznsmFnAROQMaa-w9NqqY_MTB) released.

This version adds a new optimizer argument -initial. It's meant for setting the initial population and the choices are:

-random: the default value, creates a random initial population
-script: reads the values set in the script and makes one of the initial population members use those values. The rest of the population is random. (thanks Seedmanc for the suggestion)
-<log_file_name>: Read a log file (or files, using a wildcard *) and create the initial population from the pareto front of the results. If the pareto front is larger than the population size the chosen algorithm will prune the results (but the best result is always included). If the pareto front is smaller than the population size the rest will be filled with random individuals.


NOTE: -initial only works with the algorithm SPEA2 for now, I will add the support for the rest in the next version.


Also some much needed bug fixes:
-can now read and understand a negative value as the variable's original value in the script (thanks Seedmanc and Boulder for the report)
-optimizer will now stop when all parameter combinations have been tested. The population size will be set to number of combinations if it is larger than that (in order to avoid duplicate parameters). Thanks ChaosKing for the report.
-duplicate parameter combinations are no longer possible in the initial population (also thanks to ChaosKing for the report)

[zorr]

AvisynthOptimizer-0.9.10-beta (https://drive.google.com/open?id=1pD9qguKcAkvVNTjpfJeixcukf4yXKtAI) released.

The -initial argument now works with the algorithms NSGA-II and mutation as well.

Some more bug fixes: the input script argument without any path component now works (thanks Seedmanc for the bug report). Also the wildcard didn't work with the -initial argument but does work now.

[ChaosKing]

If you still plan to add VS support someday, there is a new filter now which can calculate a VMAF, PSNR, SSIM and MS-SSIM score (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF)

[Boulder]

It would be really nice, high bitdepth support is already a big plus and also those other scores could be useful.

[Boulder]

Running this kind of downsizing test with only one frame will hit the "DUPLICATE PARAMS" phase quite soon after the process starts. It seems to try the same values over and over again. Setting sensitivity to false helps a bit but the safest way is to sample multiple frames which will then increase the runtime quite a lot.

In this case, I used -1000..1000 as the range, normally I would go for -100..100 but it has the same problem.

TEST_FRAMES = 1 # how many frames are tested
MIDDLE_FRAME = 0 # middle frame number

source=FFVideoSource("c:\avisynthoptimizer\churchill.avi",colorspace="yv12")

b = 330/1000.0 # optimize b = _n_/1000.0 | -1000..1000 | b
c = 330/1000.0 # optimize c = _n_/1000.0 | -1000..1000 | c

denoised=BicubicResize(source, 1280, 536, b=b, c=c).LanczosResize(1920, 800)

c selected for mutation
b selected for mutation
mutated c with 38.399986267089844, value 610 -> 609
mutated b with 38.399986267089844, value -149 -> -148
param values after resolve: b -148 c 609
DUPLICATE PARAMS -148 609
change counts: b 13c 19
b selected for mutation
c selected for mutation
mutated b with 38.399986267089844, value -149 -> -149
mutated c with 38.399986267089844, value 610 -> 610
param values after resolve: b -149 c 610
DUPLICATE PARAMS -149 610
change counts: b 13c 19
c selected for mutation
b selected for mutation
mutated c with 38.399986267089844, value 610 -> 610
mutated b with 38.399986267089844, value -149 -> -149
param values after resolve: b -149 c 610
DUPLICATE PARAMS -149 610
change counts: b 13c 19
b selected for mutation
c selected for mutation
mutated b with 38.399986267089844, value -149 -> -150
mutated c with 38.399986267089844, value 610 -> 609
param values after resolve: b -150 c 609
DUPLICATE PARAMS -150 609

[zorr]

Running this kind of downsizing test with only one frame will hit the "DUPLICATE PARAMS" phase quite soon after the process starts. It seems to try the same values over and over again.

That's weird, the v0.9.9-beta should have fixed that kind of problem. I will try your script and see what I can find. You're running the latest version right?

[Boulder]

That's weird, the v0.9.9-beta should have fixed that kind of problem. I will try your script and see what I can find. You're running the latest version right?

Yes, I'm running v0.9.10-beta. Producing a 2-frame video clip and setting sensitivity to false works much better, about 14000 iterations done without the issue. This is what appears constantly:
Mutating 1 params by 30,0 %
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2

[zorr]

AvisynthOptimizer v0.9.11-beta (https://drive.google.com/open?id=15tPadak_g9LeU9igeGEvabgldxhF-NqQ) released.

This version mainly fixes bugs and makes some minor improvements.

So there was a bug in the JMetal metaheuristics library which I found and fixed, it affected SPEA2 algorithm's "archive truncation". There is a chance that SPEA2 will give better results after this fix, I haven't done a proper test yet to confirm.

Boulder reported a bug which made the sensitivity estimation go haywire, it happened when two consecutive population generations had the exact same runtimes. That one is fixed as well.

Also there were too many "mutation failed 1000 times..." log messages so I removed those. The situation is normal when almost all combinations have been tried already.

Series results count was sometimes reported as zero when autorefreshing chart started reading the log file at the same time, thanks for ChaosKing for the logs.

I made a text change to make it a bit clearer what the best and worst results are about: "Best result" -> "Best run", "Worst result" -> "Worst run". By comparing the best and worst you can determine how reliably the runs achieve a similar result.

Finally, the optimizer will stop running if it's unable to change parameter's value in the script (earlier it caused an error message in an infinite loop).

[zorr]

@Boulder, can you try the latest version and report if it fixes all your issues? You should now be able to use 100.0 as the divider as well.


TEST_FRAMES = 1 # how many frames are tested
MIDDLE_FRAME = 0 # middle frame number


When I tried your script I had to change MIDDLE_FRAME to 1 in order to get clip length of one frame. With zero it returned the whole clip. Does it work differently for you?

[Boulder]

@Boulder, can you try the latest version and report if it fixes all your issues? You should now be able to use 100.0 as the divider as well.
Seems to work fine, thank you.

When I tried your script I had to change MIDDLE_FRAME to 1 in order to get clip length of one frame. With zero it returned the whole clip. Does it work differently for you?

For me it worked as I was feeding a one-frame clip in the optimizer to test the resizing parameters. In Avisynth syntax, Trim(n,0) means that you get all the frames from frame n until the end of the clip. I was just thinking that the middle frame must be inside the clip range and in Avisynth, counting starts from zero.

[Boulder]

Gah, the issue appeared again quite close to the end. I set only -iters 100000 to test. At least it didn't happen right away like it did in the earlier version.

change counts: b 17c 17
b selected for mutation
c selected for mutation
mutated b with 1.463081955909729, value -68 -> 10
mutated c with 1.463081955909729, value 34 -> 100
param values after resolve: b 10 c 100
DUPLICATE PARAMS 10 100
change counts: b 17c 17
b selected for mutation
c selected for mutation
mutated b with 1.463081955909729, value -68 -> -13
mutated c with 1.463081955909729, value 34 -> -79
param values after resolve: b -13 c -79
DUPLICATE PARAMS -13 -79
change counts: b 17c 17
c selected for mutation
b selected for mutation
mutated c with 1.463081955909729, value 34 -> 100
mutated b with 1.463081955909729, value -68 -> 74
param values after resolve: b 74 c 100
DUPLICATE PARAMS 74 100
change counts: b 17c 17
c selected for mutation
b selected for mutation
mutated c with 1.463081955909729, value 34 -> 100
mutated b with 1.463081955909729, value -68 -> -30
param values after resolve: b -30 c 100

[Seedmanc]

I'm trying to add MRecalculate to optimization with the following (partial) script:
doRecalc = true # optimize doRecalc = _n_ | false,true | doRecalc
smooth=1 # optimize smooth=_n_ | 0,1 ; filter:doRecalc | smooth
RblockSize = 12 # optimize RblockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:Rdivide 0 > 8 2 ? ; filter:doRecalc | RblockSize
RsearchAlgo = 5 # optimize RsearchAlgo = _n_ | 0..5 D ; filter:doRecalc | RsearchAlgo
RsearchRange = 4 # optimize RsearchRange = _n_ | 1..30 ; filter:doRecalc | RsearchRange
Rdivide=0 # optimize Rdivide=_n_ | 0..2 D ; max:RblockSize 8 >= 2 0 ? ; filter:doRecalc | Rdivide
Roverlap=4 # optimize Roverlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:RblockSize 2 / ; filter:doRecalc | Roverlap
Rdct = 0 # optimize Rdct = _n_ | 0,1,2,3,4,5,6,7,8,9,10 D ; filter:doRecalc | Rdct
Rmeander = true # optimize Rmeander = _n_ | false,true ; filter:doRecalc | Rmeander
RscaleCSAD = 2 # optimize RscaleCSAD = _n_ | -2..2 ; filter:doRecalc | RscaleCSAD
thsad = 200 # optimize thsad = _n_ | 0..10000 ; filter:doRecalc | thsad

bv = doRecalc ? MRecalculate(super_render, bv, thsad=thsad, smooth=smooth, blksize=RblockSize, search=RsearchAlgo, searchparam=RsearchRange, truemotion=false, overlap=Roverlap, dct=Rdct, divide=Rdivide, meander=Rmeander, scaleCSAD=RscaleCSAD) : bv
fv = doRecalc ? MRecalculate(super_render, fv, thsad=thsad, smooth=smooth, blksize=RblockSize, search=RsearchAlgo, searchparam=RsearchRange, truemotion=false, overlap=Roverlap, dct=Rdct, divide=Rdivide, meander=Rmeander, scaleCSAD=RscaleCSAD) : fv


but Optimizer fails with
java.lang.Exception: Unknown element doRecalc
at avisynthoptimizer.parser.RPNParser.push(RPNParser.java:124)
at avisynthoptimizer.AviSynthOptimizer.filterValue(AviSynthOptimizer.jav
a:3709)
What could be wrong? I tried replacing filter with max, not helpful. For now I had to get rid of filtering by doRecalc, but that's wasting iterations.

[zorr]

AvisynthOptimizer v0.9.12-beta (https://drive.google.com/open?id=1kjo-GK7Pt5aRpFpDwBwc7K4lF7l5lCeD) released.

Some minor bug fixes and improvements:


set the logging threshold higher for reporting the mutation failures, now has to be at least 100 000 failed attempts before logging starts (thanks @Boulder for the report)
bugfix: conflict resolvation didn't support variables which had boolean types in min/max/filter definition, it does now. This is related to the bug @Seedmanc reported above, thanks! Also related: the optimizer will stop running if it's unable to resolve conflicts in 100 000 tries.
avsr version updated to latest v0.1.9. It has a cool new -frames argument which I'm going put to good use, but the new feature (automatic frame selection) using it is not finished yet. Thanks @Groucho2004!

[zorr]

I'm trying to add MRecalculate to optimization with the following (partial) script:
doRecalc = true # optimize doRecalc = _n_ | false,true | doRecalc
smooth=1 # optimize smooth=_n_ | 0,1 ; filter:doRecalc | smooth

but Optimizer fails


The first problem here was that the optimizer didn't support variables with boolean types (like the variable "doRecalc"). The latest version v0.9.12-beta does so grab that one.

The second problem is that the filter rejects all the values when doRecalc==false, therefore the conflict resolvation cannot finish (it expects to find some combination of parameters which are valid).

How to fix this? You have to return at least one valid value for every parameter. Using variable RblockSize as an example, the filter could be like this:
filter:doRecalc x 16 ? x ==

The logic here is that when doRecalc is true the rest of the equation becomes (x == x) which is always true so it accepts all the values. When doRecalc is false the equation becomes (x == 16) accepting one value. Since the MRecalculate is disabled the value we choose doesn't matter but we have to choose something anyway.

If the parameter already has a filter, you can add min or max definition (easier to separate the logic than try to make one more complex filter).

Min:
min:doRecalc 4 64 ?

Max:
max:doRecalc 64 4 ?

The min will set 64 as the minimum value when doRecalc is false, which leaves only the value 64 as valid. The max is similar but sets the maximum as 4.

So it's a bit of work to add all these min/max/filter definitions, I will have to think if there's an easier way to support this kind of disabling of parametes.

Another way is to make a separate script for the MRecalculate-version. Optimize both scripts and see which one gave better results. I tried this and to my surprise the script without MRecalculate was better. I think it might be because there are much more parameters to optimize in the MRecalculate-script and therefore the search doesn't get anywhere close to optimal results in the same iteration count.

[Boulder]

I got this one with the latest version. It happened quite close to the end (once again those resizing tests of mine) and seems to happen with every run. I fired up five concurrent optimizations and three have now failed with the error around the ~39000th iteration.

java.lang.IllegalArgumentException: bound must be positive
at java.util.Random.nextInt(Unknown Source)
at avisynthoptimizer.AviSynthOptimizer.selectRandomWeightedByChangeCount(AviSynthOptimizer.java:2493)
at avisynthoptimizer.AviSynthOptimizer.mutateResult(AviSynthOptimizer.java:5049)
at avisynthoptimizer.nsga_ii.RangeMutation.execute(RangeMutation.java:93)
at avisynthoptimizer.nsga_ii.RangeMutation.execute(RangeMutation.java:25)
at avisynthoptimizer.spea2.DynamicSPEA2.reproduction(DynamicSPEA2.java:103)
at org.uma.jmetal.algorithm.impl.AbstractEvolutionaryAlgorithm.run(AbstractEvolutionaryAlgorithm.java:60)
at java.lang.Thread.run(Unknown Source)

[zorr]

I got this one with the latest version. It happened quite close to the end (once again those resizing tests of mine) and seems to happen with every run. I fired up five concurrent optimizations and three have now failed with the error around the ~39000th iteration.


Thanks for that report, I released v0.9.13-beta (https://drive.google.com/open?id=1rNrcqoqjkTpFaVaEm71tRsqUiwUdE6q7) where this is fixed.

When I changed the mutation failures logging threshold higher I introduced a bug where the code wanted to mutate more parameters than exists. Nothing good will happen when you try that. :)

[Seedmanc]

Something is still wrong, before 0.9.13 it was doing like 60 iterations overnight where it previously done thousands, now it just failed after 13 (says "java.lang.NumberFormatException: For input string: "true"), here's the script https://pastebin.com/HZH4Ac8v
zorr, previously you mentioned that every parameter involving other parameters in filters/minmaxes should "mirror" that dependency in the description of that parameter as well to avoid bias. In my example I tie a lot of recalculate params to doRecalc, but don't tie it to them, will that affect the script, and if so, how?
Also, in my script, I filter overlap by blocksize/2, but don't do filter blocksize by overlap, same question.

Finally, could you add a != to a list of supported operations? I was trying to limit DCT=1 to blocksizes less or equal to 24 by doing filter:blockSize 24 > 1 -1 ? x !=. Is there a better way?

[zorr]

Something is still wrong, before 0.9.13 it was doing like 60 iterations overnight where it previously done thousands, now it just failed after 13

I'm looking into this, meanwhile here's a link to v0.9.12 (https://drive.google.com/open?id=1kjo-GK7Pt5aRpFpDwBwc7K4lF7l5lCeD) if you don't have it anymore.

[zorr]

Something is still wrong, before 0.9.13 it was doing like 60 iterations overnight where it previously done thousands, now it just failed after 13 (says "java.lang.NumberFormatException: For input string: "true")

Thanks again for the report. I missed one part where boolean types were still not supported in the conflict resolvation. That's why it crashed with NumberFormatException, but it also affected the choice of parameters and pretty much never set doRecalc=false (at least in my test). Version 0.9.14-beta (https://drive.google.com/open?id=1AwQd_7OG5BHda6UE_XsSmv5Tffaya17Z) fixes this.

The slowness I think might be due to always doing the MRecalculate, in my tests while most runs took less than half a second, there were occasional long ones taking as long as 274 seconds. If it sticks with those slow parameter combinations it would only run 13 tests per hour which is in the same ballpark as your runs. You can look in the log files if there are individual slow runs which could explain it.

zorr, previously you mentioned that every parameter involving other parameters in filters/minmaxes should "mirror" that dependency in the description of that parameter as well to avoid bias. In my example I tie a lot of recalculate params to doRecalc, but don't tie it to them, will that affect the script, and if so, how?

The result of the script execution (quality and speed) would be the same without the doRecalc-dependency so it's only trimming down the number of combinations and not introducing any bias.

Also, in my script, I filter overlap by blocksize/2, but don't do filter blocksize by overlap, same question.

This will introduce some bias. Basically every time blocksize and overlap conflict, it's the overlap that will get changed. Here's an example case: let's say blocksize is 16 and overlap is 8. Only one parameter is mutated (we're in the final stretch of the optimization) and overlap is selected for mutation. If the overlap is trying to become bigger (let's say 12) the blocksize is in conflict and now the overlap will be rolled back to 8. If it's trying to become smaller it can still do that, so that means overlap can only change into one direction.

Finally, could you add a != to a list of supported operations? I was trying to limit DCT=1 to blocksizes less or equal to 24 by doing filter:blockSize 24 > 1 -1 ? x !=. Is there a better way?

The support for != is also in the latest version so filter:blockSize 24 > 1 -1 ? x != should now work. I couldn't come up with anything cleaner. I think I should also add some boolean operators (at least AND and OR).

[EDIT] Almost forgot, you can save 1-3 milliseconds per frame if you do the scaling outside FrameEvaluate. Something like this:

divisor = 4
scaled_orig = orig_yv12.bilinearresize(orig_yv12.width/divisor,orig_yv12.height/divisor)
scaled_inter = inter_yv12.bilinearresize( inter_yv12.width/divisor, inter_yv12.height/divisor)

global total = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(scaled_orig, scaled_inter)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""")

[zorr]

The result of the script execution (quality and speed) would be the same without the doRecalc-dependency so it's only trimming down the number of combinations and not introducing any bias.


I may have to retract that. It doesn't cause bias immediately but when a result with doRecalc=false gets mutated so that doRecalc becomes true then all the parameters for MRecalculate are having the values specified in the filter/min/max. Those values are then going to be tested a lot more than others. It might be a good idea to test what kind of results you get with and without the doRecalc dependency.

I'm running your script with the doRecalc dependency enabled and this time the MRecalculate is winning, all except the fastest two pareto front results use it.

https://i.postimg.cc/W1NjS4KK/seedmanc05.png

Top 3 results, running on a clip of Frozen:

9.941789 1900 super_pad=48 super_pel=4 super_sharp=2 super_rfilter=3 blockSize=12
searchAlgo=2 searchRange=2 searchRangeFinest=6 divide=2 overlap=4 badSAD=4045
badRange=27 negBadRange=true meander=false temporal=true trymany=true dct=1
scaleCSAD=-2 doRecalc=true smooth=0 RblockSize=8 RsearchAlgo=5 RsearchRange=3
Rdivide=0 Roverlap=0 Rdct=10 Rmeander=true RscaleCSAD=1 thsad=0 maskScale=4

9.941031 1670 super_pad=48 super_pel=4 super_sharp=2 super_rfilter=3 blockSize=12
searchAlgo=0 searchRange=2 searchRangeFinest=6 divide=2 overlap=4 badSAD=545
badRange=2 negBadRange=true meander=false temporal=true trymany=true dct=1
scaleCSAD=-2 doRecalc=true smooth=1 RblockSize=8 RsearchAlgo=2 RsearchRange=3
Rdivide=0 Roverlap=0 Rdct=7 Rmeander=true RscaleCSAD=2 thsad=59 maskScale=1

9.940445 1650 super_pad=48 super_pel=4 super_sharp=2 super_rfilter=3 blockSize=12
searchAlgo=0 searchRange=2 searchRangeFinest=6 divide=1 overlap=4 badSAD=860
badRange=5 negBadRange=true meander=false temporal=true trymany=true dct=1
scaleCSAD=-2 doRecalc=true smooth=0 RblockSize=8 RsearchAlgo=2 RsearchRange=3
Rdivide=0 Roverlap=0 Rdct=0 Rmeander=true RscaleCSAD=2 thsad=62 maskScale=1

[Boulder]

AvisynthOptimizer v0.9.12-beta (https://drive.google.com/open?id=1kjo-GK7Pt5aRpFpDwBwc7K4lF7l5lCeD) released.

Some minor bug fixes and improvements:


set the logging threshold higher for reporting the mutation failures, now has to be at least 100 000 failed attempts before logging starts (thanks @Boulder for the report)


The mutation failures still pop up near the end and apparently slow down the process. Is it possible to tell the optimizer to just test all the possible combinations? As I have only two parameters to adjust, there's a finite amount of them and I want to see all of them tested anyway. It would also be nice to have a graph of such cases with the two parameters as x- and y-axis.

[zorr]

The mutation failures still pop up near the end and apparently slow down the process. Is it possible to tell the optimizer to just test all the possible combinations? As I have only two parameters to adjust, there's a finite amount of them and I want to see all of them tested anyway.

I have pondered about such functionality myself. As of now it's not possible and in the end it's getting harder and harder to find a mutation which hasn't already been tried before. It would be much simpler to just iterate all the combinations in order. I'll implement that as a new algorithm variation.

What's the divider you're using with your two parameters (ie how many combinations are you testing)?


It would also be nice to have a graph of such cases with the two parameters as x- and y-axis.

And the color of the x,y position would represent the value? The chart library I'm using doesn't support such a visualization but I think I can find a way to implement that.

[Boulder]

I have pondered about such functionality myself. As of now it's not possible and in the end it's getting harder and harder to find a mutation which hasn't already been tried before. It would be much simpler to just iterate all the combinations in order. I'll implement that as a new algorithm variation.

What's the divider you're using with your two parameters (ie how many combinations are you testing)?
The amount of combinations is a little over 40000 (two parameters, and possible values -100..100).



And the color of the x,y position would represent the value? The chart library I'm using doesn't support such a visualization but I think I can find a way to implement that.
Yes, that should do it. I suppose it is possible to change the colors on the fly as we don't know where the top will be, or did you think the min-max values would determine what color grade to use? At least in my resizing tests, the values have hit a quite narrow range compared to the min-max range.

[Boulder]

As many tests have been run with MVTools, is there a generic way to optimize the parameters for motion estimation? I mean, is the "add fake noise" - "denoise" the way or is MFlow or MCompensate better in such cases? As the resizing optimization is a rather simple task, it would be interesting to optimize the motion search parameters for my denoising function as well. Of course, SSIM may not be the optimal measuring method there but currently VMAF is Vapoursynth-only.

[zorr]

AvisynthOptimizer v0.9.15-beta (https://drive.google.com/open?id=1u6iyVXsTvGLZaoR8RD9rfepK8RdLQxJy) released.

This version adds the "exhaustive" algorithm which tests all the valid parameter combinations (unless the number of combinations is greater than 2147483647, in which case it "only" tries the first 2147483647 combinations. :) When running the exhaustive algorithm the number of runs is one by default, you can still set it higher by providing the -runs argument. The other optimization arguments are not relevant.

I also added operators "and" and "or" to conflict resolvation parser. I couldn't use the more familiar && and || since the "|" is already used for separating the definition parts.

There's also a bugfix: the filter dependency did not find any valid values when the parameter was defined as a range.

@Boulder I will work on the 2D heatmap-style visualization next.

[Boulder]

Thank you, the exhaustive algorithm will be very useful :)

[Boulder]

There's a small bug with the selection of frames. For example, I've set to test 20 frames of a clip (TEST_FRAMES = 20, MIDDLE_FRAME = 10) but when I open the script used for calculations, I get 19 frames in VDub.

[StainlessS]

Should TEST_FRAMES be an odd number ? (otherwise MIDDLE _FRAME aint middle frame).
With TEST_FRAMES=20, MIDDLE _FRAME=10, so 10 frames prior to middle frame and 9 after.
(maybe it just rounds to 9 frames before, and 9 frames after, where frame 0 is skipped, is it frame 0 missing ?).

[zorr]

There's a small bug with the selection of frames. For example, I've set to test 20 frames of a clip (TEST_FRAMES = 20, MIDDLE_FRAME = 10) but when I open the script used for calculations, I get 19 frames in VDub.

That's odd, it works for me. I verified this by opening the script in VDub and looking the File / File Information window where you can see the length in frames.

The formula is clip.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)

Doing the math we get clip.Trim(10 - 10 + 1, 10 + 10) which is clip.Trim(1, 20)

Using Trim with positive arguments it returns the frames between the arguments including the first and last frame, that would be frames from 1 to 20 and that's 20 frames. Even using MIDDLE_FRAME 9 we would get 20 frames, this time from 0 to 19.

Perhaps it's not very intuitive if you want the first frames. For that use case a plain Trim(0, -TEST_FRAMES) would probably be better.

[Boulder]

That's odd, it works for me. I verified this by opening the script in VDub and looking the File / File Information window where you can see the length in frames.

The formula is clip.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)

Doing the math we get clip.Trim(10 - 10 + 1, 10 + 10) which is clip.Trim(1, 20)

Using Trim with positive arguments it returns the frames between the arguments including the first and last frame, that would be frames from 1 to 20 and that's 20 frames. Even using MIDDLE_FRAME 9 we would get 20 frames, this time from 0 to 19.

Perhaps it's not very intuitive if you want the first frames. For that use case a plain Trim(0, -TEST_FRAMES) would probably be better.
Thanks, that simple command works fine. I always create a noncompressed testclip for my resize tests by selecting random frames with SelectEvery, and the amount varies between 4-20 frames depending on how much time I have :)

[Seedmanc]

zorr, in your experience, did you notice any relation between the resulting searchRange and searchRangeFinest? I'm trying to figure out which one should normally be larger than the other, but the observations are inconclusive.

Have you tried optimizing scripts with the MCompensate trick yet? I've just started and already found that overlap=0 crashes mvtools.

[Boulder]

I have a small request regarding the log file: would it be possible to have the result of the best score written as the last item like it is in the command prompt output? When I run multiple consecutive optimizations with a for loop, the information is lost and I need to use Excel to get the parameters to use in the final script for encoding.

[Seedmanc]

Or maybe even sort the entire log by SSIM upon finishing the pass, been thinking about this as well.

[Boulder]

Or maybe even sort the entire log by SSIM upon finishing the pass, been thinking about this as well.

True, that would also be a good option.

[pinterf]

I've just started and already found that overlap=0 crashes mvtools.
Big thanks for the report, regression in v37, fixed in v38!

[Boulder]

Big thanks for the report, regression in v37, fixed in v38!

Hey, while you are here.. could you consider porting the VMAF filter to Avisynth+? :) Currently it's Vapoursynth-only, and it could be really useful with the optimizer.

https://github.com/Netflix/vmaf

[zorr]

zorr, in your experience, did you notice any relation between the resulting searchRange and searchRangeFinest? I'm trying to figure out which one should normally be larger than the other, but the observations are inconclusive.

This is a good opportunity for me to introduce the latest feature: heat map visualization. To answer your question let's make a heatmap of these two parameters. The command is:

optimizer -mode evaluate -vismode seriesheatmap -map searchRange searchRangeFinest -log "../scripts/some_script*.log"

It will display a heat map where the brightest colors represent the best results. With my logs (a total of 77902 results) it looks like this:

https://i.postimg.cc/3xLCbnGj/search-Range-search-Range-Finest-top100.png

It's a bit difficult to see where the best results are so let's focus on the best 10% of results by adding -top 10 to the command:

https://i.postimg.cc/dtYB77vC/search-Range-search-Range-Finest-top10.png

Ok, so it looks like most of the good results have searchRange=2 and searchRangeFinest is not that important.

But wait, let's see another set of results which are using a different source video and also a different script (a total of 271729 results):

https://i.postimg.cc/QdNJbRFx/search-Range-search-Range-Finest-top100-bridge.png

Hmm... this one was using -top 100 but we need to focus on the best results again, using -top 20:

https://i.postimg.cc/Ghxjvch5/search-Range-search-Range-Finest-top20-bridge.png

Oh, this time the best results have searchRangeFinest 1 or 2 and searchRange is not that important.

So I guess there's no fixed rule. Maybe it's possible to get good results in both ways, but the results here are a good indication the the most optimal settings sometimes need searchRange larger than searchRangeFinest and sometimes the opposite.

Here's the latest version v0.9.16-beta (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) which includes the heat map visualization.


Have you tried optimizing scripts with the MCompensate trick yet? I've just started and already found that overlap=0 crashes mvtools.

Sorry, haven't gotten around to do that yet. I did do some experiments several months ago.

[zorr]

I have a small request regarding the log file: would it be possible to have the result of the best score written as the last item like it is in the command prompt output? When I run multiple consecutive optimizations with a for loop, the information is lost and I need to use Excel to get the parameters to use in the final script for encoding.

I use the -mode evaluate for this purpose. It can search for and show the best result (and the whole pareto front) from one or multiple log files. You don't need to save the values either, just add -scripts best and you get a script with the best settings already set.

Oh, and maybe you already noticed but the latest version (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) has the heat map visualization you asked for. :) Since your script only has two parameters you don't need to specify the -map argument (the default is taking the fist two parameters from the script). It works with the autorefresh feature so if you want to watch the process you can call it like this:

optimizer -mode evaluate -vismode heatmap -autorefresh true

If you want to analyze multiple runs then use -vismode seriesheatmap.

https://i.postimg.cc/7YdhmRYr/b-c-top20.png

[Boulder]

I use the -mode evaluate for this purpose. It can search for and show the best result (and the whole pareto front) from one or multiple log files. You don't need to save the values either, just add -scripts best and you get a script with the best settings already set.Hmm, there must some bug there as I've tried that and -scripts bestofrun, but there are no scripts to check out after the analysis is complete. The work folder is empty and the only scripts in the optimizer folder are the original ones used for the analysis.

Oh, and maybe you already noticed but the latest version (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) has the heat map visualization you asked for. :)
Thank you very much for this, it looks really useful for my purposes. Much easier to see the ballpark for the sane values to use if I run a quick test for some episode of a series first :)

[zorr]

Hmm, there must some bug there as I've tried that and -scripts bestofrun, but there are no scripts to check out after the analysis is complete. The work folder is empty and the only scripts in the optimizer folder are the original ones used for the analysis.


Thanks for the report. I tried to reproduce this but didn't find any issues. The scripts should be written into the same directory where the original script file was. And the original script folder is read from the log files. The first line starting with #script has the script path that will be used. What does that line look like in your log files?

# script D:\optimizer\bin/script.avs
# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
19.70072 1980 b=-7 c=-93
19.797161 1980 b=41 c=66
19.828617 1980 b=-86 c=-27
19.700043 1970 b=28 c=-56
19.80134 1970 b=-75 c=-67
...

[Boulder]

Running optimizer test2.avs -alg exhaustive -scripts best

# script C:\AvisynthOptimizer/test2.avs
# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
0.987764 70 b=-80 c=10
0.987764 70 b=-79 c=10
0.98775 70 b=-78 c=10
0.987745 70 b=-77 c=10

At the end of the run, I get this information, but there is no script with that best data available. Test2.avs is like it is saved by me.
Pareto front:
0.987969 70ms b=-68 c=52
0.987969 70ms b=-68 c=49
0.987969 70ms b=-72 c=45
0.98794 60ms b=-54 c=59


The heatmap is indeed very useful. I can easily see which values for b I can safely leave out and reduce the amount of combinations by a fair amount :)

[zorr]

Running optimizer test2.avs -alg exhaustive -scripts best

# script C:\AvisynthOptimizer/test2.avs
# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
0.987764 70 b=-80 c=10
0.987764 70 b=-79 c=10
0.98775 70 b=-78 c=10
0.987745 70 b=-77 c=10

At the end of the run, I get this information, but there is no script with that best data available. Test2.avs is like it is saved by me.
Pareto front:
0.987969 70ms b=-68 c=52
0.987969 70ms b=-68 c=49
0.987969 70ms b=-72 c=45
0.98794 60ms b=-54 c=59


Ok now I see the problem. The -scripts is an argument that only works with -mode evaluate. So after your optimization is done (or even during it) you should call:

optimizer -mode evaluate -scripts best


The heatmap is indeed very useful. I can easily see which values for b I can safely leave out and reduce the amount of combinations by a fair amount :)

That's nice! You can also use a filter to reduce the number of tried combinations by forcing the values to be divisible by 10 (or any other number). For example:

b = 33/100.0 # optimize b = _n_/100.0 | -100..100 ; filter:x 10 % 0 == | b

[Boulder]

Ok now I see the problem. The -scripts is an argument that only works with -mode evaluate. So after your optimization is done (or even during it) you should call:

optimizer -mode evaluate -scripts best

Thanks, will try that. What if I have multiple log files, like I usually do after the for loop procedure.

[zorr]

Thanks, will try that. What if I have multiple log files, like I usually do after the for loop procedure.

You can specify the log files with the -log argument and using the wildcard *. Example:

optimizer -mode evaluate -log "./scripts/part_of_filename*.log"

If your logs are from the same script then they should all start with the script name and you can use -log "./path/script_name*.log". Or maybe you put all the log files into the same directory, then you can use -log "./path_to_directory/*.log". Maybe you only want to analyze the files from script "abc" created in this month, then you can use -log "./path/abc*2018-12*.log". And so on.

[Boulder]

Cheers, that will smooth things out a lot. It's a bit of a pain putting everything through Excel. I have to do that already too many times at work these days :D

[Seedmanc]

How do I filter/limit the boolean or string values? Suppose I want to have a variable always be false when another var is below 1. Usually I'd write it as filter:Clevel 1 > NULL true ? x != so that when Clevel is above 1, x is compared to something that it is guaranteed not to be, thus resulting in an always true statement, allowing both true and false. But AvsOptim doesn't know what NULL is or whatever else I try to put instead of that.

[zorr]

How do I filter/limit the boolean or string values? Suppose I want to have a variable always be false when another var is below 1. Usually I'd write it as filter:Clevel 1 > NULL true ? x != so that when Clevel is above 1, x is compared to something that it is guaranteed not to be, thus resulting in an always true statement, allowing both true and false. But AvsOptim doesn't know what NULL is or whatever else I try to put instead of that.

There are a couple of ways to approach this. The simplest way is probably returning true whenever Clevel > 1 (all values of x are valid in that case) and also return true when x == false (no matter what Clevel is, x is allowed to be false). In infix form it's

(Clevel > 1) or (x == false)

and translated to reverse polish notation

Clevel 1 > x false == or

Note that in your original filter and this one false is the only allowed value when Clevel is below 2. If you want it false when Clevel is below 1 then use Clevel > 0 or Clevel >= 1.

[Boulder]

I came up with an idea to reduce the amount of needed iterations, works at least for the resizer test that I run in exhaustive mode.

My results seem to be a bell-shaped curve, so it would be safe to skip the rest of the iterations of b against constant c. Thus, when the SSIM result of a pair is lower than the previous result, start the next round of varying b against c.

[zorr]

I came up with an idea to reduce the amount of needed iterations, works at least for the resizer test that I run in exhaustive mode.

My results seem to be a bell-shaped curve, so it would be safe to skip the rest of the iterations of b against constant c. Thus, when the SSIM result of a pair is lower than the previous result, start the next round of varying b against c.

Yes that would work. Your problem is the kind of problem that can be solved with a hill climbing (https://en.wikipedia.org/wiki/Hill_climbing) algorithm. The mutation algorithm works as hill climbing with population 1 and a small mutation amount. Try these parameters:

-alg mutation -pop 1 -runs 1 -mutcount 1 -mutamount 0.01

The search will start at some random location and gradually move towards the optimum. The heat map looks like this:

https://i.postimg.cc/mkjVCgRV/boulder-hillclimbing.png

The heat map will show a "cross" when the optimum has been reached (the optimum being located at the center of the cross).

[Boulder]

Try these parameters:

-alg mutation -pop 1 -runs 1 -mutcount 1 -mutamount 0.01

The heat map will show a "cross" when the optimum has been reached (the optimum being located at the center of the cross).

Thanks, worked fine at least with my test of 5 episodes. The first five were run with those settings and the last five with the exhaustive algorithm.


Run 1 best: 4.952373 570 b=-54 c=52
Run 2 best: 4.934916 600 b=-56 c=58
Run 3 best: 4.9637823 590 b=-46 c=34
Run 4 best: 4.97074 600 b=-41 c=60
Run 5 best: 4.8731537 570 b=-67 c=57
Run 6 best: 4.952373 570 b=-54 c=52
Run 7 best: 4.934916 600 b=-56 c=58
Run 8 best: 4.9637823 600 b=-47 c=34
Run 9 best: 4.97074 580 b=-41 c=60
Run 10 best: 4.8731537 590 b=-67 c=57


The results of a hillclimb series looks like this, so there's probably room for improvement. There were 2000 iterations (default?) there but the best result would probably have been found with less. The exhaustive test needs 2856 iterations at the moment - b is -75..25 and c is 15..70. If the best result hits any of those borders, I run a short optimization run with min-max limits close to the border in question.

https://i.postimg.cc/ZK9x76Sp/hillclimb.png

[zorr]

The results of a hillclimb series looks like this, so there's probably room for improvement. There were 2000 iterations (default?) there but the best result would probably have been found with less. The exhaustive test needs 2856 iterations at the moment - b is -75..25 and c is 15..70. If the best result hits any of those borders, I run a short optimization run with min-max limits close to the border in question.

Yes it looks like the optimal result can be found with less iterations. In my example it was found within 300 iterations but it could vary (it's a randomized process after all). You can set the number of iterations with the argument -iters.

I think this is where the dynamic iteration count could be useful because it can stop the optimizing process when no better results are found. I ran some tests and found settings where the optimum was found in 10 out of 10 runs (using b and c range -100..100). Try these:

-alg mutation -iters dyn -dyniters 12 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

The mutation amount is large (0.1) in the beginning and small (0.01) in the end, and looks like this:

https://i.postimg.cc/9QwCfFVs/boulder-dynamic.png

It takes on average about 100 iterations per run. That's 28 times faster than the exhaustive algorithm, not bad! :)

[Boulder]

Try these:

-alg mutation -iters dyn -dyniters 12 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

It takes on average about 100 iterations per run. That's 28 times faster than the exhaustive algorithm, not bad! :)

Thanks, I'll definitely test that with the next batch of encodes I have. A hundred iterations sounds tremendous, as I can then use more frames for analysis to make sure I get a good allround result.

[Dogway]

Is it possible to resume a test that finished too early and take logs into consideration to refine future mutations?
I'm using the last dynamic mode posted above but it finished too early.

I might have understood it wrong but in my tests the best result is always the one with least change, naturally we want a change since we are denoising. The following always ends with a smdegrain(tr=2,thSAD=100,contrasharp=true,blksize=8,overlap=4,divide=0,refinemotion=true,lsb=true)
sigma = 20*20 # optimize sigma = _n_*20 | 100..600 | sigma
blockSize = 8 # optimize blockSize = _n_ | 4,8,16,32 ; min:divide 0 > 8 2 ? ; filter:overlap 2 * x <= | blockSize
overlap = 4 # optimize overlap = _n_ | 4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 8 2 ? % 0 == | overlap
tr = 3 # optimize tr = _n_ | 2..4 | tr
divide = 0 # optimize divide = _n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
denoised=smdegrain(tr=tr,thSAD=sigma,contrasharp=true,prefilter=2,blksize=blockSize,divide=divide,overlap=overlap,refinemotion=true,lsb=true)

[zorr]

Is it possible to resume a test that finished too early and take logs into consideration to refine future mutations?

There's no real resume support, but you can take the best results from the logs as the initial population with the -initial argument. Take a look here (https://forum.doom9.org/showthread.php?p=1856877#post1856877). I can see the value in resuming the optimization, for example when Windows suddenly decides that it's mandatory to install some updates and reboot the computer while I'm running an optimization with 50000 iterations... So I will probably implement that at some point.

I'm using the last dynamic mode posted above but it finished too early.

Those settings are not really suitable for the general case, they were handcrafted to a specific kind of optimization which is easier than most. You probably need a lot more iterations than those settings provide. You should perhaps try the default optimization settings first and see if you get better results with those.

I might have understood it wrong but in my tests the best result is always the one with least change, naturally we want a change since we are denoising.

When doing denoising you need to make a clip with added noise, then remove the noise and finally compare the denoised clip with the original. See here (https://forum.doom9.org/showthread.php?p=1851670#post1851670) for an example. I have thought about other ways (https://forum.doom9.org/showthread.php?p=1856031#post1856031) to do it but this is currently the most reliable way.

Can you show what kind of values you are returning from the script?

[ChaosKing]

When doing denoising you need to make a clip with added noise, then remove the noise and finally compare the denoised clip with the original.

I would be nice if we would have also halo and "other crap" simulators to easily test (optimize) other kinds of filters.

[Dogway]

When doing denoising you need to make a clip with added noise, then remove the noise and finally compare the denoised clip with the original. See here (https://forum.doom9.org/showthread.php?p=1851670#post1851670) for an example. I have thought about other ways (https://forum.doom9.org/showthread.php?p=1856031#post1856031) to do it but this is currently the most reliable way.

I see, that makes more sense although it kind of becomes a synthetic test since normally we don't have a ground truth to compare to. I'll be doing grain modeling with GrainFactory instead since it resembles more film grain.

I thought on using the dynamic mode because if I only use iterations I felt it was being stucked into some ranges.
Leaving it at low iterations and 5 run cycles it would get out of said ranges but now it would totally ignore past cycle logs which is undesired. I will try to do some other tests with low cycles and high iterations.

[Dogway]

Following your indications and still get the least-change variables.

Best result:
smdegrain(tr=1,thSAD=100,contrasharp=0,prefilter=0,blksize=32,overlap=12,divide=0)

Not sure if I'm doing something wrong. If you need my scripts please tell me and I upload them.

[zorr]

Not sure if I'm doing something wrong. If you need my scripts please tell me and I upload them.

Yes that's a good idea. I will take a look.

[EDIT] I have a suspicion of what the problem is. If your video is very grainy then the degraining process cannot remove a lot of noise in order to look like the original. This add noise -> remove noise -> compare only works well when the original video doesn't have much (or any) noise. You could try finding a part of the video where there is very little noise or perhaps even use a completely different video (with very little noise but similar content and movement) as the source in the optimization.

[Dogway]

Here's the link with the 3 clips. A slightly modified version of smdegrain to test contrasharp and divide. Prefilter has a few dependencies so you might want to comment it.

[zorr]

Here's the link with the 3 clips.

Thanks. I don't know why but my 7zip says it doesn't support the compression method. I downloaded the latest version but apparently it needs a reboot in order to install and I'm in the middle of optimization process so I don't want to do it now. Can you provide the files in another zip format?

[StainlessS]

Zorr, My version 7zip from April 2018 opens Dogway zip no problem, yours must be reasonably old version.

Latest update 2 days ago, 2018 Dec 30:- https://www.7-zip.org/download.html

No reboot required here (x64 version).
I think about 12 -> 18 months ago, some fancy new algo was added to 7zip.

For 7zip using zip archive, should perhaps set to Normal compression,
and whatever you wish for 7z archives.

[zorr]

Here's the link with the 3 clips. A slightly modified version of smdegrain to test contrasharp and divide.

Is your modified version uploaded somewhere? Looks like the latest released SMDegrain version (v3.1.2.100s) doesn't have the divide argument. And it would be a good idea to use the exact same function anyway so we can compare results.

The source video doesn't have any grain so it should work well...

I'm going to have to delay the testing a bit because I also need to update MVTools2 and the optimization task I'm running uses an older version. I don't want to do anything radical while it's running... should be done tomorrow.

[zorr]

I would be nice if we would have also halo and "other crap" simulators to easily test (optimize) other kinds of filters.

The halo removal is something I have thought about. Using a synthetic test is probably not useful there because the effect is unique in every video so it needs customized parameters.

The way I would do it: take a single frame (or even a small part of the frame) where the halo effect is clearly visible and use Photoshop or whatever to remove it manually (using the clone stamp tool for example). Then use a halo removal filter in AviSynth and optimize its parameters so that it replicates the result. The same parameters should then work well for the whole video.

[Dogway]

Here (https://www.sendspace.com/file/3orm7i) it is, in the rush I forgot to include it. This is not a release or anything, I modified it a bit to test divide and integer type data for contrasharpening (the generic "val" data type wasn't suited for the variable). I skipped prefilter=3 because I can't make dfttest work on my end. The denoised version was denoised with: smdegrain(tr=3,thSAD=250,contrasharp=40,prefilter=2,blksize=16,overlap=8,divide=0,refinemotion=true,lsb=true). Which is (maybe?) a good starting point for grainy 1080p, in lack of better denoisers for avs+ like BM3D.

Whenever you have time is ok, I use the mvtools2 from pinterf github.

[zorr]

I see, that makes more sense although it kind of becomes a synthetic test since normally we don't have a ground truth to compare to.

I agree that it's a synthetic test. But there's something you can do to improve SMDegrain that doesn't rely on such a synthetic test. Since SMDegrain relies on MVTools you can optimize its arguments using motion compensation (MAnalyze + MCompensate). If the motion compensation is high quality then SMDegrain should benefit from that also.

Does anyone know whether MDeGrain uses pixel-based motion compensation (like MFlowInter) or the block based (like MCompensate)? I think it's the latter but I'm not sure.

Another open question: if you find good parameters for MCompensate (or MFlowInter) do they work equally well with MDeGrain? Even if they're not similar there's the MAnalyze (and MRecalculate) part which should work in identical way for all use cases.

[ChaosKing]

The way I would do it: take a single frame (or even a small part of the frame) where the halo effect is clearly visible and use Photoshop or whatever to remove it manually (using the clone stamp tool for example). Then use a halo removal filter in AviSynth and optimize its parameters so that it replicates the result. The same parameters should then work well for the whole video.

Ahhh why didn't I think of what! And testing one-two frame(s) should be enough :)

[Dogway]

Yes I dabbled with MCompensate back then, we used it for all kinds of filtering I guess I wanted to keep things simple and fast at that time but I had some beta version with it included. I want to run some tests with avsoptimizer and see where it takes me.
I think it uses MCompensate internally, you can see the vectors with MMask. In my opinion MFlowInter artifacts are less forgivable but I have been long out of the loop so things might have changed.

[Boulder]

Regarding denoising, do you have any plans for the Vapoursynth support that was briefly discussed when the optimization method was published?

[zorr]

This is not a release or anything, I modified it a bit to test divide and integer type data for contrasharpening (the generic "val" data type wasn't suited for the variable).
...
Whenever you have time is ok, I use the mvtools2 from pinterf github.

All right, figured out the problem. In the script you're comparing the denoised clip to the noisy clip when you should compare it to the original (without added noise). It was trying hard to come up with parameters that don't remove any noise. :)

So just add a line to load the original clip:
orig=ffvideosource("source.avi")

and then calculate SSIM between the original and denoised. Remove
global ssim = SSIM_FRAME(source, denoised)
and replace with
global ssim = SSIM_FRAME(orig, denoised)

I'm running a small test with such modifications and seems like it prefers tr=4 (as it should).

[zorr]

Zorr, My version 7zip from April 2018 opens Dogway zip no problem, yours must be reasonably old version.

Still couldn't open the 7zip after the update. Perhaps the package was corrupted during download. The .zip opened without problems.

[zorr]

Regarding denoising, do you have any plans for the Vapoursynth support that was briefly discussed when the optimization method was published?

I haven't looked at Vapoursynth yet. I know almost nothing about it so it's hard to say how much effort it would take. Can you recommend a tutorial / site to get one started with VapourSynth?

[ChaosKing]

The official site (http://www.vapoursynth.com/doc/) and for a quickstart my portable pack https://forum.doom9.org/showthread.php?t=175529 <- there are some bat ecnode and VS editor examples
It's very similar to avisynth.

[Dogway]

yes, that makes more sense. I guess I got confused since adding synthetic grain seems counterintuitive... off to some tests.

[StainlessS]

EDIT:
Can you recommend a tutorial / site

Perhaps a succinct Python3 intro would be easiest way into VS,
here "Learn X in Y Minutes" for Python3:- https://learnxinyminutes.com/docs/python3/
[already linked on Vapoursynth home site]

EDIT: Python 2 is somewhat different to Python 3, so give tutorials on that a wide berth.

[ChaosKing]

Here are some basics:

from vapoursynth import core
import somescript as xy


clip = core.d2v.Source(r'E:\test.d2v') # r means raw string, so u can copy paste a windows path. Alternatively use (without r) 'E:\\test.d2v'
video = clip

# call a function from our imported script
clip = xy.Filter(clip)
clip = somescript.Filter(clip) #alternative

# call a plugin function. All plugins have a namespace.
clip = core.dfttest.DFTTest(clip, sigma=5)
clip = clip.dfttest.DFTTest(sigma=5) #alternative

#compare
clip = core.std.StackHorizontal([video.text.Text("original"), clip])
clip.set_output()



EDIT
Oh and it would be also very easy to run Avisynth plugins/scripts within VS

clip = core.d2v.Source(r'E:\test.d2v')
clip_distorted = core.avsw.Eval('FineDehalo(darkstr =0)', clips=[clip], clip_names=["last"], slave_log=r"D:\debug.txt", avisynth =r"D:\AVSPLUS_x86\Avisynth.dll")

vmaf.VMAF(clip_reference, clip_distorted) # https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF

[Wolfberry]

From vapoursynth.com/doc/pythonreference.html#get_core (http://www.vapoursynth.com/doc/pythonreference.html#get_core)
core
Gets the singleton Core object. If it is the first time the function is called, the Core will be instantiated with the default options. This is the preferred way to reference the core.

get_core([threads = 0, add_cache = True])
Deprecated, use the core attribute instead.

Get the singleton Core object. If it is the first time the function is called, the Core will be instantiated with the given options. If the Core has already been instantiated, all options are ignored. Setting threads to a value greater than zero overrides the autodetection.
It's not recommended to use get_core as it is deprecated.
Just use vs.core instead of vs.get_core() (also simpler).
Alternatively, you can use "from vapoursynth import core".

[ChaosKing]

Thx, I updated it in my post. One line less now :-)

[Boulder]

I haven't looked at Vapoursynth yet. I know almost nothing about it so it's hard to say how much effort it would take. Can you recommend a tutorial / site to get one started with VapourSynth?

As others have posted, it's quite easy to grasp once you get the basics of Python code structure.

This one is something that could provide a nice enhancement for VS-based optimization compared to SSIM:
http://forum.doom9.org/showthread.php?t=175862.

[Dogway]

Based on some tests I guess prefiltering is a gimmick lol, aside so many filtering conventions sticked to our brains for so long (like blocksize 16 for 1080p). Thanks for the tool it takes some of the voodoo-guessing out of the equation. I think implementing VMAF is worth here, after reading the Netflix VMAF article a few weeks ago it seemed that SSIM did some bad assumptions, in my case contrasharpening was defaulting to 0 or false which in my opinion wasn't making justice to what I observed. That or if your port it I might start using VapourSynth occasionally for VMAF and BM3D, I just don't like how it requires you to write more, and script readability. Also something for x264 like SSIM Plus would be cool if someone knows about a free open source alternative.

[zorr]

Thanks @ChaosKing and @StainlessS, I have now read the Python tutorial, the official VapourSynth site and ChaosKing's FATPACK thread. I found a SSIM plugin for VapourSynth as well.

Is there a plugin for measuring runtime (preferably per frame)?

I suppose Python's standard I/O routines can be used to write stuff to a file during the script evaluation?

[StainlessS]

Maybe Groucho has better code for timing stuff.

But some stuff from RT_Stats below. [EDIT: Use as you will]


AVSValue __cdecl RT_LocalTimeString(AVSValue args, void* user_data, IScriptEnvironment* env) {
const bool file = args[0].AsBool(true);
SYSTEMTIME st = { 0 };
if(file) {
DWORD tick=GetTickCount();
while(GetTickCount()==tick) // Wait until system clock goes TICK (prevent two separate calls returning same time)
Sleep(0);
}
GetLocalTime(&st);
char bf[64];
if(file) {
sprintf(bf,"%4d%02d%02d_%02d%02d%02d_%03d",st.wYear,st.wMonth,st.wDay,st.wHour,st.wMinute,st.wSecond,st.wMilliseconds);
} else {
sprintf(bf,"%4d-%02d-%02d %02d:%02d:%02d.%03d",st.wYear,st.wMonth,st.wDay,st.wHour,st.wMinute,st.wSecond,st.wMilliseconds);
}
return env->SaveString(bf);
}


RT_LocalTimeString(Bool "file"=True)
Returns current local time as a string.
Where digits Y=Year, M=Month, D=Day, H=Hour, M=Minute, S=Second, m=millisecond.
When bool file==False, then string in format "YYYY-MM-DD HH:MM:SS.mmm"
When bool file==True (Default) string is in format "YYYYMMDD_HHMMSS_mmm"
Also when file==True, function first waits until the system tick count is incremented (about every 10ms)
before inquiring system time. This is to prevent 2 consecutive calls returning the same time string.
Perhaps useful for temporary filename generation.



AVSValue __cdecl RT_Timer(AVSValue args, void* user_data, IScriptEnvironment* env) {
return double(clock()) / double(CLOCKS_PER_SEC);
}


double __cdecl RT_TimerHP_Lo(void) {
LARGE_INTEGER liPerfCounter = {0,0};
LARGE_INTEGER liPerfFreq = {0,0};
bool bStat = true;
DWORD_PTR dwpOldMask = SetThreadAffinityMask(GetCurrentThread(), 0x01);
Sleep(0);
if ((QueryPerformanceFrequency(&liPerfFreq) == 0) || (QueryPerformanceCounter(&liPerfCounter) == 0))
bStat = false;
double tim;
if(!bStat) { // High precision NOT available.
static clock_t first_time32=0;
static bool done32=false;
const clock_t t = clock();
if(!done32) {
first_time32 = t;
done32=true;
}
tim = double(t-first_time32) / double(CLOCKS_PER_SEC); // Fallback, low rez timer
} else { // High precision IS available.
static __int64 first_time64=0;
static bool done64 = false;
if(!done64) {
first_time64 = liPerfCounter.QuadPart;
done64=true;
}
tim = (double)(liPerfCounter.QuadPart-first_time64) / (double)liPerfFreq.QuadPart;
}
SetThreadAffinityMask(GetCurrentThread(), dwpOldMask);
Sleep(0);
return tim; // return double for C client
}


AVSValue __cdecl RT_TimerHP(AVSValue args, void* user_data, IScriptEnvironment* env) {
return RT_TimerHP_Lo(); // Implicit type conversion to AVSValue float
}


RT_Timer()
Returns time in seconds since start of process.
Usage:
s=RT_Timer()
Test()
e=RT_Timer()
Str=RT_String("Test Start=%.2f End=%.2f Time=%.2f secs",s,e,e-s)
SubTitle(str)
RT_Debug(str)

***
***
***

RT_TimerHP()
Returns a higher precision time than RT_Timer (If available on your system, otherwise uses same routine as RT_Timer).
Note, where RT_Timer returns time elapsed in seconds since start of process, RT_TimerHP is not defined to return anything
in particular, the only thing that is meaningful is the difference between returns from 2 calls to RT_TimerHP.
DO NOT mix times from RT_Timer with RT_TimerHP, results may be meaningless.
Usage:
s=RT_TimerHP() Test() e=RT_TimerHP()
Str=RT_String("Test Start=%.3f End=%.3f Time=%.6f secs",s,e,e-s)
SubTitle(str)



High precision timer quite often called HPET in bios setup, usually defaults to OFF. [EDIT: Presumably, High Precision Event Timer]

EDIT:

env->AddFunction("RT_LocalTimeString", "[file]b",RT_LocalTimeString, 0);
env->AddFunction("RT_Timer", "",RT_Timer, 0);
env->AddFunction("RT_TimerHP", "",RT_TimerHP, 0);

[Groucho2004]

Maybe Groucho has better code for timing stuff.
Your 'RT_TimerHP_Lo' function looks pretty good in terms of precision and resolution (and somewhat familiar :D).

Edit: Déjà vu...

[StainlessS]

Your 'RT_TimerHP_Lo' function looks pretty good in terms of precision and resolution (and somewhat familiar :D).

Edit: Déjà vu...

Yep, If's I recalls correctly, me robbed some of that from your PM (I was wantin' to keep that a secret :D )

[Groucho2004]

Yep, If's I recalls correctly, me robbed some of that from your PM (I was wantin' to keep that a secret :D )Ah, yes. Alzheimer slowly kicking in.

[zorr]

But some stuff from RT_Stats below. [EDIT: Use as you will]

Thanks, I forgot to mention I need it in a VapourSynth script. :) I guess this VapourSynth discussion is a bit off topic here so I made a new thread (https://forum.doom9.org/showthread.php?t=176002) in the VapourSynth section.

[Dogway]

I wasn't convinced with leaving sharpness at 0, I run some comparisons with VMAF and other algorithms in MSU VQMT and indeed SSIM (and PSNR) are the only ones that favor blurry filters.

[Seedmanc]

So my assumptions during MFlow testing were correct, the metricts favor blur, hence the sharp=0 (and possibly pel=4) bias. Maybe you could try downscaling before calculation as well.


zorr, is there a length limit for the combination of filters and limits for a single parameter?
Whenever I add more params to dependencies than I have here

Clevel = 8/10.0 # optimize Clevel = _n_/10.0 | 0..10 ; min:super_pel2 1 > 2 0 ? super_sharp2 0 > 2 0 ? super_rfilter2 0 > 2 0 ? blockSize2 4 > 2 0 ? searchAlgo2 0 > 2 0 ? searchRange2 1 > 2 0 ? searchRangeFinest2 1 > 2 0 ? divide2 0 > 2 0 ? overlap2 0 > 2 0 ? CdoRecalc true == 2 0 ? truemotion2 true == 2 0 ? max | Clevel
I get "Unknown element badSA2" (I was adding badSAD2 after truemotion2, strange how it's skipping a letter while keeping the digit instead of just cutting the name).
Same here, add CRscaleCSAD2 to
CdoRecalc=true # optimize CdoRecalc=_n_ | false,true ; filter:Clevel 1 > Csmooth 0 > CRsearchAlgo 0 > CRsearchRange 1 > CRdivide 0 > CRoverlap 0 > Cthsad 0 > CRtruemotion true == CRmeander true == x false == or | CdoRecalc
and get "Unknown element CRscaleCSA"

[zorr]

I wasn't convinced with leaving sharpness at 0, I run some comparisons with VMAF and other algorithms in MSU VQMT and indeed SSIM (and PSNR) are the only ones that favor blurry filters.

That's a great find and good to know. This screenshot from the MSU page shows per pixel SSIM values and the resulting image is clearly blurry.

https://i.postimg.cc/Z5hXXbVK/ssim-precise.jpg

Also the SSIM index for the pixel is calculated using a 8x8 neighbourhood (fast method) or using a gaussian weighting function (high quality method). That explains why it's not able to "see" fine details.

I'm running some Vapoursynth tests now and there we also have GMSD (http://www4.comp.polyu.edu.hk/~cslzhang/IQA/GMSD/GMSD.pdf) which gave much better results than SSIM in my denoising test (best SSIM result had ringing artifacts). I haven't tried VMAF yet.

It would be great to have these quality metrics as Avisynth plugins too.

[zorr]

So my assumptions during MFlow testing were correct, the metricts favor blur, hence the sharp=0 (and possibly pel=4) bias. Maybe you could try downscaling before calculation as well.

Yes you were correct. But I think that to counter a blurry quality measurement you should upscale and not downscale.

zorr, is there a length limit for the combination of filters and limits for a single parameter?

There shouldn't be, I will take a look. Just curious though, what is that filter doing? :)

[EDIT] I tried the Clevel = ... line and it worked. But there are some small fixes in the parser code which are not released yet, so maybe only my development version accepts it. You should check that there are no line breaks anywhere, the optimizer doesn't support continuing lines with \ like Avisynth does.

Oh and if I'm interpreting it correctly you're trying to take the maximum of many numbers. That's not going to work with only one max function since it only takes the top two numbers from the stack.

[Seedmanc]

It's just a part of my ever-increasing MFlow script that now does the initial upsample, follows by the MCompensate trick and then tries to add recalculation to it (over 40 params now). However it always chooses not to do the latter part.
Since I have all the params of MCompensate tied to the weight at which I overlay the MCompensated version over the original one (so as not to try combination when the weight is under 0.2) I tried to follow the no-bias rule and tie the weight itself back to them. Same with the doRecalc switch later.

But the fact that max doesn't work with more than 2 arguments is a disappointing discovery, that means I won't be able to do the many-to-one parameter tying. Is the situation the same for the OR operator which I'm using to combine multiple boolean values?

[zub35]

I wasn't convinced with leaving sharpness at 0, I run some comparisons with VMAF and other algorithms in MSU VQMT and indeed SSIM (and PSNR) are the only ones that favor blurry filters.

Try MSU Blurring. It will show not only blur but also sharp.

Using MSU_Blurring can improve SSIM:
(SSIM+SSIM*BLUR)/(SSIM+1)

BLUR - if encode_blur < source_blur then (encode_blur / source_blur)
BLUR - if encode_blur > source_blur then (source_blur / encode_blur)

UPD: Fix calculation of the value, relative to SSIM. The lower the SSIM, the lower the effect of the blur on the final value. Example:
SSIM=0.9 BLUR=0.5 - Result=0.7105
SSIM=0.5 BLUR=0.9 - Result=0.6333
Previously, in both situations it was 0.7

p.s. There is also a B-SSIM metric that fixes SSIM problems for blurry images.
http://images2.imagebam.com/13/9d/d9/b736951096302774.png
Fig.b SSIM = 0.8241, B-SSIM = 0.6434
Fig.c SSIM = 0.7276, B-SSIM = 0.4526

[zorr]

It's just a part of my ever-increasing MFlow script that now does the initial upsample, follows by the MCompensate trick and then tries to add recalculation to it (over 40 params now). However it always chooses not to do the latter part.

It's possible that you've hit the curse of dimensionality. Every new parameter multiplies the number of possible combinations. At some point the search space gets so ridiculously large that no algorithm can find good solutions. Or it could be something else. :)

I tried doing MFlow + MRecalculate at the same time and there MRecalculate brought some extra quality. I haven't tried all three at the same time (and my method of combining MCompensate is different than yours).

I also tried MFlow + MCompensate in phases, I first optimized MFlow only and once I had optimal values I "freezed" them. I then added the MCompensate and only optimized its parameters. That way the number parameters stays lower and the search is easier. It might however result in little less optimal result than what could be possible doing them both at the same time.

I tried to follow the no-bias rule and tie the weight itself back to them.

I'm not actually sure if it's less biased to set some default values to MCompensate when it's not used because then those default values appear more often than others. You could try running with and without "no bias" and see which one gives better results.

But the fact that max doesn't work with more than 2 arguments is a disappointing discovery, that means I won't be able to do the many-to-one parameter tying. Is the situation the same for the OR operator which I'm using to combine multiple boolean values?

Yes it's the same with OR and all the other functions except "?" which takes three arguments. That's the way this reverse polish notation works. But it's not a limitation of reverse polish notation, it can do anything the infix form can and do it without parenthesis (which is the reason it's used, it's so easy to parse).

So if you need the max of say, three values, you can do it this way:
max(a,b,c) == max(a, max(b,c))

In rpn it's even simpler: a b c max max. So you just add (n-1) max functions to max n numbers. The same works with OR as well.

[zorr]

Try MSU Blurring. It will show not only blur but also sharp.

Using MSU_Blurring can improve SSIM:
(SSIM + MSU_Blurring*) / 2
* if encode_blur < source_blur then [encode_blur / source_blur]
* if encode_blur > source_blur then [source_blur / encode_blur]

That's an interesting approach, do you have link for more detailed explanation?
You could also calculate the blurriness and optimize for both SSIM and blurriness. The pareto front will then have results with increasing SSIM and blurriness and you can choose which one looks best.

Of course we'd need a blurriness detection in Avisynth to do that. I found a promising method which is simple to implement and should give good results: variance of the laplacian (https://www.pyimagesearch.com/2015/09/07/blur-detection-with-opencv/). The laplacian filter is easy to calculate with MaskTools and StainlessS's RT_YPlaneStdev can be used to calculate the variance.

p.s. There is also a B-SSIM metric that fixes SSIM problems for blurry images.

I took a look at B-SSIM (https://www.researchgate.net/publication/265164680_B-SSIM_Structural_Similarity_Index_for_Blurred_Videos) and looks like it's using almost the same method as the "variance of the laplacian", it just replaces laplacian with sobel and takes the standard deviation instead of variance. The SSIM value is then multiplied by (2*SIf*SIh)/(SIf^2 + SIh^2) where SIf is the stddev(sobel) of original frame and SIh is the same for the modified frame. Looks like that can also be implemented in a simple script.

[EDIT] Just remembered that SSIM value can be negative, the valid range is -1 .. 1. So by multiplying a really bad SSIM with < 1 you can actually make the metric better.

[zorr]

B-SSIM seemed too intriguing to pass so I went ahead and implemented it. Or at least a variation of it. The original uses the Sobel filter for edge detection, I'm using the Scharr operator (https://en.wikipedia.org/wiki/Sobel_operator#Alternative_operators) instead because it's more accurate. And by the way the "sobel" in MaskTools is not actually sobel, it detects edges but the mask is incorrect and gives sub-par results.

Other difference is that I'm not taking the maximum of standard deviations from the whole clip sample, I just take one from each frame. Taking the maximum would need a two pass algorithm and I wanted to avoid that. The difference to the original shouldn't be that large.

In my tests this seems to work as it should giving lower scores to blurrier frames.

Anyway, here's a sample code to compare a blurred clip to the original, showing both SSIM and B-SSIM(mod).

# source clip
AVISource("d:\process2\1 deinterlaced.avi")

ConvertToYV16() # for MaskTools
orig = last

blurred = orig.GaussianBlur(1.5)

scharr_orig = scharr(orig)
scharr_blurred = scharr(blurred)

final = ScriptClip(last, """
global ssim = SSIM_FRAME(orig, blurred)
bssim = BSSIM_MOD(scharr_orig, scharr_blurred, ssim)
return orig.SubTitle("SSIM "+String(ssim)+" BSSIM "+String(bssim))
""")

return final

function scharr(clip c) {
scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), u=1, v=1)
return scharr
}

function BSSIM_MOD(clip scharr_orig, clip scharr_alt, float ssim) {
stddev_orig = RT_YPlaneStdev(scharr_orig)
stddev_alt = RT_YPlaneStdev(scharr_alt)

mul = (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)
return mul*ssim
}


The implementation needs StainlessS's RT_Stats plugin (https://forum.doom9.org/showthread.php?t=165479) for the standard deviation.

Seedmanc and Dogway, can you try the B-SSIM variation and see if it fixes the blurring issue?

[ChaosKing]

An additional "toy" for comparing video qualitiy: https://github.com/fdar0536/vapoursynth-butteraugli

[Seedmanc]

I don't think I understand how to use the new method. I tried incorporating it into my script and thing went wrong I think, the pareto graphs look weird (http://puu.sh/CG9K0/35bddd7660.png) and the "best" results now are terribly slow and very different from anything I've gotten before.
Where should I get a x64 version of GaussianBlur? How do you determine the radius that should be used there?

Here's the script https://pastebin.com/c65dhju2, I launch it like F:\avsoptim\optimizer.bat F:\avsoptim\genBC.avs -iters 7h -runs 2 -pop 24

[StainlessS]

Where should I get a x64 version of GaussianBlur?
I think its in the VariableBlur whatsit:- http://avisynth.nl/index.php/AviSynth%2B_x64_plugins

[Reel.Deel]

I think its in the VariableBlur whatsit:- http://avisynth.nl/index.php/AviSynth%2B_x64_plugins

Actually at the moment is not on there, the download link points to the 32-bit version. I did add it on there but someone edited a few things and changed the link :mad:

Also I don't recall a 64-bit version of the latest VariableBlur (v0.7), only v.05 is available:


Binary: http://www.mediafire.com/download/0z0hl43za6bwlb4/VariableBlur05_x64.7z

Source: http://www.mediafire.com/download/hzwtctzyu5vw9vc/variableblur05_x64src.7z

[StainlessS]

Hi RD, glad that you are still lurking about :)

Just checked, I got v0.5 only on x64.

OK, had a bit more of a look, here is v0.7 (x86 only) with source, but its got lots of asm.
http://www.mediafire.com/file/1ugcb886dxq3tdo/variableblur_0.7.zip/file

[zorr]

I don't think I understand how to use the new method. I tried incorporating it into my script and thing went wrong I think, the pareto graphs look weird (http://puu.sh/CG9K0/35bddd7660.png) and the "best" results now are terribly slow and very different from anything I've gotten before.

Sorry, I should have made a better example. You don't actually need to blur anything, it was there just to demonstrate how B-SSIM score gets lower if you increase the blurring.

Replace the lines 135-146 with this and it should work.

scharr_orig = scharr(orig_yv12)
scharr_inter = scharr(inter_yv12)
FrameEvaluate(last, """
#global ssim = SSIM_FRAME(scaled_orig, scaled_inter)
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global bssim = BSSIM_MOD(scharr_orig, scharr_inter, ssim)
global ssim_total = ssim_total + (bssim == 1.0 ? 0.0 : bssim)
""")

[zorr]

New release, new name! AvisynthOptimizer is called "Zopti" from now on. Why? 'Cause it supports VapourSynth as well, that's why. Download the version 1.0-beta here (https://drive.google.com/open?id=1ZJ-4ETdp2fFlL-YXZcesEikMi98TTB84).

There's some other improvements as well, here's a complete list:


added support for VapourSynth scripts
Linux support (with VapourSynth)
added zoptilib python module to ease script writing
added support for VMAF output logs
improved heat map visualization:
-updates optimization progress to title bar when using autorefresh
-shows correct visuals also when smallest value is best
-added latest results visualization
-heat maps can be saved to image sequence with the -animation argument
-initial window size scaled to full screen
improved scatter chart visualization:
-improved zooming (-top)
-better legend position when smallest value is best
uses resolve for initial population when using randomized arguments fails 100 times
more error handling options when script execution fails: new argument -errors with options:
-stop (default, stops execution)
-ignore (continues optimization)
-log (continues optimization and logs errors into a separate error log)
checks that input arguments are all valid for the chosen -mode
improved error handling and error reporting in reverse polish notation parser
new option -priority for setting the optimizer process priority (on Windows only), options are:
-"": same as /NORMAL, default
-lowest: same as /LOW
-lower: same as /BELOWNORMAL
-higher: same as /ABOVENORMAL
-highest: same as /HIGH
removed bias from the random number generation


A few notes about upgrading: I renamed the optimizer.ini to zopti.ini and the optimizer.bat to zopti.bat. If you already have the .ini you can just rename it to zopti.ini and it will work.

I have used the -priority option to set the thread priority lower because when VapourSynth/MVTools is running a heavy task the other programs can become sluggish with the default priority. Priority option is Windows only for now, Linux probably doesn't even need it.

The error handling options are useful when running the VapourSynth MVTools which is currently not as robust as Pinterf's version. With -errors ignore or -errors log you can keep running the optimization even when the plugin crashes. I still recommend running with the default error handling option (which stops the execution) unless you are expecting errors.

I will make another thread to the VapourSynth section to tell more about the VapourSynth-specific stuff.

[Seedmanc]

Is this version abandoned and will any development only happen for the VS fork? I don't feel like learning a whole new language just to continue processing video as I did before just fine with AVS.

For some reason I was unable to find this thread in the thread list for Avisynth Development section, it said "Displaying 12 of 12 threads". Now that I bumped it, it's displayed there and says "13 of 13". How do I see all threads at once?

[wonkey_monkey]

In your forum user Control Panel, go to Edit Options, then find and change "Default Thread Age Cut Off."

[zorr]

Is this version abandoned and will any development only happen for the VS fork? I don't feel like learning a whole new language just to continue processing video as I did before just fine with AVS.

No, the same version works for both AviSynth and VapourSynth. So most of the improvements will affect both versions (new algorithms etc). The only VapourSynth-specific thing is the Zoptilib helper library which is not really needed. I wish I was able to write a similar helper for AviSynth but it would take a guru like StainlessS to accomplish that. :)

I haven't made much progress lately, so it's quiet on both fronts.

[StainlessS]

Oh do stop it. https://www.cosgan.de/images/smilie/konfus/s030.gif

[zorr]

Perhaps the only way to make it faster and threadsafe in Avisynth would be to use an array as you suggested.

After some tests I've found that Avisynth doesn't have array variables, only array constants. You cannot store values into an array after it has been created so currently arrays are not an option.

Or is it possible to make all the frames create their own log file and combine them when it's time to calculate the total score?

It would be easy to create a separate log file for each frame but that wouldn't make anything faster.

I tried some other ways. You can create a new variable for each frame (using Eval) and store the metric there. It doesn't solve the synchronization problem though, there are sometimes missing variables when it is time to combine the results. So that method only works without Prefetch. It would be useful for Avisynth to have a callback function which would be triggered when all the frames have been processed.

I also thought that maybe there's no need for any extra variables at all, we can simply read the metric from all the frames at the last frame using the offset parameter of propGetFloat(). The good news is that it actually works and is faster than the default method which writes to a file at each frame. The bad news is that the speedup is simply due to writing to the file only once (saves about 1,6ms per frame) and there is no additional speedup when using Prefetch. Here's the code:

TEST_FRAMES = 50 # how many frames are tested
MIDDLE_FRAME = 100 # middle frame number

RawSourcePlus("D:/optimizer/test/flower/flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)
#return noisy

#denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

sigma = 400/100.0 # optimize sigma = _n_/100.0 | 100..800 | sigma
bt = 4 # optimize bt = _n_ | -1..5 | blockTemporal
blockSize = 32 # optimize blockSize = _n_ | 2..64 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 0..32 ; max:blockSize 2 / | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

last = GMSD(source, denoised, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

Prefetch(4)

return last


I should also note that the previous GMSD version is not deterministic when used with Prefetch, it's rare but sometimes the last frame is not finished last and one of the frames is left out of the total sum. This version doesn't have that problem.

By the way this is a bit off-topic on this Vapoursynth forum. If any mods happen to read this it might be a good idea to move the posts starting from Worby's (https://forum.doom9.org/showthread.php?p=1938947#post1938947) to the Avisynth Zopti thread (https://forum.doom9.org/showthread.php?t=175723).

[Boulder]

I'm unable to get the new version to work with this test script. It processes the same step for minutes and if you take a look at Task Manager, the I/O read bytes shows that the source file has been read multiple times. Tested with and without Prefetch at the end.

SetCacheMode(0)
orig = FFVideoSource("c:\zopti\lotr_fotr.avi")

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

downscaled_width = 1920
downscaled_height = 808

alternate = BicubicResize(orig, downscaled_width, downscaled_height, b=b, c=c).Lanczos4Resize(orig.width(),orig.height())

GMSD(alternate, orig, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

#Prefetch(threads=24, frames=1)

return last

I used the "Report thread" button to notify that some posts should be moved to the Avisynth section.

[zorr]

I'm unable to get the new version to work with this test script.

I tested it using your silverado.avi clip from the other thread as the source. It does work on my setup, albeit slowly (about 170 seconds per script). Are you using the Avisynth+ 3.7 branch?

[Boulder]

I tested it using your silverado.avi clip from the other thread as the source. It does work on my setup, albeit slowly (about 170 seconds per script). Are you using the Avisynth+ 3.7 branch?

I did this test with a 4K clip of 400 frames. The "old" GMSD script processed one step of it in about 39 seconds while the new one was still processing after waiting for over 5 minutes. I'll try testing the Silverado clip later to make sure the frame size is not an issue somehow. EDIT: the new version finished the first step in 1095 seconds :D

I'm currently using the latest Avs+.

I was wondering if AVSTimer is one reason for seemingly serialized processing despite the prefetch value set. The GRunT stuff like WriteFileIf is MT-friendly if I remember correctly.

[Boulder]

I was wondering if AVSTimer is one reason for seemingly serialized processing despite the prefetch value set. The GRunT stuff like WriteFileIf is MT-friendly if I remember correctly.

Hmm..not that either. It must then be the way Avisynth parses things as all that is left is the wait for all frames to finish to get the final result. Strange though that using 24 threads instead of disabling Prefetch doesn't seem to have any effect even when the metric calculation per frame is the slow part.

[zorr]

Hmm..not that either. It must then be the way Avisynth parses things as all that is left is the wait for all frames to finish to get the final result. Strange though that using 24 threads instead of disabling Prefetch doesn't seem to have any effect even when the metric calculation per frame is the slow part.

I also tried without AVSTimer, made no difference. And I couldn't see any improvement when using 2, 4, 8, 16 or 24 threads.

Then I took out the scaling and only left the GMSD calculation. The time is still the same and the Prefetch behaviour remains. So it's the GMSD (and probably MDSI) which is slow and doesn't get any benefit from multiple threads. I guess it's because the multithreading only accelerates plugins and GMSD is implemented as a script. I tried to include a Prefetch call into the GMSD.avsi but it doesn't allow it there. :(

[StainlessS]

I was wondering if AVSTimer is one reason for seemingly serialized processing despite the prefetch value set. The GRunT stuff like WriteFileIf is MT-friendly if I remember correctly.

Just a wild guess, but would not WriteFileIf [if done non serialized] write any log file in randon-ish order. [maybe same for avsTimer].
Something like AvsMeter may be able to look at it from 'outside' and on total throughput of frames delivered, and not necessarily in any kind of order.
[although there would also have to be some kind of serialzation when played at output - which may/will affect AvsMeter too]

[kedautinh12]

VariableBlur 0.7 (x86 & x64)
https://drive.google.com/file/d/1tiC6bzQ40JOiNkarWYDWcRmSzS-NRqX7/view?usp=sharing

[Boulder]

I also tried without AVSTimer, made no difference. And I couldn't see any improvement when using 2, 4, 8, 16 or 24 threads.

Then I took out the scaling and only left the GMSD calculation. The time is still the same and the Prefetch behaviour remains. So it's the GMSD (and probably MDSI) which is slow and doesn't get any benefit from multiple threads. I guess it's because the multithreading only accelerates plugins and GMSD is implemented as a script. I tried to include a Prefetch call into the GMSD.avsi but it doesn't allow it there. :(

Multithreading definitely works with script functions as well, my normal method of processing contains both denoising and resizing from functions outside the script itself.

Both GMSD and MDSI use ScriptClip to put things in the frame properties, I don't see any other special thing in those functions. They're mostly Masktools stuff which runs multithreaded just fine.

[Boulder]

It's definitely something in those metrics functions, just posted to the Avisynth thread. I can only think the use of ScriptClip doing something which trips the multithreading.

[zorr]

A new major release is here! Zopti now supports executing multiple scripts simultaneously which can lead to very nice performance improvement. This can be very useful when the script itself is not able to utilize all of your cores.

Here's the full release info:


added support for multithreaded execution of scripts with argument -threads
-default value of -threads is 1 meaning one script is executed at a time just like before
-all optimization algorithms (nsga-ii, spea2, mutation, exhaustive) support the the -threads option
-due to the nature of the heuristic algorithms (nsga-ii, spea2, mutation) there can be at most P threads running the scripts at the same time, where P is the population size
-exhaustive algorithm has no thread limits
-validate mode also supports -threads. there the maximum number of threads is the number of validated results (size of the pareto front)
-note: using more than one thread will make time measurements less accurate but will not change the quality measurements
avsr version update (v0.2.2)
added jMetal.log.ini to disable multithreading related JMetal logging
updated Zoptilib to version 1.0.9m
bugfix: evaluate could not parse log files when Zopti was used


Download link is at the first post. I will shortly post some data on the multithreading performance.

[zorr]

I tested the new -threads argument using the denoising tutorial script, this time using GMSD as the similarity metric and using 50 frames instead of 5. Here's the script:


SetCacheMode(0)

TEST_FRAMES = 50 # how many frames are tested
MIDDLE_FRAME = 100 # middle frame number

RawSourcePlus("D:/optimizer/test/flower/flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)

#denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

sigma = 400/100.0 # optimize sigma = _n_/100.0 | 400..600 ; filter:x 5 % 0 == | sigma
bt = 5 # optimize bt = _n_ | 3,5 | blockTemporal
blockSize = 14 # optimize blockSize = _n_ | 6,12 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 4..6 ; max:blockSize 2 / | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

last = GMSD(source, denoised, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

return last


There's also a VapourSynth version of the script which I posted to the VapourSynth Zopti thread (https://forum.doom9.org/showthread.php?p=1940864#post1940864).

The script only has 246 valid parameter combinations to test so we can try all of them using
zopti denoise_ex.avs -alg exhaustive -threads 1

It takes 179 seconds to test all of the combinations using one thread. Let's see the performance when using more threads:

https://i.postimg.cc/QdZrh7St/zopti-performance-scaling.png

The best result is about 20 seconds using 16 threads (on a Ryzen 3900X which has 12 cores / 24 threads).

Another way to look at the scaling is to calculate how much faster we get the results when using -threads:

https://i.postimg.cc/RFDj8JNv/zopti-performance-factor.png

Using 16 threads is 8,6 times as fast as using only one thread. Of course this is just one data point and I don't mean to imply that you can always get such a performance improvement. I would have liked to include more tests using more real-world usage scenarios (HD source etc) but my stock cooled processor cannot handle those for more than a few seconds, it becomes so hot that my PC shuts down. :eek:

Comparing AviSynth and VapourSynth it looks like VS has the edge when using one or just a few threads, but the differences almost vanish when using 12 or more threads. The VS is also able to utilize more than 16 cores while with AVS the performance starts to degrade. The point where more threads are just slowing down is probably dependent on the specific script, I will have to run more tests on that.

[Boulder]

Thanks zorr, looks like a very handy approach. I'll try testing it over the weekend.

[tormento]

Let's say I want to do compression/quality tests on 1-3% of a whole movie, changing the parameters on SMDegrain in a fixed range, mostly tr and thSAD.

Is there a way to batch that process? Usually I have to manually set a frame range and have multiple avs scripts.

Can you help me?

My usual script, with the parts I want to have permutations in red:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource(something)

SMDegrain (tr=4, thSAD=400, refinemotion=false, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Prefetch(6)

[zorr]

Let's say I want to do compression/quality tests on 1-3% of a whole movie, changing the parameters on SMDegrain in a fixed range, mostly tr and thSAD.

Is there a way to batch that process? Usually I have to manually set a frame range and have multiple avs scripts.

How do you evaluate the quality? Do you have a "reference" video you could compare the different settings with? Zopti needs some way to tell the quality and usually that means comparing two videos.

If you use your own eyes to measure the quality and just want to generate the different versions... well, Zopti doesn't output any video so it can't help you there. But it could generate you the different script variations if you provide a file with the parameter combinations you want, like this
1 tr=4 thSAD=40
1 tr=5 thSAD=40
1 tr=4 thSAD=50
etc.

[tormento]

How do you evaluate the quality?
I don't need automatic quality comparison, just a way to automate the changes in the script.

[zorr]

Time for a new update! This version has some major additions so I bumped the version to 1.2.0.


new option: -timeout (default is disabled). if the execution of Avisynth script takes longer than timeout seconds
-the script execution process (and subprocesses) will be terminated
-the result is set to worst possible as a penalty so it will not be considered as a valid candidate
-useful with scripts where certain parameter combinations can result in execution times several hundred times slower than usual
-example: zopti script.avs -pop 24 -iters 1000 -timeout 60
new option: -continue
-aborted optimization can be continued from the last complete generation (one generation is N iterations, where N is the population size)
-give the log file name of the aborted optimization as the value of continue
-old results up to last complete generation are copied to a new log file and optimization will continue from there
-useful if optimization was unexpectedly aborted or the CPU resources are needed for something else for a while
-example: zopti script.avs -pop 24 -iters 10000 -continue "mvtools spea2 run-01.log"
-NOTE: it is possible to change the optimization parameters (population, mutation count etc.) from those used in the continued log
-NOTE: random seed is not restored so the results will not be identical to what they would have been if execution had not been aborted
-NOTE: only works with algorithm spea2 (for now)
new visualization mode: history
-displays the best found value and all tried values of certain optimized parameter (given with -param)
-invalid results (script execution has failed OR timed out) are highlighted in orange
-also shows the phase of dynamic iteration (if applicable)
-accepts parameter -range to limit the displayed history to certain iteration range
-example: zopti -mode evaluate -vismode history -param lambda (displays all history of parameter "lambda")
-example: zopti -mode evaluate -vismode history -param lambda -range 1000 5000 (displays history from iteration 1000 to iteration 5000)
-example: zopti -mode evaluate -vismode history -param lambda -range -10000 (displays history of the last 10000 iterations)
argument -dynphases can now be zero (previously minimum was 1)
pareto front and total runtime is displayed also after mutation or exhaustive algorithm is finished
if population is based on a log file and the pareto front is smaller than the population size, the population will be filled with mutations of the pareto front instead of completely random combinations
if a log file for the optimization already exists it will be cleared in the beginning of the optimization
XChart updated to version 3.8.0 with some customization


And here's what -vismode history looks like:

https://i.postimg.cc/nLjnb7Y8/vismode-history.png

The black thick line shows the best found value of the chosen parameter. The red diamonds indicate a point where a new best result has been found and this parameter's value was changed. The white diamonds indicate a new best result but this parameter's value remained the same.

In the example optimization was finished and then restarted with different settings at around 190 000 iterations.

The -timeout was needed when I started running MVTools2 tests with all the parameters and it turned out that some combinations took an enormous amount of time to finish (typical time was 5 seconds but some combinations took over 30 minutes...). Using a timeout of 60 seconds solved that problem.

-continue is useful if you want to keep optimizing a result that is finished, you can try different settings. Also very handy when the optimization has been aborted by hardware failure or something like Windows update... you can just restart it using the same settings. This also makes it possible to run huge iteration counts little by little even if you need/want to use the computer for other things once in a while. :D

Download link updated at first post.

[Dogway]

Hey thanks a lot for the tool. I plan to use it now heavily, is it possible to combine it with VMAF? I tweaked your GMSD client script but VMAF doesn't seem to write frameProps, not at least as _PlaneVMAF.
By the way, if you need to combine/merge arrays (didn't read what for) I made this function (you need to explicitly declare the args in the call):

function ArrayAdd( val_array "a", val_array "b") {

as = ArraySize(a)-1
bs = ArraySize(b)-1
na = ""
for (i = 0, as+bs+1, 1) {

o = i - as - 1
cm = i != as+bs+1 ? "," : ""
na = na + ( i > as ? String(Eval(Format("b[{o}]"))) : \
String(Eval(Format("a[{i}]")))) + cm
}

return Eval("["+na+"]") }

[zorr]

Hey thanks a lot for the tool. I plan to use it now heavily, is it possible to combine it with VMAF? I tweaked your GMSD client script but VMAF doesn't seem to write frameProps, not at least as _PlaneVMAF.

VMAF writes its own result file (xml) and Zopti can read it directly so there's no need to read any frame props and save them to file. I haven't actually tested the Avisynth VMAF version but it should work if you set the log_path same as the Zopti file= parameter and log_fmt=0. Also set pool=0 to be consistent with the pooling method of other similarity metrics.

But before you start using it you might want to check these tests (https://forum.doom9.org/showthread.php?p=1867043#post1867043) I conducted (there's more tests in that thread) about the different similarity metrics, seems like VMAF is not very consistent and sometimes gives downright bad results.

By the way, if you need to combine/merge arrays (didn't read what for) I made this function (you need to explicitly declare the args in the call):

Thanks, might be useful some day. Arrays were considered as a solution to synchronization problem when using Prefetch() and the script tries to write the results of each frame to the same file. Reading the frame props in the end solved that problem, but I think it's not the optimal way (I think sometimes the frame props are no longer in cache when they are requested so some frames could be calculated twice).

[Dogway]

zorr, I'm trying to come up with a self resolving BBSIM function, I have troubles with SSIM_FRAME(), not only it requires 8-bit but it's output value isn't affected by the filtered clip. I tried to implement reading _PlaneSSIM property if created with vsSSIM but its output value is very different from SSIM_FRAME().

This is my attempt at normalizing PlaneSSIM to SSIM_FRAME (by /335 and then 2*bssim).
function BSSIM(clip source, clip filtered, bool "PlaneSSIM", bool "show", bool "fulls") {

show = Default(show, false)
PS = Default(PlaneSSIM, false)
fs = Default(fulls, false)

blurred = source.ex_blur(1.5*2,bifit=true,UV=1)

source8 = PS ? source : source.ConvertBits(8, dither=-1, fulls=fs)
blurred8 = PS ? blurred : blurred.ConvertBits(8, dither=-1, fulls=fs)

scharr_orig = ex_edge(source8, "pscharr",0,255)
scharr_blurred = ex_edge(blurred8, "pscharr",0,255)

source = PS ? source.ConvertBits(32, fulls=fs) : source8
blurred = PS ? blurred.ConvertBits(32, fulls=fs) : blurred8
scharr_orig = PS ? scharr_orig.ConvertBits(32, fulls=fs) : scharr_orig
scharr_blurred = PS ? scharr_blurred.ConvertBits(32, fulls=fs) : scharr_blurred

ScriptClip(source, function [source, blurred, scharr_orig, scharr_blurred, PS, show] () {

SSIM = PS ? propGetFloat(source,"_PlaneSSIM")/335 : \
SSIM_FRAME(source, blurred)
stddev_orig = RT_YPlaneStdev(scharr_orig)
stddev_alt = RT_YPlaneStdev(scharr_blurred)

bssim = SSIM * (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)
bssim = PS ? 2*bssim : bssim

propSet("_PlaneBSSIM", bssim, 0)

return show ? Subtitle("PlaneBSSIM: " + String(bssim)) : last } ) }

[zorr]

I have troubles with SSIM_FRAME(), not only it requires 8-bit but it's output value isn't affected by the filtered clip.

If you mean the input argument clip filtered then that's not a mystery, it's not used in the function at all. :)

I tried to implement reading _PlaneSSIM property if created with vsSSIM but its output value is very different from SSIM_FRAME().

There are some differences in SSIM implementations. I don't think it's that important to try to match the different implementations. The important aspect of B-SSIM is just the multiplier which scales the result based on how blurry the clip is. Although there might be a small problem with the presented algorithm as SSIM (at least in some implementations) has a range of -1 to 1 and it would scale the negative results in the wrong direction.

[Dogway]

haha, yes didn't notice. I thought 'blurred' was a preprocessing.
Well, I think this will do it. (EDIT: I need to fix vsSSIM as I borked it when optimizing it)

function BSSIM(clip source, clip blurred, bool "PlaneSSIM", bool "show", bool "fulls") {

show = Default(show, false)
PS = Default(PlaneSSIM, false) # Set to True if the clip has the _PlaneSSIM frame property
fs = Default(fulls, false)

source8 = source.ConvertBits(8, dither=-1, fulls=fs)
blurred8 = blurred.ConvertBits(8, dither=-1, fulls=fs)

scharr_orig = ex_edge(source8, "pscharr",0,255)
scharr_blurred = ex_edge(blurred8, "pscharr",0,255)

ScriptClip(source, function [source, source8, blurred8, scharr_orig, scharr_blurred, PS, show] () {

SSIM = PS ? propGetFloat(source,"_PlaneSSIM") : \
SSIM_FRAME(source8, blurred8) # only works on 8-bit
stddev_orig = RT_YPlaneStdev(scharr_orig) # only works on 8-bit (output is also 8-bit)
stddev_alt = RT_YPlaneStdev(scharr_blurred) # only works on 8-bit (output is also 8-bit)

bssim = SSIM * (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)

propSet("_PlaneBSSIM", bssim, 0)

return show ? Subtitle("PlaneBSSIM: " + String(bssim)) : last } ) }

[Dogway]

I'm running some tests for motion vectors and I have a few questions.
-How do you terminate the process? Normally I 'ctrl+c' on the console but then I have to kill each avsr64.exe process
-What is the best approach to simply run a bunch of random iterations without any refinement? I simply used low iterations with high number of runs, but 'runs' doesn't allow multithreading, and 'timeout' finishes the whole thing. The problem here is that some settings have more weight than others so it can bias the refinement so I want to give equal probability to all of them to draw some conclusions (ie. in Excel), at least for the initial tests.
SET RUNS= 4
SET ALG= mutation
SET POPULATION= 4
SET INITIAL= random
SET ITERS= dyn
SET MUTAMOUNT= 0.5 0.01
SET MUTCOUNT= 60% 1
SET CROSSPROB= 0.1
SET CROSSDIST= 20
SET SENSITIVITY= TRUE
SET DYNPHASES= 1
SET DYNITERS= 8
SET ERRORS= STOP
SET PRIORITY=
SET THREADS= 4
SET TIMEOUT= 410
SET CONTINUE=

java -jar Zopti.jar "..\Zopti_test.avs" ^
-alg %ALG% -initial %INITIAL% -iters %ITERS% -dyniters %DYNITERS% -dynphases %DYNPHASES% -pop %POPULATION% -runs %RUNS% -mutcount 1 -mutamount %MUTAMOUNT% -timeout %TIMEOUT% -threads %THREADS%
I don't quite understand the settings marked in red though. Maybe they are related to what I want to do.

-Also I tried to kill some iterations as soon as some frames are rendered longer than necessary so I played a bit with WriteFileIf() but didn't find what keyword is for timeout. Tested with:
WriteFileIf(resultFile, "avstimer > 4500.0", """ 0.0 9999999"""+Chr(10)+""" "stop " """, append=true)

[zorr]

-How do you terminate the process? Normally I 'ctrl+c' on the console but then I have to kill each avsr64.exe process

Currently you cannot easily terminate the process (and all the subprocesses that were started by Zopti). I'll have to look if I can implement a way to detect the termination and shut down the subprocesses as well.

-What is the best approach to simply run a bunch of random iterations without any refinement? I simply used low iterations with high number of runs, but 'runs' doesn't allow multithreading, and 'timeout' finishes the whole thing.

The best approach (with no bias) is to run with a high population count and the same number of iterations, one run only. That way all the iterations are random and not refined in any way. Also there will be no duplicate parameter combinations, Zopti takes care of that.

The problem here is that some settings have more weight than others so it can bias the refinement so I want to give equal probability to all of them to draw some conclusions (ie. in Excel), at least for the initial tests.

I see what you mean. But it's also difficult to make any conclusions in that early phase of the optimization, unless you only have a couple of parameters you want to vary. With one or two parameters your best bet is to run exhaustive search, you can then make nice visualizations from those. Zopti has a heatmap visualization for two parameters, for example. And exhaustive search can also be "sparse" in a way that you test for example only every 10th value.

SET RUNS= 4
SET ALG= mutation
SET POPULATION= 4
SET INITIAL= random
SET ITERS= dyn
SET MUTAMOUNT= 0.5 0.01
SET MUTCOUNT= 60% 1
SET CROSSPROB= 0.1
SET CROSSDIST= 20
SET SENSITIVITY= TRUE
SET DYNPHASES= 1
SET DYNITERS= 8
SET ERRORS= STOP
SET PRIORITY=
SET THREADS= 4
SET TIMEOUT= 410
SET CONTINUE=
...

I don't quite understand the settings marked in red though. Maybe they are related to what I want to do.

This (https://forum.doom9.org/showthread.php?p=1853726#post1853726) is the official "reference post" of the optimizer arguments. It doesn't have the later additions though but covers the ones you needed clarification about.

-Also I tried to kill some iterations as soon as some frames are rendered longer than necessary so I played a bit with WriteFileIf() but didn't find what keyword is for timeout. Tested with:
WriteFileIf(resultFile, "avstimer > 4500.0", """ 0.0 9999999"""+Chr(10)+""" "stop " """, append=true)

If you're playing with MVTools there are some parameter combinations that can take a very, very long time before they even return the first frame. The -timeout argument was implemented just for that purpose, to stop the execution of the AviSynth script if it takes longer than the specified time. The termination is immediate as it simply kills the whole process and can start executing another one instead.

[zorr]

I implemented a shutdown hook to kill all the running subprocesses if Zopti is terminated (thanks for the idea Dogway). I had some other changes in the pipeline as well, so here's a new release:

new option: retry
-tries to run the script again this many times if execution fails
-can be useful if the script uses plugins which are not 100% reliable and can crash
-default value is zero (do not retry)
-example: zopti script.avs -pop 24 -iters 10000 -retry 4
new visualization mode: line
-draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
-accepts parameter -range to limit the displayed values to certain range
-black line shows the best values, blue line indicates the number of results per value
-a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
-example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
support for custom output properties written by the script (only in VapourSynth scripts for now)
better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
option -continue now also supported by exhaustive algorithm (when using more than one thread)
order of runs in a visualization is now based on file name, not by modified date
all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
visualization mode seriespareto no longer contains the global pareto line
VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
bugfix: fixed a memory leak in XChart (at least partially)


Here's what -vismode line looks like:

https://i.postimg.cc/mDb5mW9t/lambda.png

Download link updated at first post.

[Dogway]

Thanks for the update!!

I got some issues with sensitivity option. Probably my optimization range in the script wasn't too broad or something and I was getting this error message after DYNITERS iterations using "mutation" as algo, and "dyn" as iterations.

Parameter sensitivity estimation with 256 result combinations
java.lang.ArrayIndexOutOfBoundsException: Index -700 out of bounds for length 23
00
at avisynthoptimizer.Parameter.updateAverages(Parameter.java:954)
at avisynthoptimizer.Parameter.updateSensitivityEstimation(Parameter.jav
a:970)
at avisynthoptimizer.SensitivityEstimation.estimateParameterSensitivity(
SensitivityEstimation.java:102)
at avisynthoptimizer.AviSynthOptimizer.optimize_mutation_multithreaded(A
viSynthOptimizer.java:4726)
at avisynthoptimizer.AviSynthOptimizer.main(AviSynthOptimizer.java:620)
ERROR: Index -700 out of bounds for length 2300

C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0>rem ERROR: Series name
>phase 1,00< has already been used. Use unique names for each series!!!


By the way on another note I was trying to evaluate motion vectors as a general case without the intervention of client functions like MFlowFPS, or MDegrain by using the SAD mask in MMask (kind=1) and using the average value as the frame metric. Well this didn't yield good results, it tends to optimize to a large searchRange while for example MDegrain with a searchRange over 2 smears the content. So first I'm locking searchRange to 2 and optimizing for MDegrain with the MDegrain client.
Also most of the hardest settings to optimize are those of truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global... they show bad correlation overall.


Here's my Excel chart for the first randomized data samples. You can easily see that any other than DCT=0 is slow and bad quality, so you can remove those entries and continue guessing settings that can be locked down. Ideally one would run some multivariate analysis but I'm still studying statistics.

Google Spreadsheets (https://docs.google.com/spreadsheets/d/1MNrZe-cjzZGuOucvK7U-LpxKr_yIxoRIUKHep_mbxn0)

[zorr]

I got some issues with sensitivity option. Probably my optimization range in the script wasn't too broad or something and I was getting this error message

That looks intriguing, can I take a look at your script so that I can try to replicate this error?

By the way on another note I was trying to evaluate motion vectors as a general case without the intervention of client functions like MFlowFPS, or MDegrain by using the SAD mask in MMask (kind=1) and using the average value as the frame metric. Well this didn't yield good results

Yeah you have to be careful what you optimize for. If there's a way to cheat a good result Zopti most likely finds a way to do that. So it's always a good idea to only optimize for the final result and not some intermediate metric.

Also most of the hardest settings to optimize are those of truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global... they show bad correlation overall.

Most of the settings are only relevant after the most important ones are found and the search starts to focus.

Note that you don't need truemotion if you set lambda, lsad, pnew, plevel and global in your script as those values will overwrite whatever truemotion sets them to.

Here's my Excel chart for the first randomized data samples. You can easily see that any other than DCT=0 is slow and bad quality

I wouldn't be so sure. :) Maybe you've already seen this (https://forum.doom9.org/showthread.php?p=1942410#post1942410) but I did some pretty extensive tests with MVTools frame interpolation and found that dct=1 was consistently best, it was the winner in three test runs with at least 100000 iterations each. It could very well be that dct=0 is the best for MDegrain and also it may have more to do with the specific video, as even I haven't done tests with multiple videos to find out which parameters are consistently best with most source material (or perhaps none are, or perhaps it depends on the resolution or content type).

In any case 100 tests is not enough to say anything sure. I try to make conclusions on the best value only after Zopti has consistently selected it as the best one, after which it can be locked down and another tests can focus on finding the best values for the remaining unlocked parameters. MVTools has so many parameters that it's a real challenge to optimize them due to enormous search space, that's why it has to be done meticuously and using large iteration counts.

Ideally one would run some multivariate analysis but I'm still studying statistics.

That sounds interesting, let me know if you find a good method to analyze the results.

[Dogway]

The error didn't happen to me in my last edit, I think it was related to the parameter range AND the "filter:" used. It will happen again I guess so I will post the offending script here.

Currently I'm benchmarking MDegrain since MFlowFPS was a bit too complex and as you say DCT=1 was the best. DCT=1 and searchAlgo=3 are the slowest settings in MAnalyse aside of a high searchRange, so it would be difficult to find a pareto front with them. In the case of MDegrain I don't feel they are critical (did some runs) so I exclude them.

Note that you don't need truemotion if you set lambda, lsad, pnew, plevel and global in your script as those values will overwrite whatever truemotion sets them to.
Oh I though it was the other way around, truemotion overriding those settings.

I don't know how you can run so many iterations, I'm testing with 150 frames and as you can see it takes about 2 to 5 minutes per iteration (taking out DCT=1 and search=3). I run 8 scripts at the same time each for a thread on my CPU.
What I did, instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain.

Here's my reference clip (https://www.sendspace.com/file/jn69kp) (a sample (https://www.sendspace.com/file/se2g3g)from @tormento).
setmemorymax(2048)
DGSource("brazil.dgi",cl=0,ct=40,cr=0,cb=40)

ConvertBits(16,fulls=true)

pre=smdegrain(tr=1,mode="temporalsoften",blksize=16,thSAD=900,LFR=400,prefilter=3,DCTFlicker=true,contrasharp=false,refinemotion=true).ex_unsharp(0.5, 1920.0/8, 0.0)
smdegrain(tr=2,mode="MDegrain",blksize=16,prefilter=pre,thSAD=400,LFR=200,contrasharp=true,refinemotion=true)
ConvertBits(8,dither=1)

And this is my Zopti script with refined ranges after about 400 random iterations, and a few locked ones after a week of tests:
setmemorymax(16384/8)
DGSource("brazil-ref.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=last
DGSource("brazil.dgi",cl=0,ct=40,cr=0,cb=40)

AssumeFPS(24000,1001)
RequestLinear(30)

thSAD = 285 # optimize thSAD = _n_ | 230..290 ; filter:x 5 % 0 == | thSAD
thSADC = 200 # optimize thSADC = _n_ | 140..250 ; filter:x 10 % 0 == | thSADC
thSADR = 280 # optimize thSADR = _n_ | 150..300 ; filter:x 10 % 0 == | thSADR
BlkSize = 16
BlkSizeR = 8
overlap = 4
overlapR = 4
pel = 1
sharp = 2 # optimize sharp = _n_ | 2 | sharp
scaleCSAD = 2
trymany = false
truemotion = false # optimize truemotion = _n_ | true,false | truemotion
truemotionR = true # optimize truemotionR = _n_ | false,true | truemotionR
temporal = true # optimize temporal = _n_ | true,false | temporal


# Settings that depend on source noise
# lsad the noisier the lower
# sigma the noisier the higher
# badSAD

# TRUEMOTION SETTINGS
lambda = 440 # optimize lambda = _n_ | 400..900 ; filter:x 10 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 1320 # optimize lambdaR = _n_ | 960..2000 ; min:lambda ; max:lambda 3 * ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114 # optimize pnew = _n_ | 100..200 ; filter:x 2 % 0 == | pnew
pnewR = 136 # optimize pnewR = _n_ | 100..250 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 100 # optimize pzero = _n_ | 28..100 ; filter:x 2 % 0 == | pzero
lsad = 6000 # optimize lsad = _n_ | 1000..8000 ; filter:x 100 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79 # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1 # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 8 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2 # optimize badrange = _n_ | 0..50 ; filter:x 2 % 0 == | badrange
badSAD = 1350 # optimize badSAD = _n_ | 1100..2200 ; filter:x 50 % 0 == | badSAD
dct = 0 # optimize dct = _n_ | 0 | dct
dctre = 9 # optimize dctre = _n_ | 0,2,3,6,7,9 | dctre
rfilter = 3
searchAlgo = 1 # optimize searchAlgo = _n_ | 1 | searchAlgo
searchAlgoR = 4 # optimize searchAlgoR = _n_ | 0,1,2,4,5 | searchAlgoR
searchRange = 15 # optimize searchRange = _n_ | 2..17 | searchRange
searchRangeR = 13 # optimize searchRangeR = _n_ | 2..18 | searchRangeR
searchRangeFinest = 17# optimize searchRangeFinest = _n_ | 9..20 | searchRangeFinest
sglobal = true # optimize sglobal = _n_ | true,false | sglobal

trim(1,12)+\
trim(60,96)+\
trim(173,285-12)

C=ConvertBits(16,fulls=false)
pre=C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true


superfilt = MSuper(pre, hpad=16, vpad=16, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
superR = MSuper(C, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false, chroma=false)
superRe = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
bak2 = MAnalyse(superfilt, isb=true, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 = MAnalyse(superfilt, isb=true, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 = MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 = MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe, bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : bak
bak1 = Recalculate ? MRecalculate(superRe, bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : bak
fwd1 = Recalculate ? MRecalculate(superRe, fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR , thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe, fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR , thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=true)

global frame_count = FrameCount()

GMSD(src.ConvertBits(32, fulls=false, fulld=true), ConvertBits(32, fulls=false, fulld=true), show=0)

global total = 0.0
global GMSD = 0.0
global GMSD_total = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
global GMSD_total = GMSD_total + GMSD
""",local=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (GMSD, time)
global delimiter = "; "
resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "GMSD_total", append=true)

#Prefetch(4)
return last

My last benchmark will be to test with a different clip and check what parameters depend on grain/thSAD, and what parameters are independent and coincide.
In any case IMO degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc. They divide the film in scenes and collect metrics for each one (they even encode by scenes). I tried without success to modify StainlessS' ScSelect_HBD() to know beforehand the last and next scene changes so filters can use accumulated per-scene metrics.

[Boulder]

It's often mentioned that dct=5 would be the most optimal choice, so it would be interesting to see how it fares. Also a bigger value for pel seems to change things a lot, at least when viewing things with MShow.

[Dogway]

That sounds interesting, let me know if you find a good method to analyze the results.

In lack of MANOVA we can try Excel Data Analysis tool, my Excel version is too old though. Will try to do in python or something.
A correlation matrix builds a variable x variable sized matrix showing the correlations between them. Simply go to Data -> Data Analysis -> Correlation. I don't think Google Spreadsheets has analysis tools.

@Boulder., in my tests results differ a bit whether I'm optimizing for MFlowFPS or MDegrain. For MFlowFPS DCT=1 and pel=2 looks like a no brainer even for 1080p. For MDegrain I found that DCT=0 and DCTR=9 is more optimal, and pel=2 doesn't have a big impact. DCT=5 if any would be more useful for RefineMotion.

EDIT: Correlation matrix for 210 random iterations (also in Google Spreadsheets (https://docs.google.com/spreadsheets/d/1pJwM6opK6BX7GKT2qzbVqrUyHL1-fZEuUQ_3GTrJkes/edit?usp=sharing)). This (correlation calculation) can work to tune the Sensitivity internals.
https://i.imgur.com/wOECyat.png

[DTL]

" truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global."

I can recommend to look also at scaleCSAD adjustment. It work only if chroma=true. It is too coarse now in 2.7.45 like +-2 integer including 0 (only 5 valid positions total) but may significantly change out MPEG encoder speed (like greatly influence SAD values). It looks even not 'auto-normalizing' after changing scaleCSAD value so you mostly probably need to re-adjust thSAD in MDegrain (and may be other thresholds).
Depending on relative amount of luma/chroma noise and its spectrum it looks may give visibly different results at different footages.

I even thinking of making it float param because difference between 0 and -1 may be too large. -1 may give less out MPEG speed but looks like smooth many. Though float calculation will visibly lower the speed on CPU-processing because it is frequent enough calculation.

"degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc."

In best case - yes. Because each scene may use different film ISO or camera gain and have different noise levels. Current approach is to set more tr and more thSAD cause slower processing and also may degrade more details on low noise scenes.

[Dogway]

Currently it's locked at a value of 2, this has shown increased metrics in most tests unless the clip is crap of course. It might be also useful to check its correlation to thSAD.

The problem with chroma motion vectors is that you might not want them to impact SAD much. I once did a test with normalized chroma planes (saturating bitdepth range) and yes it protected more and thus it denoised less in areas where to the eye is more noticeable, specially for anime. So what is this? YUV is YCbCr, Cb plane is blue opposing while Cr is red opposing. Human visual system is much less sensitive to the blue color than to red, so I wouldn't want to give much importance to the U plane.

I have even thought on adding a hue plane to process motion vectors from. It makes more sense as 2 totally different colors side to side will be very far in values and raise SAD, but this also requires scaling thSADC and other settings as well.

[DTL]

"Currently it's locked at a value of 2"

It is really very great deviation from default 0. It either mean default 0 is wrong or something else. Also 2 is currently max allowed value and not allow any more adjustment if it continue to increase quality (may be 3 is even better ?).

Default 0 also mean it have nominal adjustment range of +-2 around 'best recommended value'. So if new 'best default is 2' (for YV12 typical inputs ?) so it mean for good adjustment range we need 2(+-2) ?

As for documentation:
Fine tune chroma part weight in SAD calculation (since 2.7.18.22)


Possible values for luma:chroma SAD ratio
-2: 4:0.5
-1: 4:1
0: 4:2 (default, same as the native ratio for YV12)
1: 4:4
2: 4:8

So 'practical default 2' mean the chroma SAD is very important and its weight need to be increased more (higher 4:8) ?

Actual internal output SAD calculation from MAnalyse:
outSAD = SAD_Y + ScaleSADChroma(SAD_UV, eff_scale)

Where (from file def.h)

static MV_FORCEINLINE sad_t ScaleSadChroma(sad_t sad, int effective_scale) {
// effective scale: 1 -> div 2
// 2 -> div 4 (YV24 default)
// -2 -> *4
// -1 -> *2
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);
}


and effective_scale is calculated from scaleCSAD as

// New experiment from 2.7.18.22: keep LumaSAD:chromaSAD ratio to 4:2

// luma SAD : chroma SAD
// YV12 4:(1+1) = 4:2 (this 4:2 is the new standard from 2.7.18.22 even for YV24)
// YV16 4:(2+2) = 4:4
// YV24 4:(4+4) = 4:8

// that means that nSCD1 should be normalize not by subsampling but with user's chromaSADscale

// YV12 YV16 YV24
// nLogXRatioUV 1 1 0
// nLogYRatioUV 1 0 0
// effective_chromaSADscales: (shift right chromaSAD)
// chromaSADscale=0 -> 0 1 2 // default. YV12:no change. YV24: chroma SAD is divided by 4 (shift right 2)
// =1 -> -1 0 1 // YV12: shift right -1 (=left 1, =*2) YV24: divide by 2 (shift right 1)
// =2 -> -2 -1 0 // YV12: shift right -2 (=left 2, =*4) YV24: no change
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV));
effective_chromaSADscale -= chromaSADscale; // user parameter to have larger magnitude for chroma SAD
// effective effective_chromaSADscale can be -2..2.
// when chromaSADscale is zero (default), effective_chromaSADscale is 0..2


user input 2 -> eff_scale = -2 (for YV12) and ScaleSADChroma return saduv << 2 that is 4 * saduv, so total output SAD = SAD_Y + 4*SAD_UV. If is not enough so may be need SAD = SAD_Y + (5,6,7, ..?)*SAD_UV ?

Unfortunately current 'bitshift' scale method not allow fine adjustment so if even simply enable value +3 and add <<3 processing it will make only SAD = SAD_Y + 8*SAD_UV next adjustment step that may be too high. So for compatibility reason with old scripts may be only a way is to add new param like 'scaleCSADfine' that is true float multiplier and can be adjusted in good range like from 0.1 to 10.0. With default to 1.0.
So total new formula will be

static MV_FORCEINLINE sad_t ScaleSadChroma(sad_t sad, int effective_scale) {
if (chroma_finetune == 1.0f)
{ // high speed calculation for compatibility with old scripts
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);
}
else
{ // new fine tuning in big range lower speed calculation
sad_t imm_uvsad;
if (effective_scale == 0) imm_uvsad = sad;
else if (effective_scale > 0) imm_uvsad = sad >> effective_scale;
else
imm_uvsad = sad << (-effective_scale);

return (sad_t)(imm_uvsad * chroma_finetune);
}
}


Unfortunately additional internal 'adaptation' of UV sad scale factor to pixel format makes things a bit more complex. Also it looks current docs do not mention it is really internally 'adapted' to input pixel format. So no simple one formula available for all possible input formats. Also it looks even in 2022 the practical 'lumaSAD/chromaSAD' ratio is still in 'experimental' state and still may changes in the very high range: like from +4*SAD_UV and more to +0.25*SAD_UV and may be less (for YV12 input).
Current format of output data from MAnalyse do not allow to output luma and chroma SAD separately - it may help to script writers for more complex processing. So we can only adjust internal processing of luma/chroma SAD values ratio with external adjustment param and with simple linear mixing function. But it not only output SAD but also total SAD using at search (with 'onCPU' mode - only available for 2.7.45) of MVs at each step. So it affect both MVs search and MDegrain client degrain operation.

Addition: I now find a bug with 'reversing direction' of scaleCSAD adjustment in new development version so my 'better denoise' also moves to scaleCSAD > 0. Though with some not very detailed compare it looks it more blur the most of frame and sharp textures. Also it was tested with not internal MAnalyse MVs search engine but with NVIDIA's encoder hardware ME.

[zorr]

The error didn't happen to me in my last edit, I think it was related to the parameter range AND the "filter:" used. It will happen again I guess so I will post the offending script here.

I took a look at your current script, didn't manage to run it yet because I think it needs the latest test version of AviSynth in order to run propNumElements (in ex_MinBlur) outside ScriptClip. But I had some strange errors earlier when I used the same way to write the frame props to a file, there may be rare errors when the info about the last frame is not written as the last line of the file. I implemented another way to write all the props only at the last frame of the clip, you can see an example of that here (https://forum.doom9.org/showthread.php?p=1940863#post1940863).

If you encounter the bug again look for the offending script in the zopti work folder and see if the result file has a wrong order of result lines (the others don't matter but the last line should start with "stop").

Oh I though it was the other way around, truemotion overriding those settings.

That's easy to test, run a script twice and only change the truemotion true/false, you should get the exact GMSD score either way.

I don't know how you can run so many iterations, I'm testing with 150 frames and as you can see it takes about 2 to 5 minutes per iteration (taking out DCT=1 and search=3). I run 8 scripts at the same time each for a thread on my CPU.

I try to make the script runtimes fast. In the thread I mentioned earlier I cropped the video into a smaller one (528x580) where the interesting and challenging motion was located at. I also cut a segment out of it, it was only 29 frames in total. If the script can handle the difficult part, it should be able to handle the rest of the scene as well. The runtimes were from 2 seconds upto around 25 seconds in the later parts of the optimization (it mostly depends on what kind of speed the scripts in the pareto front have). In the beginning of the optimization there can be parameter combinations which run for 30 minutes or longer so I used a timeout of 60 seconds to stop the scripts which take too long.

I have a 12 core / 24 thread CPU and 64GB RAM to be able to run many scripts at the same time. I do some tests to find out how many threads saturates the cores nicely, also important to check that the script are not limited by having too little RAM as that will definitely cause slowdowns.

Oh, and you need patience. :) For large tests I use a dynamic iteration count as that's a good way to guarantee that the optimization process is not limited by the iteration count too much. So sometimes the runs take a week, or more. Sometimes I stop the run in order to use the CPU for something else and use the -continue option to proceed with the same test run. The largest iteration count so far was almost 700k.

What I did, instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain.

That's a good way to do it, that way you can remove real noise and not some synthetic one which may look very different. There are many kinds of noise reduction algorithms and it's because different noise needs a different algorithm (or at least the optimal settings are different). The synthetic noise tests may still be valuable to try to find new and perhaps surprising parameter combinations, those can then be tested with real noise.

In any case IMO degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc. They divide the film in scenes and collect metrics for each one (they even encode by scenes). I tried without success to modify StainlessS' ScSelect_HBD() to know beforehand the last and next scene changes so filters can use accumulated per-scene metrics.

That would indeed be awesome, to drive the filter settings with frame props. Determining those settings from some kind of noise profile could also be optimized with Zopti but someone would have to do the hard work of optimizing all the scenes of the reference clip by hand first.

[DTL]

"instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain."

May be also good to try with real video camera to make 2 shots with low and high gain settings (with disabled in-camera noise reduction) and use low-gain recording as reference. Though with moving picture it is not easy make 2 identical shots for PSNR-like metric to work. But it makes natural noise distribution in both luma and chroma channels. I not sure if many or any 'noise addition' plugins may generate real natural photon-shot noise with it complex enough params (like moving from Poisson distribution at low photons number to 'Natural-Gaussian' with high photons number) + some 'read-noise' of sensors.
Though for static scene it may be much simple like make interval shooting with digital camera from good tripod with RAW files writing.

[zorr]

It's often mentioned that dct=5 would be the most optimal choice, so it would be interesting to see how it fares. Also a bigger value for pel seems to change things a lot, at least when viewing things with MShow.

Before I locked DCT=1 I did three runs with all the choices and DCT=5 was the second best. The comparison is not completely fair though, as you can see the DCT=1 was tested with almost 70000 combinations and DCT=5 with "only" about 35000. That's the nature of the optimization as it tries to focus on the most promising results it has found so far.

There were some DCT=5 results in the pareto front as it was able to beat DCT=1 in speed.

https://i.postimg.cc/xd36gdpF/MFlow-FPS-DCT.png

[zorr]

In lack of MANOVA we can try Excel Data Analysis tool, my Excel version is too old though. Will try to do in python or something.
A correlation matrix builds a variable x variable sized matrix showing the correlations between them. Simply go to Data -> Data Analysis -> Correlation. I don't think Google Spreadsheets has analysis tools.

Thanks for the tip, I have Excel so I can try that. But first we have to stop and think what this correlation matrix is actually telling us. So we're talking about the Pearson correlation coefficient (https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) which is a number between -1 and 1. If the value is -1 or 1 then the two populations can be perfectly described by a linear equation. If the value is zero the populations definitely cannot be described that way. So a large absolute correlation implies that the two variables have a very simple relation with one another. So for example if we calculate correlation between time(ms) and DCT (which in your chart is very large, 0.81) what would mean that by increasing the DCT the script runs slower. DCT correlates negatively with GMSD so by increasing the DCT the GMSD should decrease (and smaller GMSD means better quality in this context). I think that is true to some extent but we cannot just assume that we can get the best quality by using the maximum DCT value. Also since DCT=1 is very slow the relation of slower script and larger DCT does not hold very well either.

What about correlations with two parameters, what do they mean? So for example correlation between blockSize and DCT. Well, that only tells what values you have tested and how randomized they really were. If there was a strong correlation between any two parameters it would mean the population was not random.

The correlations between parameters might be useful if they're only calculated from the results in the pareto front after the optimization is finished. We might see interesting relations on how two change one variable in relation to another in order to stay at the pareto front.

There are of course other ways to visualize and analyze the pareto front, Zopti can create a heatmap of two parameters and display only the values in the pareto front, brightness reflecting the GMSD score for example.

If you really want to get into the relations between two parameters, you should lock down all the parameters except for two and do an exhaustive search and see the heatmap, the result can be very complex like this one below:

https://i.postimg.cc/ZqJ5XfjK/halos-ghosts-judder-ex4-2021-05-09-22-57-38-optimize-exhaustive-run-01-1.png

This (correlation calculation) can work to tune the Sensitivity internals.

The sensitivity estimation is such a dumb idea that it may actually be genious. You take the last N results from the log and do a NxN comparison (ie you compare every result with all the others). You calculate the average change in the primary result (in this case GMSD) and how much the parameter's value was changed. You calculate this for every parameter. The value tells how sensitive the GMSD is to changes in this parameter and the mutation algorithm can utilize this information to decide how much each parameter should be mutated on average. I think because the number of comparisons is so large (for example using 100 last results you get almost 10000 comparisons) the estimated sensitivity is useful even though we're disregarding which other parameters also changed.

I guess the idea could be extended to consider how much (and in which direction!) each parameter should mutate assuming these other parameters are also mutated, but I have to think about that more.

[zorr]

"Currently it's locked at a value of 2"

It is really very great deviation from default 0. It either mean default 0 is wrong or something else. Also 2 is currently max allowed value and not allow any more adjustment if it continue to increase quality (may be 3 is even better ?).


I also noticed that ScaleCSAD=2 is the best option in my MFlowFPS tests. I went further and visualized the best result of every ScaleCSAD value, trymany is also included because it was another head scratcher.

https://i.postimg.cc/4yTrJJZf/allparams-scale-CSAD-trymany.png

So there seems to be a strong correlation with larger ScaleCSAD -> better result.

DTL, since you know how to interpret the MVTools source could you take a look at trymany, what does it do and why trymany=true gives worse results than trymany=false?

My notes about these two parameters from the other thread were


trymany=false is surprising, or perhaps I don't understand what it means. The docs say "try to start searches around many predictors". So it sounds like trymany is doing more work to try to find the best possible block matches but somehow this ruins quality. Perhaps it's the same phenomenon we saw with the large search range (which also destroys quality).
scaleCSAD=2 means the luma:chroma ratio used in SAD calculations is 4:8 so chroma is twice as important as luma. That's a bit of a suprise. In my script the video is converted to YV24 before going into MVTools so chroma already has the same resolution as luma.

[DTL]

"what does it do"

It is long to type - better next day. It is night here and better to try to sleep somehow.

"scaleCSAD=2 means the luma:chroma ratio used in SAD calculations is 4:8 so chroma is twice as important as luma. That's a bit of a suprise. In my script the video is converted to YV24 before going into MVTools so chroma already has the same resolution as luma."

It is shorter to type I hope:

4:8 for YV12 not mean UV is twice more weighted.
Initially at imaging device that is RGB naturally (film or 3chip/3tube camera) we have 3 RGB full-band channels and they all have equal noise level (about equal because red and blue photons may be different in number ?).
When YV12 that is YUV 4:2:0 created from full-band RGB we have different band channels:
Y channel is full-band and 'initial noise level' that create SAD_Y of max absolute value.
U and V channels are half banded (as bt.601 require without anti-gibbs additional high frequency attetuation) that is about half of noise too. If you can read SAD for level 1 (half sized) you will see it is about half smaller in compare with level 0 (full size - output level). Or you can resize input frame 2 time smaller and look at the SAD value in MShow().
Also SAD values are absoluted and each UV block in 4:2:0 is 1/2 linear size and 1/4 of samples count. So if it even have same relative 'noise level' it will produce only 1/4 of SAD addition to full output sum.

So with simple YV12 luma + chroma addition (and equal noise, that is not true).
outSAD = SAD_Y + 1/4 (SAD_Y as SAD_U) + 1/4 (SAD_Y as SAD_V) . Because noise level in 1/4 sized U and V blocks also lower in compare with Y full-band channel - the real proportion is even worse: may be close to
outSAD = SAD_Y + 1/8 (SAD_Y as SAD_U) + 1/8 (SAD_Y as SAD_V).

Now MAnalyse have build-in ScaleSADchroma() function to tweak relative weight of U+V SADs in output sum. With 4:8 you get only x4 multiplier to SAD_U + SAD_V:

outSAD = SAD_Y + (1/4..1/8 (SAD_Y as SAD_U) + 1/4..1/8 (SAD_Y as SAD_V))x4. result is about SAD_Y + (1..1/2)SAD_U + (1..1/2) SAD_V - it is only close to 1:1 proportion of luma to chroma SADs in output sum.
Also it may be not correct to count U and V channels 'equal' to Y channel in this summing - may be it need to be something about 'medium geometric'. So we again have a filed for experiments:

Example of possible 'common SAD' calculation:
outSAD= SAD_Y + Chroma_weight * SQRT(SAD_U^2 + SAD_V^2));

Also the output SAD may be calculated not in YUV colour space but in something like HSV with different weigthing of H and S channels.

But it looks still no good science exist what is best proportion required. May be because with some footages the chroma noise may be too large in compare with luma noise - so it require to lower chroma weight. So current default 4:2 for YV12 still make chroma weight about 1/4 of luma or less. And increasing of scaleCSAD to 2 only make weights of SAD_Y and sum of SAD_U + SAD_V about equal.

[Dogway]

I knew your latest script version but it wasn't working correctly on my end, some error on non existant frame property in last frame (let me check tomorrow and I report). Also I wanted to track per frame times to debug or tune my script. I tried to modify it without success maybe you can sorta add the array trick you had in mind, ArrayAdd() is now internal. The good news is that your script gave some insight into accumulated stats for per scene metrics, I wanted to avoid globals because I'm very stubborn, but maybe I should give in lol

I was outside so I couldn't add a note on the correlation matrix. This is a random data set so it doesn't imply a coherent relationship for example between pnew and pzero, it just compares random sequences except for the first two columns GMSD and ms. Those are the ones that matter until I find other ways, or maybe you can find out since you have more math background. In my script GMSD values are inverted so 1 is reference quality, therefore higher DCT implies lower quality in the chart, same for other settings.
What I aim to find is a real correlation between settings, one way would be to limit the dataset to the first quantile or maybe pick only the pareto front. But I'm not sure if comparing by pairs is enough, raising thSAD might imply a more dimensional behaviour where thSADC needs to be lowered, maybe same for thSADR and who knows what else (lambda, badSAD, etc), everything is on multivariate analysis. In that regard sensitivity works because it's as simple as it gets, straight forward, but having those interrelations can also be useful.

index 0 frameprops was implemented in 3.7.1 test 22, or 24 or so, can't remember. It's a game changer in my opinion. Either way you can use other similar prefilter, I wanted to be fair with the comparison since you would never run such a clip without a prefilter.

I have 32Gb of RAM yet I'm using max memory of 2048 for each script, I'm not sure if more is necessary, but my CPU is old, I know things get a bit slower in MT but still faster in the big picture.

The problem I see is that a simple AddGrainC() can't mimic real grain. You would need to use something more like GrainFactory3mod, and then encode it so DCT takes the grain to lower frequencies (DCTFlicker as Didée called it). It's an option sure. But I have reserves with synthetic grain, in filmed content grain is the actual image not a transparent layer over it, grain models the detail contours so I like to think of grain as a convolution layer. I half made (https://forum.doom9.org/showthread.php?p=1959080#post1959080) a SAD analysis tool I need to finish it but it's a manual task. When knowing scene bounds beforehand it can run just once and save some time.

DTL, an idea, render a CGI animation with high samples and low samples. With low samples you get film like grain (not sure if it respects grain distribution though), then encode to h264.
By the way in my random tests trymany gave better scores but was too slow so I defaulted to false.

[kedautinh12]

I think f3kgrain will make a good addgrain :D
https://github.com/kedaitinh12/AVSPlus-Scripts/blob/master/f3kgrain_v0.4MOD.avsi

[DTL]

"not sure if it respects grain distribution though"

If you can do something in math - you can get set of samples values and perform statistical analysis: if it good or bad fit natural photon noises: Poisson distribution properties - https://en.wikipedia.org/wiki/Poisson_distribution
Gaussian - https://en.wikipedia.org/wiki/Normal_distribution

It is also one thing to add to MDegrain - it currently compare current frame with ref frame. But when looking for median area value it is better to compare current frame with 'median' value. So current algoriphm skips some blocks that fits in the thSAD from median value but falls out of thSAD check if current and ref blocks are on the different sides of the 'distribution curve'.

It can be partially workarounded with multi-pass MDegrainin - 1st pass moves block samples values closer to its median and next passes can add to averaging more blocks. But it is very slow. Faster it make with statistical analysis of the currently loaded pool of blocks in tr-scope around current frame in 1-pass.

Now about trymany:

"trymany, what does it do"

It is all inside PseudoEPZSearch() in PlanesofBlocks.cpp https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/PlaneOfBlocks.cpp#L901

For each round of search at each level it is possible to be predictors:
1. Zero - its current block position with zero move. Starting predictor at each round/level.
2. Global - it is median vector of motion of the whole frame.
3. Hierarchical - it is the only interconnecting between levels predictor - spatially interpolated MV of the prevoius level (to 4 surrounding blocks).
4. 3 surrounding blocks predictors (left, top (found at current level), bottom (interpolated from prev level) + their median = 4 more predictors
5. Temporal - same block MV from previous frame
Total 3+4+1=up to 8 predictors.

And at the end - the Refine() function that actually perform the requested search with requested radius around _requested_ position.

If trymany disabled:
Check performed for each 1..8 predictors once (at its MV value) and progressively collected best (lowest) 'cost' value (also best SAD value in parallel).
After it - Refine() is called once to perform actual search around best found 'prediction' position.

If trymany enabled:
After each check of 1..8 predictors called Refine() search function to try to get better MV (cost) around each predictor.
Finally Refine() called for the best of the best found MV (at the predictors + refining stage). Its output SAD value is written to output array.

cost = SAD + penalty * SAD; (for some predictors penalty is individual setting - like zero, global. new is general penalty for other predictors and searches (refining))

This may be better for motion interpolation task but skips some best SAD values. So sometime setting all penalties to zero may give better denoising results with lower thSAD value. But if it cause to worse MVs search - it may cause error-blending distortions (?).

" why trymany=true gives worse results than trymany=false?"

Generally trymany should give best possible MVs but its actual results may depend on both MAnalyse settings and input footage.

"I half made a SAD analysis tool I need to finish it but it's a manual task."

It is possibly good to use MAnalyse as SAD calculating engine - it have at least many good speed optimizations. Just set levels to 1 and search radius to 0 and it should return SAD of the zero predictor. But current Avisynth looks like not allow to use MAnalyse output as input to other filters ? Though it can output result to binary file and it can be parsed after somehow loading in the Avisynth or by external software.

"in filmed content grain is the actual image not a transparent layer over it,"

I think of natural photon-shot noise model as of addition of noise-distributed random values to some zero-noise source patch (flat or textured). So the processing of MDegrain is real physics based - it average many equal patches with added random noise with bipolar symmetric (enough symmetric at high photon-numbers and Natural-Gaussian distribution - but it is not exactly valid for dark low-photons areas with Poisson distribution ?) distribution. So after averaging enough many patches (large enough tr-value of MDegrain) you can finally got initial noiseless patch looking.

So it is important for synthetic noise generator to have natural random values distribution. Not 'flat' in some total deviation range.

For best possible results the thSAD value must be inifinitely big so all possible max noise deviations are totally compensated. But SAD also used in block-matching processing so too high thSAD cause errors-blending at moving areas. May be it is also good to add some thSAD modulation depending on the MVs length (like increasing thSAD at the areas with low or zero MVs that may be hint of static areas).

[Dogway]

@zorr: True! how odd, truemotion arg doesn't override the truemotion settings, while at it I also tested truemotionR and temporal, and they didn't output a different GMSD metric. I also thought about the correlation matrix, there's a way to know for certain how they relate each other, this is me thinking as I haven't done any further research. You have to lock the GMSD score. Yes, you can't so the procedure is to shoot a few thousand randomized iterations and pick the first three blocks with same and best scores. That is pick 5 rows with score 95.0 +- 0.001 (ie out of 100.0), pick 7 rows with score 86.0, and 5 (or whatever matched) rows of score 80. Do the correlation matrix of each one, fit a regression for the 3 correlation points to check if the relationship is linear or not (always relative to the GMSD score of course).

About the issue I have with your newer version, the script seems to be processed until the end then in the result file I get this error message:
Error getting frame property "_PlaneGMSD": property is not set
(C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0\work\Zopti_test2s - MDegrain3 - copia_1642960023349_5_1642960023509.avs, line 102)
(C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0\work\Zopti_test2s - MDegrain3 - copia_1642960023349_5_1642960023509.avs, line 106)
And this is the script: pastebin (https://pastebin.com/qB7rmHuX)

@DTL, I have read about Poisson distribution but haven't worked with it. So you say that MDegrain is a gaussian weighted median of some sorts? My SAD analysis tool is basically a flat patch searcher, I got the patch size from Neat video, I think it was 128x128, but I will try to check with your suggestion, I didn't know MAnalyse could output SAD value (?), interesting. Decoupling thSAD from block matching would also make sense or the modulation you talk about as it makes total sense, it would make the use of prefilters less necessary.

I will try to render a few low sample shots for MDegrain tests and post here when I get some time.

[DTL]

"MDegrain is a gaussian weighted median of some sorts? "

It is simple weighted averaging tool. Currently - blocks averaging with or without overlapping. Gaussian is expected noise distribution of some natural sources. As I read somewhere in math wiki - with high enough samples (photons) number the Poisson distribution is close to Gaussian. And as natural video cameras typically works with low to medium photon numbers per sample it mean at low samples values its noise deviation should be closer to Poisson distribution and with high samples values - to Gaussian distribution.

"I didn't know MAnalyse could output SAD value (?)"

It is the natural MAnalyse output format:
x, y, SAD (all integers 32bit signed). The output file have some short 'header' and list of all blocks data in that format.

outfile

Name of file to write motion vectors data, or an empty string (nothing written). This data may be used by some external program or may be by next MVTools versions for second pass coding, etc. Produced binary file has a header (MVAnalysisData structure, see MVInterface.h source code), and the data sequence:
◾Frame number,
◾vector data (Vx, Vy, SAD) of every block,
◾next valid frame number,
◾this frame vector data,
◾and so on…

So as DX12_ME hardware do not output SAD - it was added separate SAD compute shader to calculate SAD for complete output. Before compute shader version it perform call to SearchMVs() of GroupofPlanes to calculate SAD only and fill output structure with SAD values.

The mean frame SAD you can see in MShow() output with showsad=true. Though it is rendered in output framebuf for visual reading - not returned as some variable to script for processing.
I sometime use MShow(showsad=true) to check median SAD to adjust initial thSAD in MDegrain - it should be a bit more minimum or the degraining is about disabled (weights of most ref blocks = 0).

May be it also possible to get access in script environment to the output of MAnalyse data currently passed to MDegrain (and all other clients filters) and fetch x,y,SAD data for analysis. May be ask pinterf to patch Avisynth to allow some data fetching from this 'pseudoclip' ?
Or add some 'clip-source' converter filter to mvtools to 'convert' MAnalyse output to 'filter-readable/script-readable clip' inside AVS+ environment with Eval() (?) function. Like convert to virtual clip of BlckX (number of blocks per width) width and BlckY (number of blocks per height) height clip of RGB48 format and you can get x,y,SAD triplets as R,G,B 16bit sample components. RGB96 looks not supported as pixel_fmt in current AVS+ to read full 32bit signed values - but >16bit SAD is mostly possible only for HBD and large block sizes. Typical 'good' SAD values are about thSAD values and good fit in 16bit signed.

[zorr]

It is long to type - better next day. It is night here and better to try to sleep somehow.

Yes, do get some sleep, I'm not in a hurry. :)

4:8 for YV12 not mean UV is twice more weighted.

Thanks for the detailed explanation, but my test clip in the MFlowFPS tests was converted to YV24 before it was processed with MVTools so it doesn't apply there.

I found this in your other message, is this basically the chroma scaling algorithm currently used in MVTools?


// YV12 YV16 YV24
// nLogXRatioUV 1 1 0
// nLogYRatioUV 1 0 0
// effective_chromaSADscales: (shift right chromaSAD)
// chromaSADscale=0 -> 0 1 2 // default. YV12:no change. YV24: chroma SAD is divided by 4 (shift right 2)
// =1 -> -1 0 1 // YV12: shift right -1 (=left 1, =*2) YV24: divide by 2 (shift right 1)
// =2 -> -2 -1 0 // YV12: shift right -2 (=left 2, =*4) YV24: no change
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV));
effective_chromaSADscale -= chromaSADscale; // user parameter to have larger magnitude for chroma SAD
// effective effective_chromaSADscale can be -2..2.
// when chromaSADscale is zero (default), effective_chromaSADscale is 0..2


That would mean in my test the effective_chromaSADscale was 2 so chroma really was weighted twice as high as luma.

Also it may be not correct to count U and V channels 'equal' to Y channel in this summing

Yes I agree, notice how the range of typical values in U and V channels is much narrower than the luma range, this will make the SAD of U and V effectively smaller. This may be the real reason why chroma needs to be scaled higher - at least when the chroma channels are good enough quality and their SAD values are reliable and can help finding the matching blocks.

https://i.postimg.cc/YSHpxTcC/luma-chroma-histogram.png


So current default 4:2 for YV12 still make chroma weight about 1/4 of luma or less. And increasing of scaleCSAD to 2 only make weights of SAD_Y and sum of SAD_U + SAD_V about equal.

Just wondering about this, so in order to get maximum possible chroma weight the input should be converted to YV24? Perhaps the ScaleCSAD should work so that it's applied on top of the internal autoscaling (4:2 for YV12, 1:1 for YV24 etc), then the chroma scale would behave the same no matter which input format is chosen.

There's of course the possibility to pre-process the clip so that saturation is increased, this will then give additional weight to chroma SAD values.

[DTL]

"my test clip in the MFlowFPS tests was converted to YV24 before it was processed with MVTools so it doesn't apply there."

If your YV24 is not natural RGB-full-band sourced but upconverted from typical distribution YV12 - it 'half-banded full-sized' chroma. It still have 'noise level' lower in compare with true full-band YV24. Though its SAD_UV from full-size UV may be bigger in compare with YV12 UV blocks.

"is this basically the chroma scaling algorithm currently used in MVTools?"

Yes - it is from latest 2.7.45 release. It is 'chroma weight selection in addition operation of saduv to total output SAD'. Yes - it may directly mean lumaSAD is _never_ any 'autoscaled/normalized' (with any scaleCSAD setting and any pixel format). So any addition of chroma can only increase total output SAD.

"in order to get maximum possible chroma weight the input should be converted to YV24? "

As I see with YV24 the nLogXRatioUV/nLogYRatioUV both 0. So initial
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV)); = 2
and even providing scaleCSAD = 2 will result
effective_chromaSADscale -= chromaSADscale; of only 0
and 0 result with passing SAD_U+SAD_V to output sum unchanged.
Translating eff_sclale to actual scale function is in https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/def.h#L123 . So it looks for YV24 it is impossible with current max scaleCSAD value of 2 to get any amplification to SAD_U+SAD_V over 1.
It looks the internal functions are adjusted in such way that in YV12 and in YV24 scaleCSAD=2 will result in 'equal luma + sum on chromas' balance. And default scaleCSAD=0 mean in both YV24 and YV12 it is about 1:1/4 (it marked as 4:2 as 4: (1+1) but each U and V is 1/4 in relation with 1 Y). Though UV in YV24 have 4x more samples and _may_ even in upconverted from YV12 'half-banded' state give bigger SAD value (not sure and not tested). The purpose of internal functions may be to output more or less equal SAD values from MAnalyse to MDegrain in any input format from YV24 to YV12 (no need to significantly re-adjust thSAD and other thresholds).

" the possibility to pre-process the clip so that saturation is increased, this will then give additional weight to chroma SAD values."

With high-saturated colours it will simply clip to min/max UV values and it will be close to useless. Only will help on low-saturated areas.

[zorr]

I knew your latest script version but it wasn't working correctly on my end
...
maybe you can sorta add the array trick you had in mind, ArrayAdd() is now internal

Yes I think I need to look into that. Hopefully ArrayAdd() is thread-safe. :)

What I aim to find is a real correlation between settings, one way would be to limit the dataset to the first quantile or maybe pick only the pareto front. But I'm not sure if comparing by pairs is enough, raising thSAD might imply a more dimensional behaviour where thSADC needs to be lowered, maybe same for thSADR and who knows what else (lambda, badSAD, etc), everything is on multivariate analysis.

I agree, definitely there are interactions between three or more parameters and that's what makes analyzing the results difficult.

I have thought about how to find the best values for any input clip and I think a good approach may be simply to collect lots of optimization results with different videos and then run an automatic test which tries each of them and reports the best one found.

The problem I see is that a simple AddGrainC() can't mimic real grain. You would need to use something more like GrainFactory3mod, and then encode it so DCT takes the grain to lower frequencies (DCTFlicker as Didée called it). It's an option sure. But I have reserves with synthetic grain, in filmed content grain is the actual image not a transparent layer over it, grain models the detail contours so I like to think of grain as a convolution layer.

Definitely if synthetic grain is used it should be as realistic as possible. It doesn't really matter if adding it to original clean clip takes a lot time, it can be "burned in" and then read directly as a source clip while the reference clip is the original. I think it's worth trying, it doesn't need to replicate real noise perfectly. Even if we had it perfect it would only be perfect for one clip so the algorithm will have to handle some variation in the noise profile anyway.

By the way in my random tests trymany gave better scores but was too slow so I defaulted to false.

The clip I used might be a special case as it has a very challenging, repetitive background which is bound to confuse the motion analysis.

[zorr]

"trymany, what does it do"

Thanks for that, I got the basic idea. :thanks:

This may be better for motion interpolation task but skips some best SAD values. So sometime setting all penalties to zero may give better denoising results with lower thSAD value. But if it cause to worse MVs search - it may cause error-blending distortions (?).

I think it has to do with the repetitive moving background in the test video. It's a difficult case on purpose and may need special settings compared to a typical case.

[zorr]

@zorr: True! how odd, truemotion arg doesn't override the truemotion settings

It says this on the manual page:

truemotion - This is a preset of these parameters values. It allows easy to switch default values of all "true motion" parameters at once. Set it true for true motion search (high vector coherence), set it false to search motion vectors with best SAD. Default is true since v1.4.10. In any case you can tune each parameter individually.

I take this to mean that whatever truemotion is set to, the parameter value (if set) will always take precedence.

I also thought about the correlation matrix, there's a way to know for certain how they relate each other, this is me thinking as I haven't done any further research. You have to lock the GMSD score. Yes, you can't so the procedure is to shoot a few thousand randomized iterations and pick the first three blocks with same and best scores. That is pick 5 rows with score 95.0 +- 0.001 (ie out of 100.0), pick 7 rows with score 86.0, and 5 (or whatever matched) rows of score 80. Do the correlation matrix of each one, fit a regression for the 3 correlation points to check if the relationship is linear or not (always relative to the GMSD score of course).

That sounds interesting. If you'd like to test this idea with a lot of data I can send you a result file with almost 700k results, that should be enough. :)

About the issue I have with your newer version, the script seems to be processed until the end then in the result file I get this error message:
Error getting frame property "_PlaneGMSD": property is not set

I was able to replicate the error (had to remove FluxSmooth and Luma_Rebuild though and use show=false instead of show=0 in GMSD, maybe we have different versions?). So the error happens because GMSD writes the frame props to the first clip in the argument list, and you want them in the second (the processed clip). So just switch the parameters around. But then there was another error:

RequestLinear: internal error (frame not cached)!

And that went away be commenting out RequestLinear(30). Perhaps there's a better way to handle it, because the processing took very long time at the last frame - it was probably recalculating all the frames which no longer were in the cache.

[zorr]

May be it also possible to get access in script environment to the output of MAnalyse data currently passed to MDegrain (and all other clients filters) and fetch x,y,SAD data for analysis. May be ask pinterf to patch Avisynth to allow some data fetching from this 'pseudoclip' ?
Or add some 'clip-source' converter filter to mvtools to 'convert' MAnalyse output to 'filter-readable/script-readable clip' inside AVS+ environment with Eval() (?) function. Like convert to virtual clip of BlckX (number of blocks per width) width and BlckY (number of blocks per height) height clip of RGB48 format and you can get x,y,SAD triplets as R,G,B 16bit sample components. RGB96 looks not supported as pixel_fmt in current AVS+ to read full 32bit signed values - but >16bit SAD is mostly possible only for HBD and large block sizes. Typical 'good' SAD values are about thSAD values and good fit in 16bit signed.

I was thinking the same and actually asked Pinterf (https://forum.doom9.org/showthread.php?p=1946741#post1946741) in the MVTools thread. No response though, perhaps he was too busy to even think about it. :)

This would be a great addition and would make testing new motion vector algorithm ideas much easier.

[zorr]

As I see with YV24 the nLogXRatioUV/nLogYRatioUV both 0. So initial
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV)); = 2
and even providing scaleCSAD = 2 will result
effective_chromaSADscale -= chromaSADscale; of only 0

That's not what I got. The value of chromaSADscale for YV24 is 0 when ScaleCSAD is 2 so effective_chromaSADscale -= chromaSADscale; is 2-0 = 2.

With high-saturated colours it will simply clip to min/max UV values and it will be close to useless. Only will help on low-saturated areas.

If you look at the range of U and V in the histogram you can see that it really is low-saturated, in this clip the chroma could easily be scaled by four and it still would not get clipped (and that's in addition to the ScaleCSAD multiplier 2). In my experience this is typical - chroma values are using much smaller range than luma. But of course you have to be careful and check before you do the upscaling.

But it would be better to allow larger ScaleCSAD values in MVTools at least in order to do some experiments - it could be a very simple and cheap way to get better motion vectors. Scaling the chroma by adjusting saturation can be used as a workaround to see if it really helps.

[DTL]

"=2"

eff_chroma is not final multiplier - it is bitshift control value. It looks you not look into source: https://github.com/pinterf/mvtools/blob/mvtools-pfmod/Sources/def.h#L123

// effective scale: 1 -> div 2
// 2 -> div 4 (YV24 default)
// -2 -> *4
// -1 -> *2
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);


To have sum of SAD_U + SAD_V to be multiplied by 2 you need eff_chroma control value of 1 (bitshift 1 to the left).

The comment also shows: YV24 scaleCSAD=0 is converted to eff_scale = 2 that mean (sum_of_sad_U+V) /4 . Even increasing user-control param scaleCSAD to 2 only can lower eff_scale to 0 that mean passing sum_of_sad_U+V unchanged (= multiplier 1).

[Boulder]

It says this on the manual page:

truemotion - This is a preset of these parameters values. It allows easy to switch default values of all "true motion" parameters at once. Set it true for true motion search (high vector coherence), set it false to search motion vectors with best SAD. Default is true since v1.4.10. In any case you can tune each parameter individually.

I take this to mean that whatever truemotion is set to, the parameter value (if set) will always take precedence.


Truemotion=true works like a preset in an encoder - it sets up a bunch of predefined values which you can then choose to override if necessary.

In fact, it might not be a bad idea to have a real preset functionality if something generic is found with all these tests.

[zorr]

eff_chroma is not final multiplier - it is bitshift control value. It looks you not look into source: https://github.com/pinterf/mvtools/blob/mvtools-pfmod/Sources/def.h#L123

Oh right, I did look at the code but I wasn't paying attention. Also I had a misunderstanding what the value of chromaSADscale is. It's just the ScaleCSAD parameter (defaults to 0 if not given in script).


// effective scale: 1 -> div 2
// 2 -> div 4 (YV24 default)
// -2 -> *4
// -1 -> *2
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);


To have sum of SAD_U + SAD_V to be multiplied by 2 you need eff_chroma control value of 1 (bitshift 1 to the left).

I think you mean control value of -1. :)

The comment also shows: YV24 scaleCSAD=0 is converted to eff_scale = 2 that mean (sum_of_sad_U+V) /4 . Even increasing user-control param scaleCSAD to 2 only can lower eff_scale to 0 that mean passing sum_of_sad_U+V unchanged (= multiplier 1).

Yes that's true. The algoritm only ever allows you to get 4 times the chroma weight of YV12 and in YV24 that means the chroma doesn't get any scaling as it already has 4 times the weight of YV12 chroma.

In the documentation the chroma weight of YV24 is written as 4:8 implying that chroma has double the weight compared to luma. That is justified by considering that the chroma has two channels which both contribute to the final SAD value. In the source code it's clarified: 4 : (4+4) = 4:8.

I just may have to run some chroma weighting tests to see if the results can indeed be improved by scaling the chroma even further. Or perhaps the scaling should be more fine-tuned like you suggested and the best scaling factor lies somewhere from 0.5 to 1.0 for YV24.

[zorr]

First chroma scaling results are in. I did a quick test to see if keeping all the previous optimal parameters of these tests (https://forum.doom9.org/showthread.php?p=1945684#post1945684) and only scaling the chroma using Tweak(sat=chromaScale/100.0, coring=false) would give better results. The scaling range was 50..200 -> from 0.5 to 2.0. There are only 151 different values so they can all be tested with exhaustive algorithm.

https://i.postimg.cc/zGqCGzt8/halos-ghosts-judder8b-blocksize8-2022-01-25-01-22-05-optimize-exhaustive-run-01.png

Looks like the parameters were so tightly optimized for the original chroma that any changes to it result in clearly worse score.

The next step is to try to run the optimization with more unlocked parameters to see if that allows finding a better chroma scale and better GMSD score.

[DTL]

But it would be better to allow larger ScaleCSAD values in MVTools at least in order to do some experiments - it could be a very simple and cheap way to get better motion vectors. Scaling the chroma by adjusting saturation can be used as a workaround to see if it really helps.

Here is new build with scaleCSADfine added new parameter - https://drive.google.com/file/d/1OAqXVBvrX3a6KVbY6nH5-TJQrED2sli9/view?usp=sharing . It is float muliplier with currently unlimited range. Now both scaleCSAD and scaleCSADfine works at once.

The tests shows that if adjusting scaleCSAD it is required to adjust thSAD in MDegrain at about same proportion (with >1 scaling). Such changing scaleCSAD from 0 to 2 with YV12 input require change of thSAD from 175 to 350 to keep about the same 'degrain level'. The requirement to adjust >1 parameter at once makes analysis much harder/longer. It is now require to make many encodings with different luma/chroma SAD ratio and adjust thSAD to get about the same MPEG output speed for each tested ratio and check for the quality of fine textures.

Default internal value for scaleCSADfine is 1.0 so it do not change old processing.

[DTL]

New update with many new params to optimize: https://drive.google.com/file/d/1BUP0sQ3zRinr0tVKfIumPUqLK6r5GTCF/view?usp=sharing
Based on commit - https://github.com/DTL2020/mvtools/commit/ee38893b26054dde0369ae2e142e76d1c2b0e6bc
(Windows-7 compatible build. No DX12 features.)

MDegrainN new added params:
1. adjSADzeromv (1.0 - default, no op) - possible SAD multiplier for zero-move blocks (before thSAD processing for getting block's weighting value). Float value. Recommended values: 0.9..0.4-. Possible medium values 0.75..0.5. Allow to increase degraining at static areas.

Example: setting 0.5 result thSAD for zero move blocks (static) will be thSAD*2.

2. adjSADcohmv (1.0 - default, no op) - possible SAD multiplier for blocks in coherent moving areas (before thSAD processing). Float value. Recommended values: 0.9..0.4-. Possible medium values 0.75..0.5. Allow to increase degraining at big enough coherent moving areas of much larger 1 block_size size (like camera pan movement over non-changing scene).

3. thCohMV (-1 default, no op) - threshold to detect if block's move vector is equal to surround blocks (top,left,right, down) move vectors. -1 - disables this part of processing (faster), 0 - lowest working value. Recommended values 0..4. Possible range - 0..unlimited int. Too high values will create error-blended blocks (like with too high thSAD value).

Example: setting to 4 allow SAD_of_MVs of 4 surrounding blocks with current block (x_cur, y_cur) to be up to 4.


int iabs_dc_x = SADABS(v_upper.x - x_cur) + SADABS(v_left.x - x_cur) + SADABS(v_right.x - x_cur) + SADABS(v_lower.x - x_cur);
int iabs_dc_y = SADABS(v_upper.y - y_cur) + SADABS(v_left.y - y_cur) + SADABS(v_right.y - y_cur) + SADABS(v_lower.y - y_cur);

if ((iabs_dc_x + iabs_dc_y) <= ithCohMV)
{
block_sad = (sad_t)((float)block_sad * fadjSADcohmv);
}
where int SADABS(int x) { return (x < 0) ? -x : x; }


Setting adjSADzeromv = 1.0 and adjSADcohmv = 1.0 disables new processing (faster, compatible with old scripts).

Still only C-version (no SIMD acceleration) so may be visibly slower if enabed new added processing.

Typical my processing settings now (for Full HD footage from 3chip video cam ENG/EFP type):
YV12 format:
MAnalyse(old args..., scaleSCAD=2)
MDegrainN(old args..., thSAD=250, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=4)

Using non-default new arguments assumed to allow much lower 'general' thSAD control value (about half of typical old) to keep more details on arbitrary moving objects (and skin details) while keeping noise and static areas low for better MPEG compression and looking. Possible target content type: low movement TV series (culture, nature). May visibly fail if many moving and changing its view at move time objects. For high-move footages with complex scene change recommended to disable adjSADcohmv (1.0 - default or keep thCohMV to default -1) or not set adjSADcohmv too low.

Task for auto-optimization: found sensitivity to new arguments and typical good/best values.

[zorr]

New update with many new params to optimize
...
MDegrainN new added params:
1. adjSADzeromv (1.0 - default, no op)
2. adjSADcohmv (1.0 - default, no op)
3. thCohMV (-1 default, no op)
...
Task for auto-optimization: found sensitivity to new arguments and typical good/best values.

I haven't done MDegrain tests yet, would have to come up with a good noisy / clean reference video pair. Dogway had MDegrain tests under way, perhaps he can test the new options as well.

By focusing on MFlowFPS / MBlockFPS it's possible to find good parameters for MAnalyse which will then benefit pretty much all MVTools operations including MDegrain.

Doing MDegrain properly would mean calculating optimal MAnalyse parameters for many different frame deltas, I think there may be some variation depending on the delta.

[DTL]

"there may be some variation depending on the delta."

It may be additional big field of experiments: currently only thSAD in only MDegrainN can be interpolated between 2 points - start and end in tr-length. But may be other params is also good to make interpolated. The new added adjustments for thSAD internal processing too. Currently MDegrainN have about 12 params to adjust in smooth gradation - already large enough set for manual testing the best combo.
Most hard to adjust are dependent like thSAD/adjSADzeromv/adjSADcohmv.

Also it is good to make the optimization target like 'minimum output x264-encoded filesize (with fixed set of params for x264)'. To select some balance between image quality and final encoded filesize. Typically the best image quality may not be equal to minimum encoded filesize.

[zorr]

I was able to replicate the error (had to remove FluxSmooth and Luma_Rebuild though and use show=false instead of show=0 in GMSD, maybe we have different versions?). So the error happens because GMSD writes the frame props to the first clip in the argument list, and you want them in the second (the processed clip). So just switch the parameters around. But then there was another error:

RequestLinear: internal error (frame not cached)!

And that went away be commenting out RequestLinear(30). Perhaps there's a better way to handle it, because the processing took very long time at the last frame - it was probably recalculating all the frames which no longer were in the cache.

I now have a better solution, it works reliably and is faster and makes the script simpler.

It requires a new version of Zopti which is ready to go but it's getting late here so I'll do the release tomorrow along with more detailed explanation.

[zorr]

This release introduces a better way for Avisynth scripts to write the results and fixes a couple of bugs. I recommend the upgrade especially if you're using more than one thread, there was a rare but possible bug which could result in flawed results.


supports a new way for an Avisynth script to write the result file
-first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
-then come the per frame result lines in any order, using the same format as before
-do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
-this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
-old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
-new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value

Download link updated at first post.

Here's an example of the new way to write results, based on Dogway's script from here (https://forum.doom9.org/showthread.php?p=1961928#post1961928). Only showing the part which does the writing.

resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write the number of frames to tell the optimizer how many result lines to expect
WriteFileStart(resultFile, "FrameCount()")

GMSD(src.ConvertBits(32, fulls=false, fulld=true), ConvertBits(32, fulls=false, fulld=true), show=false)

global GMSD = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
""",local=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (GMSD, time)
global delimiter = "; "
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")


This new method to write results has been completely reliable in my tests, I have been running slightly modified script by Dogway for a couple of days using 12 threads and have over 26000 results so far and no problems. The same script was very slow by reading the GMSD frameProps at the last frame causing many frames to be evaluated twice and also unreliable using the original method because the order of lines written cannot be guaranteed.

[Dogway]

Thanks for the update! Eager to test. I was a bit let down with the idea of having to run thousands of runs for the correlation test so took an hiatus.
This was my accumulated runs (random or not). Pitty that the interesting part of the convex hull didn't get sampled. I merged all the result files by hand though since they were independent.
https://i.imgur.com/0HY3EkUl.png

This was my correlation coefficients with the above sample data with the method I explained earlier. The correlation median of similar GMSD (high) score blocks. Greyed out are not reliable enough due to few data samples, as they need to be totally random otherwise you get a bias.
http://i.imgur.com/TI6Y73ch.png (https://i.imgur.com/TI6Y73c.png)

With values over +-0.6 you can satisfy linear correlation y = ax+b, you can fill that into "min:" and "max:" filter with some bias offset (ie. min: 1.3 * lambda + 100 max: 1.3 * lambda + 500)

[zorr]

Boulder, I might as well report some findings so far with your MDegrain script as I have plenty of data on it. Some changes I made to the script:


I used FFVideoSource instead of DGSource. I tried with DGSource and noticed it reports errors (popup windows) when used with a large number of threads. I will investigate this more later.
removed the preprocessing filters (ex_FluxSmoothST and ex_Luma_Rebuild) because my Avisynth version didn't support reading frameProps outside ScriptClip (I could try adding those later though).
the clip brazil-ref you shared probably already had the trimming baked into it because the clips were out of sync with the trim lines so I removed them and verified that the clips are in sync after that
I removed the optimization of truemotion and truemotionR and set them to false as their value has no effect on the result
used the new method to write results which was introduced in Zopti v1.2.2


[EDIT] Removed visualizations of bad data

I'm using dynamic iteration count with -dyniters 360 and -dynphases 10, with these setting my earlier MFlowFPS script took about 100000 iterations to finish. It's currently altering between phase 0,1 and 0,3 so it's quite early but we can already see some trends.

The -vismode heatmap is also useful when trying to determine if the search ranges are large enough (or too large).

[EDIT] Removed conclusions based on bad data.

Also looks like we have different GMSD scores for some reason, perhaps something to do with the trimming.
[EDIT] Oh boy, just noticed you have another set of trims further down which balances the first ones. So I have been calculating almost meaningless results. :o

[Dogway]

Haha, yes no issues. It happened to me as well after a few days.

Some observations, prefiltering is a MUST with such noisy sources. Maybe you can use vanilla MinBlur(2).
This is an updated recommendation from my old SMDegrain examples, you need Zs_RF_Shared.avsi which is nice for old AVS+ versions:

pre=fluxsmootht(2).MinBlur(2).Dither_Luma_Rebuild(3)

If MinBlur() is utterly slow (it uses medianblur()) use MinBlur(1).
Luma_Rebuild is also very recommended specially for movies. There are some movies like comedies that are very bright, but generally some brightneing of the low range is recommended, this also expands the luma/chroma range to PC levels. With this I think we are mostly on the same page.

Another thing is that searchRange is bound to searchAlgo, it means different things for different algos, therefore I normally run random iterations to know which range suits which algo. I had some conclusions after much testing, at least for MDegrain and this sample. In all cases, all pareto front (and most of my higher random results) was composed of the next values so they can be locked. searchRange is a bugger, I actually want to set it to 2 because higher than that smears shading but GMSD keeps scoring high for higher values.

BlkSize = 16
BlkSizeR = 8
overlapR = 4
pel = 1 # 1 for HD
sharp = 2 # optimize sharp = _n_ | 2 | sharp
lvl = 1
rfilter = 3
scaleCSAD = 2
trymany = false
truemotion = false
truemotionR = false

dct = 0 # optimize dct = _n_ | 0 | dct
dctre = 9 # optimize dctre = _n_ | 9 | dctre
searchAlgo = 1 # optimize searchAlgo = _n_ | 1 | searchAlgo
searchAlgoR = 4 # optimize searchAlgoR = _n_ | 4 | searchAlgoR
searchRange = 13 # optimize searchRange = _n_ | 13 | searchRange


These can be optimized. These are very refined ranges so expand them 25% more
thSAD = 285 # optimize thSAD = _n_ | 235..285 ; filter:x 5 % 0 == | thSAD
thSADC = 200 # optimize thSADC = _n_ | 140..250 ; filter:x 10 % 0 == | thSADC
thSADR = 280 # optimize thSADR = _n_ | 150..300 ; filter:x 10 % 0 == | thSADR
overlap = 4 # optimize overlap = _n_ | 4,8 | overlap
temporal = true # optimize temporal = _n_ | true,false | temporal


# TRUEMOTION SETTINGS
lambda = 440 # optimize lambda = _n_ | 440 | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 1320 # optimize lambdaR = _n_ | 960..2000 ; min:lambda ; max:lambda 3 * ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114 # optimize pnew = _n_ | 114 | pnew
pnewR = 136 # optimize pnewR = _n_ | 100..250 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 100 # optimize pzero = _n_ | 100 | pzero
lsad = 6000 # optimize lsad = _n_ | 1000..8000 ; filter:x 100 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79 # optimize plevel = _n_ | 79 | plevel

# lambda is not used for global predictor
pglobal = 8 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2 # optimize badrange = _n_ | 0..50 ; filter:x 2 % 0 == | badrange
badSAD = 1350 # optimize badSAD = _n_ | 1100..2200 ; filter:x 50 % 0 == | badSAD
searchRangeR = 13 # optimize searchRangeR = _n_ | 2..18 | searchRangeR
searchRangeFinest = 17# optimize searchRangeFinest = _n_ | 9..20 | searchRangeFinest
sglobal = true # optimize sglobal = _n_ | true,false | sglobal

[zorr]

Some observations, prefiltering is a MUST with such noisy sources. Maybe you can use vanilla MinBlur(2).

I assume you have already optimized the prefiltering with Zopti? :) If not, that's something it could do, either separately or (more thoroughly) combined with the optimization of other parameters.

And if you already have a good prefiltering figured out it's not a problem if it's slow. You can "bake it in" by saving the prefiltered clip as lossless video (Lagarith, Huffyuv etc) and just load the pre-filtered clip in the script you want to optimize. I installed the latest Avisynth+ and already tested this, it saves about 13 seconds per iteration in your script (running a single thread).

I just started running your original MDegrain script again, this time with the prefiltering included, no stupid trim issues and even using the original DGSource (doesn't seem to have problems with the latest AviSynth).

This is an updated recommendation from my old SMDegrain examples, you need Zs_RF_Shared.avsi which is nice for old AVS+ versions:

pre=fluxsmootht(2).MinBlur(2).Dither_Luma_Rebuild(3)

So this works better than
pre=ConvertBits(16,fulls=false).ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)?

I had some conclusions after much testing, at least for MDegrain and this sample.

Thanks, let's see if I come to the same conclusions after running about 20k iterations...

[Dogway]

hehe No I haven't run prefiltering through Zopti, one at a time. But you need some "dumb" temporal prefiltering and fluxsmooth is very good for that, minblur is the same but spatially. By Didée's words you want to soften the edges (not to mention flat areas!) otherwise "you'll end up with pretty big SADs wherever there's an edge. Means, little to nothing will happen on edges". It's been time tested.

You said you can't run my prefiltering, so I assembled an equivalent using old filters included in Zs_RF_Shared.avsi. But now that you're using new AVS+, use the "ex_" prefiltering since they are optimized. If you want to test this area I have some defaults at SMDegrain(). It's dependant on source grain, I would ditch dfttest (slow) and ex_knlmeans (slow and not as good as BM3D).

Thanks for testing, do an integrity check/run before running so nothing is mangled.

[kedautinh12]

I don't know why but in TemporalDegrain2 postFFT KNLMEANSCL faster than BM3D

[zorr]

Thanks for testing, do an integrity check/run before running so nothing is mangled.

It's looking good, very similar GMSD scores compared to your data. Also checked the clips are in sync with Interleave(), this time returning them at the end of the script and not at the start. :)

https://i.postimg.cc/qMKm38fD/mdegrain-fast.png

I will post some observations next.

[Dogway]

Thank you, very beautiful curve. Grabbing only the data samples of the inflection area can be useful for some study as it's where most of the things are going on. Not sure how to isolate those.
When I run my tests ex_MinBlur() was some kind of broken for HBD, so you might have more accurate results.

[zorr]

All right, let's see some visualizations again with correct data. I will remove the previous shots from a couple of posts up so nobody gets wrong information.

So there are still some parameters which have their best values at or very near the edge of the search range which could be extended. It's easy to see these by -vismode line.

https://i.postimg.cc/TwtTXz2S/lambda.png
Lambda is at the minimum allowed and the trend points to better results with lower lambda still.

https://i.postimg.cc/vTDb80cm/lsad.png
lsad is not at the very edge but there's a similar (although noisy) trend as with lambda.

https://i.postimg.cc/8knTXg0f/pnew.png
Looks like pnew is at the top of a hill, but might be worthwile to check if there's really a downward slope with smaller values.

https://i.postimg.cc/nLZx9FCg/pnewR.png
pnewR is quite noisy but there's a clear trend, smaller values should be tested.

https://i.postimg.cc/5NYbZKh7/pzero.png
There's no slope near pzero but there are almost as good results to the right. It might help the other parameters to find new optimal values if we allow more breathing room for pzero.

https://i.postimg.cc/wTCgHcH6/search-Range-Finest.png
searchRangeFinest is quite jumpy, we could try our luck and extend the range a bit to the left, there should be faster combinations at least.

https://i.postimg.cc/fRYZbtTY/thSAD.png
thSAD is also almost the edge, could try to extend the range upwards and get rid of the smallest values.

I will let this run continue a bit further and then restart with the extended or moved ranges.

[Dogway]

The results align well to my pareto front.
http://i.imgur.com/BU9V8muh.png (https://i.imgur.com/BU9V8mu.png)
If I'm correct the blue line tells how many results contain said value right?

These graphs are good to find a pareto front for a certain task, like MDegrain in this case with this clip.
But to better understand the correlation between settings we need a correlation study (without bias)

[zorr]

If I'm correct the blue line tells how many results contain said value right?

Yes. The axis on the right displays the scale for the number of results.

These graphs are good to find a pareto front for a certain task, like MDegrain in this case with this clip.

You don't need these graphs for the pareto front, Zopti simply lists it whenever you use the -mode evaluate. :) But the graphs certainly will help in other ways.

But to better understand the correlation between settings we need a correlation study (without bias)

Yes that could certainly be useful. Calculating correlations is not very hard, it could be built into Zopti. I think Principal Component Analysis (PCA) could also be useful.

[Dogway]

Sure, everything has to do with the covariance between the 2 sets. Have never worked with PCA though. Will start my stats studies in March, by the way bought your recommended book (physical), thanks for the suggestion!

[zorr]

by the way bought your recommended book (physical), thanks for the suggestion!

Where's my commission! (just kidding) :)

Decided to stop the run and made some adjustments to the search ranges. After 55k iterations it was still going back and forth between phases 0.0 and 0.3 (and a short visit to 0.4). Looks like there's some action in the flat top part also.

https://i.postimg.cc/zBR8Xd0p/mdegrain-fast-2.png

I will start a new one but I will do a MCompensate test before that.

MCompensate is actually the most direct way to evaluate the quality of the motion vectors. So I think the optimization of MSuper, MAnalyse and MRecalculate could be done with MCompensate and then the same parameters should work with MDegrain, MFlowFPS etc where the quality of motion vectors is harder (or slower) to evaluate. This should of course be tested so after we have arrived with a robust set of parameters for MDegrain for this source clip we could try to reproduce them using MCompensate.

[Dogway]

I explored that route and I don't recommend it. I started with the same, comparing motion vectors from a HFR clip and a motion interpolated one and that doesn't translate well into MDegrain. With MDegrain you can get away with DCT>1 with motion interpolation probably not (too slow to read pareto). Also searchRange for MDegrain is not recommended (smearing) while with motion interpolation helps a lot.

I stepped back and started with the basic MDegrain, the only thing I care is parameter correlation. Once I get that I will expand into motion interpolation. Maybe in a few days if you haven't already I will try to do some runs with randomized iterations using new version 1.2.2 and do my correlation matrix. Another run to verify the correlation matrix. This is fine if for example you want to change searchAlgo in a filter then you want to default searchRange to a different number so the filter always gives out optimal values for all clips, as I'm currently making a SceneStats() filter to output that, scene stats and make automatic filtering decisions by these stats.


EDIT: I did some correlation studies only for the thSADs given the rest is equal (using my highest scored settings) and found thSADR is directly proportional until the critical point, that is, if you use too much thSAD, thSADR is also going to go wrong and best score is inversely proportional (-1.9x+682). Since we don't know the critical point and this is also subjective given the user desired amount of denoising I think a positive correlation is fine.
thSADR = round(exp(-101./(thSAD*0.83)) * 360)
Also studied thSADC (I needed to set chroma=true, plane=4, and in GMSD V=true):
thSADC = round(thSAD * 0.755 * (0.25*exp(scaleCSAD*0.693)))

For this example my results were (GMSD score of 146.09013):
thSAD = 245
thSADR= 219
thSADC= 185
It might vary depending on other parameters but I would consider these still first class parameters where all the rest should conform to.

By the way a question, is it possible to tell sensitivity to only care for score and not score * speed?

[zorr]

I explored that route and I don't recommend it. I started with the same, comparing motion vectors from a HFR clip and a motion interpolated one

So you had a clip with high frame rate and tried to interpolate the missing frames from a lower frame rate clip? In that case what I'm suggesting is different.

MCompensate can be used to take for example frame n+1 and using motion vectors to rearrange it into frame n. If the vectors are perfect then GMSD(n, MCompensate(n+1)) will be zero. Also the parameters of MCompensate have nothing to optimize (recursion is the only one that could be experimented with but it would only benefit MCompensate so it should be at default value 0). This means the results can directly tell the quality of the motion vectors.

and that doesn't translate well into MDegrain. With MDegrain you can get away with DCT>1 with motion interpolation probably not (too slow to read pareto). Also searchRange for MDegrain is not recommended (smearing) while with motion interpolation helps a lot.

If what you're saying is true then MDegrain does not want as good motion vectors as possible for the best results, it wants something else. It would be interesting to know what exactly it is and why.

My mental image of how MDegrain works is that it creates those new frames using the same process as MCompensate and takes averages of pixels in temporal dimension (or ignores them if SAD is too bad). It also has the thSAD2, thSADC2 parameters to give the closest frames more weight in the average calculation. It should be possible to recreate the exact functionality using MCompensate and Average(). Or better because each motion vector delta could use different set of parameters for MAnalyze etc.

I stepped back and started with the basic MDegrain, the only thing I care is parameter correlation. Once I get that I will expand into motion interpolation. Maybe in a few days if you haven't already I will try to do some runs with randomized iterations using new version 1.2.2 and do my correlation matrix. Another run to verify the correlation matrix. This is fine if for example you want to change searchAlgo in a filter then you want to default searchRange to a different number so the filter always gives out optimal values for all clips, as I'm currently making a SceneStats() filter to output that, scene stats and make automatic filtering decisions by these stats.

I can help you with the correlation matrix by calculating you more data. Since the tests require purely random data perhaps I could add an option for that in Zopti. Of course since you already have set specific ranges for the parameters it's not completely random but assuming those ranges cover all sensible use cases it should work. And it would be too difficult to find any real correlations with full ranges anyway.

EDIT: I did some correlation studies only for the thSADs given the rest is equal (using my highest scored settings)
...
It might vary depending on other parameters

Just be careful not to make too strong conclusions. Some parameters like the search algorithm can have big effects on the optimal range of other parameters. But since you have the option to choose which algorithms to allow in your functions you could make correlations that work for those.

By the way a question, is it possible to tell sensitivity to only care for score and not score * speed?

I'm not sure I follow, the sensitivity only cares for the first optimized parameter, which usually is the quality. If you want to use the sensitivity calculations perhaps I should give an option to limit how many results to include, right now it's the last two generations that are used. However it would take too long to calculate the sensitivity from much larger result set because it's O(N^2) algorithm.

[DTL]

If the vectors are perfect then GMSD(n, MCompensate(n+1)) will be zero. "

It is unlikely for natural scenes. The 'vector' compensation only cover more or less good Translate transform of possible basic Scale/Rotate/Translate transforms of object. Also object (block) may change its looking because of other reasons (like simply change of incident light colour (3 components also - hue/sat/intensity)). So in current mvtools the possible compensation is very limited. In the better new versions it is possible to add more at least basic transforms to search and compensate - scale and rotate (at least 1 of 3 axises rotate). So it is not possible to expect ideal result from the still very limited tool.

In addition to 'basic' transforms may be other like Skew/Perspective

https://desktop.arcgis.com/en/arcmap/10.3/tools/coverage-toolbox/GUID-D81DC3FC-6A37-4C30-BBD3-C9B23F874ECE-web.gif

I hope in some future there will be more advanced version of mvtools with search/measure/compensate for more number of transforms. But it is very computational expensive so speed may be 100 or 10000 times slower - not very practical. It is a task for the current high speed massive multithreaded compute accelerators and future (if we will see it in current dying civilization).
Also the multi-transforms search is not simply linear sequence of several transforms search - it is multi-pass refining and even slower.

It still may be some research which of the other transforms are the most valueable with natural scenes and is worth to put developers resources to implement next in addition to current Translate-transform only in mvtools. I think it may be Rotate transform.
Also the Translate transform may be treated as special case of Rotate transform with very low angle and very large radius (distance to origin). It looks the Translate transform was most useful for MPEG coders and enough cheap in computing at old hardware so it become first and main transform to compensate. But it still not cover all natural transform cases.

So the full transform structure of block to compensate will be much more complex like:
1. Translate MV (dx,dy)
2. Rotate angle(s) (rz,(rx,ry)), +rotate origin x,y ?
3. Scale (sx,sy)
4. Skew (..x,..y)
5. Perspective (kx,ky)
6. Lighting (YUV/HSB/...)
...

" MDegrain works is that it creates those new frames using the same process as MCompensate and takes averages of pixels in temporal dimension (or ignores them if SAD is too bad). It also has the thSAD2, thSADC2 parameters to give the closest frames more weight in the average calculation. It should be possible to recreate the exact functionality using MCompensate and Average(). "

MDegrain works on per-block basis. For each block of current frame it fetches block of ref frame and shift (translates) to align with current block and check for SAD - it SAD below thSAD - it calculate non-zero averaging weight and use this block in averaging process (actually all ref blocks are used but blocks with zero weight add nothing to output result - it is to make processing a bit simple, so processing speed do not depends on number of zero-weighted blocks). But with overlap enabled there is more blocks shifted a bit (more than simple (frame_width/block_sizeH)x(frame_height/block_sizeV)). So I not sure if MCompensate may give exact result.

The real sad truth about current mvtools-mdegrain: it currently mostly tool to detect static and somehow moving blocks and sometime if moving block is mostly Translate-transformed only - it can also add it to averaging and degrain. But if moved (transformed) block is somehow other way transformed and still have SAD below thSAD - it also averaged but processing cause blurring because the transform is not good (totally/best) reversed (compensated).

[Dogway]

I did two different tests (for optical flow purposes both). One simply output the SAD mask to give a SAD score (script here https://pastebin.com/4QTLzrVy), and second took an HFR clip from Youtube, tonemaped it, downscaled it to 1080p. Then compared the original frames from the newly interpolated ones. The first test didn't translate well for MDegrain and not much either for MFlowFPS() so I think the second test is more suited.

For MFlowFPS() for example you want higher search ranges because the primary issue with it is high motion objects so you want to extend the search range. With MDegrain() I observed that anything higher than searchRange=2 smears shading a lot specially with truemotion=false, so I'm trying to find a sweet spot with truemotion=true settings. Yes you can use low frequency restoration (as seen in SMDegrain) but that will also restore DCTFlicker. With MCompensate() you still need the higher searchRange to do the motion compensation and probably DCT=1 will also be worth it there where for MDegrain it doesn't make such a visible change to be worth the performance hit, let's say MDegrain is more indulgent with the motion vectors.
So to sum it up:
Motion Interpolation clients: high fidelity vectors + high search range (blending okeish)
MDegrain: medium fidelity vectors + low search range (blending or smearing not bad)

By the way I'm refining by chunks, simply to get some correlations but I actually managed to raise the score to 146.1026

Check with this as a starting template:
thSAD = 245
thSADR = 219
thSADC = 185
BlkSize = 16
BlkSizeR = 8
overlap = 8
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = true
truemotionR = true
temporal = true

# TRUEMOTION SETTINGS
lambda = 310
lambdaR = 960
pnew = 114
pnewR = 136

pzero = 134
lsad = 6000
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79
lvl = 1

# lambda is not used for global predictor
pglobal = 8

badrange = 2
badSAD = 1350
dct = 0
dctre = 9
rfilter = 3
searchAlgo = 1
searchAlgoR = 4
searchRange = 8
searchRangeR = 11
searchRangeFinest = 12
sglobal = true


I can help you with the correlation matrix by calculating you more data. Since the tests require purely random data perhaps I could add an option for that in Zopti. Of course since you already have set specific ranges for the parameters it's not completely random but assuming those ranges cover all sensible use cases it should work. And it would be too difficult to find any real correlations with full ranges anyway.

Thanks. Currently I'm running these small tests to release new SMDegrain but later I want to do the full correlation map. It only makes sense if the score is in the highs, so all data samples below a given threshold (ie. if a known setting hampers scores in all cases) are discarded. Then I divide data samples by same score blocks so the correlation can be done at several points for more certainty, and finally do the median (I like the interquantile mean though). I'm not going to say it's a perfect solution but it's a start, after that you only know how much linearity correlation there is, but not the linear factor so I will try to do some regressions as I did above with thSAD. For me thSAD is a first class setting so thSADR and thSADC are derived from it, other first class settings include searchAlgo, searchrange and DCT. I'm trying to look for more of this kind, probably also lambda, pnew and pzero. Don't forget to lock thSADC (my bad) as GMSD score only uses Y plane.

I'm not sure I follow, the sensitivity only cares for the first optimized parameter, which usually is the quality. If you want to use the sensitivity calculations perhaps I should give an option to limit how many results to include, right now it's the last two generations that are used. However it would take too long to calculate the sensitivity from much larger result set because it's O(N^2) algorithm.

So currently it only optimizes the score? I thought it was also looking at the time to find the best score/time relation (pareto front)

[DTL]

[It looks 'pelsearch' is control radius at level=0, edited]

Also if using sub-pel processing modes it may be good to compare different interpolation algoriphms. Built-it 0,1,2 not looks as best. The current versions allow to provide external pelclip with any other upsizing algoriphm.
Also the MSuper(sharp=2) is documented as 'for sharper Wiener interpolation (6 tap, similar to Lanczos)' - but its kernel not looks like sinc-based (natural Lanczos kernel):
https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/Interpolation.cpp#L2094

pDst[i] = std::min(max_pixel_value, std::max(0, ((pSrc[i - 2]) + (-(pSrc[i - 1]) + (pSrc[i] << 2)
+ (pSrc[i + 1] << 2) - (pSrc[i + 2])) * 5 + (pSrc[i + 3]) + 16) >> 5));

So in the current development build I will try other sinc-based kernel sub-pel shifting at processing time. The users of old builds may try to provide 'pelclip' externally resized. But the best interpolation algoriphm may greatly depends on the 'conditioning' of source to process. With 'badly conditioned' sources sinc-based may give too much ringing and worse results. It is one of the sad place when going to the best quality sub-pel transformations refining/compensation. Most of other transforms compensating like Rotate is mostly based on sub-pel samples recalculating and even can not have any 'full-pel' search/compensate mode available at all. So the interaction of the sub-pel interpolation algoriphm with actual spectral distribution of the source footage will greatly change the performance of any sub-pel-based search/compensation degrainers.

[Dogway]

Thank you DTL, this might explain why fine details get warped in high motion right? I wonder what are the alternatives by looking into other optical flow solutions, some search should always be performed at finest level. In the code 'searchparam' is clamped to 'pelsearch' at finest level, maybe that's why 'pelsearch' optimizes to a high number like 17. I'm curious to know if then at finest level 'pelsearch' is no-op.

For resizing lately I'm digging blackman resize, it's like lanczos but with less ringing so it's ok to use 6 or 8 taps and get a sharp output with moderate ringing. I searched for blackmanminlobe but it's only a thing of fmtconv, couldn't find any literature about it. I have no idea how the internal Wiener kernel is supposed to look like, maybe it's worth providing a pelclip in this case.


EDIT: reading your previous post, it's interesting how the internals work, as always the simplicity of the concept stands out. One question, for many of those transformations shouldn't the block size be adaptive? Also as I see it it's practically a bunch of matrix operations, maybe this task can be accelerated by the GPU? I think with the implementation of scaling and horizontal transforms the motion estimation would greatly improve.

[DTL]

"why fine details get warped in high motion right?"

It depends on both motion speed and other reasons.

Currently if all previous levels fail to produce really best motion vector - the refining at finest level (after checking all possible predictors) is limited to 1 full-pel (distance between samples in input source). So it speed of movement > 1 sample per the time between frames it may fail to reach best position (at the last refining stage). Ofcourse it is expected the all previous stages (predictors + hierarchical refinement) will provide already best vector and no more refining is needed. But it may not every time be true.

Other reasons - the fast motion may cause also other transformations (like Skew because of rolling-shutter in some cameras and many more) that can not be compensated with current engine (using only 'solid-Translate' block compensating). And if thSAD is set too high - it will cause averaging/blending of not ideally all-transforms compensated blocks and blur of details.

Some idea to think about: Is it possible to separate SAD value increasing from non-ideal reverse-transform (transforms compensating) of blocks from natural noise SAD value increasing ? Currently engine can not determine if ref block is not perfectly match in shape/position (total geometric compensation is done) the src block and may perform failed blending. The single SAD-based decision of either put block in blending or not looks far from perfect for 'ideal denoiser'. May be other metric is possible to detect if 2 noised blocks are not good compensated so it is better to pass more noise and keep more details (skip badly compensated ref block from blending).

Currently single scalar SAD value control too much in the denoising engine and this cause fails sometime. Need to add more complex metrics/algoriphms.
In the latest testbuilds the testing of MVs length was added in algoriphm of selecting block to blending or not. Idea was: Typically static and very slow moving blocks are less distorted and simple Translate compensation can be enough (that equal to zero MV for full static block). Though if lighting changes it is also not best decision. Or other symmetrical transforms like Skew may give zero Translate MV value, or slow rotation around center, etc.

"maybe that's why 'pelsearch' optimizes to a high number like 17"

Oh - that is strange. The documentation also says: "and pelsearch is the radius parameter at finest (pel) level. " . And 'pelsearch' should be equal to 'pel' default. But at debugging I typically got searchparam =1 passed to PlaneofBlocks->SearchMVs at level=0 even if using searchparam=2 (default). Need to check it again. If pelsearch of 17 is passed to level=0 search even with pel=1 it will cause extremely great loss of speed.

Well - it looks the 'pelsearch' can really set very high search radius at level=0. But I never use it. So the first part of post https://forum.doom9.org/showthread.php?p=1963476#post1963476 looks invalid and I will delete it.

If large 'pelsearch' value really make better result - it mean the previous levels of hierarchical search significantly fails to provide good predictor for final search more or less frequently.

[zorr]

If the vectors are perfect then GMSD(n, MCompensate(n+1)) will be zero. "

It is unlikely for natural scenes.

Indeed. It's so unlikely that if the GMSD score 0.0 (perfect match) is returned it is assumed to be an error and the worst possible score is returned for that frame instead. It's not a theoretical situation either - at least some older versions of MVTools sometimes returned the reference frame instead of the MCompensated frame in rare error cases which resulted in that perfect score so I had to add this check. I haven't seen it happen in more recent versions of MVTools but it's better to be safe. So if you have wondered what this line
global GMSD = (GMSD == 0.0 ? 1.0 : GMSD)
is doing in the script, this is the reason.

The 'vector' compensation only cover more or less good Translate transform of possible basic Scale/Rotate/Translate transforms of object.
...
So it is not possible to expect ideal result from the still very limited tool.

I wasn't implying that perfect score would be expected, just that the GMSD score can tell a lot about the quality of the motion vectors. And especially it should be a good match for MDegrain.

Also the Translate transform may be treated as special case of Rotate transform with very low angle and very large radius (distance to origin).

Yes this is probably why the translate alone can be very successful. Most of the motion shown in video is so slow that curved motion paths can be quite accurately approximated with linear segments where transform is all you need.

[DTL]

" Most of the motion shown in video is so slow that curved motion paths can be quite accurately approximated with linear segments where transform is all you need. "

It may about work with very low tr values - like for sequential frames only (tr about 1). But for good degrain we need tr as large as possible and with increasing frame-distance rotation quickly cause great mismatching of ref and src blocks (SAD increases and far-tr blocks got zero weight and stops add to the output in good case or start to smooth outout in worse case). So increasing of tr value do not add to degraining any more - only waste time.
Currerntly MDegrain check current block vs all blocks in tr-scope. I calculate for block 8x8 and rotation around center - the qpel shift at the middle of edge starts with rotation angle about 3.6 degrees (arctg((1/4)/4)). It looks about 3..4 degrees is the lowest granularity for rotation angle for rotation search. For blocksize 16x16 it about 2 times lower - about 1.5 degrees.

Also rotation (around Z-axis) is not only about curved path - the curved path may be OK if block keeps it orientation relative to frame sampling grid.

It looks complex transforms quickly make increasing of tr-value useless for getting better degrain result. So best results with 'large' tr values typically got at the scenes with lots of static areas. Where the motion-search engine simply confirms the block is static or compensates the very slight residual movements of camera at tripod.

[zorr]

I did two different tests (for optical flow purposes both). One simply output the SAD mask to give a SAD score (script here https://pastebin.com/4QTLzrVy), and second took an HFR clip from Youtube, tonemaped it, downscaled it to 1080p. Then compared the original frames from the newly interpolated ones. The first test didn't translate well for MDegrain and not much either for MFlowFPS() so I think the second test is more suited.

I have also thought about using the SAD mask alone as the optimization criteria. But there's something weird about the output of MMask(kind=1). I think it should output one color brightness per block (the SAD of that block) but the output is not a low-resolution grid but a blurred one. Perhaps it's doing some kind of interpolation as well. DTL, can you take a look at it? :)

MCompensate should be a much better match for MDegrain. Neither of them do motion interpolation whereas MFlowFPS does. It's more difficult task to create good vectors for motion interpolation where the coherence of vectors is more important.

For MFlowFPS() for example you want higher search ranges because the primary issue with it is high motion objects so you want to extend the search range. With MDegrain() I observed that anything higher than searchRange=2 smears shading a lot specially with truemotion=false

The main difference is that while MDegrain can be picky and only accept high quality vectors (with low SAD value) MFlowFPS has no choice, it has to use the best motion vectors that were found. Therefore MFLowFPS needs more extreme settings to find those good enough vectors. Those hard-to-find vectors MFlowFPS needs may still be too low quality to be useful for MDegrain so it makes no sense to even try that hard, therefore lower search range works better.

Yes you can use low frequency restoration (as seen in SMDegrain) but that will also restore DCTFlicker.

Sorry I'm not familiar with the concept of low frequency restoration and DCTFlicker, can you elaborate those?

With MCompensate() you still need the higher searchRange to do the motion compensation and probably DCT=1 will also be worth it there where for MDegrain it doesn't make such a visible change to be worth the performance hit, let's say MDegrain is more indulgent with the motion vectors.

Yes agreed and I see your point why MCompensate may not be that useful for finding vectors for MDegrain. What should be done is have MCompensate also reject too high SAD blocks (with the thSAD parameter) but I'm not sure how to tell which thSAD value to use. Perhaps we could calculate optimal parameters for a few different thSAD values and SMDegrain could use those depending on which thSAD user has selected?

By the way I'm refining by chunks, simply to get some correlations but I actually managed to raise the score to 146.1026

So overlap was changed from 4 to 8, also pzero is outside the original range. That's why I like to start the optimization with all parameters unlocked and with quite large ranges... it does take longer to get results though.

Thanks. Currently I'm running these small tests to release new SMDegrain but later I want to do the full correlation map. It only makes sense if the score is in the highs, so all data samples below a given threshold (ie. if a known setting hampers scores in all cases) are discarded. Then I divide data samples by same score blocks so the correlation can be done at several points for more certainty, and finally do the median (I like the interquantile mean though). I'm not going to say it's a perfect solution but it's a start, after that you only know how much linearity correlation there is, but not the linear factor so I will try to do some regressions as I did above with thSAD. For me thSAD is a first class setting so thSADR and thSADC are derived from it, other first class settings include searchAlgo, searchrange and DCT. I'm trying to look for more of this kind, probably also lambda, pnew and pzero.

Ok so if you provide the script I can run some results and give you the output. Do you think it would help if Zopti had a setting where it only tries random settings (still making sure they pass the restrictions stated in the script, of course)? Right now you can do that by creating a large population but it's not optimal, for example the creation of new random settings is single threaded and I think the resolve phase should kick in immediately if it's known that it always takes over 100 tries to find random parameters which pass the restrictions. And I think doing the rest of your process in Zopti could also be possible, not sure how you take the median though.

Don't forget to lock thSADC (my bad) as GMSD score only uses Y plane.

Good catch, I'll disable it for the next run.

So currently it only optimizes the score? I thought it was also looking at the time to find the best score/time relation (pareto front)

Yes the pareto front is based on both quality and speed (or whichever properties you have defined in your script). However the sensitivity estimation (I think we were discussing that and not pareto) only cares about the first property, usually it's the quality. The sensitivity estimation is just used to select which parameters are mutated and you can turn that feature off if you want.

[zorr]

But for good degrain we need tr as large as possible and with increasing frame-distance rotation quickly cause great mismatching of ref and src blocks (SAD increases and far-tr blocks got zero weight and stops add to the output in good case or start to smooth outout in worse case). So increasing of tr value do not add to degraining any more - only waste time.

Hmm.. perhaps this could be tested on script level. Create several rotated and/or scaled versions of the frame and do motion interpolation between original and the rotated/scaled future/past frames. Then you'd need to select the best motion interpolated block from all of these variations based on the SAD (using MMask). Some blending would be needed in order to avoid harsh edges. And oh boy that would be slow and take tons of memory!

[Dogway]

https://i.imgur.com/iLn521l.png
This is the pruning I plan to do with the results. I only care where the action is taking place (the blue shaded area), but for correlation analysis I will only take the upper half (yellow shaded).

I will divide the yellow area in horizontal slices of same GMSD score and do the correlations.

If that area is full of say pnew=114 by refinement convergence I'm not going to take any stat from it (it's biased), so I need pnew to get there by randomness. If it happens to be all pnew=114 I would know then that it's a parameter that is first class and probably shouldn't be exposed (no quality/performance gain from it).

When all correlations and stats are done we test the results with another clip.
I will probably run the Spearman's correlation and mutual information, RDC or CoS. I read about PCA but it only describes the orthogonal vectors of the covariance ellipsoid, I'm not sure what to do with that. With high Spearman's correlations I will go one by one, plot it and fit a function.


Let me cook the script.

EDIT: The script. You might want to exclude 1 from DCT and DCTre since it will slow down things a lot, same for 3 for searchAlgo and searchAlgoR. Also starting iterations with script settings so we have a high score reference.

setmemorymax(16384/8)
DGSource("Brazil-cleanREF.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=AssumeFPS(24000,1001)
DGSource("Brazil-prefiltered.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
pre=AssumeFPS(24000,1001)
DGSource("brazil2.dgi",cl=0,ct=40,cr=0,cb=40)
AssumeFPS(24000,1001)


thSAD = 245 # optimize thSAD = _n_ | 150..300 ; filter:x 5 % 0 == | thSAD
thSADR = 219 # optimize thSADR = _n_ | 150..300 ; filter:x 5 % 0 == | thSADR
thSADC = 185
BlkSize = 16
BlkSizeR = 8
overlap = 8
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = true
truemotionR = true # optimize truemotionR = _n_ | false,true | truemotionR
temporal = true # optimize temporal = _n_ | true,false | temporal

# TRUEMOTION SETTINGS
lambda = 310 # optimize lambda = _n_ | 100..2000 ; filter:x 20 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 960 # optimize lambdaR = _n_ | 300..3000 ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114 # optimize pnew = _n_ | 20..200 ; filter:x 2 % 0 == | pnew
pnewR = 136 # optimize pnewR = _n_ | 20..200 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 134 # optimize pzero = _n_ | 0..200 ; filter:x 2 % 0 == | pzero
lsad = 6000 # optimize lsad = _n_ | 1000..8000 ; filter:x 50 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79 # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1 # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 8 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2 # optimize badrange = _n_ | 0..50 | badrange
badSAD = 1350 # optimize badSAD = _n_ | 0..3000 ; filter:x 10 % 0 == | badSAD
dct = 0 # optimize dct = _n_ | 0..10 | dct
dctre = 9 # optimize dctre = _n_ | 0..10 ; filter:x 1 != | dctre
rfilter = 3
searchAlgo = 1 # optimize searchAlgo = _n_ | 0..5 ; filter:x 3 != | searchAlgo
searchAlgoR = 4 # optimize searchAlgoR = _n_ | 0..5 ; filter:x 3 != | searchAlgoR
searchRange = 8 # optimize searchRange = _n_ | 0..20 ; filter: dct 0 > dct 5 < and x 5 < true ? | searchRange
searchRangeR = 11 # optimize searchRangeR = _n_ | 0..20 | searchRangeR
searchRangeFinest = 12# optimize searchRangeFinest = _n_ | 0..20 | searchRangeFinest
sglobal = true # optimize sglobal = _n_ | true,false | sglobal


trim(1,12)+\
trim(60,96)+\
trim(173,285-12)


C=ConvertBits(16,fulls=false)

Pre = pre #C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true

superfilt = MSuper(pre, hpad=16, vpad=16, sharp=sharp, rfilter=rfilter, pel=pel, mt=true) # all levels for MAnalyse
superR = MSuper(C, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=true, chroma=false)
superRe = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=true)
bak2 = MAnalyse(superfilt, isb=true, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 = MAnalyse(superfilt, isb=true, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 = MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 = MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=true, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe, bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
bak1 = Recalculate ? MRecalculate(superRe, bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
fwd1 = Recalculate ? MRecalculate(superRe, fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe, fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=true, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=true)



resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFileStart(resultFile, "FrameCount()")

GMSD(ConvertBits(32, fulls=false, fulld=true), src.ConvertBits(32, fulls=false, fulld=true), Y=true,U=false,V=false)

global GMSD = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
""",local=false)

global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

global delimiter = "; "
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")



Sorry I'm not familiar with the concept of low frequency restoration and DCTFlicker, can you elaborate those?

You apply a highpass on the smeared clip and add that to a lowpassed reference.
This seems good in principle but low frequency detail is not stable in the reference (grainy) clip because as per Didée's words DCT compression propagates grain detail into the low frequencies which he calls DCT flicker. A further DCTFlicker pass is necessary, you can do this as temporal denoising the lowpassed clip, use a debanding filter or a mix of both.

I made it into a function.
function ex_LFR (clip c, clip ref, int "LFR", int "UV") {

w = c.width ()
h = c.height()
LFR = Default( LFR, 300*(w/1920.)) # Default is 300 for 1080p
UV = Default( UV, 1)

LFR = max(LFR,50)
Fs = max(w,h) * 2 # Frequency sample rate is resolution * 2 (for Nyquist)
k = sqrt(log(2)/2) * LFR # Constant for -3dB
LFR = Fs / ( k * 2 * pi ) # Frequency Cutoff for Gaussian Sigma

ex_makeadddiff(c, c.ex_gaussianblur(LFR, UV=UV), ref.ex_gaussianblur(LFR,UV=UV), UV=UV) }



https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/Interpolation.cpp#L2094

pDst[i] = std::min(max_pixel_value, std::max(0, ((pSrc[i - 2]) + (-(pSrc[i - 1]) + (pSrc[i] << 2)
+ (pSrc[i + 1] << 2) - (pSrc[i + 2])) * 5 + (pSrc[i + 3]) + 16) >> 5));


Is it possible to convert that to fmtc_resample impulse coefficients to have a look?

[DTL]

In the same file it is commented as:
https://github.com/pinterf/mvtools/blob/d8bdff7e02c15a28dcc6e9ef2ebeaa9d16cc1f56/Sources/Interpolation.cpp#L1879
// so called Wiener interpolation. (sharp, similar to Lanczos ?)
// invarint simplified, 6 taps. Weights: (1, -5, 20, 20, -5, 1)/32 - added by Fizick


Have more ideas how to keep fine low-contrast details from blur:
It may blur because 2-frames motion search engine may return noise-distorted motion vectors if we even have totally static frame. It can be checked with static frame clip + AddGrain and look at motion vectors with MShow().
The 'penalty' settings in MAnalyse may be also attempt to decrease noise-based false-motion but MAnalyse have typically only access to pair of frames and can not perform more complex motion vectors processing in time-dimension.
Only MDegrainN have access to all motion vectors in tr-scope and only at this stage we can make additional refining (post-processing) of motion vectors before usage in motion compensating and weighted-blending.
Current ideas: To make 'denoising' of motion vectors in tr-scope either using low-pass filtering or +additional threshold-based correction of too deviated motion vectors from mean (calculated from low-pass filtering of a sequence of motion vectors in the all tr-scope for current block). But it need some thinking how to control low-pass filtering from user-params without overloading user with lots of new params to adjust (one way may be to provide impulse coefficients of filter as array of params).

[Dogway]

Thanks
imp=[0.03125, 0, -0.15625, 0, 0.625, 1.0, 0.625, 0, -0.15625, 0, 0.03125]
a=fmtc_resample(1280*2,634*2, css="420", kovrspl=2, cnorm=true, center=true, kernel="impulse",impulse=imp)

Looks like a softer version of spline64. I don't know, normally one wants the pelclip of the motion search to be on the soft side, after all we do prefiltering to clean edges. But for the pelclip of the MSuper to be fed to MDegrain it could be improved with for example blackman taps=8, or should they use identical scaler?

I managed to clean some blotches from the ex_lfr() function by running it through a limiter. It's not perfect but there's an improvement, in any case I'm testing with anime which might be easier. Another option is to run ex_SmoothGrad() but maybe a little slow in the global scope.

ex_makeadddiff(c, lpc, !DCT ? lpr : ex_limitdif(lpc, lpr, thr=0.3, elast=4.0, UV=UV), UV=UV)

[DTL]

should they use identical scaler?"

They can use different sub-sample interpolation.

For MAnalyse it can be tuned to best motion vector search and for MDegrain it must be tuned for best sub-sample image data interpolation. Because with pel >1 it directly go in the output image result. But best interpolation is depend on input data spectrum. Unfortunately we still have no any one standard on interpolation processing of digital moving pictures data. I think it is typically implied to be something about sinc-based interpolation from times of hybrid analog/digital mixed systems. So for 'best' sharpness it can go up to 'pure sinc' like SincResize/SincLin2Resize. Unfortunately for small block-based interpolation it may be too truncated sinc kernel and too highly distort edges of small block with small margings (to have fast small kernel convolution) with Gibbs. So I currently set Lanczos-weighted sinc into SO=5 MAnalyse (still not finally debugged / fine tuned).

Typically user must adjust interpolation method of MDegrain to current input content to check if it not too distort it and max possible sharpness reached. If too smooth interpolation used - it will also cause decreasing sharpness on any moved blocks (down to half-sample moves with small sharp details).

Currently the interpolation is provided via 'super' clip and it can be different for MAnalyse and for final output MDegrain. Also currently SO=5 MAnalyse uses internal runtime interpolation for SAD calculation stage using about Lanczos 8 samples kernel (+-4) to decrease data traffic from memory.

[Dogway]

While testing in deep_resize() I noticed SincResize with 4 taps applies some kind of local contrast on lower frequencies so a small deconvolution had to be performed. For utmost sharpness this is fine but for speed I found blackmanresize to be visually more pleasant with 8 taps than lanczos4 which showed some ringing and aliasing.

[DTL]

"SincResize with 4 taps applies some kind of local contrast on lower frequencies "

*In a perfect world* SincResize must have as much taps as possible. But it require *perfectly conditioned content* (+processing in the conditioning domain that is not standard defined - it may be either linear data or system-transfer transformed data). So the selection of interpolation method is greatly depends on input content - for some badly conditioned may be only Bilinear/Bicubic is good. Or some non-linear (nnedi ?).
The internal math of MSuper/MDegrain is about linear processing and for best results it may be good to feed in with linear (that is typically RGB) data, not system-transformed. Also the linear averaging operation on system-transformed data is not very good. The natural photon noise is distributed in linear domain and after conversion to non-linear transfer its distribution become distorted. So linear averaging may be less effective ?
Also the interpolation operation for sub-sample precision processing better works (less ringing and other distortions) in its target conditioning domain that may be linear (may depend on the source of current digital moving picture content).

So better workflow of MAnalyse/MDegrain may be:
1. Feed standard system-transformed YCrCb data to MAnalyse (via first MSuper filter). Motion search may better work with system-transfered data ?
2. Feed linear RGB data to MDegrain (via second MSuper filter producing second super clip).
3. Transform/pack back to system format (transfer/matrix/chroma_subsampling) when required.

Much less questions is if we process with pel=1 (not going into sub-sample precision) but it produce less quality with moving fine detailed objects (make more blurring and/or distort motion).

[zorr]

Let me cook the script.

EDIT: The script. You might want to exclude 1 from DCT and DCTre since it will slow down things a lot, same for 3 for searchAlgo and searchAlgoR. Also starting iterations with script settings so we have a high score reference.

Thanks, I will get started tomorrow, getting a bit late today. :) So I assume these are using the same clips as before and you are also loading the preprocessed clip?

[zorr]

Dogway, I did a test run with the script, just 24 results. Some observations:

Your best settings resulted in GMSD score 146.05907, not 146.1026. Not sure where the difference is coming from. It could be that the prefiltered clip is different somehow. I saved it as Lagarith from VDub using the same format as the source and DGIndexNV is not able to read that one so I left it as Lagarith.
There was already a better combination with score 146.07826.
It took 1h 34 minutes to run those 24 tests. Mainly because there were some settings that took almost an hour and a half to process. So I guess we should consider adding a timeout, I don't think anyone is patient enough to wait that long for 150 frames let alone for a whole movie... at that rate a 25 fps 90 minute movie would take 56 days. :eek:
Searchalgos 6 and 7 are pure horizontal and vertical exhaustive searches, they might be useful for some specific clips but never in general use. Those could be dropped.

[Dogway]

I encoded the prefiltered clip with transparent settings in x264, with --opencl (makes a slight difference). Now I realize and forgot to change "--input-range tv" to "pc", don't know if that makes a difference since it's only a tag. Also check avstp.dll is not in plugin path.
SET PROFILE= high
SET BITDEPTH= 8
SET PRESET= veryslow
SET TUNE= film
SET ME= umh
SET QP= 8

I repeated the run and got again 146.10242, a bit lower because some value rounding per my rules.
146.10242 90230 thSAD=245 thSADR=220 truemotionR=true temporal=true lambda=320 lambdaR=960 pnew=114 pnewR=136 pzero=134 lsad=6000 plevel=79 pglobal=8 badrange=2 badSAD=1350 dct=0 dctre=9 searchAlgo=1 searchAlgoR=4 searchRange=8 searchRangeR=11 searchRangeFinest=12 sglobal=true

I don't know why the difference, I guess you are using my (latest) GMSD version (https://github.com/Dogway/Avisynth-Scripts/blob/master/EX%20mods/SimilarityMetrics.avsi). In any case I upload my prefiltered clip here (https://www.mediafire.com/file/okkn37nq9kd6gzi/Brazil-prefiltered.mkv/file). If you still get some different scores I wouldn't worry since we only care about relative scores.

For the high times, did you take out DCT and DCTre=1 and searchAlgo and searchAlgo=3? Yes I forgot about those (vertical and horizontal) while expanding the ranges, I edited my script (https://forum.doom9.org/showthread.php?p=1963517#post1963517) (also with below DCT rules).
I gave a look at my charts with those taken out and probably the culprit is DCT (DCTre seems fine), anything in 0 < DCT < 5 is several times slower. My fastest one with that was 525510ms, too slow. The only way to alleviate times with those DCT is to lower searchRange below 5. In any case if you can send me the output samples, I'm also going to run tests and fill the charts so we can have a broader look even with higher DCT values.
I also realized that very long times can add a bias to the correlation because those ones will add less samples to the data set. In any case though as I said I will pick only the parallelogram of the inflection area of the pareto front, so high times will naturally be taken out.

EDIT: Got a new record (and this is using random iterations, so I'm a bit surprised):
146.10953 314810ms thSAD=220 thSADR=175 truemotionR=true temporal=false lambda=160 lambdaR=2420 pnew=64 pnewR=22 pzero=122 lsad=2650 plevel=39 pglobal=12 badrange=29 badSAD=2700 DCT=10 DCTre=7 searchAlgo=0 searchAlgoR=2 searchRange=13 searchRangeR=11 searchRangeFinest=8 sglobal=false


EDIT2: I did a random run of 230 iterations, calculated the parallelogram section and all DCT with 1, 3 or 4 were out (because of very bad quality/speed ratio). I think it is safe to assume to take those out so instead of lowering searchRange for those we can simply drop them. With this it will run fast enough (DCT=2 a bit slowish though).

From this:
dct = 0 # optimize dct = _n_ | 0..10 | dct
searchRange = 8 # optimize searchRange = _n_ | 0..20 ; filter: dct 0 > dct 5 < and x 5 < true ? | searchRange
to this:
dct = 0 # optimize dct = _n_ | 0,2,5,6,7,8,9,10 | dct
searchRange = 8 # optimize searchRange = _n_ | 0..20 ; filter: dct 2 == x 5 < true ? | searchRange

[zorr]

For the high times, did you take out DCT and DCTre=1 and searchAlgo and searchAlgo=3?

Yes the script version I used already had a filter to reject those.

probably the culprit is DCT (DCTre seems fine), anything in 0 < DCT < 5 is several times slower. My fastest one with that was 525510ms, too slow. The only way to alleviate times with those DCT is to lower searchRange below 5.

I run some 700 random results without timeouts, only dct=1, dctre=1 and searchAlgo 3 rejected. Here's a table of the fastest and slowest time per dct (also included a colum with the dctre value of the slowest result to see if it's always 2-4).

dct fastest slowest slowest dctre
0 33470 191010 2
1 - - -
2 373470 7703930 0
3 294230 8109110 3
4 291710 7540350 3
5 38190 270050 0
6 45160 300320 9
7 41280 390440 2
8 40100 342090 2
9 39090 248980 10
10 40250 291160 2


So yes looks like dct 2-4 (and 1 even though it wasn't tested) are much slower. But there can be pretty slow results with the other dct values as well, 390 seconds (with dct=7) is already 6,5 minutes. And if we allow that much time there are already results in dct 2-4 which run faster than that so rejecting them outright might seem unfair.

I guess the real question is can the dct 2-4 be competitive in such a restricted time limit. In the pareto front of those 700:ish results none used dct or dctre 2-4 so that indicates they might not be, but it's a bit limited amount of data for my taste.


The distribution of runtimes looks like this:

https://i.postimg.cc/PJJYP6Cy/runtime-distribution.png

35% is under 100 seconds, 62% is under 200 seconds, 68% is under 300 seconds. [EDIT these were incorrect for 200 and 300 seconds]

Another way to look at this graph is to think of the surface area as the total time to calculate results. The vast majority of time is going for those last few % with the slowest execution times.

With a time limit that problem could be solved in a fair manner, doesn't matter which dct is used. On the other hand with a time limit the ones which fail to execute under the limit are just wasted time as we will not get a result for them. But with a limit of say 200 seconds means 38% of results is wasted like this. Also in the pareto front the best (and slowest) executed in 169 seconds so looks like we should be able to get pretty good results in under 200 seconds. I think we could calculate which time limit maximizes the number of useful results...

I also realized that very long times can add a bias to the correlation because those ones will add less samples to the data set. In any case though as I said I will pick only the parallelogram of the inflection area of the pareto front, so high times will naturally be taken out.

The purpose of this exercise is not to be completely fair but to find correlations which you can use to automate some parameters. :) The range we use in the search is already "not fair". We can and should also add other limiting factors to maximize the number of useful results because nobody will use the kind of parameters that take two months to denoise a movie.

EDIT: Got a new record (and this is using random iterations, so I'm a bit surprised):
146.10953 314810ms thSAD=220 thSADR=175 truemotionR=true temporal=false lambda=160 lambdaR=2420 pnew=64 pnewR=22 pzero=122 lsad=2650 plevel=39 pglobal=12 badrange=29 badSAD=2700 DCT=10 DCTre=7 searchAlgo=0 searchAlgoR=2 searchRange=13 searchRangeR=11 searchRangeFinest=8 sglobal=false

In my MCompensate test I had to adjust the allowed parameter ranges multiple times to arrive at the optimal results. I think the same may apply here and we still don't quite know what the optimal range is.

EDIT2: I did a random run of 230 iterations, calculated the parallelogram section and all DCT with 1, 3 or 4 were out (because of very bad quality/speed ratio). I think it is safe to assume to take those out so instead of lowering searchRange for those we can simply drop them. With this it will run fast enough (DCT=2 a bit slowish though).

It's not quite that simple, I had results with dct=2 searchRange 15 faster than searchRange 2, for example. Here are the fastest results with dct=2:

145.35135 373470 thSAD=160 thSADR=230 truemotionR=true temporal=false lambda=1620 lambdaR=1020 pnew=70 pnewR=28 pzero=68 lsad=2400 plevel=25 pglobal=13 badrange=3 badSAD=2540 dct=2 dctre=0 searchAlgo=6 searchAlgoR=7 searchRange=2 searchRangeR=19 searchRangeFinest=3 sglobal=false
145.52867 428370 thSAD=220 thSADR=155 truemotionR=true temporal=false lambda=1200 lambdaR=400 pnew=120 pnewR=196 pzero=178 lsad=4650 plevel=53 pglobal=18 badrange=39 badSAD=1810 dct=2 dctre=10 searchAlgo=7 searchAlgoR=6 searchRange=15 searchRangeR=8 searchRangeFinest=1 sglobal=false
145.22943 428830 thSAD=265 thSADR=285 truemotionR=false temporal=false lambda=1920 lambdaR=2800 pnew=96 pnewR=34 pzero=84 lsad=3750 plevel=67 pglobal=7 badrange=14 badSAD=2480 dct=2 dctre=4 searchAlgo=6 searchAlgoR=6 searchRange=2 searchRangeR=8 searchRangeFinest=4 sglobal=true
145.55482 472610 thSAD=205 thSADR=155 truemotionR=true temporal=false lambda=1320 lambdaR=2840 pnew=88 pnewR=36 pzero=86 lsad=6100 plevel=75 pglobal=19 badrange=0 badSAD=220 dct=2 dctre=10 searchAlgo=6 searchAlgoR=5 searchRange=7 searchRangeR=9 searchRangeFinest=3 sglobal=true
145.43944 479630 thSAD=160 thSADR=205 truemotionR=true temporal=true lambda=1640 lambdaR=380 pnew=140 pnewR=50 pzero=84 lsad=1500 plevel=79 pglobal=19 badrange=23 badSAD=1670 dct=2 dctre=0 searchAlgo=7 searchAlgoR=2 searchRange=10 searchRangeR=8 searchRangeFinest=3 sglobal=false
145.43701 504800 thSAD=170 thSADR=195 truemotionR=false temporal=false lambda=1840 lambdaR=2060 pnew=144 pnewR=76 pzero=196 lsad=5750 plevel=37 pglobal=8 badrange=15 badSAD=2990 dct=2 dctre=10 searchAlgo=0 searchAlgoR=6 searchRange=1 searchRangeR=0 searchRangeFinest=7 sglobal=true
145.41245 525230 thSAD=260 thSADR=230 truemotionR=true temporal=false lambda=580 lambdaR=1440 pnew=100 pnewR=102 pzero=106 lsad=2550 plevel=21 pglobal=14 badrange=40 badSAD=720 dct=2 dctre=4 searchAlgo=7 searchAlgoR=7 searchRange=20 searchRangeR=4 searchRangeFinest=2 sglobal=true
145.64351 562840 thSAD=175 thSADR=150 truemotionR=true temporal=false lambda=340 lambdaR=2980 pnew=50 pnewR=182 pzero=86 lsad=3950 plevel=47 pglobal=12 badrange=36 badSAD=2720 dct=2 dctre=10 searchAlgo=4 searchAlgoR=7 searchRange=1 searchRangeR=19 searchRangeFinest=17 sglobal=false

I will look at the GMSD issue next. Didn't know that you had your own version of it.

[EDIT] I get GMSD 146.10782 with your prefiltered clip. Closer, but still a small difference. Going to try your GMSD version next.

[zorr]

Dogway, I switched to your latest GMSD version, the score is now 146.10785 so only a very minor change (got 146.10782 with standard GMSD). Also tried upgrading to latest Avisynth+ 3.7.2 test 3, that didn't have any effect. MVTools is already the latest version. You're using 64bit version as well, right? It's a bit mysterious but I guess we can go forward even with these slight differences.

The issue with DGSource came back, it seems to only appear when there are enough simultaneous threads using it. I run the tests using 12 threads and each script loads three sources with DGSource. Three out of 12 scripts give incorrect results because one or more of those DGSources pops an error message "Failed to create video decoder (2)" but runs anyway and the GMSD score drops to around 112. I guess I'm going to have to load at least one of the sources with FFVideoSource to avoid this issue. Or I could concatenate two of those sources as one file and split it in the script.

I thought about how to calculate the timeout which gives most results in a given time, it's some kind of infinite fraction which I don't know how to solve so I wrote a simple simulator instead. It takes 100 million samples per timeout using the distribution of execution times I collected and returns the average time per result. The accuracy is about 10-50ms between different runs. The results for 100 - 300 seconds looks like this:

https://i.postimg.cc/bJTChSjf/average-runtime-per-timeout.png

The best average runtime is about 220 seconds using a timeout of 170 seconds. I think using at least 200 would be better so we don't rule out too many high quality combinations. Perhaps the margin should be even larger, what do you think?

Using the timeout of course has a little problem that combining mine and your results is more difficult as the execution times are a bit different on my system and yours. But we can calculate the speed ratio and scale the timeout accordingly.

Do you think that using a timeout is good or do you prefer just dropping dct 2-4? I'll do whatever you think is better, the data is generated for you anyway. :)

[Dogway]

I don't like timeouts much because first you waste time and second how much is too much? you might be ruling out some combinations that we might need.
One solution would be to look at the frame to frame render time, so no time is wasted. Say, abort if the mean time of the 5 first render times are above x, in this case I would set it to a rather high-ish threshold like 500000ms/150

In any case I prefer to limit my times by settings so I know what's going on. If something makes the script too slow, search that up and write a rule. Aside from the settings noted above searchAlgo=5 slow things a bit, like I get 300000ms or some higher on my CPU with DCT=0, but still I think I may want to run with it. So if you know you want to try DCT=2 and searchAlgo=5, just make sure both don't coincide. I mean, this is some kind of a global scope test so we want (a bit) high times in, insanity times out.
One note; whenever possible I observed it's better (more random) to define the range as "0,1,2,4,5" than write a rule like "filter: x 3 !=". Maybe you can have a look if that's correct.

The case where DCT=2 and searchRange=15 was fastest might be due to searchRangeFinest.
Practically the main performance hoggers are DCT and searchAlgo and in second term, searchRange and searchRangeFinest (related to the previous). At least that I know of from which we are trying. I suspect something else is going on with badSAD and/or badRange. If you want tomorrow I can run a test bench for DCT=2 and see what causes high or low times for this one.

EDIT: I wanted to combine results, but if we get different scores it might not mean much. Yes, I'm on x64, latest on everything included DGDecNV (just in case). Check if you have latest ExTools and fmtconv, and the order of the GMSD() clips, it shouldn't matter but just in case, it's defined as GMSD(clip dist, clip ref), so I use GMSD(last, src) references.


Latest script version with the new rules, you can set timeout to 500000ms I doubt you will reach it often or hope so.
And one last thing, is it ok speed wise to use MDSI(), with or without downsample=true? I say because GMSD only cares about structure (edges), but searchRangeFinest affects shading which is low frequency.
setmemorymax(16384/8)
DGSource("Brazil-cleanREF.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=AssumeFPS(24000,1001)
DGSource("Brazil-prefiltered.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
pre=AssumeFPS(24000,1001)
DGSource("brazil2.dgi",cl=0,ct=40,cr=0,cb=40)
AssumeFPS(24000,1001)


thSAD = 220 # optimize thSAD = _n_ | 150..300 ; filter:x 5 % 0 == | thSAD
thSADR = 175 # optimize thSADR = _n_ | 150..300 ; filter:x 5 % 0 == | thSADR
thSADC = 185
BlkSize = 16
BlkSizeR = 8
overlap = 8
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = true
truemotionR = true # optimize truemotionR = _n_ | true,false | truemotionR
temporal = false # optimize temporal = _n_ | true,false | temporal

# TRUEMOTION SETTINGS
lambda = 160 # optimize lambda = _n_ | 100..2000 ; filter:x 20 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 2420 # optimize lambdaR = _n_ | 300..3000 ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 64 # optimize pnew = _n_ | 20..200 ; filter:x 2 % 0 == | pnew
pnewR = 22 # optimize pnewR = _n_ | 20..200 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest, and searchRangeFinest with badRange)
pzero = 122 # optimize pzero = _n_ | 0..200 ; filter:x 2 % 0 == | pzero
lsad = 2650 # optimize lsad = _n_ | 1000..8000 ; filter:x 50 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 39 # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1 # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 12 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 29 # optimize badrange = _n_ | 0..50 | badrange
badSAD = 2700 # optimize badSAD = _n_ | 0..3000 ; filter:x 10 % 0 == | badSAD
dct = 10 # optimize dct = _n_ | 0,2,5,6,7,8,9,10 | dct
dctre = 7 # optimize dctre = _n_ | 0,2,3,4,5,6,7,8,9,10 | dctre
rfilter = 3
searchAlgo = 0 # optimize searchAlgo = _n_ | 0,1,2,4,5 ; filter: dct 2 == x 5 != true ? | searchAlgo
searchAlgoR = 2 # optimize searchAlgoR = _n_ | 0,1,2,4,5 | searchAlgoR
searchRange = 13 # optimize searchRange = _n_ | 0..20 ; filter: dct 2 == x 5 < true ? | searchRange
searchRangeR = 11 # optimize searchRangeR = _n_ | 0..20 | searchRangeR
searchRangeFinest = 8# optimize searchRangeFinest = _n_ | 0..20 | searchRangeFinest
sglobal = false # optimize sglobal = _n_ | true,false | sglobal


trim(1,12)+\
trim(60,96)+\
trim(173,285-12)


C = ConvertBits(16,fulls=false)
Pre = pre #C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true

superfilt = MSuper(pre, hpad=16, vpad=16, sharp=sharp, rfilter=rfilter, pel=pel, mt=false) # all levels for MAnalyse
superR = MSuper(C, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false, chroma=false)
superRe = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
bak2 = MAnalyse(superfilt, isb=true, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 = MAnalyse(superfilt, isb=true, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 = MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 = MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, truemotion=truemotion, lambda=lambda, pelsearch=searchRangeFinest, pzero=pzero, badSAD=badSAD, badrange=badrange, temporal=temporal, lsad=lsad, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe, bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
bak1 = Recalculate ? MRecalculate(superRe, bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : bak
fwd1 = Recalculate ? MRecalculate(superRe, fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe, fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, truemotion=truemotionR, lambda=lambdaR, thSAD=thSADR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=false)



resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFileStart(resultFile, "FrameCount()")

GMSD(ConvertBits(32, fulls=false, fulld=true), src.ConvertBits(32, fulls=false, fulld=true), Y=true,U=false,V=false)

global GMSD = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
""",local=false)

global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

global delimiter = "; "
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")

[DTL]

Starting experiment with denoising of static-image clip I found some feature of current mvtools algoriphm - it sometime converts luma (value) noise into 'phase' noise. I.e. 2D errors of MVs cause shifting of parts of image in 2D space. Also it looks the 'MVs coherency' algorithm in MAnalyse cause random noised blocks to have same 2D error in nearby areas so large enough parts of image got phase-noise shift.

LoadPlugin("mvtools2.dll")
LoadPlugin("AddGrainC.dll")
LoadPlugin("RawSourcePlus.dll")
RawSourcePlus("out8.raw", 308,308, "Y8")
Loop(100)
ConvertToYV12()

ColorYUV(gain_y=-150,off_y=70)

frsrc=last

AddGrain(50)
#return last
padd=8

tr = 2 # Temporal radius
super1 = MSuper (pel=1, chroma=true)
multi_vec1 = MAnalyse (super1, overlap=0, multi=true, search=3, searchparam=2, delta=tr,chroma=true,mt=false)
frp=MDegrainN (super1, multi_vec1, tr, thSAD=1000, thSAD2=1000-1, thSCD1=2000, thSCD2=2000, mt=false)

super = MSuper(pel=4, mt=false)
forward_vec1 = MAnalyse(super, isb = false, search=3, searchparam=2, chroma=false, delta = 1, mt=false, levels=0)
frmsh=MShow(super,showsad=true, forward_vec1, thSCD1=2000, thSCD2=2000, scale=10)

frp=AddBorders(frp, padd,padd, padd, padd)
frsrc=AddBorders(frsrc, padd,padd, padd, padd)

difclip=Subtract(frp, frsrc)
difclip=ColorYUV(difclip, gain_y=300, off_y=-150)

StackHorizontal(difclip, frsrc, frp, frmsh)

out8.raw zoneplate file is https://drive.google.com/file/d/1gjAswPsugMI9EXrea4FQhaGwCcNZ8j1o/view?usp=sharing

Result is
https://i4.imageban.ru/out/2022/02/15/36e5a09f1cdf646140e3b533c866f647.png

at some of frames. It is visible as difference clip shows great correlation with low frequencies parts of the source (not noised original). Also the MVs errors (all MVs here are errors of search) at the low freq parts of image have great correlation though it is static image with random noise.



So do the used GMSD metric more or less sensitive to such types of distortions ? The simple metrics like PSNR will treat such phase-noise as more or less great distortion because significant parts of image is shifted.

The phase-noised image looks good denoised, but may still be not very nice to mpeg compressor.

[zorr]

I don't like timeouts much because first you waste time and second how much is too much? you might be ruling out some combinations that we might need.
One solution would be to look at the frame to frame render time, so no time is wasted. Say, abort if the mean time of the 5 first render times are above x, in this case I would set it to a rather high-ish threshold like 500000ms/150

Yes you need to be careful with the timeout and set it high enough. I already had an idea why the timeout wouldn't rule out important combinations, I guess I should share my data.

https://i.postimg.cc/YSTkrdNW/mdegrain2.png

This is from the random population. You can see the pareto line ends at 170 seconds. Considering you're only going to use the population "where the action is taking place" that would be results with time around 30-80 seconds. With more results the "action area" is going to move towards the top and left meaning the execution times will be even faster. Considering that perhaps a timeout of 300 seconds would surely not prune anything that's really needed.

And when it comes to wasting time, the timeout is of course meant to do the opposite and save time. Even with the timeout of 500 seconds it's going to be a huge chunk (see the professional-quality graph below). The waste is only really there if we reject results that would have been useful if we hadn't aborted.

https://i.postimg.cc/xTZ6FB9M/runtime-distribution-saved2.jpg

The idea of looking at the per frame timings is a good one, I have already investigated a similar idea. The idea was to try to predict if the total quality would be close to the best value found, based on the first frames using statistics. In the end the benefit of the prediction was so small that it wasn't worth the added complexity. This was a bit different situation though because with quality you want to be absolutely sure you don't throw away a good result, with speed there's more potential because there is a much larger spread of values. But I'd like to collect some stats on how well the first few frames can predict the total runtime before I go down that path. Preliminary data indicates the per frame execution times vary a lot so it may not be a sure thing.

In any case I prefer to limit my times by settings so I know what's going on. If something makes the script too slow, search that up and write a rule. Aside from the settings noted above searchAlgo=5 slow things a bit, like I get 300000ms or some higher on my CPU with DCT=0, but still I think I may want to run with it. So if you know you want to try DCT=2 and searchAlgo=5, just make sure both don't coincide. I mean, this is some kind of a global scope test so we want (a bit) high times in, insanity times out.

I like that too because then we'll learn more about the performance characteristics of MVTools. I'm just afraid that the rules to control the runtime may become really complex, and there's also the risk of pruning combinations that would have good enough performance.

One note; whenever possible I observed it's better (more random) to define the range as "0,1,2,4,5" than write a rule like "filter: x 3 !=". Maybe you can have a look if that's correct.

That's correct. When the parameter's value is checked after mutation and it's not valid according to the filter then the closest valid value is chosen instead. This introduces bias. Why a random valid value is not used is that it would no longer respect the original mutation amount. We could just try the mutation again and hope to hit a valid value but that would be very inefficient.

I suspect something else is going on with badSAD and/or badRange.

It's definitely a factor, but only on those frames where the motion vectors are difficult to find and SAD is considered "bad" on one or more blocks. The search is done only to those bad blocks so it may be skipped completely or in extreme cases calculated for all blocks. So the performance depends not only on the chosen settings but also on the contents of the frame (and previous, next frames).

EDIT: I wanted to combine results, but if we get different scores it might not mean much. Yes, I'm on x64, latest on everything included DGDecNV (just in case). Check if you have latest ExTools and fmtconv, and the order of the GMSD() clips, it shouldn't matter but just in case, it's defined as GMSD(clip dist, clip ref), so I use GMSD(last, src) references.

I'll try updating ExTools, that one is a bit older than your latest.

In the mean time I started another optimization with revised ranges based on the first test (below the changes). This is still using overlap = 4 so perhaps even better results can be found with overlap = 8.

# parameter sharp no longer optimized (only had one possible value)
# thSAD range extended from 230..290 to 230..330
# thSADR range extended from 150..300 to 120..300
# lambda range moved from 400..900 to 100..600 (offset -300)
# lambdaR range moved from 960..2000 to 760..1800 (offset -200), maximum changed from "lambda 3 *" to "lambda 6 *"
# pnew range moved from 100..200 to 40..140 (offset -60)
# pnewR range extended from 100..250 to 0..250
# pzero range moved from 28..100 to 58..130 (offset 30)
# lsad range changed from 1000..8000 to 100..6000
# badrange range extended from 0..50 to 0..70
# badSAD range extended from 1100..2200 to 1100..2400
# dctre options changed from 0,2,3,6,7,9 to 0,6,7,9
# optimization of thSADC disabled as GMSD doesn't evaluate the chroma by default


The best result found so far is 146.21552.

https://i.postimg.cc/Twq2YZR4/mdegrain-fast2.png

Latest script version with the new rules, you can set timeout to 500000ms I doubt you will reach it often or hope so.

Based on the latest results you might want to adjust the ranges of pnewR and lsad to include smaller values, the best result has pnewR=2 and lsad=200. The other parameters have the best value inside your ranges.

And one last thing, is it ok speed wise to use MDSI(), with or without downsample=true? I say because GMSD only cares about structure (edges), but searchRangeFinest affects shading which is low frequency.

Not sure, I haven't done performance comparisons between GMSD and MDSI. But I say if MDSI better represents the quality it should be used.

I did a couple of graphs to show the vast difference in speed of dct 2-4 versus others. These are from the same random population. First all the paretos grouped by dct:

https://i.postimg.cc/s22DDnBj/groupby-dct-all.png

dct 2-4 are in their own group far away from the global pareto. This also tells us that a timeout of 500 seconds would actually cut out the best results of dct 2-4. Here you can also see that the filter:x 1 != has caused dct 0 and dct 2 have more samples than the others.

https://i.postimg.cc/VknXWk4r/groupby-dct-5-10.png

Only dct 5-10 here, they are all grouped quite close together. The top quality results are from dct 6 and 9.

https://i.postimg.cc/s23GP9vq/groupby-dct-0.png

Only dct 0 in this one, it dominates the pareto of faster results.

I'm thinking dct 1-4 might not have any representation in the "action" area and therefore we might as well leave them out. Or do another run where they are the only dct values allowed.

[Dogway]

It depends on the (final) inflection curve width, I need enough samples in the horizontal dimension. The slowest "low" pareto time must match the fastest "high" pareto (high and low of the inflection curve), this width segment has also to be applied on the "high" pareto to extract the correct parallelogram of the inflection curve.

I don't think your plot is a good example as it's basically a rectangular corner, there's no convex hull. The only point where things are going on are in the 146.03 slice.

Here's my (no rules) test with a sectioned high to low pareto (still not enough samples to form a convex hull though). In this case I didn't shift-expand the section but I would need to include 250 more seconds of samples to the right (300-50=250).
https://i.imgur.com/jS5k208.png

In the end these are my times with my CPU and all DCT included, so adjust depending on how the inflection curve is gonna end, if you know beforehand otherwise you are biasing. The correlations are going to be as good as the sampling method.

I like that too because then we'll learn more about the performance characteristics of MVTools. I'm just afraid that the rules to control the runtime may become really complex, and there's also the risk of pruning combinations that would have good enough performance.

Check my samples spreadsheet. You can easily see many things I noted before, the fastest with DCT 1, 2, 3, or 4 is DCT=3 with 432230ms, and that's only because DCTRe is 0 and all the ranges are below 5. Do you really want to set that rule? The score was crap anyway.
And among the 76 samples in those DCT there wasn't a single interesting score below 1400s!
https://docs.google.com/spreadsheets/d/10tUQAW-EbP_mTXconeXelYOS-4HA54VTug3aenayonE/edit?usp=sharing

GMSD by default uses "downsample=true", so it is safe to assume MDSI works well also with "downsample=true", and it will be faster. I updated the metrics script yesterday for using downsample when clip is HD. You'd need to adjust timeout if you are going to use it.

[zorr]

It depends on the (final) inflection curve width, I need enough samples in the horizontal dimension. The slowest "low" pareto time must match the fastest "high" pareto (high and low of the inflection curve), this width segment has also to be applied on the "high" pareto to extract the correct parallelogram of the inflection curve.

All right, I trust that you know what you are doing. :)

there's no convex hull.

Just a clarification, you probably know this but the pareto front is not a convex hull even though it's drawn in a similar manner. Actually the pareto boundary is not a straight line between the pareto front points, it should be drawn as two orthogonal lines. I draw it as a straight line to make it easier for the eyes. And naturally convex hull covers all sides which is not the case here. The pareto front is formed from the results which are non-dominated (no other result is at least as good in quality/speed and better in either quality or speed).

Check my samples spreadsheet.

Tried to but it requires access rights, I sent the request.

GMSD by default uses "downsample=true", so it is safe to assume MDSI works well also with "downsample=true", and it will be faster. I updated the metrics script yesterday for using downsample when clip is HD. You'd need to adjust timeout if you are going to use it.

MDSI might be better but perhaps we should try to evaluate which one better represents the quality in this case. We could do a medium length run with both and check which one found the better looking result. In my earlier similarity metric tests I concluded that GMSD is probably better for most cases but it may depend. One fun test would be to optimize for both GMSD and MDSI and visually evaluate the pareto front. That might be even more fair test than two different runs.

I tried your latest ExTools but it didn't change the GMSD score (and it shouldn't have because it's not used when the preprocessed clip is just loaded from the disk). Also updated MaskTools2, same story. I thought I found the culprit when I noticed I have IQA_downsample.avsi which has a different implementation of _IQA_downsample() but that wasn't it either.

Since there is still small variation between our scores I think we should do a small validation test, say 10 random results. I can send you the log of my results and you can repeat the same tests using -mode validate.

A couple of notes about your latest script: I wonder why truemotionR is optimized, perhaps because MRecalculate doesn't have lsad and pnew which are controlled by truemotion? Anyway I tested it and changing it true/false didn't have any effect on the score.

I noticed the plevel has wrong range, the valid values are 0..2.

The ranges look good except I would extend the range of pnewR to zero and lsad to 50 (can't be zero because that's illegal value and must be divisible by 50). This is based on my latest run, for example here's pnewR.

https://i.postimg.cc/520fQsxy/line-pnew-R.png

You mentioned earlier that GMSD favored something that caused smearing. Do you have an example (script + which frame) to demonstrate that? If it looks like I think it does then there might be a way to boost it so that GMSD considers it a more severe artifact.

[Boulder]

In my regular "downscale - upscale" method, GMSD favours a slightly softer result compared to MDSI. I often use it for more grainy sources where the resulting average bitrate would be very high.

[Dogway]

Yes, I checked MDSI and it's pretty much also edge based too. GMSD fared better for this cases in some tests I did.

Send me the test scripts so I can do a -mode validate but I won't be able to check until the night.
I fixed already your range observations, I think I referenced an old documentation for plevels.

[zorr]

Send me the test scripts so I can do a -mode validate but I won't be able to check until the night.

I generated the test cases using your latest script with my modifications. I made a package with the script and the result log. You can download the zip from here (https://drive.google.com/file/d/1szzudGoyxO2N-5pMhQOfPD3m7P9TFzws/view?usp=sharing). Also included is the latest Zopti jar for version 1.2.3 WIP since the validate mode in previous versions only validates the pareto front and here we want to validate all results. Just replace the jar file in Zopti bin folder.

You can run the validation with command
zopti -mode validate -log "path/to/file/mdegrain_validate run-01.log" -threads 10 -count 1 -types avg

Adjust file location and number of threads. Also change the first line of the mdegrain_validate run-01.log file to point to correct .avs file location.

There are 10 results, including some with dct=2. One of them takes about 45 minutes to run even with searchRange=4.

The validation writes two files, in this case they have the same information because count is 1. Usually validation is done with multiple measurements and the other file contains all individual results and the other one contains the average/min/max/median of those multiple results.

If the quality is different than what is found in the log Zopti writes a warning line to console output and in the end the number of warnings. I expect 10 warnings but the interesting thing will be how different the results are.

[Dogway]

Ok, went ahead to check and didn't read your example call haha, it's taking a while (single threaded) but so far 5 validations and in all there's a score difference of 0.005, always lower. I don't know what to think about it... You are running latest GMSD, AVS+, DGSource, MVTools2, fmtconv, and latest ExTools.

Do you want me to re run them with your call or is it not necessary?

[zorr]

Ok, went ahead to check and didn't read your example call haha, it's taking a while (single threaded) but so far 5 validations and in all there's a score difference of 0.005, always lower.

That's interesting, also helpful if the difference is almost a constant. I wonder if it's because you're running Intel and I have AMD, after all floating point math is not exact in different architectures. The delta is quite large though so it must be some cumulative effect.

I don't know what to think about it... You are running latest GMSD, AVS+, DGSource, MVTools2, fmtconv, and latest ExTools.

GMSD Similarity Metrics Pack v2.6 (16-02-2022)
AVS+ AviSynth+ 3.7.2 (r3620, master, x86_64)
DGSource 239.0.0.0
MVTools2 2.7.45.0 (20210608)
fmtconv r28
ExTools ExTools v7.5 (12-02-2022)

Do you want me to re run them with your call or is it not necessary?

The result will be the same so no need to run again but get a couple more results if you have the time so that we can confirm the quality delta.

[Dogway]

Yep, all latest except AVS+ version, I use r3622, but I was also thinking on float point precision from the beginning, didn't know you had an AMD, did a search on AVX512 in Expr docs but it's not used anywhere so we are on par with that (AVX2 top here).
The difference is not exactly 0.005, it fluctuates a tiny bit but never above the third digit. Probably I would consider it safe to sum 0.005 to my scores.

Validation results so far:
145.82434 85382 thSAD=265 thSADR=245 temporal=true lambda=1740 lambdaR=760 pnew=174 pnewR=24 pzero=100 lsad=6100 plevel=1 pglobal=0 badrange=16 badSAD=2300 dct=8 dctre=0 searchAlgo=2 searchAlgoR=4 searchRange=8 searchRangeR=11 searchRangeFinest=6 sglobal=false
146.01328 85692 thSAD=220 thSADR=270 temporal=true lambda=160 lambdaR=980 pnew=74 pnewR=48 pzero=100 lsad=250 plevel=2 pglobal=4 badrange=50 badSAD=80 dct=6 dctre=6 searchAlgo=0 searchAlgoR=1 searchRange=8 searchRangeR=0 searchRangeFinest=2 sglobal=false
146.00676 97915 thSAD=220 thSADR=265 temporal=false lambda=900 lambdaR=700 pnew=98 pnewR=18 pzero=188 lsad=1700 plevel=0 pglobal=20 badrange=33 badSAD=1290 dct=10 dctre=9 searchAlgo=4 searchAlgoR=2 searchRange=10 searchRangeR=1 searchRangeFinest=0 sglobal=false
145.94038 109009 thSAD=255 thSADR=190 temporal=false lambda=400 lambdaR=960 pnew=100 pnewR=78 pzero=42 lsad=6800 plevel=0 pglobal=10 badrange=26 badSAD=2380 dct=10 dctre=6 searchAlgo=1 searchAlgoR=0 searchRange=1 searchRangeR=11 searchRangeFinest=20 sglobal=true
145.94707 187095 thSAD=195 thSADR=185 temporal=false lambda=1080 lambdaR=2800 pnew=68 pnewR=78 pzero=134 lsad=100 plevel=2 pglobal=7 badrange=10 badSAD=110 dct=7 dctre=10 searchAlgo=0 searchAlgoR=1 searchRange=15 searchRangeR=16 searchRangeFinest=17 sglobal=false
145.62733 209594 thSAD=155 thSADR=235 temporal=false lambda=1960 lambdaR=1620 pnew=108 pnewR=68 pzero=98 lsad=2100 plevel=0 pglobal=2 badrange=33 badSAD=930 dct=8 dctre=4 searchAlgo=2 searchAlgoR=5 searchRange=13 searchRangeR=18 searchRangeFinest=2 sglobal=true
145.80855 212743 thSAD=215 thSADR=195 temporal=false lambda=860 lambdaR=1420 pnew=146 pnewR=12 pzero=172 lsad=3000 plevel=2 pglobal=0 badrange=46 badSAD=2650 dct=10 dctre=4 searchAlgo=5 searchAlgoR=2 searchRange=20 searchRangeR=15 searchRangeFinest=12 sglobal=true
145.86792 501016 thSAD=225 thSADR=300 temporal=true lambda=520 lambdaR=360 pnew=192 pnewR=156 pzero=162 lsad=6600 plevel=1 pglobal=14 badrange=20 badSAD=170 dct=2 dctre=7 searchAlgo=0 searchAlgoR=0 searchRange=4 searchRangeR=13 searchRangeFinest=4 sglobal=true


Yours for comparison:
145.82846 84450 thSAD=265 thSADR=245 temporal=true lambda=1740 lambdaR=760 pnew=174 pnewR=24 pzero=100 lsad=6100 plevel=1 pglobal=0 badrange=16 badSAD=2300 dct=8 dctre=0 searchAlgo=2 searchAlgoR=4 searchRange=8 searchRangeR=11 searchRangeFinest=6 sglobal=false
146.01889 85180 thSAD=220 thSADR=270 temporal=true lambda=160 lambdaR=980 pnew=74 pnewR=48 pzero=100 lsad=250 plevel=2 pglobal=4 badrange=50 badSAD=80 dct=6 dctre=6 searchAlgo=0 searchAlgoR=1 searchRange=8 searchRangeR=0 searchRangeFinest=2 sglobal=false
146.01184 98740 thSAD=220 thSADR=265 temporal=false lambda=900 lambdaR=700 pnew=98 pnewR=18 pzero=188 lsad=1700 plevel=0 pglobal=20 badrange=33 badSAD=1290 dct=10 dctre=9 searchAlgo=4 searchAlgoR=2 searchRange=10 searchRangeR=1 searchRangeFinest=0 sglobal=false
145.94518 109910 thSAD=255 thSADR=190 temporal=false lambda=400 lambdaR=960 pnew=100 pnewR=78 pzero=42 lsad=6800 plevel=0 pglobal=10 badrange=26 badSAD=2380 dct=10 dctre=6 searchAlgo=1 searchAlgoR=0 searchRange=1 searchRangeR=11 searchRangeFinest=20 sglobal=true
145.95306 178040 thSAD=195 thSADR=185 temporal=false lambda=1080 lambdaR=2800 pnew=68 pnewR=78 pzero=134 lsad=100 plevel=2 pglobal=7 badrange=10 badSAD=110 dct=7 dctre=10 searchAlgo=0 searchAlgoR=1 searchRange=15 searchRangeR=16 searchRangeFinest=17 sglobal=false
145.63268 197430 thSAD=155 thSADR=235 temporal=false lambda=1960 lambdaR=1620 pnew=108 pnewR=68 pzero=98 lsad=2100 plevel=0 pglobal=2 badrange=33 badSAD=930 dct=8 dctre=4 searchAlgo=2 searchAlgoR=5 searchRange=13 searchRangeR=18 searchRangeFinest=2 sglobal=true
145.8131 203980 thSAD=215 thSADR=195 temporal=false lambda=860 lambdaR=1420 pnew=146 pnewR=12 pzero=172 lsad=3000 plevel=2 pglobal=0 badrange=46 badSAD=2650 dct=10 dctre=4 searchAlgo=5 searchAlgoR=2 searchRange=20 searchRangeR=15 searchRangeFinest=12 sglobal=true
145.87228 907400 thSAD=225 thSADR=300 temporal=true lambda=520 lambdaR=360 pnew=192 pnewR=156 pzero=162 lsad=6600 plevel=1 pglobal=14 badrange=20 badSAD=170 dct=2 dctre=7 searchAlgo=0 searchAlgoR=0 searchRange=4 searchRangeR=13 searchRangeFinest=4 sglobal=true
145.73233 934690 thSAD=265 thSADR=255 temporal=false lambda=1100 lambdaR=1360 pnew=36 pnewR=170 pzero=10 lsad=2050 plevel=2 pglobal=4 badrange=35 badSAD=320 dct=2 dctre=2 searchAlgo=4 searchAlgoR=1 searchRange=4 searchRangeR=10 searchRangeFinest=10 sglobal=false
146.09076 2676410 thSAD=250 thSADR=290 temporal=true lambda=480 lambdaR=2740 pnew=64 pnewR=24 pzero=40 lsad=1700 plevel=0 pglobal=2 badrange=17 badSAD=530 dct=2 dctre=9 searchAlgo=1 searchAlgoR=2 searchRange=4 searchRangeR=10 searchRangeFinest=19 sglobal=false


By the way your last one took a lot of time, dct=2 and searchRange=4, so the culprit could be searchRangeFinest. In any case for me it's fine to drop DCT from 2 to 4 if you want, or limit searchRangeFinest but those didn't get good scores anyway.

I'm going to let it optimize overnight with the last refined script version (sans DCT 2 ~ 4), and check the results tomorrow. If you can share your highest score parameters so far (146.21552) so I can include it in the following runs for pareto reference.


EDIT: My results, took about 8h so I guess I can do 1000 iterations per night. One question though, do you know how I can continue with this "optimization" without creating a new one?

https://i.imgur.com/awrSxonl.png

The difference in time can roughly be mapped like this:
time = time/100.
time > 178.04 ? 170.306*pow(time,0.25045)-436.243 : \
1.0849*time-7.839
time = time*100

[zorr]

Yep, all latest except AVS+ version, I use r3622

Where did you get that one? I'm using the latest 3.7.2 test 3 version pinterf shared in AviSynth+ thread Vol.2, the Github only has 3.7.1 versions.

By the way your last one took a lot of time, dct=2 and searchRange=4, so the culprit could be searchRangeFinest.

I did about 2140 results using the latest script, including dct=2 but limiting the seachRange and searchAlgo. The script runtime graph now looks like this:

https://i.postimg.cc/sDcXbYSy/runtime-distribution-dct2.png

Not surprisingly the slowest results have all dct=2. The fastest dct=2 was 498 seconds so the lump at 500 is where dct=2 results start. The limits are mostly restricting the time to stay below 1000 seconds but some of them are higher still. Here's a zip (https://drive.google.com/file/d/1hrKFyTsFevCRm8gQIA_RvpTGHaGO2X_i/view?usp=sharing) of the dct=2 results sorted by time if you want to try to figure out what makes them slow.

In any case for me it's fine to drop DCT from 2 to 4 if you want, or limit searchRangeFinest but those didn't get good scores anyway.

Yes I think that's a reasonable option right now. We can come back to DCT 1-4 later. I know they have potential because in my MFlowFPS tests DCT=1 was the top quality result.

If you can share your highest score parameters so far (146.21552) so I can include it in the following runs for pareto reference.

146.21703 95570 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dctre=6 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true

One question though, do you know how I can continue with this "optimization" without creating a new one?

Not possible at the moment, I will make a Zopti version which supports -continue with the mutation algorithm, at least with the initial random generation. But the risk of getting duplicates is pretty low so you can just start a new one and concatenate the result files manually later.

The difference in time can roughly be mapped like this:
time = time/100.
time > 178.04 ? 170.306*pow(time,0.25045)-436.243 : \
1.0849*time-7.839
time = time*100

I'd like to get more data for this curve fitting function. We can do it the other way, if you send me your 1000 results I can run the validation on them.

[Dogway]

Where did you get that one? I'm using the latest 3.7.2 test 3 version pinterf shared in AviSynth+ thread Vol.2, the Github only has 3.7.1 versions.

From here (https://forum.doom9.org/showthread.php?p=1963492#post1963492)I think.

Yes I think that's a reasonable option right now. We can come back to DCT 1-4 later. I know they have potential because in my MFlowFPS tests DCT=1 was the top quality result.

I also think they have potential but not as much for MDegrain client as for MFlowFPS.

146.21703 95570 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dctre=6 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true

Do you have DCT and searchAlgo values for this?

I'd like to get more data for this curve fitting function. We can do it the other way, if you send me your 1000 results I can run the validation on them.

OK, they uploaded here (https://www.mediafire.com/file/v1c0rskswsoez7o/Zopti_test2s_-_MDegrain6_-_Random_%25282022-02-18_00-34-31%2529_optimize_mutation_pop-500_mutcount-60-1_mutamount-0.5-0.01_iters-500_run-01.log/file).
The fitting is pretty basic given I only had like 8 samples and 1 of them was an outlier. But for short to medium times it seemed to follow a linear function.

[zorr]

From here (https://forum.doom9.org/showthread.php?p=1963492#post1963492)I think.

Looks the same link I used. What does Version() output?

146.21703 95570 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dctre=6 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true

Do you have DCT and searchAlgo values for this?

dct = 0
searchAlgo = 1

These may also be relevant

BlkSize = 16
BlkSizeR = 8
overlap = 4
overlapR = 4
pel = 1
sharp = 2
scaleCSAD = 2
trymany = false
truemotion = false
truemotionR = false

OK, they uploaded here (https://www.mediafire.com/file/v1c0rskswsoez7o/Zopti_test2s_-_MDegrain6_-_Random_%25282022-02-18_00-34-31%2529_optimize_mutation_pop-500_mutcount-60-1_mutamount-0.5-0.01_iters-500_run-01.log/file).

Thanks, I'll do the validation after I have run about 10k results without dct 1-4, started already.

The fitting is pretty basic given I only had like 8 samples and 1 of them was an outlier. But for short to medium times it seemed to follow a linear function.

I will do a scatter plot and try to fit some functions.

I have thought about adding a conversion mode to Zopti, you could for example convert the result file into CSV or JSON format for easier importing to other software. Also sorting by GMSD or time could be useful. Do you need anything like that?

[Dogway]

Yes, I actually have a better global picture of what's going on when I add to Excel and color code the values. Then I can simply sort by columns and check the relationships at the same time for different parameters (like sectioned by DCT...), prune those that don't need, etc. Just like in the Google spreadsheets.
CSV output would be very useful because I always have to remove "param_name=". I have a regex for that but better if it's built internally.
As for sorting it could also be useful, currently I can sort in Notepad2 with Alt+O, but this only takes into account the first column (GMSD).

This is my version()
https://i.imgur.com/uAoTfdO.png

[zorr]

One question though, do you know how I can continue with this "optimization" without creating a new one?

The option -continue now works with mutation algorithm for the initial random generation population. Get the latest jar here (https://drive.google.com/file/d/1M_dIRPR2yfHVps_9DG_gy_shlLn2F5DW/view?usp=sharing).

I have about 5.5k results now.

https://i.postimg.cc/tCfCPMZR/mdegrain-latest-fast-dct-5k.png

[Dogway]

Thanks! I will leave it tonight using the -continue flag.

The curve is shaping up nicely, is there any reason you use different values for population and iterations?

Also if you can share your top results with overlap=8, as we are testing with that one locked. At the time I did some study on overlap and found that in average it was slower and with worse scores, but we can go back to it after the correlations, same with DCT 1~4.

By the way, I think that rather than trying to map GMSD and time from your logs to mine or the other way around, it would be better to section an area (the 'action' area) and validate those.

[zorr]

The curve is shaping up nicely, is there any reason you use different values for population and iterations?

The mutation algorithm works by creating an offspring for every member of the population and then selecting the best 50% which brings the population back to original size. That is also done in the first generation so N*2 solutions will be generated, where N is the population size. And that's the number of iterations. I guess it also works if iterations is smaller than N*2 but I wanted to be precise. So in a nutshell if you want X random results you set the iterations to X and population size to X/2 in the mutation algorithm.

Also if you can share your top results with overlap=8, as we are testing with that one locked.

My best overlap=8 result currently is from the ongoing random run, 146.19359. The earlier larger runs were using overlap=4.

At the time I did some study on overlap and found that in average it was slower and with worse scores, but we can go back to it after the correlations, same with DCT 1~4.

Overlap=8 was slower and worse, or overlap=4? Anyway I think overlap is one of those hard to predict parameters, the best option may depend on the source material.

By the way, I think that rather than trying to map GMSD and time from your logs to mine or the other way around, it would be better to section an area (the 'action' area) and validate those.

Hmm yes, of course that means after the area is sectioned and validated the section could be elsewhere and not anymore in the 'action' area. Also if the area is not large enough it might introduce selection bias. But I get your point, it's a good idea as long as the section is selected carefully. I guess there's no need to do the speed validation run of your 1000 results?

I have finished 10k results and continued, now over 11k. I think 100k would be a nice target number, takes at least two weeks though...

There are already some trends visible, especially if you add the average score per parameter value (the green line). The average might actually be a better indicator when using purely random samples.

https://i.postimg.cc/pXbKp8cS/mdegrain-random-th-SAD.png

[Dogway]

This is my highest with overlap=8
146.20343 235920 thSAD=245 thSADR=220 temporal=false lambda=250 lambdaR=780 pnew=76 pnewR=2 pzero=120 lsad=300 plevel=9 pglobal=17 badrange=20 badSAD=1350 dct=0 dctre=6 searchAlgo=1 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=true
Just changed overlap for your highest score.

The sectioned 'action' area is going to be shifted to the right to the same amount as highest time minus lowest time of the 'action' area. So all slices have the same horizontal dimension (and enough room) and minimizes bias.
The 'validation' is only to take my samples instead of merging them manually while trying to map time and scores. You don't save time but do save randomness.

[zorr]

Zopti v1.2.3 is ready. This one has a new conversion feature to help importing Zopti data into other software. Also improvements to -vismode line and some other minor things. Details below:

new mode: convert
-converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
-writes a new file and doesn't remove the original
-does not overwrite an existing file (error message given in such situation)
-supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
-sort direction can be specified by adding a + or - after the field name, for example "-sort time-" sorts by time, descending
-ascending direction (+) is the default if no + or - is given
-multiple sort criterias supported, for example "-sort dct GMSD-" sorts by dct (ascending) and the rows with the same dct value by GMSD (descending)
-output format: CSV
-specified with -format csv or by not giving format option
-target file name is the same as the original .log file but file ending changes from .log to .csv
-csv file includes title row with the names of result values and optimized parameters
-can be opened directly (for example) in Excel
-uses the default decimal format of the OS with floating point values
-output format: zopti
-specified with -format zopti
-keeps the original Zopti format and header rows, only sorts the results when option -sort is given
-filename is "converted " + original file name
-example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
-example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
improvements to visualization mode: line
-now supports option -types which controls which aggregate lines are displayed in the chart
-valid values are "best", "worst", "average", "median", "count" and "samples"
-best = best result per parameter value
-worst = worst result per parameter value
-average = average result per parameter value
-median = median result per parameter value
-count = number of samples per parameter value
-samples = display all results as small dots
-separate multiple values with space, for example "-types best average"
-default value is "best count" which will display the best value per parameter value and the number of samples per parameter value
-a legend is shown if more than one type is selected
display of option -groupby improved, pareto fronts can no longer be cut off the screen
option -continue also supported with mutation algorithm
-but only for the generation of the initial random population
-useful if you only want to test independent random samples with a large population
validation mode behavior change: it now validates all the results in the log file and not just the pareto front


Download link updated at the first post.

[Ceppo]

In love with the project, waiting for the documentation!

[zorr]

Dogway, you can now download 100k random mdegrain results (https://drive.google.com/file/d/11dvn95o5j7e5F8WArh77fO_d4PJXttbR/view?usp=sharing). :) The best result was surprisingly good 146.22812.

https://i.postimg.cc/fbrPntM8/mdegrain-100k.png


I'm working on a new sample histogram visualization, here's an example of parameter thSAD:

https://i.postimg.cc/FzQDN08V/mdegrain-100k-samplehist.png

[Dogway]

Thanks! Nice convex graph!

Here are the isolated action areas (https://www.mediafire.com/file/icjrwezwarsm5yc/mdegrain_100k_-_ACTION_AREAS.zip/file) if you also want to run some tests. And here the Excel chart (https://www.mediafire.com/file/5ox04pviij5h5tg/ZORR_100K.xlsx/file).
Initially I was only going to work with the action area between convex inflection and convex top, but that yielded only 60 samples, so I also include the expanded action area (blue shaded (https://forum.doom9.org/showthread.php?p=1963517#post1963517)) between convex bottom and convex top, this results in 683 samples which is better to work with. The GMSD of the bottom sample is 145.967 so I guess it's fine enough for the correlations to still be meaningful.

As anchor points I used:
146.08434 34410 TOP
146.16876 47800 AREA1 BOTTOM
145.9666 28280 AREA2 BOTTOM

[zorr]

Dogway, I took your action area 2 samples (683 results) and decided to do a correlation test of my own. I was curious how much the results change if we crop the video by 10%, 20%, 30% etc. The cropping is done by taking the same amount on each edge, so for example crop by 10% takes 5% from left, right, top and bottom edges. Also important to set downsample=true parameter to GMSD, otherwise it will automatically be false when cropping drops the resolution to SD (happens at 50% or more cropping). I calculate the correlation between original and cropped results and also the average time. The cropping was done within the script so there's a small performance hit from the cropping itself which could be avoided.

Here's the results:

https://i.postimg.cc/mDg2ksQy/crop-correlation.png


crop correlation average GMSD average time time %
no crop 1,0000 146,0399465 48027,01 100,00 %
10 % 0,9935 145,9842366 40753,73 84,86 %
20 % 0,9849 145,9095277 33480,41 69,71 %
30 % 0,9751 145,8502522 26815,10 55,83 %
40 % 0,9479 145,8150252 21069,68 43,87 %
50 % 0,8983 145,8019399 16655,15 34,68 %
60 % 0,8704 145,7848522 14720,74 30,65 %
70 % 0,8553 145,7376085 14241,59 29,65 %


Above 0,7 correlation is already considered strong so these are all pretty good but I would be more comfortable with something like above 0,97 correlation which would already give a nice 44% performance improvement using 30% crop, or perhaps even 40% with 0,95 correlation and more than twice as fast.

Here's what crops 10%, 20%, 30% and 70% look like in a scatter plot:

https://i.postimg.cc/NfDtyVjN/crop-correlation-scatter.png

You can download an Excel with all the calculations here (https://docs.google.com/spreadsheets/d/1KJw2PwpLt11Nv5r1XwTJPaVb4h-RwMmR/edit?usp=sharing&ouid=106586920922614226357&rtpof=true&sd=true).

[Dogway]

Cropping to save time right? 10% already achieves a nice perf up, 30% looks like the limit. This can come very handy when optimizing for MFlow. I actually cropped my interpolation sample to the "motion area", to maximize the worst scenario case.

Currently I'm finishing SceneChange and SceneStats filter and after that I'm free to dedicate the needed time to work on the correlations, I plan to implement, as I fear it's not already in any Python package, the CoS (https://www.researchgate.net/publication/318889643_A_Copula_Statistic_for_Measuring_Nonlinear_Dependence_with_Application_to_Feature_Selection_in_Machine_Learning) (Copula Statistic) correlation which seems to be the state-of-the-art algo for multivariate analysis.

[zorr]

Yes, cropping could be used to save time. I'm also going to investigate how to select the most important frames, again saving time. I have a theory that the frames with the largest variance in GMSD should be used. Zopti could first determine the variance and then select the frames based on that.

[Dogway]

That's certainly true. But it just happens that MFlowInter and MFlowFps have an unfortunate looking failure case, which the MCompensate doesn't have. So even with (actually, especially with) bad vectors MCompensate will look better. Of course you can't use MCompensate to generate an inbetween frame, it can only recreate a complete frame using other frames.

But if we first create the inbetween frame with MFlowInter / MFlowFps and then use MCompensate to reconstruct that frame using two nearby frames something magical happens...

https://i.postimg.cc/wvDbnFMP/mflowinter-mcompensate.gif

Here we see the original frame (orig), a reconstructed frame from MFlowInter (inter) and finally the MCompensated frame (final).

Interesting, I would like to know more about it, how you mix MFlowFPS output with MCompensate, is that with a badvectors mask?

I have been a bit busy with latest repo scripts but I'm almost done, I think I can start with the correlations later this week.
I run some visualizations in -vismode and found it very useful (didn't know how to write the calls). Analyzing only the sectioned area I found the following to be meaningful.
searchAlgo searchRange (optimal)
0 4
1 5
2 3
4 5
5 4

searchAlgoR searchRangeR (optimal)
0 4
1 1
2 6
4 6
5 6

searchAlgo 4 for qual/perf
searchAlgo 2 for qual
searchAlgoR 0 for qual/perf
searchAlgoR 4 for qual
searchRange 3 for qual
searchRangeR0 for qual

dct 0 for qual
dctre 4~5 for qual

pglobal 4 for qual/perf
pglobal 6 for qual
# although 10~13 achieved the best next results

plevel 0 for qual
sglobal true for qual

temporal true for qual/perf
temporal false for qual
pnew 20~36 for qual (and prolly qual/perf)

lambda 100-287 for qual/perf
lambda 664-851 for qual
badSAD 1798-2095 for qual and qual/perf

thSAD 250-290 for qual and qual/perf (in this clip)
lsad 50-844 is the most constant best

These are loose but safe observations given the results.
For future tests we can lock plevel, sglobal, and probably temporal, dct and dctre.
Others like pzero, pnew, searchRangeFinest, etc need to wait for the correlations.

One Zopti option I couldn't make to work was -mapres 200x200, it always opens at max screen height but only for -vismode. I use jdk-16.0.1 on Win7.

Also something looked iffy with -vismode line, checking on 'dct':
zopti -mode evaluate -autorefresh false -vismode line -param dct -mapres 200x200 -log "results.log"
https://i.imgur.com/lf5CNOIl.png

It shows values 0, 1, 2, 3, 4, 5 and 6, but we skipped dct=1 and also included up to 9.

-------------------

On another note when running an optimization I get this error after population count runs out:
Parameter sensitivity estimation with 256 result combinations
java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
at avisynthoptimizer.Parameter.updateAverages(Parameter.java:954)
at avisynthoptimizer.Parameter.updateSensitivityEstimation(Parameter.java:972)
at avisynthoptimizer.SensitivityEstimation.estimateParameterSensitivity(SensitivityEstimation.java:102)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:315)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:38)
at org.uma.jmetal.util.evaluator.impl.MultithreadedSolutionListEvaluator.lambda$evaluate$1(MultithreadedSolutionListEvaluator.java:36)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(UnknownSource)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

I use the last avs template we came to use with the next batch script:
SET RUNS= 1
SET ALG= spea2
SET POPULATION= 8
SET INITIAL= script
SET ITERS= dyn
SET MUTAMOUNT= 0.3 0.01
SET MUTCOUNT= 60% 1
SET CROSSPROB= 0.1
SET CROSSDIST= 20
SET SENSITIVITY= TRUE
SET DYNPHASES= 1
SET DYNITERS= 8
SET ERRORS= STOP
SET PRIORITY=
SET THREADS= 8
SET TIMEOUT= 1440
SET CONTINUE=

java -jar Zopti.jar "C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.2\Tests\Zopti_test2s - MDegrain7 - Random.avs" ^
-alg %ALG% -initial %INITIAL% -sensitivity %SENSITIVITY% -iters %ITERS% -dyniters %DYNITERS% -dynphases %DYNPHASES% -pop %POPULATION% -runs %RUNS% ^
-mutcount %MUTCOUNT% -mutamount %MUTAMOUNT% -timeout %TIMEOUT% -threads %THREADS%


I tested with several options, halving threads, doubling population, etc. With both Zopti 1.2.2 and 1.2.3.

[zorr]

Interesting, I would like to know more about it, how you mix MFlowFPS output with MCompensate, is that with a badvectors mask?

First you create two new frames with MCompensate, one from the previous original frame (Mprev) and one from the next original frame (Mnext). The target frame for both is the new interpolated frame (inter) in between them created by MFlowFPS. Then you create a pixel mask where luma is 255 where difference of Mprev and inter is smaller than the difference of Mnext and inter. You could use this mask directly to take pixels from Mprev and Mnext but it looks better if the mask is processed a little bit. I have tried downscaling and upscaling back to original size with point resize so that bigger blocks are selected and also blurring so that the edges are not harsh. This is something that Zopti could optimize but I never got very far with that experiment.

I'll post my old script in the next post (too large to fit here).

One Zopti option I couldn't make to work was -mapres 200x200, it always opens at max screen height but only for -vismode. I use jdk-16.0.1 on Win7.

-mapres is meant for -vismode heatmap only and it sets the maximum resolution of the heat map, not in pixels but in how many rows and columns it has. If you want to see the visualizations in specific size you can use the -shot parameter, for example -shot 200x200 creates a screenshot with that size (or approximately that, it also depends on the natural dimensions of the chart).

Also something looked iffy with -vismode line, checking on 'dct':
...
It shows values 0, 1, 2, 3, 4, 5 and 6, but we skipped dct=1 and also included up to 9.

Yes that's a bug I'm aware of. It's showing the internal mapping of the values into continuous range. I have tried to display the original labels but there seems to be a bug with the XChart library and it's not working the way it's supposed to.

On another note when running an optimization I get this error after population count runs out:

Thanks for the report, I will try to reproduce it.

[zorr]

Here's an old script I used to test the MCompensate concept. It's using Corrector plugin and I'm not sure if there's a 64 bit version, you can easily replace it with Expr (the code is commented so you know what it's supposed to do).
The code is old and ugly, didn't spend time cleaning it up. :)


TEST_FRAMES = 10
MIDDLE_FRAME = 50

# source clip
AVISource("d:\process2\1 deinterlaced.avi")

# original framerate
FPS_NUM = 50
FPS_DEN = 1
AssumeFPS(FPS_NUM, FPS_DEN)

#return last

# needed for some parameter combinations
#ConvertToYV24()
ConvertToYV16() # RemoveGrain only works in planar colorspaces

orig = last

weight = 1.0/3.0
avg = Average(trim(orig,2,0), weight, trim(orig,1,0), weight, orig, weight)
#return avg


# preprocessing - blur and/or degrain
#searchClip = orig
#searchClip = last.Blur(1.58).Blur(1.58)
searchClip = RemoveGrain(orig, 16)
searchClip = RemoveGrain(searchClip, 23)


super_pad = 16 #- optimize super_pad = _n_ | 0,4,6,8,12,16,18,24,32,48 | super_pad
super_pel = 4 #- optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 #- optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 4 #- optimize super_rfilter = _n_ | 0..4 | super_rfilter
super_search = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, searchClip)
super_render = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, orig, levels=1)

blockSize = 8 #- optimize blockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 4 #- optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 1 #- optimize searchRange = _n_ | 1..30 | searchRange
searchRangeFinest = 12 #- optimize searchRangeFinest = _n_ | 1..60 | searchRangeFinest
lambda = 830*(blockSize*blockSize)/(8*8) #- optimize lambda = _n_ | 0..20000 ; filter:x 10 % 0 == | lambda
lsad=7752 #- optimize lsad=_n_ | 8..20000 ; filter:x 8 + 10 % 0 == | LSAD
pnew=0 #- optimize pnew=_n_ | 0..256 | pnew
plevel=2 #- optimize plevel=_n_ | 0..2 | plevel
overlap=0 #- optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=0 #- optimize overlapv=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlapv
divide=0 #- optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true #- optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 5836 #- optimize badSAD = _n_ | 4..10000 ; filter:x 4 + 10 % 0 == | badSAD
badRange = 28 #- optimize badRange = _n_ | 4..100 | badRange
badAlgo = 0 #- optimize badAlgo = _n_ | 0,1 | badAlgo
meander = false #- optimize meander = _n_ | false,true | meander
temporal = true #- optimize temporal = _n_ | false,true | temporal
trymany = true #- optimize trymany = _n_ | false,true | trymany
dct = 5 #- optimize dct = _n_ | 0,1,2,3,4,5,6,7,8,9,10 D | dct
scaleCSAD = 0 #- optimize scaleCSAD = _n_ | -2..2 | scaleCSAD

delta = 1
useChroma = true
badRange = (badAlgo == 0 ? badRange : -badRange)
bv = MAnalyse(super_search, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
fv = MAnalyse(super_search, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)



threshold = 16320 # max for thSCD1
maskScale = 1 #- optimize maskScale = _n_ | 1..600 | maskScale
mask_fps = 2 #- optimize mask_fps = _n_ | 0..2 | mask_fps
inter = orig.MFlowFPS(super_render, bv, fv, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold, thSCD2=100)

### mcompensate pass
interpolated = mcompensate_pass(inter.ConvertToYV16(), orig)
### mcompensate pass

#return Interleave(inter.selectOdd().ConvertToYUY2(), interpolated) # compare result from MFlowFPS and MCompensate
#return Interleave(orig.ConvertToYUY2(), interpolated) # MFlowFPS + MCompensate final clip


fps_only = interpolated #inter.SelectOdd()



# second pass
super_search3 = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only)
super_render3 = MSuper(hpad=super_pad, vpad=super_pad, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, fps_only, levels=1)
bv3 = MAnalyse(super_search3, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
fv3 = MAnalyse(super_search3, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, scaleCSAD=scaleCSAD, divide=divide, badSAD=badSAD, badrange=badRange, meander=meander, temporal=temporal, trymany=trymany, dct=dct)
inter3 = fps_only.MFlowFPS(super_render3, bv3, fv3, num=FPS_NUM*2, den=FPS_DEN, mask=mask_fps, ml=maskScale, thSCD1=threshold, thSCD2=100)

### second mcompensate pass
interpolated2 = mcompensate_pass(inter3.ConvertToYV16(), orig)
### second mcompensate pass



#return Interleave(trim(orig,1,0), interpolated2.ConvertToYV16()) # compare original and twice interpolated frames


fps_only2 = interpolated2 # inter2.SelectOdd()


delimiter = "; "

inter_yv12 = fps_only2.ConvertToYV12()
orig_yv12 = orig.ConvertToYV12()
avg_yv12 = avg.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
avg_yv12 = avg_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

global total = 0.0
global ssim_avg = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global ssim_avg = SSIM_FRAME(inter_yv12, avg_yv12)
ssim = (ssim > ssim_avg) ? ssim : 0.0
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig_yv12, inter_yv12, avg_yv12, delta, frame_count")

# NOTE: AvsTimer call should be before the WriteFile call
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
resultFile = "perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

function mcompensate_pass(clip input, clip orig) {

searchClip2 = RemoveGrain(input, 19) #- optimize RemoveGrain(input, _n_) | 11,12,19,20 D | mrmgrain
searchClip2 = RemoveGrain(searchClip2, 0) #- optimize RemoveGrain(searchClip2, _n_) | 0..24 D | mrmgrain2

msuper_pad = 16 #- optimize msuper_pad = _n_ | 6,8,12,16,18 | msuper_pad
msuper_pel = 4 #- optimize msuper_pel = _n_ | 2,4 | msuper_pel
msuper_sharp = 2 #- optimize msuper_sharp = _n_ | 0..2 | msuper_sharp
msuper_rfilter = 0 #- optimize msuper_rfilter = _n_ | 0..4 | msuper_rfilter
super_search2 = MSuper(hpad=msuper_pad, vpad=msuper_pad, pel=msuper_pel, sharp=msuper_sharp, rfilter=msuper_rfilter, searchClip2)
super_render2 = MSuper(hpad=msuper_pad, vpad=msuper_pad, pel=msuper_pel, sharp=msuper_sharp, rfilter=msuper_rfilter, input, levels=1)

mblockSize = 8 #- optimize mblockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:mdivide 0 > 8 2 ? ; filter:moverlap moverlapv max 2 * x <= | mblockSize
msearchAlgo = 6 #- optimize msearchAlgo = _n_ | 0..7 D | msearchAlgo
msearchRange = 1 #- optimize msearchRange = _n_ | 1..10 | msearchRange
msearchRangeFinest = 1 #- optimize msearchRangeFinest = _n_ | 1..60 | msearchRangeFinest
mlambda = 3290*(mblockSize*mblockSize)/(8*8) #- optimize mlambda = _n_ | 0..40000 ; filter:x 10 % 0 == | mlambda
mlsad=10042 #- optimize mlsad=_n_ | 8..20000 ; filter:x 8 + 10 % 0 == | mLSAD
mpnew=20 #- optimize mpnew=_n_ | 0..256 | mpnew
mplevel=2 #- optimize mplevel=_n_ | 1..2 | mplevel
moverlap=4 #- optimize moverlap=_n_ | 4 | moverlap
moverlapv=4 #- optimize moverlapv=_n_ | 4 | moverlapv
mdivide=0 #- optimize mdivide=_n_ | 0..2 ; max:mblockSize 8 >= 2 0 ? moverlap 4 % 0 == 2 0 ? min | mdivide
mglobalMotion = false #- optimize mglobalMotion = _n_ | false,true | mglobalMotion
mbadSAD = 656 # optimize mbadSAD = _n_ | 4..20000 ; filter:x 4 - 500 % 0 == | mbadSAD
mbadRange = 4*10 # optimize mbadRange = _n_*10 | 1..10 | mbadRange
mbadAlgo = 1 #- optimize mbadAlgo = _n_ | 0,1 | mbadAlgo
mmeander = true #- optimize mmeander = _n_ | false,true | mmeander
mtemporal = true #- optimize mtemporal = _n_ | false,true | mtemporal
mtrymany = false #- optimize mtrymany = _n_ | false,true | mtrymany
mdct = 5 #- optimize mdct = _n_ | 0,2,5,6,9,10 D | mdct
mscaleCSAD = -2 #- optimize mscaleCSAD = _n_ | -2..2 | mscaleCSAD

delta = 1
useChroma = true
mbadRange = (mbadAlgo == 0 ? mbadRange : -mbadRange)
bv2 = MAnalyse(super_search2, isb = true, blksize=mblockSize, search=msearchAlgo, searchparam=msearchRange, pelsearch=msearchRangeFinest, chroma=useChroma, delta=delta, lambda=mlambda, lsad=mlsad, pnew=mpnew, plevel=mplevel, global=mglobalMotion, overlap=moverlap, overlapv=moverlapv, scaleCSAD=mscaleCSAD, divide=mdivide, badSAD=mbadSAD, badrange=mbadRange, meander=mmeander, temporal=mtemporal, trymany=mtrymany, dct=mdct)
fv2 = MAnalyse(super_search2, isb = false, blksize=mblockSize, search=msearchAlgo, searchparam=msearchRange, pelsearch=msearchRangeFinest, chroma=useChroma, delta=delta, lambda=mlambda, lsad=mlsad, pnew=mpnew, plevel=mplevel, global=mglobalMotion, overlap=moverlap, overlapv=moverlapv, scaleCSAD=mscaleCSAD, divide=mdivide, badSAD=mbadSAD, badrange=mbadRange, meander=mmeander, temporal=mtemporal, trymany=mtrymany, dct=mdct)


threshold = 16320 # max for thSCD1
f1 = MCompensate(input, super_render2, fv2, thSCD1=threshold, thSAD=threshold)
b1 = MCompensate(input, super_render2, bv2, thSCD1=threshold, thSAD=threshold)

#return StackHorizontal(inter, f1, b1)
#return Interleave(inter.SelectOdd().Subtitle("inter"), f1.SelectOdd().Subtitle("f1"), b1.SelectOdd().Subtitle("b1"))
#return inter.SelectOdd()

inter = input.SelectOdd()
f1 = f1.SelectOdd()
b1 = b1.SelectOdd()

# Corrector needs YUY2
f1c = f1.ConvertToYUY2()
b1c = b1.ConvertToYUY2()
interc = inter.ConvertToYUY2()
origc = orig.ConvertToYUY2()

# take pixel from the frame (b2 or f2) whose value is closest to inter (processing Y U and V individually)
comb = Corrector(interc, interc, b1c, f1c, b1c, f1c, mode=1, th=255)

#f1 = f1.ConvertToYV16()
#b1 = b1.ConvertToYV16()
comb_yv16 = comb.ConvertToYV16()

# create masks where value is 255 when combined pixel has the same value as source
mask_f1 = mt_lutxy(f1, comb_yv16, yexpr=mt_polish("((x==y)?255:0)"), U=-128, V=-128)
mask_b1 = mt_lutxy(b1, comb_yv16, yexpr=mt_polish("((x==y)?255:0)"), U=-128, V=-128)

# downscale masks, essentially generate averages for how many times f1 and b1 had correct colors
downScale = 0 #- optimize downScale = _n_ | 0,2,4,8,16,32 | downScale
scaled_f1 = downscale(mask_f1, orig, downScale)
scaled_b1 = downscale(mask_b1, orig, downScale)

# create new mask with value 255 where scaled_f1 mask has larger value and value 0 otherwise
mask_combined = mt_lutxy(scaled_f1, scaled_b1, yexpr=mt_polish("((x>y)?255:0)"), U=-128, V=-128)

# refine mask: remove single pixels where all 8 neighbours are opposite color
# phase 1: calculate average of 9x9 neighbourhood
mask_combined_1 = mask_combined.ConvertToYV16().mt_edge(mode="1 1 1 1 1 1 1 1 1 9", thY1=0, thY2=255, Y=3, U=-128, V=-128)
# phase 2: change pixel color to opposite if average is in certain range
mask_filled = mt_lutxy(mask_combined, mask_combined_1, yexpr=mt_polish("((x==255)?((y<30)?0:255):((y>225)?255:0))"), U=-128, V=-128)

# scale mask back to original size
mask_combined = mask_combined.PointResize(orig.Width, orig.Height)
mask_filled = mask_filled.PointResize(orig.Width, orig.Height)

#return Interleave(mask_combined, mask_filled)
#return mask_combined

mask_final = mask_filled

#mask_f1 = mask_f1.ConvertToYUY2()
#mask_b1 = mask_b1.ConvertToYUY2()
mask_final = mask_final.ConvertToYUY2()

blurAmount = 155*10 #- optimize blurAmount = _n_*10 | 101..200 | blurAmount
# mask_final = blurrr(mask_final, blurAmount)
mask_final = mask_final.GaussianBlur(varY=blurAmount)

#return mask_final

# create final result by combining b1 and f1 with the mask
interpolated = Overlay(b1, f1, mask=mask_final, output="YUY2")

#final = Interleave(origc.Subtitle("orig"), f1c.Subtitle("f1"), b1c.Subtitle("b1"), comb.SubTitle("comb"), mask_f1.Subtitle("mask f1"), mask_b1.Subtitle("mask b1"),mask_final.Subtitle("final mask"), interc.Subtitle("inter"), mask_combined.Subtitle("final"))
final = Interleave(origc.Subtitle("orig"), interc.Subtitle("inter"), interpolated.Subtitle("final"))
#return mask_final
#return interc
return interpolated
#return final

}


function downscale(clip c, clip orig, int scale, int "currScale") {
String SSS = """
currScale = Default(currScale, 2)
targetWidth = orig.Width/currScale
targetWidth = (targetWidth % 2 == 1) ? targetWidth+1 : targetWidth
c = c.BilinearResize(targetWidth, orig.Height/currScale)
return (currScale==scale) ? c : downscale(c, orig, scale, currScale*2)
"""
return scale >= 2 ? Eval(SSS) : c
}

function blurrr(clip c, int times) {
String SSS = """
c = c.Blur(1.58)
return times==1 ? c : blurrr(c, times-1)
"""
return times>0 ? Eval(SSS) : c
}

[Dogway]

Thanks for the script, I will try to clean it up and add into comparisons for the interpolation tests.

I ran a test to confirm my above assertions that we can lock plevel, pnewR, temporal, sglobal, dct, and dctre. I took all my best scores from pareto of your 100K test and also some internal tests and simply swapped those values, then ran a validation. In all cases but one (it preferred dctre=7) I got same or better score with times same or better in average, the "hyphen" depicts a new pareto front.

INPUT
146.1719 56870 thSAD=290 thSADR=275 temporal=TRUE lambda=680 lambdaR=1420 pnew=38 pnewR=0 pzero=22 lsad=3900 plevel=0 pglobal=13 badrange=43 badSAD=1860 dct=0 dctre=5 searchAlgo=2 searchAlgoR=5 searchRange=3 searchRangeR=6 searchRangeFinest=0 sglobal=TRUE
146.13617 41180 thSAD=285 thSADR=195 temporal=TRUE lambda=260 lambdaR=1820 pnew=20 pnewR=0 pzero=64 lsad=7500 plevel=0 pglobal=12 badrange=49 badSAD=2230 dct=0 dctre=5 searchAlgo=4 searchAlgoR=4 searchRange=2 searchRangeR=5 searchRangeFinest=19 sglobal=TRUE
146.1149 37630 thSAD=300 thSADR=250 temporal=TRUE lambda=420 lambdaR=2360 pnew=22 pnewR=0 pzero=86 lsad=2500 plevel=0 pglobal=5 badrange=44 badSAD=1470 dct=0 dctre=5 searchAlgo=0 searchAlgoR=0 searchRange=1 searchRangeR=2 searchRangeFinest=6 sglobal=TRUE
146.08434 34410 thSAD=215 thSADR=295 temporal=TRUE lambda=200 lambdaR=1800 pnew=48 pnewR=0 pzero=40 lsad=1250 plevel=0 pglobal=5 badrange=26 badSAD=1920 dct=0 dctre=5 searchAlgo=4 searchAlgoR=0 searchRange=3 searchRangeR=1 searchRangeFinest=6 sglobal=TRUE
146.00838 29060 thSAD=285 thSADR=205 temporal=TRUE lambda=240 lambdaR=1340 pnew=52 pnewR=0 pzero=192 lsad=7850 plevel=0 pglobal=19 badrange=20 badSAD=1740 dct=0 dctre=5 searchAlgo=0 searchAlgoR=1 searchRange=0 searchRangeR=0 searchRangeFinest=9 sglobal=TRUE
146.15271 376230 thSAD=260 thSADR=210 temporal=TRUE lambda=480 lambdaR=1660 pnew=86 pnewR=0 pzero=42 lsad=1850 plevel=0 pglobal=12 badrange=49 badSAD=1950 dct=0 dctre=5 searchAlgo=2 searchAlgoR=1 searchRange=16 searchRangeR=13 searchRangeFinest=0 sglobal=TRUE
146.19511 107890 thSAD=245 thSADR=220 temporal=TRUE lambda=240 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=15 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
146.20343 235920 thSAD=245 thSADR=220 temporal=TRUE lambda=250 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=17 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=1 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
-146.19547 68920 thSAD=245 thSADR=220 temporal=TRUE lambda=260 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=13 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE

OUTPUT
-146.18489 60372 thSAD=290 thSADR=275 temporal=TRUE lambda=680 lambdaR=1420 pnew=38 pnewR=0 pzero=22 lsad=3900 plevel=0 pglobal=13 badrange=43 badSAD=1860 dct=0 dctre=5 searchAlgo=2 searchAlgoR=5 searchRange=3 searchRangeR=6 searchRangeFinest=0 sglobal=TRUE
146.17502 49104 thSAD=285 thSADR=195 temporal=TRUE lambda=260 lambdaR=1820 pnew=20 pnewR=0 pzero=64 lsad=7500 plevel=0 pglobal=12 badrange=49 badSAD=2230 dct=0 dctre=5 searchAlgo=4 searchAlgoR=4 searchRange=2 searchRangeR=5 searchRangeFinest=19 sglobal=TRUE
-146.17696 37867 thSAD=300 thSADR=250 temporal=TRUE lambda=420 lambdaR=2360 pnew=22 pnewR=0 pzero=86 lsad=2500 plevel=0 pglobal=5 badrange=44 badSAD=1470 dct=0 dctre=5 searchAlgo=0 searchAlgoR=0 searchRange=1 searchRangeR=2 searchRangeFinest=6 sglobal=TRUE
145.85498 39661 thSAD=215 thSADR=295 temporal=TRUE lambda=200 lambdaR=1800 pnew=48 pnewR=0 pzero=40 lsad=1250 plevel=0 pglobal=5 badrange=26 badSAD=1920 dct=0 dctre=5 searchAlgo=4 searchAlgoR=0 searchRange=3 searchRangeR=1 searchRangeFinest=6 sglobal=TRUE
-146.12015 31993 thSAD=285 thSADR=205 temporal=TRUE lambda=240 lambdaR=1340 pnew=52 pnewR=0 pzero=192 lsad=7850 plevel=0 pglobal=19 badrange=20 badSAD=1740 dct=0 dctre=5 searchAlgo=0 searchAlgoR=1 searchRange=0 searchRangeR=0 searchRangeFinest=9 sglobal=TRUE
146.16965 136785 thSAD=260 thSADR=210 temporal=TRUE lambda=480 lambdaR=1660 pnew=86 pnewR=0 pzero=42 lsad=1850 plevel=0 pglobal=12 badrange=49 badSAD=1950 dct=0 dctre=5 searchAlgo=2 searchAlgoR=1 searchRange=16 searchRangeR=13 searchRangeFinest=0 sglobal=TRUE
146.19472 68707 thSAD=245 thSADR=220 temporal=TRUE lambda=240 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=15 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
-146.197 115782 thSAD=245 thSADR=220 temporal=TRUE lambda=250 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=17 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=1 searchAlgoR=5 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE
146.19452 69217 thSAD=245 thSADR=220 temporal=TRUE lambda=260 lambdaR=780 pnew=76 pnewR=0 pzero=120 lsad=300 plevel=0 pglobal=13 badrange=20 badSAD=1350 dct=0 dctre=5 searchAlgo=2 searchAlgoR=4 searchRange=7 searchRangeR=12 searchRangeFinest=9 sglobal=TRUE

This is only an insight for optimizations, for correlations I need all the values to draw a bigger picture, will start tomorrow.

For some reason the error I reported "fixed" after I changed the variable ERROR (https://forum.doom9.org/showthread.php?p=1966188#post1966188)from STOP to CONTINUE, despite not being called in the Zopti line.

[zorr]

Thanks for the script, I will try to clean it up and add into comparisons for the interpolation tests.

This is something I also want to do... whenever I get the time. :) I've been thinking about a kind of MVToolsHQ library with focus on quality and using every trick in the book, optimized by Zopti.

I ran a test to confirm my above assertions that we can lock plevel, pnewR, temporal, sglobal, dct, and dctre. I took all my best scores from pareto of your 100K test and also some internal tests and simply swapped those values, then ran a validation. In all cases but one (it preferred dctre=7) I got same or better score with times same or better in average, the "hyphen" depicts a new pareto front.

That's good to know, eliminating some parameters from the optimization always helps.

Meanwhile I've got some really promising results on the selection of frames. Looks like we can throw away 60% of the frames and still have over 0.99 correlation.

[Dogway]

This is something I also want to do... whenever I get the time. :) I've been thinking about a kind of MVToolsHQ library with focus on quality and using every trick in the book, optimized by Zopti.

Yep, so far our tests have been quite focused, truemotion=true, refinemotion=true. But I think this is fine in order to win small battles first. For example I noticed that for copulas I shouldn't mix continuous with discreet distributions so I also need to lock those args that are boolean or with a narrow range like DCT.

I also found that the CoS algo uses a gaussian copula, I don't know if it's "ok" to change that distribution type but I feel more confident using a separate algo, currently looking on T-copula as our samples (slices in the section area) are rather few.

In any case I'm going to lock those and a few more args and run a test so I can actually use all the samples (I presume most will be confined around the original 'action' area).

There are also some/many args that depend on motion/vector length. This is a double-edged sword, default for high motion or for low motion? You can't have both so in my opinion for MDegrain it's better to default for low motion as some of my visual tests have shown.

[Ceppo]

@zorr
Does your creation allow to choose the best trade off between speed/quality or it's just about quality?

EDIT:
I need to speed up CQTGMC (https://forum.doom9.org/showthread.php?p=1963693#post1963693) while losing as less precision as possible.

EDIT:
I was thinking to speed up the process you could try my CDuplicate + CReplaceDuplicate(drop=true) to remove duplicate or frames very similar to each other so that only frames with a strong differences are checked.

[zorr]

@zorr
Does your creation allow to choose the best trade off between speed/quality or it's just about quality?

It finds the "pareto front" of results, which contains all the best combinations in speed/quality. For example in this image of 100k results the red dots represent the different tradeoffs between speed and quality. The fastest result is the leftmost dot and the one with highest quality is the topmost. The other red dots are other non-dominated combinations which are "best" in their own way. They're basically the best quality you can get using this much time.

https://i.postimg.cc/1XNx45Zh/mdegrain-100k-edit.png

EDIT:
I need to speed up CQTGMC (https://forum.doom9.org/showthread.php?p=1963693#post1963693) while losing as less precision as possible.

Optimizing QTGMC (or a variation of it) directly is probably too hard, as Zopti needs some kind of reference frames to determine the quality. In theory you could make a reference by having a progressive video which you then interlace and throw away half the fields and then try to reconstruct the original frames with QTGMC, but since the algorithm also does noise removal the similarity metric is affected negatively by that.

What you can do however is to improve MVTools which QTGMC is mostly based on, if you can find better settings for MVTools it will also improve the quality and/or speed of QTGMC.

EDIT:
I was thinking to speed up the process you could try my CDuplicate + CReplaceDuplicate(drop=true) to remove duplicate or frames very similar to each other so that only frames with a strong differences are checked.

That might work but we can't remove frames freely, in order to do proper motion compensation MVTools needs two previous and two next frames also. I'm studying how to remove the insignificant frames by trying all combinations of them and comparing the result to the original calculation which uses all frames. Of course the same limitation on which frames can be removed applies there as well.

[zorr]

Yep, so far our tests have been quite focused, truemotion=true, refinemotion=true.

Truemotion is just a preset for variables lambda, lsad, plevel, pnew and global. We're optimizing all of those individually so the preset value has no effect.

What's refinemotion?

But I think this is fine in order to win small battles first. For example I noticed that for copulas I shouldn't mix continuous with discreet distributions so I also need to lock those args that are boolean or with a narrow range like DCT.

Definitely you have to start from the small battles as even they can be overwhelming at first. You could also consider all distributions discrete since Zopti doesn't support anything else. All the real-valued parameters need to be discretized.

But some of the parameters are also non-continuous, like the dct you mentioned. There's no expectation that changing dct one bigger will change the results in certain direction as it's simply switching to a different algorithm. Those might cause problems with any statistical methods. I haven't read about copulas before so I don't know how those behave though.

currently looking on T-copula as our samples (slices in the section area) are rather few.

Well nothing's stopping you from using more samples, there are 100k of them after all. :) I do get that you want to focus on the higher quality results but even there you could widen the range and get more statistically robust results using more samples. Or do several tests using different amount of samples.

In any case I'm going to lock those and a few more args and run a test so I can actually use all the samples (I presume most will be confined around the original 'action' area).

Ok, let me know when you find a good range, I can provide more results.

There are also some/many args that depend on motion/vector length. This is a double-edged sword, default for high motion or for low motion? You can't have both so in my opinion for MDegrain it's better to default for low motion as some of my visual tests have shown.

Yes for noise removal to be effective relatively low motion is required (and noticing the noise in high motion frames is much more difficult anyway).

I stumbled upon this new correlation algorithm called "distance correlation (https://en.wikipedia.org/wiki/Distance_correlation)" which can reveal not just linear dependencies but pretty much any kind of relationship. That could also be useful metric.

[Dogway]

I had a break to do some Zopti and finally did a random run with the locked params, all in all only optimizing the next:
lambda,lambdaR,pnew,pnewR,pzero,lsad,pglobal,badSAD,badrange

Very interesting results:
https://i.imgur.com/fbX0Hml.png


That diagonal blob happened by chance showing a strong relationship between the params so it looks like they are all mostly correlated in some way which makes it perfect to construct qual/perf trade-off presets.

I don't know what kind of distributions they follow, actually looking at "-vismode line" it doesn't show any pattern whatsoever even in the original sectioned AREA so I will do it "by hand" with a scatter plot matrix. 1K is not much so I will go up to around 3K.

First correlate obvious params; lambda with lambdaR, pnew to pnewR, badSAD to badrange, and all the penalties. lsad might be also related to lambda. Prune outliers and finally correlate all of them.

After this I will run again a clean 3K random run with a synthetic clip by adding grain with cretindesalpes' new plugin and reassess the results.

At the end I will add in the variables, everything should be automatized by 4 variables; thSAD, tr, clip/scene motion and qual/perf preset.
There are settings that depend on other things but they should correlate with the above 4, for example badrange is in pixel units, therefore depends on input_size*pel, same for searchRange and searchRangeFinest. lambda depends on blksize, such that by default is 1000*blksize*blksizeV/64, with 1000 being substituted with my correlation function which depends on thSAD and probably lsad.

I'm creating now a global clip stats parse filter from the output of SceneStats() log file, so one can know how much motion has the whole clip (in average) so people can set up better MVTools settings. A per-scene is also possible but some people might not want to run MVTools in a runtime environment.
A SAD analysis tool is also in the works, would like to do with expressions since MVTools is so slow.


EDIT: I was testing the new additions on -vismode line and I find them great, specially the average mode. Typically I like median more as it's more robust but it creates spikes. Do you think IQM could be added? it mixes the best of both worlds. Simply create a ranking order, prune the the first and last 25% of the samples and calculate the average of the remaining. There are softer versions though like the Trimean or Winsorized mean.

[zorr]

I had a break to do some Zopti and finally did a random run with the locked params
...
everything should be automatized by 4 variables; thSAD, tr, clip/scene motion and qual/perf preset


Looks interesting and hopefully you'll find those correlations.

After this I will run again a clean 3K random run with a synthetic clip by adding grain with cretindesalpes' new plugin and reassess the results.

Yes that's a good idea also, looks like the grain is very realistic and we can have a "perfect" reference clip.

I'm creating now a global clip stats parse filter from the output of SceneStats() log file, so one can know how much motion has the whole clip (in average) so people can set up better MVTools settings.

Would be interesting to figure out the best settings automatically based on the scene stats. That would involve a lot of testing though.

EDIT: I was testing the new additions on -vismode line and I find them great, specially the average mode. Typically I like median more as it's more robust but it creates spikes. Do you think IQM could be added?

Sure, IQM sounds very simple. There are some subtle rules when the number of results is not divisible by 4 but those are not a problem.

[anton_foy]

Yes that's a good idea also, looks like the grain is very realistic and we can have a "perfect" reference clip.
Just made a simulation of a high ISO uhd clip with and without noise if someone is interested. I recorded a clean plate with very high ISO in low light to isolate as noise and then applied it onto a clean noise free clip and matched the noise to another real noisy clip.

[DTL]

For low self-noise silicon-imaging new cameras most of noise is photon shot. It have poisson distribution on low levels and gauss at medium and high. Film grain may have different distribution. So for degrain/denoise work we need to use different noise simulators. For film intermediate image capturing and direct electronic low self-noise image capturing cameras. The film is some mix of both photon shot noise with film grain.

[Dogway]

I finished the 3K run, I'm going to plot the scatter plot matrix. I tried showing the optimize animation but all I got was rendering png's to the animation folder. Is there a way to show the animation in the scatter plot window?
I used:
zopti -mode evaluate -autorefresh true -animation 1

I also noticed the CSV output is rounding score to 3 decimals

@DTL: I haven't read the implementation papers but most content that is to denoise use to be bluray films, specially non-digital, so most of the grain would be a mix of shot noise (low light), negative stock grain, and positive print film stock grain. Each stock has its own characteristics but probably targeting a 5207+2383/2393 would work for films in the last 30 years. Grain is better (more filmic) implemented in log space by the way.

EDIT: Here the 3K plot with traced limits
https://i.imgur.com/9l7k6x4m.png (https://i.imgur.com/9l7k6x4.png)

EDIT: Here the scatter plot matrix of the first 6 patches out of 18. Maybe I need more samples.
https://i.imgur.com/qYtSZIAh.png (https://i.imgur.com/qYtSZIA.png)



EDIT: By the way, I took the liberty to optimize bicubic's b and c coefficients and for upscaling it always optimized to Catmull-Rom, although I like Precise better (adds some acutance). For downscaling it optimized to -0.83,0.14 for halving, and -1.00,0.16 for 1080p to 720p. My reference was a tuned SSIM_downsample().