Introducing Zopti (ex AvisynthOptimizer)

[Boulder]

I can't get it to work, I have the same problem.

source=DGSource("test.dgi",fulldepth=true)
b = -0.6 # optimize b = _n_ | -1..1 | b
c = 0.3 # optimize c = _n_ | -1..1 | c
filtered=BicubicResize(source, 1920, 800, b=b, c=c).Lanczos4Resize(3840, 1600)

I'm testing optimizing the downsizing parameters as mentioned earlier in this thread.

EDIT: changing to "b = 0.6 # optimize b = _n_ | -1..1 | b" got past the error.

[zorr]

I can't get it to work, I have the same problem.

source=DGSource("test.dgi",fulldepth=true)
b = -0.6 # optimize b = _n_ | -1..1 | b
c = 0.3 # optimize c = _n_ | -1..1 | c
filtered=BicubicResize(source, 1920, 800, b=b, c=c).Lanczos4Resize(3840, 1600)

I'm testing optimizing the downsizing parameters as mentioned earlier in this thread.

EDIT: changing to "b = 0.6 # optimize b = _n_ | -1..1 | b" got past the error.

I have located the bug, the current version cannot read negative number values. Will be fixed in the next version.

Also note that floats are not supported, so with range -1..1 the optimizer will only try values -1, 0 and 1. You can get around this limitation by using something like this:

c = 30/100.0 # optimize c = _n_ | -100..100 | c

which would result in range -1.0 .. 1.0 with steps of 0.01.

[Boulder]

I got it to run with this script, but the iterations all have the same SSIM value. If I open the script in VirtualDub2 and run it, the perFrameResults.txt is there but it is not produced when the script is run with optimizer.bat. I'm using the latest Avisynth+ build with your special plugins in x86 mode. If I run in x64 mode, I get the error message "java.lang.NumberFormatException: For input string: "I don't know what 'ssim' means." even though the plugins are in place and I have both environments of Avisynth+ installed.

source=DGSource("test.dgi",fulldepth=true)

b = 33/100 # optimize b = _n_/100 | -100..100 | b
c = 33/100 # optimize c = _n_/100 | -100..100 | c
denoised=BicubicResize(source, 1920, 800, b=b, c=c).Lanczos4Resize(3840, 1600)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
global ssim = SSIM_FRAME(source, denoised)
global ssim = (ssim == 1.0 ? 0.0 : ssim)
global ssim_total = ssim_total + ssim
""")

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "perFrameResults.txt" # output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last

[zorr]

I got it to run with this script, but the iterations all have the same SSIM value.
b = 33/100 # optimize b = _n_/100 | -100..100 | b

When you divide by 100 you must use a floating point number as the divisor (100.0), otherwise it's integer division and most results will then end up as 0 which explains the same SSIM value.

If I open the script in VirtualDub2 and run it, the perFrameResults.txt is there but it is not produced when the script is run with optimizer.bat.

That is normal, the optimizer changes the output file name and location, the files are at <optimizer_install_dir>/work.

If I run in x64 mode, I get the error message "java.lang.NumberFormatException: For input string: "I don't know what 'ssim' means." even though the plugins are in place and I have both environments of Avisynth+ installed.

I haven't tested x64 version myself so this could be a bug. Did you set the x64 mode with "-arch x64"?

[Boulder]

When you divide by 100 you must use a floating point number as the divisor (100.0), otherwise it's integer division and most results will then end up as 0 which explains the same SSIM value.
Thanks, I think this should be mentioned this in the tutorial post. Float works only in the first parameter, not in the range call. In the range call you get "java.lang.NumberFormatException: For input string: "-100.0".

I haven't tested x64 version myself so this could be a bug. Did you set the x64 mode with "-arch x64"?It was already in the ini file as per my first test with a clean ini.

[zorr]

AvisynthOptimizer v0.9.9-beta (https://drive.google.com/open?id=1S92SJDbTznsmFnAROQMaa-w9NqqY_MTB) released.

This version adds a new optimizer argument -initial. It's meant for setting the initial population and the choices are:

-random: the default value, creates a random initial population
-script: reads the values set in the script and makes one of the initial population members use those values. The rest of the population is random. (thanks Seedmanc for the suggestion)
-<log_file_name>: Read a log file (or files, using a wildcard *) and create the initial population from the pareto front of the results. If the pareto front is larger than the population size the chosen algorithm will prune the results (but the best result is always included). If the pareto front is smaller than the population size the rest will be filled with random individuals.


NOTE: -initial only works with the algorithm SPEA2 for now, I will add the support for the rest in the next version.


Also some much needed bug fixes:
-can now read and understand a negative value as the variable's original value in the script (thanks Seedmanc and Boulder for the report)
-optimizer will now stop when all parameter combinations have been tested. The population size will be set to number of combinations if it is larger than that (in order to avoid duplicate parameters). Thanks ChaosKing for the report.
-duplicate parameter combinations are no longer possible in the initial population (also thanks to ChaosKing for the report)

[zorr]

AvisynthOptimizer-0.9.10-beta (https://drive.google.com/open?id=1pD9qguKcAkvVNTjpfJeixcukf4yXKtAI) released.

The -initial argument now works with the algorithms NSGA-II and mutation as well.

Some more bug fixes: the input script argument without any path component now works (thanks Seedmanc for the bug report). Also the wildcard didn't work with the -initial argument but does work now.

[ChaosKing]

If you still plan to add VS support someday, there is a new filter now which can calculate a VMAF, PSNR, SSIM and MS-SSIM score (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF)

[Boulder]

It would be really nice, high bitdepth support is already a big plus and also those other scores could be useful.

[Boulder]

Running this kind of downsizing test with only one frame will hit the "DUPLICATE PARAMS" phase quite soon after the process starts. It seems to try the same values over and over again. Setting sensitivity to false helps a bit but the safest way is to sample multiple frames which will then increase the runtime quite a lot.

In this case, I used -1000..1000 as the range, normally I would go for -100..100 but it has the same problem.

TEST_FRAMES = 1 # how many frames are tested
MIDDLE_FRAME = 0 # middle frame number

source=FFVideoSource("c:\avisynthoptimizer\churchill.avi",colorspace="yv12")

b = 330/1000.0 # optimize b = _n_/1000.0 | -1000..1000 | b
c = 330/1000.0 # optimize c = _n_/1000.0 | -1000..1000 | c

denoised=BicubicResize(source, 1280, 536, b=b, c=c).LanczosResize(1920, 800)

c selected for mutation
b selected for mutation
mutated c with 38.399986267089844, value 610 -> 609
mutated b with 38.399986267089844, value -149 -> -148
param values after resolve: b -148 c 609
DUPLICATE PARAMS -148 609
change counts: b 13c 19
b selected for mutation
c selected for mutation
mutated b with 38.399986267089844, value -149 -> -149
mutated c with 38.399986267089844, value 610 -> 610
param values after resolve: b -149 c 610
DUPLICATE PARAMS -149 610
change counts: b 13c 19
c selected for mutation
b selected for mutation
mutated c with 38.399986267089844, value 610 -> 610
mutated b with 38.399986267089844, value -149 -> -149
param values after resolve: b -149 c 610
DUPLICATE PARAMS -149 610
change counts: b 13c 19
b selected for mutation
c selected for mutation
mutated b with 38.399986267089844, value -149 -> -150
mutated c with 38.399986267089844, value 610 -> 609
param values after resolve: b -150 c 609
DUPLICATE PARAMS -150 609

[zorr]

Running this kind of downsizing test with only one frame will hit the "DUPLICATE PARAMS" phase quite soon after the process starts. It seems to try the same values over and over again.

That's weird, the v0.9.9-beta should have fixed that kind of problem. I will try your script and see what I can find. You're running the latest version right?

[Boulder]

That's weird, the v0.9.9-beta should have fixed that kind of problem. I will try your script and see what I can find. You're running the latest version right?

Yes, I'm running v0.9.10-beta. Producing a 2-frame video clip and setting sensitivity to false works much better, about 14000 iterations done without the issue. This is what appears constantly:
Mutating 1 params by 30,0 %
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2
mutation failed 1000 times, increasing mutation count to 2

[zorr]

AvisynthOptimizer v0.9.11-beta (https://drive.google.com/open?id=15tPadak_g9LeU9igeGEvabgldxhF-NqQ) released.

This version mainly fixes bugs and makes some minor improvements.

So there was a bug in the JMetal metaheuristics library which I found and fixed, it affected SPEA2 algorithm's "archive truncation". There is a chance that SPEA2 will give better results after this fix, I haven't done a proper test yet to confirm.

Boulder reported a bug which made the sensitivity estimation go haywire, it happened when two consecutive population generations had the exact same runtimes. That one is fixed as well.

Also there were too many "mutation failed 1000 times..." log messages so I removed those. The situation is normal when almost all combinations have been tried already.

Series results count was sometimes reported as zero when autorefreshing chart started reading the log file at the same time, thanks for ChaosKing for the logs.

I made a text change to make it a bit clearer what the best and worst results are about: "Best result" -> "Best run", "Worst result" -> "Worst run". By comparing the best and worst you can determine how reliably the runs achieve a similar result.

Finally, the optimizer will stop running if it's unable to change parameter's value in the script (earlier it caused an error message in an infinite loop).

[zorr]

@Boulder, can you try the latest version and report if it fixes all your issues? You should now be able to use 100.0 as the divider as well.


TEST_FRAMES = 1 # how many frames are tested
MIDDLE_FRAME = 0 # middle frame number


When I tried your script I had to change MIDDLE_FRAME to 1 in order to get clip length of one frame. With zero it returned the whole clip. Does it work differently for you?

[Boulder]

@Boulder, can you try the latest version and report if it fixes all your issues? You should now be able to use 100.0 as the divider as well.
Seems to work fine, thank you.

When I tried your script I had to change MIDDLE_FRAME to 1 in order to get clip length of one frame. With zero it returned the whole clip. Does it work differently for you?

For me it worked as I was feeding a one-frame clip in the optimizer to test the resizing parameters. In Avisynth syntax, Trim(n,0) means that you get all the frames from frame n until the end of the clip. I was just thinking that the middle frame must be inside the clip range and in Avisynth, counting starts from zero.

[Boulder]

Gah, the issue appeared again quite close to the end. I set only -iters 100000 to test. At least it didn't happen right away like it did in the earlier version.

change counts: b 17c 17
b selected for mutation
c selected for mutation
mutated b with 1.463081955909729, value -68 -> 10
mutated c with 1.463081955909729, value 34 -> 100
param values after resolve: b 10 c 100
DUPLICATE PARAMS 10 100
change counts: b 17c 17
b selected for mutation
c selected for mutation
mutated b with 1.463081955909729, value -68 -> -13
mutated c with 1.463081955909729, value 34 -> -79
param values after resolve: b -13 c -79
DUPLICATE PARAMS -13 -79
change counts: b 17c 17
c selected for mutation
b selected for mutation
mutated c with 1.463081955909729, value 34 -> 100
mutated b with 1.463081955909729, value -68 -> 74
param values after resolve: b 74 c 100
DUPLICATE PARAMS 74 100
change counts: b 17c 17
c selected for mutation
b selected for mutation
mutated c with 1.463081955909729, value 34 -> 100
mutated b with 1.463081955909729, value -68 -> -30
param values after resolve: b -30 c 100

[Seedmanc]

I'm trying to add MRecalculate to optimization with the following (partial) script:
doRecalc = true # optimize doRecalc = _n_ | false,true | doRecalc
smooth=1 # optimize smooth=_n_ | 0,1 ; filter:doRecalc | smooth
RblockSize = 12 # optimize RblockSize = _n_ | 4,6,8,12,16,24,32,48,64 ; min:Rdivide 0 > 8 2 ? ; filter:doRecalc | RblockSize
RsearchAlgo = 5 # optimize RsearchAlgo = _n_ | 0..5 D ; filter:doRecalc | RsearchAlgo
RsearchRange = 4 # optimize RsearchRange = _n_ | 1..30 ; filter:doRecalc | RsearchRange
Rdivide=0 # optimize Rdivide=_n_ | 0..2 D ; max:RblockSize 8 >= 2 0 ? ; filter:doRecalc | Rdivide
Roverlap=4 # optimize Roverlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:RblockSize 2 / ; filter:doRecalc | Roverlap
Rdct = 0 # optimize Rdct = _n_ | 0,1,2,3,4,5,6,7,8,9,10 D ; filter:doRecalc | Rdct
Rmeander = true # optimize Rmeander = _n_ | false,true ; filter:doRecalc | Rmeander
RscaleCSAD = 2 # optimize RscaleCSAD = _n_ | -2..2 ; filter:doRecalc | RscaleCSAD
thsad = 200 # optimize thsad = _n_ | 0..10000 ; filter:doRecalc | thsad

bv = doRecalc ? MRecalculate(super_render, bv, thsad=thsad, smooth=smooth, blksize=RblockSize, search=RsearchAlgo, searchparam=RsearchRange, truemotion=false, overlap=Roverlap, dct=Rdct, divide=Rdivide, meander=Rmeander, scaleCSAD=RscaleCSAD) : bv
fv = doRecalc ? MRecalculate(super_render, fv, thsad=thsad, smooth=smooth, blksize=RblockSize, search=RsearchAlgo, searchparam=RsearchRange, truemotion=false, overlap=Roverlap, dct=Rdct, divide=Rdivide, meander=Rmeander, scaleCSAD=RscaleCSAD) : fv


but Optimizer fails with
java.lang.Exception: Unknown element doRecalc
at avisynthoptimizer.parser.RPNParser.push(RPNParser.java:124)
at avisynthoptimizer.AviSynthOptimizer.filterValue(AviSynthOptimizer.jav
a:3709)
What could be wrong? I tried replacing filter with max, not helpful. For now I had to get rid of filtering by doRecalc, but that's wasting iterations.

[zorr]

AvisynthOptimizer v0.9.12-beta (https://drive.google.com/open?id=1kjo-GK7Pt5aRpFpDwBwc7K4lF7l5lCeD) released.

Some minor bug fixes and improvements:


set the logging threshold higher for reporting the mutation failures, now has to be at least 100 000 failed attempts before logging starts (thanks @Boulder for the report)
bugfix: conflict resolvation didn't support variables which had boolean types in min/max/filter definition, it does now. This is related to the bug @Seedmanc reported above, thanks! Also related: the optimizer will stop running if it's unable to resolve conflicts in 100 000 tries.
avsr version updated to latest v0.1.9. It has a cool new -frames argument which I'm going put to good use, but the new feature (automatic frame selection) using it is not finished yet. Thanks @Groucho2004!

[zorr]

I'm trying to add MRecalculate to optimization with the following (partial) script:
doRecalc = true # optimize doRecalc = _n_ | false,true | doRecalc
smooth=1 # optimize smooth=_n_ | 0,1 ; filter:doRecalc | smooth

but Optimizer fails


The first problem here was that the optimizer didn't support variables with boolean types (like the variable "doRecalc"). The latest version v0.9.12-beta does so grab that one.

The second problem is that the filter rejects all the values when doRecalc==false, therefore the conflict resolvation cannot finish (it expects to find some combination of parameters which are valid).

How to fix this? You have to return at least one valid value for every parameter. Using variable RblockSize as an example, the filter could be like this:
filter:doRecalc x 16 ? x ==

The logic here is that when doRecalc is true the rest of the equation becomes (x == x) which is always true so it accepts all the values. When doRecalc is false the equation becomes (x == 16) accepting one value. Since the MRecalculate is disabled the value we choose doesn't matter but we have to choose something anyway.

If the parameter already has a filter, you can add min or max definition (easier to separate the logic than try to make one more complex filter).

Min:
min:doRecalc 4 64 ?

Max:
max:doRecalc 64 4 ?

The min will set 64 as the minimum value when doRecalc is false, which leaves only the value 64 as valid. The max is similar but sets the maximum as 4.

So it's a bit of work to add all these min/max/filter definitions, I will have to think if there's an easier way to support this kind of disabling of parametes.

Another way is to make a separate script for the MRecalculate-version. Optimize both scripts and see which one gave better results. I tried this and to my surprise the script without MRecalculate was better. I think it might be because there are much more parameters to optimize in the MRecalculate-script and therefore the search doesn't get anywhere close to optimal results in the same iteration count.

[Boulder]

I got this one with the latest version. It happened quite close to the end (once again those resizing tests of mine) and seems to happen with every run. I fired up five concurrent optimizations and three have now failed with the error around the ~39000th iteration.

java.lang.IllegalArgumentException: bound must be positive
at java.util.Random.nextInt(Unknown Source)
at avisynthoptimizer.AviSynthOptimizer.selectRandomWeightedByChangeCount(AviSynthOptimizer.java:2493)
at avisynthoptimizer.AviSynthOptimizer.mutateResult(AviSynthOptimizer.java:5049)
at avisynthoptimizer.nsga_ii.RangeMutation.execute(RangeMutation.java:93)
at avisynthoptimizer.nsga_ii.RangeMutation.execute(RangeMutation.java:25)
at avisynthoptimizer.spea2.DynamicSPEA2.reproduction(DynamicSPEA2.java:103)
at org.uma.jmetal.algorithm.impl.AbstractEvolutionaryAlgorithm.run(AbstractEvolutionaryAlgorithm.java:60)
at java.lang.Thread.run(Unknown Source)

[zorr]

I got this one with the latest version. It happened quite close to the end (once again those resizing tests of mine) and seems to happen with every run. I fired up five concurrent optimizations and three have now failed with the error around the ~39000th iteration.


Thanks for that report, I released v0.9.13-beta (https://drive.google.com/open?id=1rNrcqoqjkTpFaVaEm71tRsqUiwUdE6q7) where this is fixed.

When I changed the mutation failures logging threshold higher I introduced a bug where the code wanted to mutate more parameters than exists. Nothing good will happen when you try that. :)

[Seedmanc]

Something is still wrong, before 0.9.13 it was doing like 60 iterations overnight where it previously done thousands, now it just failed after 13 (says "java.lang.NumberFormatException: For input string: "true"), here's the script https://pastebin.com/HZH4Ac8v
zorr, previously you mentioned that every parameter involving other parameters in filters/minmaxes should "mirror" that dependency in the description of that parameter as well to avoid bias. In my example I tie a lot of recalculate params to doRecalc, but don't tie it to them, will that affect the script, and if so, how?
Also, in my script, I filter overlap by blocksize/2, but don't do filter blocksize by overlap, same question.

Finally, could you add a != to a list of supported operations? I was trying to limit DCT=1 to blocksizes less or equal to 24 by doing filter:blockSize 24 > 1 -1 ? x !=. Is there a better way?

[zorr]

Something is still wrong, before 0.9.13 it was doing like 60 iterations overnight where it previously done thousands, now it just failed after 13

I'm looking into this, meanwhile here's a link to v0.9.12 (https://drive.google.com/open?id=1kjo-GK7Pt5aRpFpDwBwc7K4lF7l5lCeD) if you don't have it anymore.

[zorr]

Something is still wrong, before 0.9.13 it was doing like 60 iterations overnight where it previously done thousands, now it just failed after 13 (says "java.lang.NumberFormatException: For input string: "true")

Thanks again for the report. I missed one part where boolean types were still not supported in the conflict resolvation. That's why it crashed with NumberFormatException, but it also affected the choice of parameters and pretty much never set doRecalc=false (at least in my test). Version 0.9.14-beta (https://drive.google.com/open?id=1AwQd_7OG5BHda6UE_XsSmv5Tffaya17Z) fixes this.

The slowness I think might be due to always doing the MRecalculate, in my tests while most runs took less than half a second, there were occasional long ones taking as long as 274 seconds. If it sticks with those slow parameter combinations it would only run 13 tests per hour which is in the same ballpark as your runs. You can look in the log files if there are individual slow runs which could explain it.

zorr, previously you mentioned that every parameter involving other parameters in filters/minmaxes should "mirror" that dependency in the description of that parameter as well to avoid bias. In my example I tie a lot of recalculate params to doRecalc, but don't tie it to them, will that affect the script, and if so, how?

The result of the script execution (quality and speed) would be the same without the doRecalc-dependency so it's only trimming down the number of combinations and not introducing any bias.

Also, in my script, I filter overlap by blocksize/2, but don't do filter blocksize by overlap, same question.

This will introduce some bias. Basically every time blocksize and overlap conflict, it's the overlap that will get changed. Here's an example case: let's say blocksize is 16 and overlap is 8. Only one parameter is mutated (we're in the final stretch of the optimization) and overlap is selected for mutation. If the overlap is trying to become bigger (let's say 12) the blocksize is in conflict and now the overlap will be rolled back to 8. If it's trying to become smaller it can still do that, so that means overlap can only change into one direction.

Finally, could you add a != to a list of supported operations? I was trying to limit DCT=1 to blocksizes less or equal to 24 by doing filter:blockSize 24 > 1 -1 ? x !=. Is there a better way?

The support for != is also in the latest version so filter:blockSize 24 > 1 -1 ? x != should now work. I couldn't come up with anything cleaner. I think I should also add some boolean operators (at least AND and OR).

[EDIT] Almost forgot, you can save 1-3 milliseconds per frame if you do the scaling outside FrameEvaluate. Something like this:

divisor = 4
scaled_orig = orig_yv12.bilinearresize(orig_yv12.width/divisor,orig_yv12.height/divisor)
scaled_inter = inter_yv12.bilinearresize( inter_yv12.width/divisor, inter_yv12.height/divisor)

global total = 0.0
global ssim_total = 0.0
global avstimer = 0.0
frame_count = FrameCount()
FrameEvaluate(last, """
global ssim = SSIM_FRAME(scaled_orig, scaled_inter)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""")

[zorr]

The result of the script execution (quality and speed) would be the same without the doRecalc-dependency so it's only trimming down the number of combinations and not introducing any bias.


I may have to retract that. It doesn't cause bias immediately but when a result with doRecalc=false gets mutated so that doRecalc becomes true then all the parameters for MRecalculate are having the values specified in the filter/min/max. Those values are then going to be tested a lot more than others. It might be a good idea to test what kind of results you get with and without the doRecalc dependency.

I'm running your script with the doRecalc dependency enabled and this time the MRecalculate is winning, all except the fastest two pareto front results use it.

https://i.postimg.cc/W1NjS4KK/seedmanc05.png

Top 3 results, running on a clip of Frozen:

9.941789 1900 super_pad=48 super_pel=4 super_sharp=2 super_rfilter=3 blockSize=12
searchAlgo=2 searchRange=2 searchRangeFinest=6 divide=2 overlap=4 badSAD=4045
badRange=27 negBadRange=true meander=false temporal=true trymany=true dct=1
scaleCSAD=-2 doRecalc=true smooth=0 RblockSize=8 RsearchAlgo=5 RsearchRange=3
Rdivide=0 Roverlap=0 Rdct=10 Rmeander=true RscaleCSAD=1 thsad=0 maskScale=4

9.941031 1670 super_pad=48 super_pel=4 super_sharp=2 super_rfilter=3 blockSize=12
searchAlgo=0 searchRange=2 searchRangeFinest=6 divide=2 overlap=4 badSAD=545
badRange=2 negBadRange=true meander=false temporal=true trymany=true dct=1
scaleCSAD=-2 doRecalc=true smooth=1 RblockSize=8 RsearchAlgo=2 RsearchRange=3
Rdivide=0 Roverlap=0 Rdct=7 Rmeander=true RscaleCSAD=2 thsad=59 maskScale=1

9.940445 1650 super_pad=48 super_pel=4 super_sharp=2 super_rfilter=3 blockSize=12
searchAlgo=0 searchRange=2 searchRangeFinest=6 divide=1 overlap=4 badSAD=860
badRange=5 negBadRange=true meander=false temporal=true trymany=true dct=1
scaleCSAD=-2 doRecalc=true smooth=0 RblockSize=8 RsearchAlgo=2 RsearchRange=3
Rdivide=0 Roverlap=0 Rdct=0 Rmeander=true RscaleCSAD=2 thsad=62 maskScale=1

[Boulder]

AvisynthOptimizer v0.9.12-beta (https://drive.google.com/open?id=1kjo-GK7Pt5aRpFpDwBwc7K4lF7l5lCeD) released.

Some minor bug fixes and improvements:


set the logging threshold higher for reporting the mutation failures, now has to be at least 100 000 failed attempts before logging starts (thanks @Boulder for the report)


The mutation failures still pop up near the end and apparently slow down the process. Is it possible to tell the optimizer to just test all the possible combinations? As I have only two parameters to adjust, there's a finite amount of them and I want to see all of them tested anyway. It would also be nice to have a graph of such cases with the two parameters as x- and y-axis.

[zorr]

The mutation failures still pop up near the end and apparently slow down the process. Is it possible to tell the optimizer to just test all the possible combinations? As I have only two parameters to adjust, there's a finite amount of them and I want to see all of them tested anyway.

I have pondered about such functionality myself. As of now it's not possible and in the end it's getting harder and harder to find a mutation which hasn't already been tried before. It would be much simpler to just iterate all the combinations in order. I'll implement that as a new algorithm variation.

What's the divider you're using with your two parameters (ie how many combinations are you testing)?


It would also be nice to have a graph of such cases with the two parameters as x- and y-axis.

And the color of the x,y position would represent the value? The chart library I'm using doesn't support such a visualization but I think I can find a way to implement that.

[Boulder]

I have pondered about such functionality myself. As of now it's not possible and in the end it's getting harder and harder to find a mutation which hasn't already been tried before. It would be much simpler to just iterate all the combinations in order. I'll implement that as a new algorithm variation.

What's the divider you're using with your two parameters (ie how many combinations are you testing)?
The amount of combinations is a little over 40000 (two parameters, and possible values -100..100).



And the color of the x,y position would represent the value? The chart library I'm using doesn't support such a visualization but I think I can find a way to implement that.
Yes, that should do it. I suppose it is possible to change the colors on the fly as we don't know where the top will be, or did you think the min-max values would determine what color grade to use? At least in my resizing tests, the values have hit a quite narrow range compared to the min-max range.

[Boulder]

As many tests have been run with MVTools, is there a generic way to optimize the parameters for motion estimation? I mean, is the "add fake noise" - "denoise" the way or is MFlow or MCompensate better in such cases? As the resizing optimization is a rather simple task, it would be interesting to optimize the motion search parameters for my denoising function as well. Of course, SSIM may not be the optimal measuring method there but currently VMAF is Vapoursynth-only.

[zorr]

AvisynthOptimizer v0.9.15-beta (https://drive.google.com/open?id=1u6iyVXsTvGLZaoR8RD9rfepK8RdLQxJy) released.

This version adds the "exhaustive" algorithm which tests all the valid parameter combinations (unless the number of combinations is greater than 2147483647, in which case it "only" tries the first 2147483647 combinations. :) When running the exhaustive algorithm the number of runs is one by default, you can still set it higher by providing the -runs argument. The other optimization arguments are not relevant.

I also added operators "and" and "or" to conflict resolvation parser. I couldn't use the more familiar && and || since the "|" is already used for separating the definition parts.

There's also a bugfix: the filter dependency did not find any valid values when the parameter was defined as a range.

@Boulder I will work on the 2D heatmap-style visualization next.

[Boulder]

Thank you, the exhaustive algorithm will be very useful :)

[Boulder]

There's a small bug with the selection of frames. For example, I've set to test 20 frames of a clip (TEST_FRAMES = 20, MIDDLE_FRAME = 10) but when I open the script used for calculations, I get 19 frames in VDub.

[StainlessS]

Should TEST_FRAMES be an odd number ? (otherwise MIDDLE _FRAME aint middle frame).
With TEST_FRAMES=20, MIDDLE _FRAME=10, so 10 frames prior to middle frame and 9 after.
(maybe it just rounds to 9 frames before, and 9 frames after, where frame 0 is skipped, is it frame 0 missing ?).

[zorr]

There's a small bug with the selection of frames. For example, I've set to test 20 frames of a clip (TEST_FRAMES = 20, MIDDLE_FRAME = 10) but when I open the script used for calculations, I get 19 frames in VDub.

That's odd, it works for me. I verified this by opening the script in VDub and looking the File / File Information window where you can see the length in frames.

The formula is clip.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)

Doing the math we get clip.Trim(10 - 10 + 1, 10 + 10) which is clip.Trim(1, 20)

Using Trim with positive arguments it returns the frames between the arguments including the first and last frame, that would be frames from 1 to 20 and that's 20 frames. Even using MIDDLE_FRAME 9 we would get 20 frames, this time from 0 to 19.

Perhaps it's not very intuitive if you want the first frames. For that use case a plain Trim(0, -TEST_FRAMES) would probably be better.

[Boulder]

That's odd, it works for me. I verified this by opening the script in VDub and looking the File / File Information window where you can see the length in frames.

The formula is clip.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)

Doing the math we get clip.Trim(10 - 10 + 1, 10 + 10) which is clip.Trim(1, 20)

Using Trim with positive arguments it returns the frames between the arguments including the first and last frame, that would be frames from 1 to 20 and that's 20 frames. Even using MIDDLE_FRAME 9 we would get 20 frames, this time from 0 to 19.

Perhaps it's not very intuitive if you want the first frames. For that use case a plain Trim(0, -TEST_FRAMES) would probably be better.
Thanks, that simple command works fine. I always create a noncompressed testclip for my resize tests by selecting random frames with SelectEvery, and the amount varies between 4-20 frames depending on how much time I have :)

[Seedmanc]

zorr, in your experience, did you notice any relation between the resulting searchRange and searchRangeFinest? I'm trying to figure out which one should normally be larger than the other, but the observations are inconclusive.

Have you tried optimizing scripts with the MCompensate trick yet? I've just started and already found that overlap=0 crashes mvtools.

[Boulder]

I have a small request regarding the log file: would it be possible to have the result of the best score written as the last item like it is in the command prompt output? When I run multiple consecutive optimizations with a for loop, the information is lost and I need to use Excel to get the parameters to use in the final script for encoding.

[Seedmanc]

Or maybe even sort the entire log by SSIM upon finishing the pass, been thinking about this as well.

[Boulder]

Or maybe even sort the entire log by SSIM upon finishing the pass, been thinking about this as well.

True, that would also be a good option.

[pinterf]

I've just started and already found that overlap=0 crashes mvtools.
Big thanks for the report, regression in v37, fixed in v38!

[Boulder]

Big thanks for the report, regression in v37, fixed in v38!

Hey, while you are here.. could you consider porting the VMAF filter to Avisynth+? :) Currently it's Vapoursynth-only, and it could be really useful with the optimizer.

https://github.com/Netflix/vmaf

[zorr]

zorr, in your experience, did you notice any relation between the resulting searchRange and searchRangeFinest? I'm trying to figure out which one should normally be larger than the other, but the observations are inconclusive.

This is a good opportunity for me to introduce the latest feature: heat map visualization. To answer your question let's make a heatmap of these two parameters. The command is:

optimizer -mode evaluate -vismode seriesheatmap -map searchRange searchRangeFinest -log "../scripts/some_script*.log"

It will display a heat map where the brightest colors represent the best results. With my logs (a total of 77902 results) it looks like this:

https://i.postimg.cc/3xLCbnGj/search-Range-search-Range-Finest-top100.png

It's a bit difficult to see where the best results are so let's focus on the best 10% of results by adding -top 10 to the command:

https://i.postimg.cc/dtYB77vC/search-Range-search-Range-Finest-top10.png

Ok, so it looks like most of the good results have searchRange=2 and searchRangeFinest is not that important.

But wait, let's see another set of results which are using a different source video and also a different script (a total of 271729 results):

https://i.postimg.cc/QdNJbRFx/search-Range-search-Range-Finest-top100-bridge.png

Hmm... this one was using -top 100 but we need to focus on the best results again, using -top 20:

https://i.postimg.cc/Ghxjvch5/search-Range-search-Range-Finest-top20-bridge.png

Oh, this time the best results have searchRangeFinest 1 or 2 and searchRange is not that important.

So I guess there's no fixed rule. Maybe it's possible to get good results in both ways, but the results here are a good indication the the most optimal settings sometimes need searchRange larger than searchRangeFinest and sometimes the opposite.

Here's the latest version v0.9.16-beta (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) which includes the heat map visualization.


Have you tried optimizing scripts with the MCompensate trick yet? I've just started and already found that overlap=0 crashes mvtools.

Sorry, haven't gotten around to do that yet. I did do some experiments several months ago.

[zorr]

I have a small request regarding the log file: would it be possible to have the result of the best score written as the last item like it is in the command prompt output? When I run multiple consecutive optimizations with a for loop, the information is lost and I need to use Excel to get the parameters to use in the final script for encoding.

I use the -mode evaluate for this purpose. It can search for and show the best result (and the whole pareto front) from one or multiple log files. You don't need to save the values either, just add -scripts best and you get a script with the best settings already set.

Oh, and maybe you already noticed but the latest version (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) has the heat map visualization you asked for. :) Since your script only has two parameters you don't need to specify the -map argument (the default is taking the fist two parameters from the script). It works with the autorefresh feature so if you want to watch the process you can call it like this:

optimizer -mode evaluate -vismode heatmap -autorefresh true

If you want to analyze multiple runs then use -vismode seriesheatmap.

https://i.postimg.cc/7YdhmRYr/b-c-top20.png

[Boulder]

I use the -mode evaluate for this purpose. It can search for and show the best result (and the whole pareto front) from one or multiple log files. You don't need to save the values either, just add -scripts best and you get a script with the best settings already set.Hmm, there must some bug there as I've tried that and -scripts bestofrun, but there are no scripts to check out after the analysis is complete. The work folder is empty and the only scripts in the optimizer folder are the original ones used for the analysis.

Oh, and maybe you already noticed but the latest version (https://drive.google.com/open?id=1s5LbJkas6-H7X5FfCEg0MoiNyl0cEbzB) has the heat map visualization you asked for. :)
Thank you very much for this, it looks really useful for my purposes. Much easier to see the ballpark for the sane values to use if I run a quick test for some episode of a series first :)

[zorr]

Hmm, there must some bug there as I've tried that and -scripts bestofrun, but there are no scripts to check out after the analysis is complete. The work folder is empty and the only scripts in the optimizer folder are the original ones used for the analysis.


Thanks for the report. I tried to reproduce this but didn't find any issues. The scripts should be written into the same directory where the original script file was. And the original script folder is read from the log files. The first line starting with #script has the script path that will be used. What does that line look like in your log files?

# script D:\optimizer\bin/script.avs
# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
19.70072 1980 b=-7 c=-93
19.797161 1980 b=41 c=66
19.828617 1980 b=-86 c=-27
19.700043 1970 b=28 c=-56
19.80134 1970 b=-75 c=-67
...

[Boulder]

Running optimizer test2.avs -alg exhaustive -scripts best

# script C:\AvisynthOptimizer/test2.avs
# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
0.987764 70 b=-80 c=10
0.987764 70 b=-79 c=10
0.98775 70 b=-78 c=10
0.987745 70 b=-77 c=10

At the end of the run, I get this information, but there is no script with that best data available. Test2.avs is like it is saved by me.
Pareto front:
0.987969 70ms b=-68 c=52
0.987969 70ms b=-68 c=49
0.987969 70ms b=-72 c=45
0.98794 60ms b=-54 c=59


The heatmap is indeed very useful. I can easily see which values for b I can safely leave out and reduce the amount of combinations by a fair amount :)

[zorr]

Running optimizer test2.avs -alg exhaustive -scripts best

# script C:\AvisynthOptimizer/test2.avs
# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
0.987764 70 b=-80 c=10
0.987764 70 b=-79 c=10
0.98775 70 b=-78 c=10
0.987745 70 b=-77 c=10

At the end of the run, I get this information, but there is no script with that best data available. Test2.avs is like it is saved by me.
Pareto front:
0.987969 70ms b=-68 c=52
0.987969 70ms b=-68 c=49
0.987969 70ms b=-72 c=45
0.98794 60ms b=-54 c=59


Ok now I see the problem. The -scripts is an argument that only works with -mode evaluate. So after your optimization is done (or even during it) you should call:

optimizer -mode evaluate -scripts best


The heatmap is indeed very useful. I can easily see which values for b I can safely leave out and reduce the amount of combinations by a fair amount :)

That's nice! You can also use a filter to reduce the number of tried combinations by forcing the values to be divisible by 10 (or any other number). For example:

b = 33/100.0 # optimize b = _n_/100.0 | -100..100 ; filter:x 10 % 0 == | b

[Boulder]

Ok now I see the problem. The -scripts is an argument that only works with -mode evaluate. So after your optimization is done (or even during it) you should call:

optimizer -mode evaluate -scripts best

Thanks, will try that. What if I have multiple log files, like I usually do after the for loop procedure.

[zorr]

Thanks, will try that. What if I have multiple log files, like I usually do after the for loop procedure.

You can specify the log files with the -log argument and using the wildcard *. Example:

optimizer -mode evaluate -log "./scripts/part_of_filename*.log"

If your logs are from the same script then they should all start with the script name and you can use -log "./path/script_name*.log". Or maybe you put all the log files into the same directory, then you can use -log "./path_to_directory/*.log". Maybe you only want to analyze the files from script "abc" created in this month, then you can use -log "./path/abc*2018-12*.log". And so on.

[Boulder]

Cheers, that will smooth things out a lot. It's a bit of a pain putting everything through Excel. I have to do that already too many times at work these days :D