Zopti

[zorr]

Zopti is what was called AvisynthOptimizer. The name was changed because it now supports VapourSynth as well. This first post is an introduction similar to this one on the Avisynth Development forum, just changed a bit to reflect the latest features and what is possible with VapourSynth. I also cut out some unnecessary blabbering.

Let's say you want to motion compensation to double frame rate or replace a few corrupted frames. The right tool for that job is obviously MVTools. The only problem now is finding good parameters to get the best quality (because every video needs a little different settings). Ok no problem, I will just quickly adjust these... wait... there are about 60 parameters to adjust! Finding good settings would involve *a lot* of manual testing. This is where Zopti jumps in and says "Don't worry pal, I will find the best settings for you!".

But how? A VapourSynth script is first "augmented" by adding instructions on which parameters should be optimized. The instructions are written inside comments so the script will continue to work normally. The script also needs to measure the quality of the processed video using current parameter values or measure the running time (or preferably both the quality and runtime). The script needs to write these results to a specific file. There is a helper Python library "zoptilib" you can use to do the quality/runtime measurements and write the file.

Now you might be wondering how on earth can the script measure the quality. I can only think of one way: to compare the frames to reference frames and measure the similarity. The closer the similary value, the better the quality. VapourSynth has (at least) these similarity metrics available: SSIM (the classic, most well known), GMSD (seems better than SSIM) and VMAF (by Netflix). It's also possible to get MS-SSIM (multi-scale SSIM) from the VMAF plugin and I also implemented B-SSIM for AviSynth (SSIM with blurring adjustment).

Ok, but where do we get the reference frames? In case of MVTools we can use the original frames as reference frames and the script will try to reconstruct them using motion compensation (but it's not allowed to use the reference frame in the reconstruction). We can do something similar if we want to use MVTools to double the framerate: we first create the double-rate video, then remove the original frames from it, then double the framerate again and finally compare these to the original frames. Or you could reconstruct every odd frame using the even frames. This idea is not limited to MVTools, you could for example do color grading using some other software and then try to recreate the same result using VapourSynth. I'm sure the smart people here will find use cases I couldn't even dream about.

MVTools is a complex plugin and the settings can change the runtime of the script dramatically. In addition to quality we can also measure the runtime to find settings that are fast enough (for example if you want to use the script in real-time).

So if the purpose is to find the best settings, what do we consider the "best" when both quality and time are involved? For example we can have settings A with quality 99 (larger is better), time 2300ms (larger is worse) and settings B with quality 95 and time 200ms. Which one of these is better? We can use the concept of pareto domination to answer this question. When one solution is at least as good as the other solution in every objective (in this case the objectives are quality and speed) and better in at least one objective, it dominates the other solution. In this example, neither dominates the other. But if we have settings C with quality 99 and time 200ms it dominates both A and B. In the end we want to know all the nondominated solutions, which is called the pareto front. So there's going to be a list of parameters with increasing quality and runtime. You can have more than two objectives if you want, the same pareto concept works. There's a good and free ebook called "Essentials of Metaheuristics" which describes the pareto concept and much more.

Zopti reads the augmented script, verifies it and starts running a metaheuristic optimization algorithm. There are currently four algorithm choices: NSGA-II, SPEA2, mutation and exhaustive. The first two are some of the best metaheuristic algorithms available and also described in the ebook mentioned above. The mutation is my own simplistic algorithm (but it can be useful if you're in a hurry since it's the most efficient). Exhaustive will try all possible combinations, it's useful if you only have a small number of them (you could for example do an exhaustive search on one parameter value only). All the metaheuristic algorithms are working in a similar manner generating different solutions and testing them. The optimizer does these tests by creating scripts with certain parameter values and running them. The script then writes the measurements (quality/time) into a file which the optimizer reads. The metaheuristic then decides which parameters to try next based on the results. This continues until some end criteria is met.

There are three different ending criterias: number of iterations, time limit and "dynamic". Number of iterations is just that, the algorithm runs the script specific number of times. Setting a time limit can be pretty useful if you know how much time you can use for the optimization. You could for example let it run overnight for 8 hours and see the results in the morning. Dynamic variation is stopping only when it doesn't make any progress anymore. Making progress is defined by "no more pareto front members in last x iterations". This can be useful if you want to find the best possible results regardless of how long it takes.

During the optimization all tested parameters and their results are written to a log file. This log file can be "evaluated" during and after the optimization process. The evaluation basically means finding the pareto front and showing it. You can also create scripts from the pareto front in order to test them yourself. It's also possible to visualize the results of the log file in a two dimensional scatter chart. This chart highlights the pareto front and shows all the other results too. The chart can also be "autorefreshing": it loads the log file every few seconds and updates the visuals, which is a fun way to track how the optimization is progressing. Here's a gif what it looks like (obviously sped up):



The visualization has a few other bells and whistles but one I'd like to highlight here is the group by functionality: you can group the results by certain parameter's values and show a separate pareto front for each value. For example this what grouping by MVTools' blocksize looks like:



Another visualization mode draws a heat map where two parameters are chosen for the x and y axis, the color of the cell at x,y is the best result found with that parameter combination, brighter color meaning better result.



Measuring the script's runtime is not very accurate, ie it has some variation. All the other processes running at the same time are using CPU cycles and messing with the cache so you should try to minimize other activity on the computer. In order to get more accurate results you can run a validation on the finished results log file. In validation the idea is to run the pareto front results multiple times and calculate the average, median, minimum or maximum of these multiple measurements (you can decide which one(s)).

So how good is the optimizer? Let's take a look at one example. A while back there was a thread about best motion interpolation filters. There's a test video with a girl waving her hand. The best options that I know of are John Meyer's jm_fps script and FrameRateConverter. Here's a comparison gif with those two and Zopti. FramerateConverter was run with preset="slowest".



Now obviously I'm showing a bit of a cherry-picked example here. The optimizer was instructed to search the best parameters for this short 10 frame sequence. I have run most of my optimization runs using only 10 frames because otherwise the optimization takes too long. Ideally the optimizer would automatically select the frames from a longer video, I have started working on such a feature but it's not finished yet (got sidetracked to implement a scene change detector...) so for now user has to make the selection.

At this point I envision that Zopti can be an useful tool for plugin authors so they can test plugin parameters and try to search for the optimal ones. Zopti can also be a bug hunter, it has found several bugs with the Avisynth MVTools by Pinterf which he has since fixed. The VapourSynth MVTools is currently not yet as robust, but jackoneill has already fixed one bug (thanks!). At some later point Zopti could be useful for normal users, when combined with a script with limited search space so that the search will not take excessively long time. Zopti (or rather the previous version AvisynthOptimizer) has already been used by some courageous people from these forums for scaling, motion compensation, denoising and HDR to SRD tone-mapping. I'd like to thank them all for providing important feedback, feature requests and bug reports.

You can download the latest Zopti version here. I will keep this link updated.

Version history:

1.2.3 (full details here)

• new mode: convert
  -converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
  -supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
  -output formats: CSV and zopti
  -example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
  -example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
• improvements to visualization mode: line
  -now supports option -types which controls which aggregate lines are displayed in the chart
  -valid values are "best", "worst", "average", "median", "count" and "samples"
• display of option -groupby improved, pareto fronts can no longer be cut off the screen
• option -continue also supported with mutation algorithm
• validation mode behavior change: it now validates all the results in the log file and not just the pareto front

1.2.2

• supports a new way for an Avisynth script to write the result file
  -first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
  -then come the per frame result lines in any order, using the same format as before
  -do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
  -this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
  -old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
  -new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
• bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
• bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value

1.2.1

• new option: retry
  -tries to run the script again this many times if execution fails
  -can be useful if the script uses plugins which are not 100% reliable and can crash
  -default value is zero (do not retry)
  -example: zopti script.avs -pop 24 -iters 10000 -retry 4
• new visualization mode: line
  -draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
  -accepts parameter -range to limit the displayed values to certain range
  -black line shows the best values, blue line indicates the number of results per value
  -a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
  -example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
• a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
• mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
• reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
• scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
• support for custom output properties written by the script (only in VapourSynth scripts for now)
• better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
• option -continue now also supported by exhaustive algorithm (when using more than one thread)
• order of runs in a visualization is now based on file name, not by modified date
• all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
• visualization mode seriespareto no longer contains the global pareto line
• VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
• bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
• bugfix: fixed a memory leak in XChart (at least partially)

Pre-1.2.1 history removed due to character count limitation.

The next post will be a hands-on denoising tutorial. Stay tuned.

[ChaosKing]

Nice work!

But I get a file not found error :-)

java.io.FileNotFoundException: D:\Download\Zopti-1.0-beta\work\result_1549155407307_1549155407426.txt
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(Unknown Source)
        at java.io.FileInputStream.<init>(Unknown Source)
        at java.io.FileReader.<init>(Unknown Source)
        at avisynthoptimizer.file.FileUtil.readLinesFromFile(FileUtil.java:58)
        at avisynthoptimizer.AviSynthOptimizer.readVapoursynthResults(AviSynthOptimizer.java:6566)
        at avisynthoptimizer.AviSynthOptimizer.waitAndReadResults(AviSynthOptimizer.java:6587)
        at avisynthoptimizer.AviSynthOptimizer.runAVS(AviSynthOptimizer.java:6308)
        at avisynthoptimizer.AviSynthOptimizer.evaluateFitness(AviSynthOptimizer.java:5588)
        at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:190)
        at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:35)
        at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.lambda$evaluate$1(SequentialSolutionListEvaluator.java:24)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
        at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.evaluate(SequentialSolutionListEvaluator.java:24)
        at avisynthoptimizer.spea2.DynamicSPEA2.evaluatePopulation(DynamicSPEA2.java:76)
        at org.uma.jmetal.algorithm.impl.AbstractEvolutionaryAlgorithm.run(AbstractEvolutionaryAlgorithm.java:56)
        at java.lang.Thread.run(Unknown Source)
java.lang.ArrayIndexOutOfBoundsException: -1
        at java.util.ArrayList.elementData(Unknown Source)
....

My script:

from zoptilib import Zopti
output_file = r'D:\results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['ssim', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 4					# optimize sigma = _n_ | 1,10 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)
	
alternate.set_output()

java -jar .\Zopti.jar D:\z.vpy
vspipe path is set in the ini file.

[zorr]

Originally Posted by ChaosKing

Nice work!

But I get a file not found error :-)

The # output line should probably look like this:

output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"

The file name must match.

Also this line

alternate.set_output()

is not necessary, Zopti sets the output. In some cases it is critical that the correct file is set as the output. The alternate file is the correct one in this case but for some reason it doesn't work if it's set first by Zopti and then again in the script.

[ChaosKing]

Thx, works now without alternate.set_output()

[ChaosKing]

I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13

Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)

VMAF just stops:

Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.

Maybe I do something wrong. Here is my script again

from zoptilib import Zopti
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 35					# optimize sigma = _n_ | 9..40 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)

result_*.txt:

<?xml version="1.0"?>
<VMAF version="1.3.11">
	<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
	<fyi numOfFrames="100" aggregateVMAF="96.037" execFps="1.36185" timeTaken="73.4294" />
	<frames>
		<frame frameNum="0" adm2="0.983001" motion2="0" vif_scale0="0.709537" vif_scale1="0.953811" vif_scale2="0.974778" vif_scale3="0.983557" vmaf="91.1718" />
		<frame frameNum="1" adm2="0.98031" motion2="14.7504" vif_scale0="0.760672" vif_scale1="0.952192" vif_scale2="0.97319" vif_scale3="0.982214" vmaf="100" />

		<frame frameNum="98" adm2="0.992219" motion2="1.11641" vif_scale0="0.824504" vif_scale1="0.974561" vif_scale2="0.986421" vif_scale3="0.991392" vmaf="95.7912" />
		<frame frameNum="99" adm2="0.990791" motion2="3.7195" vif_scale0="0.787278" vif_scale1="0.973289" vif_scale2="0.986163" vif_scale3="0.991446" vmaf="98.8478" />
	</frames>
</VMAF>

EDIT
changed out1="SSIM: to out1="vmaf: and it works now

[zorr]

Originally Posted by ChaosKing

VMAF just stops:

Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.

I will look at the GMSD later, but the VMAF problem can be solved by renaming one of the #output parameters "vmaf" (not case sensitive).

[zorr]

Originally Posted by ChaosKing

I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13

Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)

In GMSD smaller number means better quality, so make sure you have GMSD: MIN(float) as the output. Zopti returns the sum of per frame values so otherwise it looks good to me.

[ChaosKing]

Ahh now I see it too, thx.

[ChaosKing]

My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.
This barely changed since the start (it's still running) https://i.imgur.com/4JszzyZ.png
There are (too?) many lines with height=1080

My log so far https://pastebin.com/UrKnMwWz

script

from zoptilib import Zopti
#output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"
#output_file = r'results.txt' # output out1="ssim: MAX(float)" file="results.txt"
output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"

zopti = Zopti(output_file, metrics=['gmsd', 'time']) #gmsd vmaf ssim
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[19157:19157+1]

# https://github.com/Infiziert90/getnative/blob/master/getnative.py#L112
# change format to GrayS with bitdepth 32 for descale
matrix_s = '709' if orig.format.color_family == vapoursynth.RGB else None
src_luma32 = core.resize.Point(orig, format=vapoursynth.YUV444PS, matrix_s=matrix_s)
src_luma32 = core.std.ShufflePlanes(src_luma32, 0, vapoursynth.GRAY)
orig = core.std.Cache(src_luma32)

width = 1400		# optimize width = _n_ | 1400..1920 | width
height = 800		# optimize height = _n_ | 800..1080 | height
x = ds.Debilinear(orig, width=width, height=height)
alternate = x.resize.Bilinear(orig.width, orig.height)

zopti.run(orig, alternate)

The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?

Is it possible that Zopti use only even numbers for width/height?

[zorr]

Originally Posted by ChaosKing

My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.

What optimization arguments were you using?

Originally Posted by ChaosKing

There are (too?) many lines with height=1080

Seems like the results are really good when height=1080 and that's why it's so common. Also when you only have two parameters to optimize and assuming you're using the default mutationCount "60% 1" then about halfway though it will only mutate one parameter. If the best results have height=1080 then at least half of the mutations have that value when it's mutating the width only. If you want it to explore a bit more you can set the mutationCount to 2 (or 100% which means the same in this case). Then it will mutate both parametes all the time.

Originally Posted by ChaosKing

The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?

When running the evalution on your log the pareto front is

1.5509998E-7 1190 width=1919 height=1080
  3.7165887E-7 1120 width=1920 height=1079
  4.1259636E-6 1110 width=1920 height=1077
  8.270416E-6 1100 width=1920 height=1075
  4.067268E-4 1090 width=1920 height=1034
  0.0016862862 1080 width=1702 height=1080
  0.0028691771 1070 width=1610 height=1080
  0.0066714296 1060 width=1450 height=1080

so yes, it can understand the scientific notation. There's a problem though, the best results are really close to 1920x1080 and looking at the logs that got a perfect GMSD score 0.0. But the optimizer currently interprets 0.0 as invalid, that's why it's not in the pareto front. I will have to think about how to fix that.

It's a bit suspicious that you can get a perfect similarity score. Should it happen in your use case?

Originally Posted by ChaosKing

Is it possible that Zopti use only even numbers for width/height?

Yes, there are a couple of ways to do that. The simplest is probably:

width = 2*700	# optimize width = 2*_n_ | 700..960 | width
height = 2*400	# optimize height = 2*_n_ | 400..540 | height

Or you can specify a filter which only accepts numbers divisible by 2:

width = 1400	# optimize width = _n_ | 1400..1920 ; filter:x 2 % 0 == | width
height = 800	# optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height

[ChaosKing]

I don't know if a perfect score should be possible, but because the "resize error" is very small it is not easy to detect I guess.

The native res in my case should be 1600x900 (900p). Setting width & height to
width = 1400 # optimize width = _n_ | 1500..1680 | width
height = 800 # optimize height = _n_ | 880..910 | height
leads to this, which look almost to what I expected in the first place.
https://i.imgur.com/QNbTYMW.png
(And maybe I just need to test another frame)

There is already a native resolution "detector" here https://github.com/Infiziert90/getna...r/getnative.py
I just wanted to see how Zopti will perform. I think the differences need to be emphasized more, so the resizing error becomes more visible.

Zopti could come in handy if a bicubic resizer was used on the clip. It will be easy to find the b and c values like you have shown here https://forum.doom9.org/showthread.p...44#post1859544

[ChaosKing]

Breaking news: I'm an idot
I used my already encoded and "descaled" file instead of the source file. That also explains the perfect 1080p match -_-


height = 800 # optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height
I want to optimize the runs but I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3

[zorr]

Originally Posted by ChaosKing

Breaking news: I'm an idot

Welcome to the club.

Originally Posted by ChaosKing

I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3

I always visualize the rpn notation as a stack. Every operation can take values from the stack, do some computation with them and then put the result on the top of the stack. So for example this x 2 % 0 ==:

1) x - put x on the stack, stack is "x"
2) 2 - put 2 on the stack, stack is "x 2" (top of the stack on the right)
3) % - modulus operator, take two topmost values from the stack, calculate modulus and put the result back into stack - stack is (x % 2)
4) 0 - put 0 on the stack, stack is (x % 2) 0
5) == - equals operator, take two topmost values and put true on the top of the stack if they are equal, false otherwise - stack is ((x % 2) == 0)

But actually in this case the filter is not needed. You can optimize the height only and calculate width from it:

height = 800 # optimize height = _n_ | 800..1080 | height
width = height*(16/9)

or if you want to give it a small range around the calculated value you can do

height = 800 # optimize height = _n_ | 800..1080 | height
width_offset = 0 # optimize width_offset = _n_ | -3..3 | width_offset
width = height*(16/9) + width_offset

[Boulder]

I've been trying to run my resizing test with VMAF, the sample clip is 5 separate single frames from an episode of Black Sails. My first run ended up with the b value at -100, so I enlarged the area so that both b and c can go -300..300. However, VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?

e 68 / 1,00 : 100.0 200ms b=-33 c=236
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 (no samples) c 1,000
MUTATED GENERATION 69
Mutating 1 params by 1,0 % (phase 1,00)
e 69 / 1,00 : 100.0 200ms b=-52 c=236
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 (no samples) c 1,000
MUTATED GENERATION 70
Mutating 1 params by 1,0 % (phase 1,00)
  70 / 1,00 : 100.0 210ms b=-50 c=236
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 (no samples) c 1,000
MUTATED GENERATION 71
Mutating 1 params by 1,0 % (phase 1,00)
e 71 / 1,00 : 100.0 200ms b=-42 c=246
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 c 1,000
MUTATED GENERATION 72
Mutating 1 params by 1,0 % (phase 1,00)
e 72 / 1,00 : 100.0 200ms b=-42 c=225
0 iterations remaining is this generation
No improvement in 24 iterations - stopping
Parameter sensitivity estimation with 4 result combinations
  -> (no samples) b 1,000 c 1,000

This is the script:

orig = core.ffms2.Source(source=r'blacksails.avi')
   
output_file = r'results.txt' # output out1="vmaf: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time'])	# initialize output file and chosen metrics 

b = 0/100.0					# optimize b = _n_/100.0 | -300..300 | b
c = 0/100.0					# optimize c = _n_/100.0 | -300..300 | c

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)

zopti.run(orig, alternate)

I've used this command line for these quick tests:
zopti test.vpy -alg mutation -iters dyn -dyniters 24 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

With gmsd, the run resulted in b=-77 c=5 as the optimal pair. In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.

[zorr]

Originally Posted by Boulder

VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?

I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.

Originally Posted by Boulder

With gmsd, the run resulted in b=-77 c=5 as the optimal pair.

Can you also try what SSIM gives as the optimal pair?

Originally Posted by Boulder

In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.

You can set the VMAF model with zopti.setVMAFModel(). Changing the pool won't have any effect since Zopti calculates the result from the individual frames and always just sums them up. The ci shouldn't change the scores either, it's just extra information.

[Boulder]

Originally Posted by zorr

I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.

A very quick test shows this kind of behaviour, so it looks like VMAF cannot measure the difference between the clips accurately enough. That is quite odd because I would expect that downsizing to 720p before upsizing to 4K would have a big enough effect. Maybe I'll also have to try keeping the original at 1080p and use model 0 even though my TV is 4K.

e 178 / 20301: 100.0 200ms b=-123 c=0
e 179 / 20301: 100.0 200ms b=-122 c=0
e 180 / 20301: 100.0 200ms b=-121 c=0
e 181 / 20301: 100.0 200ms b=-120 c=0
e 182 / 20301: 100.0 200ms b=-119 c=0
e 183 / 20301: 100.0 200ms b=-118 c=0
e 184 / 20301: 100.0 200ms b=-117 c=0
e 185 / 20301: 100.0 200ms b=-116 c=0
  186 / 20301: 99.98204 200ms b=-115 c=0
  187 / 20301: 99.99383 200ms b=-114 c=0
  188 / 20301: 99.9717 200ms b=-113 c=0
  189 / 20301: 99.97675 200ms b=-112 c=0
  190 / 20301: 99.96704 200ms b=-111 c=0
  191 / 20301: 99.953285 200ms b=-110 c=0
  192 / 20301: 99.944626 200ms b=-109 c=0

SSIM gives b=-59, c=31 as the optimal pair. That difference with gmsd would be expected if SSIM favours blurring more. Without testing, I assume that gmsd's result would be slightly sharper looking.

EDIT: The clip is here if you'd like to test it: https://drive.google.com/open?id=1rS...VQSg40vwtUZFNN

[zorr]

Originally Posted by Boulder

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)

Where is the resamplehq function from? I found resamplehq script but it doesn't have the same method name or arguments.

[ChaosKing]

It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122...43fe/revisions

[WorBry]

Originally Posted by Boulder

.. so it looks like VMAF cannot measure the difference between the clips accurately enough.

There's also this issue where the VMAF result is skewed by a component 'motion2=0' score for the first frame in a clip:

https://forum.doom9.org/showthread.p...61#post1864561

The only information I can find about this parameter is:

"Motion. This is a simple measure of the temporal difference between adjacent frames. This is accomplished by calculating the average absolute pixel difference for the luminance component."

https://medium.com/netflix-techblog/...c-653f208b9652

and:

"motion2 score typically ranges from 0 (static) to 20 (high-motion)"

https://github.com/Netflix/vmaf/blob...hon_library.md

Testing your Blacksails clip against itself, with VapourSynth VMAF v3 (Model=0) gives:

<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="8" aggregateVMAF="99.6711" aggregatePSNR="60" aggregateSSIM="0.99993" aggregateMS_SSIM="0.999936" execFps="2.27555" timeTaken="3.51563" />
<frames>
<frame frameNum="0" adm2="1" motion2="0" ms_ssim="0.999889" psnr="60" ssim="0.9999" vif_scale0="0.999999" vif_scale1="0.999997" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="97.4274" />
<frame frameNum="1" adm2="1" motion2="51.9751" ms_ssim="0.999997" psnr="60" ssim="0.999997" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999997" vif_scale3="0.999997" vmaf="100" />
<frame frameNum="2" adm2="1" motion2="45.4778" ms_ssim="0.999998" psnr="60" ssim="0.999999" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999996" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="3" adm2="1" motion2="45.4778" ms_ssim="0.999994" psnr="60" ssim="0.999995" vif_scale0="0.999997" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="4" adm2="1" motion2="48.0867" ms_ssim="0.99986" psnr="60" ssim="0.999837" vif_scale0="0.999999" vif_scale1="0.999996" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="100" />
<frame frameNum="5" adm2="1" motion2="71.4518" ms_ssim="0.999999" psnr="60" ssim="0.999999" vif_scale0="1" vif_scale1="0.999999" vif_scale2="0.999998" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="6" adm2="1" motion2="74.8847" ms_ssim="0.999751" psnr="60" ssim="0.999716" vif_scale0="0.999994" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="7" adm2="1" motion2="74.8847" ms_ssim="1" psnr="60" ssim="1" vif_scale0="0.999998" vif_scale1="0.999995" vif_scale2="0.999993" vif_scale3="0.999994" vmaf="100" />

FFMPEG SSIM reports lossless, 1.00000 (Inf)

[Boulder]

Originally Posted by ChaosKing

It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122...43fe/revisions

I should have the latest version, but looking at the code, I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.