Zopti

[zorr]

Zopti is what was called AvisynthOptimizer. The name was changed because it now supports VapourSynth as well. This first post is an introduction similar to this one (https://forum.doom9.org/showpost.php?p=1850994&postcount=1) on the Avisynth Development forum, just changed a bit to reflect the latest features and what is possible with VapourSynth. I also cut out some unnecessary blabbering.

Let's say you want to motion compensation to double frame rate or replace a few corrupted frames. The right tool for that job is obviously MVTools. The only problem now is finding good parameters to get the best quality (because every video needs a little different settings). Ok no problem, I will just quickly adjust these... wait... there are about 60 parameters to adjust! Finding good settings would involve *a lot* of manual testing. This is where Zopti jumps in and says "Don't worry pal, I will find the best settings for you!".

But how? A VapourSynth script is first "augmented" by adding instructions on which parameters should be optimized. The instructions are written inside comments so the script will continue to work normally. The script also needs to measure the quality of the processed video using current parameter values or measure the running time (or preferably both the quality and runtime). The script needs to write these results to a specific file. There is a helper Python library "zoptilib" you can use to do the quality/runtime measurements and write the file.

Now you might be wondering how on earth can the script measure the quality. I can only think of one way: to compare the frames to reference frames and measure the similarity. The closer the similary value, the better the quality. VapourSynth has (at least) these similarity metrics available: SSIM (https://en.wikipedia.org/wiki/Structural_similarity) (the classic, most well known), GMSD (http://www4.comp.polyu.edu.hk/~cslzhang/IQA/GMSD/GMSD.pdf) (seems better than SSIM) and VMAF (https://en.wikipedia.org/wiki/Video_Multimethod_Assessment_Fusion) (by Netflix). It's also possible to get MS-SSIM (https://www.researchgate.net/publication/4071876_Multiscale_structural_similarity_for_image_quality_assessment) (multi-scale SSIM) from the VMAF plugin and I also implemented B-SSIM (https://www.researchgate.net/publication/265164680_B-SSIM_Structural_Similarity_Index_for_Blurred_Videos) for AviSynth (SSIM with blurring adjustment).

Ok, but where do we get the reference frames? In case of MVTools we can use the original frames as reference frames and the script will try to reconstruct them using motion compensation (but it's not allowed to use the reference frame in the reconstruction). We can do something similar if we want to use MVTools to double the framerate: we first create the double-rate video, then remove the original frames from it, then double the framerate again and finally compare these to the original frames. Or you could reconstruct every odd frame using the even frames. This idea is not limited to MVTools, you could for example do color grading using some other software and then try to recreate the same result using VapourSynth. I'm sure the smart people here will find use cases I couldn't even dream about. :D

MVTools is a complex plugin and the settings can change the runtime of the script dramatically. In addition to quality we can also measure the runtime to find settings that are fast enough (for example if you want to use the script in real-time).

So if the purpose is to find the best settings, what do we consider the "best" when both quality and time are involved? For example we can have settings A with quality 99 (larger is better), time 2300ms (larger is worse) and settings B with quality 95 and time 200ms. Which one of these is better? We can use the concept of pareto domination to answer this question. When one solution is at least as good as the other solution in every objective (in this case the objectives are quality and speed) and better in at least one objective, it dominates the other solution. In this example, neither dominates the other. But if we have settings C with quality 99 and time 200ms it dominates both A and B. In the end we want to know all the nondominated solutions, which is called the pareto front. So there's going to be a list of parameters with increasing quality and runtime. You can have more than two objectives if you want, the same pareto concept works. There's a good and free ebook called "Essentials of Metaheuristics (https://cs.gmu.edu/~sean/book/metaheuristics/)" which describes the pareto concept and much more.

Zopti reads the augmented script, verifies it and starts running a metaheuristic optimization algorithm. There are currently four algorithm choices: NSGA-II, SPEA2, mutation and exhaustive. The first two are some of the best metaheuristic algorithms available and also described in the ebook mentioned above. The mutation is my own simplistic algorithm (but it can be useful if you're in a hurry since it's the most efficient). Exhaustive will try all possible combinations, it's useful if you only have a small number of them (you could for example do an exhaustive search on one parameter value only). All the metaheuristic algorithms are working in a similar manner generating different solutions and testing them. The optimizer does these tests by creating scripts with certain parameter values and running them. The script then writes the measurements (quality/time) into a file which the optimizer reads. The metaheuristic then decides which parameters to try next based on the results. This continues until some end criteria is met.

There are three different ending criterias: number of iterations, time limit and "dynamic". Number of iterations is just that, the algorithm runs the script specific number of times. Setting a time limit can be pretty useful if you know how much time you can use for the optimization. You could for example let it run overnight for 8 hours and see the results in the morning. Dynamic variation is stopping only when it doesn't make any progress anymore. Making progress is defined by "no more pareto front members in last x iterations". This can be useful if you want to find the best possible results regardless of how long it takes.

During the optimization all tested parameters and their results are written to a log file. This log file can be "evaluated" during and after the optimization process. The evaluation basically means finding the pareto front and showing it. You can also create scripts from the pareto front in order to test them yourself. It's also possible to visualize the results of the log file in a two dimensional scatter chart. This chart highlights the pareto front and shows all the other results too. The chart can also be "autorefreshing": it loads the log file every few seconds and updates the visuals, which is a fun way to track how the optimization is progressing. Here's a gif what it looks like (obviously sped up):

https://s22.postimg.cc/lgnp0apsh/animation.gif (https://postimg.cc/image/q2jt8nbbh/)

The visualization has a few other bells and whistles but one I'd like to highlight here is the group by functionality: you can group the results by certain parameter's values and show a separate pareto front for each value. For example this what grouping by MVTools' blocksize looks like:

https://s22.postimg.cc/x5rooblwx/groupby-blocksize.png (https://postimg.cc/image/hkad4d9yl/)

Another visualization mode draws a heat map where two parameters are chosen for the x and y axis, the color of the cell at x,y is the best result found with that parameter combination, brighter color meaning better result.

https://i.postimg.cc/XJXVRkjQ/heatmap-example.png

Measuring the script's runtime is not very accurate, ie it has some variation. All the other processes running at the same time are using CPU cycles and messing with the cache so you should try to minimize other activity on the computer. In order to get more accurate results you can run a validation on the finished results log file. In validation the idea is to run the pareto front results multiple times and calculate the average, median, minimum or maximum of these multiple measurements (you can decide which one(s)).

So how good is the optimizer? Let's take a look at one example. A while back there was a thread (https://forum.doom9.org/showthread.php?t=175508) about best motion interpolation filters. There's a test video with a girl waving her hand. The best options that I know of are John Meyer's jm_fps script and FrameRateConverter. Here's a comparison gif with those two and Zopti. FramerateConverter was run with preset="slowest".

https://i.postimg.cc/Qd1wM4xH/ezgif-2-c921f9e1c33e-zopti.gif

Now obviously I'm showing a bit of a cherry-picked example here. The optimizer was instructed to search the best parameters for this short 10 frame sequence. I have run most of my optimization runs using only 10 frames because otherwise the optimization takes too long. Ideally the optimizer would automatically select the frames from a longer video, I have started working on such a feature but it's not finished yet (got sidetracked to implement a scene change detector...) so for now user has to make the selection.

At this point I envision that Zopti can be an useful tool for plugin authors so they can test plugin parameters and try to search for the optimal ones. Zopti can also be a bug hunter, it has found several bugs with the Avisynth MVTools by Pinterf which he has since fixed. The VapourSynth MVTools is currently not yet as robust, but jackoneill has already fixed one bug (thanks!). At some later point Zopti could be useful for normal users, when combined with a script with limited search space so that the search will not take excessively long time. Zopti (or rather the previous version AvisynthOptimizer) has already been used by some courageous people from these forums for scaling, motion compensation, denoising and HDR to SRD tone-mapping. I'd like to thank them all for providing important feedback, feature requests and bug reports.

You can download the latest Zopti version here (https://drive.google.com/file/d/1Ox_HfyvpV1bbL2_j-jx9hGOOYQAunXQp/view?usp=sharing). I will keep this link updated.

Version history:

1.2.3 (full details here (https://forum.doom9.org/showthread.php?p=1965241#post1965241))

new mode: convert
-converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
-supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
-output formats: CSV and zopti
-example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
-example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
improvements to visualization mode: line
-now supports option -types which controls which aggregate lines are displayed in the chart
-valid values are "best", "worst", "average", "median", "count" and "samples"
display of option -groupby improved, pareto fronts can no longer be cut off the screen
option -continue also supported with mutation algorithm
validation mode behavior change: it now validates all the results in the log file and not just the pareto front


1.2.2

supports a new way for an Avisynth script to write the result file
-first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
-then come the per frame result lines in any order, using the same format as before
-do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
-this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
-old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
-new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value


1.2.1

new option: retry
-tries to run the script again this many times if execution fails
-can be useful if the script uses plugins which are not 100% reliable and can crash
-default value is zero (do not retry)
-example: zopti script.avs -pop 24 -iters 10000 -retry 4
new visualization mode: line
-draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
-accepts parameter -range to limit the displayed values to certain range
-black line shows the best values, blue line indicates the number of results per value
-a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
-example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
support for custom output properties written by the script (only in VapourSynth scripts for now)
better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
option -continue now also supported by exhaustive algorithm (when using more than one thread)
order of runs in a visualization is now based on file name, not by modified date
all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
visualization mode seriespareto no longer contains the global pareto line
VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
bugfix: fixed a memory leak in XChart (at least partially)


Pre-1.2.1 history removed due to character count limitation.

The next post will be a hands-on denoising tutorial. Stay tuned.

[ChaosKing]

Nice work!

But I get a file not found error :-)
java.io.FileNotFoundException: D:\Download\Zopti-1.0-beta\work\result_1549155407307_1549155407426.txt
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileReader.<init>(Unknown Source)
at avisynthoptimizer.file.FileUtil.readLinesFromFile(FileUtil.java:58)
at avisynthoptimizer.AviSynthOptimizer.readVapoursynthResults(AviSynthOptimizer.java:6566)
at avisynthoptimizer.AviSynthOptimizer.waitAndReadResults(AviSynthOptimizer.java:6587)
at avisynthoptimizer.AviSynthOptimizer.runAVS(AviSynthOptimizer.java:6308)
at avisynthoptimizer.AviSynthOptimizer.evaluateFitness(AviSynthOptimizer.java:5588)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:190)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:35)
at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.lambda$evaluate$1(SequentialSolutionListEvaluator.java:24)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.evaluate(SequentialSolutionListEvaluator.java:24)
at avisynthoptimizer.spea2.DynamicSPEA2.evaluatePopulation(DynamicSPEA2.java:76)
at org.uma.jmetal.algorithm.impl.AbstractEvolutionaryAlgorithm.run(AbstractEvolutionaryAlgorithm.java:56)
at java.lang.Thread.run(Unknown Source)
java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(Unknown Source)
....
My script:
from zoptilib import Zopti
output_file = r'D:\results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['ssim', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 4 # optimize sigma = _n_ | 1,10 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)

alternate.set_output()

java -jar .\Zopti.jar D:\z.vpy
vspipe path is set in the ini file.

[zorr]

Nice work!

But I get a file not found error :-)


The # output line should probably look like this:

output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"


The file name must match.

Also this line
alternate.set_output()
is not necessary, Zopti sets the output. In some cases it is critical that the correct file is set as the output. The alternate file is the correct one in this case but for some reason it doesn't work if it's set first by Zopti and then again in the script.

[ChaosKing]

Thx, works now without alternate.set_output() :D

[ChaosKing]

I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13

Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)

VMAF just stops:
Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.

Maybe I do something wrong. Here is my script again

from zoptilib import Zopti
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 35 # optimize sigma = _n_ | 9..40 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)


result_*.txt:
<?xml version="1.0"?>
<VMAF version="1.3.11">
<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="100" aggregateVMAF="96.037" execFps="1.36185" timeTaken="73.4294" />
<frames>
<frame frameNum="0" adm2="0.983001" motion2="0" vif_scale0="0.709537" vif_scale1="0.953811" vif_scale2="0.974778" vif_scale3="0.983557" vmaf="91.1718" />
<frame frameNum="1" adm2="0.98031" motion2="14.7504" vif_scale0="0.760672" vif_scale1="0.952192" vif_scale2="0.97319" vif_scale3="0.982214" vmaf="100" />

<frame frameNum="98" adm2="0.992219" motion2="1.11641" vif_scale0="0.824504" vif_scale1="0.974561" vif_scale2="0.986421" vif_scale3="0.991392" vmaf="95.7912" />
<frame frameNum="99" adm2="0.990791" motion2="3.7195" vif_scale0="0.787278" vif_scale1="0.973289" vif_scale2="0.986163" vif_scale3="0.991446" vmaf="98.8478" />
</frames>
</VMAF>

EDIT
changed out1="SSIM: to out1="vmaf: and it works now

[zorr]

VMAF just stops:
Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.


I will look at the GMSD later, but the VMAF problem can be solved by renaming one of the #output parameters "vmaf" (not case sensitive).

[zorr]

I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13

Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)


In GMSD smaller number means better quality, so make sure you have GMSD: MIN(float) as the output. Zopti returns the sum of per frame values so otherwise it looks good to me.

[ChaosKing]

Ahh now I see it too, thx.

[ChaosKing]

My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.
This barely changed since the start (it's still running) https://i.imgur.com/4JszzyZ.png
There are (too?) many lines with height=1080

My log so far https://pastebin.com/UrKnMwWz

script
from zoptilib import Zopti
#output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"
#output_file = r'results.txt' # output out1="ssim: MAX(float)" file="results.txt"
output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"

zopti = Zopti(output_file, metrics=['gmsd', 'time']) #gmsd vmaf ssim
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[19157:19157+1]

# https://github.com/Infiziert90/getnative/blob/master/getnative.py#L112
# change format to GrayS with bitdepth 32 for descale
matrix_s = '709' if orig.format.color_family == vapoursynth.RGB else None
src_luma32 = core.resize.Point(orig, format=vapoursynth.YUV444PS, matrix_s=matrix_s)
src_luma32 = core.std.ShufflePlanes(src_luma32, 0, vapoursynth.GRAY)
orig = core.std.Cache(src_luma32)

width = 1400 # optimize width = _n_ | 1400..1920 | width
height = 800 # optimize height = _n_ | 800..1080 | height
x = ds.Debilinear(orig, width=width, height=height)
alternate = x.resize.Bilinear(orig.width, orig.height)

zopti.run(orig, alternate)

The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?

Is it possible that Zopti use only even numbers for width/height?

[zorr]

My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.


What optimization arguments were you using?


There are (too?) many lines with height=1080


Seems like the results are really good when height=1080 and that's why it's so common. Also when you only have two parameters to optimize and assuming you're using the default mutationCount "60% 1" then about halfway though it will only mutate one parameter. If the best results have height=1080 then at least half of the mutations have that value when it's mutating the width only. If you want it to explore a bit more you can set the mutationCount to 2 (or 100% which means the same in this case). Then it will mutate both parametes all the time.

The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?


When running the evalution on your log the pareto front is

1.5509998E-7 1190 width=1919 height=1080
3.7165887E-7 1120 width=1920 height=1079
4.1259636E-6 1110 width=1920 height=1077
8.270416E-6 1100 width=1920 height=1075
4.067268E-4 1090 width=1920 height=1034
0.0016862862 1080 width=1702 height=1080
0.0028691771 1070 width=1610 height=1080
0.0066714296 1060 width=1450 height=1080

so yes, it can understand the scientific notation. There's a problem though, the best results are really close to 1920x1080 and looking at the logs that got a perfect GMSD score 0.0. But the optimizer currently interprets 0.0 as invalid, that's why it's not in the pareto front. I will have to think about how to fix that.

It's a bit suspicious that you can get a perfect similarity score. Should it happen in your use case?

Is it possible that Zopti use only even numbers for width/height?

Yes, there are a couple of ways to do that. The simplest is probably:

width = 2*700 # optimize width = 2*_n_ | 700..960 | width
height = 2*400 # optimize height = 2*_n_ | 400..540 | height


Or you can specify a filter which only accepts numbers divisible by 2:

width = 1400 # optimize width = _n_ | 1400..1920 ; filter:x 2 % 0 == | width
height = 800 # optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height

[ChaosKing]

I don't know if a perfect score should be possible, but because the "resize error" is very small it is not easy to detect I guess.

The native res in my case should be 1600x900 (900p). Setting width & height to
width = 1400 # optimize width = _n_ | 1500..1680 | width
height = 800 # optimize height = _n_ | 880..910 | height
leads to this, which look almost to what I expected in the first place.
https://i.imgur.com/QNbTYMW.png
(And maybe I just need to test another frame)

There is already a native resolution "detector" here https://github.com/Infiziert90/getnative/blob/master/getnative.py
I just wanted to see how Zopti will perform. I think the differences need to be emphasized more, so the resizing error becomes more visible.

Zopti could come in handy if a bicubic resizer was used on the clip. It will be easy to find the b and c values like you have shown here https://forum.doom9.org/showthread.php?p=1859544#post1859544

[ChaosKing]

Breaking news: I'm an idot :D
I used my already encoded and "descaled" file instead of the source file. That also explains the perfect 1080p match -_-


height = 800 # optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height
I want to optimize the runs but I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3

[zorr]

Breaking news: I'm an idot :D
Welcome to the club. :)

I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3

I always visualize the rpn notation as a stack. Every operation can take values from the stack, do some computation with them and then put the result on the top of the stack. So for example this x 2 % 0 ==:

1) x - put x on the stack, stack is "x"
2) 2 - put 2 on the stack, stack is "x 2" (top of the stack on the right)
3) % - modulus operator, take two topmost values from the stack, calculate modulus and put the result back into stack - stack is (x % 2)
4) 0 - put 0 on the stack, stack is (x % 2) 0
5) == - equals operator, take two topmost values and put true on the top of the stack if they are equal, false otherwise - stack is ((x % 2) == 0)

But actually in this case the filter is not needed. You can optimize the height only and calculate width from it:

height = 800 # optimize height = _n_ | 800..1080 | height
width = height*(16/9)


or if you want to give it a small range around the calculated value you can do

height = 800 # optimize height = _n_ | 800..1080 | height
width_offset = 0 # optimize width_offset = _n_ | -3..3 | width_offset
width = height*(16/9) + width_offset

[Boulder]

I've been trying to run my resizing test with VMAF, the sample clip is 5 separate single frames from an episode of Black Sails. My first run ended up with the b value at -100, so I enlarged the area so that both b and c can go -300..300. However, VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?

e 68 / 1,00 : 100.0 200ms b=-33 c=236
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 (no samples) c 1,000
MUTATED GENERATION 69
Mutating 1 params by 1,0 % (phase 1,00)
e 69 / 1,00 : 100.0 200ms b=-52 c=236
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 (no samples) c 1,000
MUTATED GENERATION 70
Mutating 1 params by 1,0 % (phase 1,00)
70 / 1,00 : 100.0 210ms b=-50 c=236
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 (no samples) c 1,000
MUTATED GENERATION 71
Mutating 1 params by 1,0 % (phase 1,00)
e 71 / 1,00 : 100.0 200ms b=-42 c=246
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 c 1,000
MUTATED GENERATION 72
Mutating 1 params by 1,0 % (phase 1,00)
e 72 / 1,00 : 100.0 200ms b=-42 c=225
0 iterations remaining is this generation
No improvement in 24 iterations - stopping
Parameter sensitivity estimation with 4 result combinations
-> (no samples) b 1,000 c 1,000

This is the script:
orig = core.ffms2.Source(source=r'blacksails.avi')

output_file = r'results.txt' # output out1="vmaf: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time']) # initialize output file and chosen metrics

b = 0/100.0 # optimize b = _n_/100.0 | -300..300 | b
c = 0/100.0 # optimize c = _n_/100.0 | -300..300 | c

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)

zopti.run(orig, alternate)

I've used this command line for these quick tests:
zopti test.vpy -alg mutation -iters dyn -dyniters 24 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

With gmsd, the run resulted in b=-77 c=5 as the optimal pair. In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.

[zorr]

VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?


I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.

With gmsd, the run resulted in b=-77 c=5 as the optimal pair.

Can you also try what SSIM gives as the optimal pair?

In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.

You can set the VMAF model with zopti.setVMAFModel(). Changing the pool won't have any effect since Zopti calculates the result from the individual frames and always just sums them up. The ci shouldn't change the scores either, it's just extra information.

[Boulder]

I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.
A very quick test shows this kind of behaviour, so it looks like VMAF cannot measure the difference between the clips accurately enough. That is quite odd because I would expect that downsizing to 720p before upsizing to 4K would have a big enough effect. Maybe I'll also have to try keeping the original at 1080p and use model 0 even though my TV is 4K.

e 178 / 20301: 100.0 200ms b=-123 c=0
e 179 / 20301: 100.0 200ms b=-122 c=0
e 180 / 20301: 100.0 200ms b=-121 c=0
e 181 / 20301: 100.0 200ms b=-120 c=0
e 182 / 20301: 100.0 200ms b=-119 c=0
e 183 / 20301: 100.0 200ms b=-118 c=0
e 184 / 20301: 100.0 200ms b=-117 c=0
e 185 / 20301: 100.0 200ms b=-116 c=0
186 / 20301: 99.98204 200ms b=-115 c=0
187 / 20301: 99.99383 200ms b=-114 c=0
188 / 20301: 99.9717 200ms b=-113 c=0
189 / 20301: 99.97675 200ms b=-112 c=0
190 / 20301: 99.96704 200ms b=-111 c=0
191 / 20301: 99.953285 200ms b=-110 c=0
192 / 20301: 99.944626 200ms b=-109 c=0

SSIM gives b=-59, c=31 as the optimal pair. That difference with gmsd would be expected if SSIM favours blurring more. Without testing, I assume that gmsd's result would be slightly sharper looking.

EDIT: The clip is here if you'd like to test it: https://drive.google.com/open?id=1rSc-XBrVSGB5VupxyLVQSg40vwtUZFNN

[zorr]

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)


Where is the resamplehq function from? I found resamplehq (http://vsdb.top/plugins/resamplehq) script but it doesn't have the same method name or arguments.

[ChaosKing]

It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122e359c37543fe/revisions

[WorBry]

.. so it looks like VMAF cannot measure the difference between the clips accurately enough.

There's also this issue where the VMAF result is skewed by a component 'motion2=0' score for the first frame in a clip:

https://forum.doom9.org/showthread.php?p=1864561#post1864561

The only information I can find about this parameter is:

"Motion. This is a simple measure of the temporal difference between adjacent frames. This is accomplished by calculating the average absolute pixel difference for the luminance component."

https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

and:

"motion2 score typically ranges from 0 (static) to 20 (high-motion)"

https://github.com/Netflix/vmaf/blob/master/resource/doc/VMAF_Python_library.md

Testing your Blacksails clip against itself, with VapourSynth VMAF v3 (Model=0) gives:

<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="8" aggregateVMAF="99.6711" aggregatePSNR="60" aggregateSSIM="0.99993" aggregateMS_SSIM="0.999936" execFps="2.27555" timeTaken="3.51563" />
<frames>
<frame frameNum="0" adm2="1" motion2="0" ms_ssim="0.999889" psnr="60" ssim="0.9999" vif_scale0="0.999999" vif_scale1="0.999997" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="97.4274" />
<frame frameNum="1" adm2="1" motion2="51.9751" ms_ssim="0.999997" psnr="60" ssim="0.999997" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999997" vif_scale3="0.999997" vmaf="100" />
<frame frameNum="2" adm2="1" motion2="45.4778" ms_ssim="0.999998" psnr="60" ssim="0.999999" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999996" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="3" adm2="1" motion2="45.4778" ms_ssim="0.999994" psnr="60" ssim="0.999995" vif_scale0="0.999997" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="4" adm2="1" motion2="48.0867" ms_ssim="0.99986" psnr="60" ssim="0.999837" vif_scale0="0.999999" vif_scale1="0.999996" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="100" />
<frame frameNum="5" adm2="1" motion2="71.4518" ms_ssim="0.999999" psnr="60" ssim="0.999999" vif_scale0="1" vif_scale1="0.999999" vif_scale2="0.999998" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="6" adm2="1" motion2="74.8847" ms_ssim="0.999751" psnr="60" ssim="0.999716" vif_scale0="0.999994" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="7" adm2="1" motion2="74.8847" ms_ssim="1" psnr="60" ssim="1" vif_scale0="0.999998" vif_scale1="0.999995" vif_scale2="0.999993" vif_scale3="0.999994" vmaf="100" />
FFMPEG SSIM reports lossless, 1.00000 (Inf)

[Boulder]

It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122e359c37543fe/revisions

I should have the latest version, but looking at the code, I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.

[zorr]

I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.

Thanks, got it working. Also had to set kernel='bicubic' since the default is spline36 in the latest version.

I ran some exhaustive tests with your clip, I set the divider to 10.0 instead of 100.0 to make it faster. The results were interesting:

https://i.postimg.cc/JzBccFr0/ssim-gmsd-vmaf2.png

SSIM and GMSD are roughly in agreement where the best settings are. SSIM had best at b=-6 c=3. Note that this is equivalent to b=-60 c=30 with your original script which is very close to your best result b=-59, c=31. GMSD's best is at b=-7 c=1, a bit further away from your result b=-77 c=5. GMSD is a bit more "focused", it seems to see the differences better than SSIM.

VMAF on the other hand gives many "perfect" results and when we look at only those the center is very close to the top left coordinates b=-30 c=30. I think VMAF is trying to say that the differences are so small that they're not perceptible to average human. Have you tried eyeballing the different settings? It is however a bit of a mystery why VMAF has the center of best results in a different place than SSIM and GMSD.

[Boulder]

To be honest, without zooming in, it's not easy to see the differences of the optimal parameters provided by the various metrics. My guess is that VMAF is not the best method for these single frame comparisons but has much better value with moving video. GMSD looks quite promising with these scaling comparisons.

[ChaosKing]

muvsfunc added a new similarity metric: MDSI(Mean Deviation Similarity Index) https://github.com/WolframRhodium/muvsfunc/commit/0e17b8ca44bde1844ce68749c70cf0cb6b1db304
MDSI is a full reference IQA model that utilize gradient similarity (GS), chromaticity similarity (CS), and deviation pooling (DP).
The lowerer the MDSI score, the higher the image perceptual quality.
Larger MDSI values indicate to the more severe distorted images, while an image with perfect quality is assessed by a quality score of zero.

@Boulder you shoud also try https://github.com/fdar0536/VapourSynth-butteraugli
It detects even the smallest change very reliably.

[ChaosKing]

I've added butteraugli and mdsi to zoptilib https://pastebin.com/511BcmNp

[Boulder]

Thanks, nice to test various options :)

[zorr]

I've added butteraugli and mdsi to zoptilib https://pastebin.com/511BcmNp

Thanks! I'm running some tests on those two. Some initial observations:

MDSI seems promising based on the authors' paper. It requires clips to be RGB. Boulder, is your source material in BT 709 colorspace? I'm not sure if Zopti should automatically do the conversation or just require RGB. MDSI by default downscales the clips by 2 which is kinda counterproductive when trying to measure the effects of scaling. I will add a method to set the downscaling value. The current MDSI implementation in muvsfunc returns "inf" for one of the frames and the script doesn't finish. Perhaps it's not ready for prime time (since it's not even released yet).

Butteraugli can indeed measure small differences, that's its main purpose. So it may not be accurate with large differences but should work well for Boulder's use case. It's also slow. I mean slooooow... one iteration (8 frames) takes 22 seconds on my machine. :scared: Butteraugli also needs RGB and it needs to be gamma corrected... if I'm interpreting it correctly clips should be in linear RGB space. How does one do that conversion with Vapoursynth?

// Value of pixels of images rgb0 and rgb1 need to be represented as raw
// intensity. Most image formats store gamma corrected intensity in pixel
// values. This gamma correction has to be removed, by applying the following
// function:
// butteraugli_val = 255.0 * pow(png_val / 255.0, gamma);
// A typical value of gamma is 2.2. It is usually stored in the image header.
// Take care not to confuse that value with its inverse. The gamma value should
// be always greater than one.
// Butteraugli does not work as intended if the caller does not perform
// gamma correction.

[WolframRhodium]

The current MDSI implementation in muvsfunc returns "inf" for one of the frames and the script doesn't finish.

Could you provide the material related to such error to me, so that I could fix it asap?

[zorr]

Could you provide the material related to such error to me, so that I could fix it asap?

Here's the script:

import vapoursynth as vs
import resamplehq as rhq
from zoptilib import Zopti

core = vs.core

orig = core.ffms2.Source(source=r'blacksails.avi')

# convert to RGB
orig = core.fmtc.resample(clip=orig, css="444")
orig = core.fmtc.matrix(clip=orig, mat="709", col_fam=vs.RGB)
orig = core.fmtc.bitdepth(clip=orig, bits=8)

zopti = Zopti(r'result.txt', metrics=['mdsi', 'time']) # initialize output file and chosen metrics

b = -30/10.0 # optimize b = _n_/10.0 | -30..30 | b
c = -30/10.0 # optimize c = _n_/10.0 | -30..30 | c

alternate = rhq.resample_hq(orig, width=1280, height=720, kernel='bicubic', filter_param_a=b, filter_param_b=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
zopti.run(orig, alternate)


You can download the source clip from here (https://drive.google.com/file/d/1rSc-XBrVSGB5VupxyLVQSg40vwtUZFNN/view).

You need the zoptilib version ChaosKing modded, here (https://pastebin.com/511BcmNp).
Oh and thanks for the muvsfunc library, the similarity metrics are perhaps the most important ingredient on making an optimizer like zopti work! :thanks:

[WolframRhodium]

You can download the source clip from here (https://drive.google.com/file/d/1rSc-XBrVSGB5VupxyLVQSg40vwtUZFNN/view).

You need the zoptilib version ChaosKing modded, here (https://pastebin.com/511BcmNp).


It may have been fixed after this commit (https://github.com/WolframRhodium/muvsfunc/commit/bbdac4695c99e686222f537ab201290e36f3788f). I also disable downsampling by default now.

Thank you for your support and recognition. Your work on Zopti is impressive.

[zorr]

It may have been fixed after this commit (https://github.com/WolframRhodium/muvsfunc/commit/bbdac4695c99e686222f537ab201290e36f3788f). I also disable downsampling by default now.

Thanks, it is indeed fixed. And the results are in:

https://i.postimg.cc/QXQvGWK0/ssim-gmsd-vmaf-butteraugli-mdsi.png

Butteraugli is not as smooth function as the others especially near the best value. It's more focused than SSIM and slightly more focused than GMSD near the best value. Best value at b=-8, c=2 which again quite nicely agrees with SSIM and GMSD. It took almost 24 hours to create that picture, hopefully I don't have to do that again... but of course I have to because this was calculated without gamma correction. But I think I now know how to do it:

# convert to linear RGB
orig = core.fmtc.resample(clip=orig, css="444")
orig = core.fmtc.matrix(clip=orig, mat="709", col_fam=vs.RGB)
orig = core.fmtc.transfer(orig, transs="709", transd='linear') # to linear RGB
orig = core.fmtc.bitdepth(clip=orig, bits=8)


MDSI is even more focused than GMSD. Best value is at b=-6, c=4, still very close to the others.

All except VMAF are giving best b within [-6 .. -8] and best c within [1 .. 4]. So for this use case it probably doesn't matter which one of those is chosen. More tests are needed to determine which one works best for other use cases.


Thank you for your support and recognition. Your work on Zopti is impressive.

Thanks. Are you planning on implementing more similarity metrics? Of course quality matters more than quantity but it's always possible to optimize for more than one similarity metric and there might be some useful combinations.

[WolframRhodium]

Are you planning on implementing more similarity metrics? Of course quality matters more than quantity but it's always possible to optimize for more than one similarity metric and there might be some useful combinations.

I will implement more if I find them practical. It's also interesting to see if those no-reference metrics work well, but I have not found good candidates.

Anyway, you can set 'downsample=False' in SSIM to skip downsampling. I think those full-reference metrics are highly correlated.

[zorr]

I will implement more if I find them practical. It's also interesting to see if those no-reference metrics work well, but I have not found good candidates.

A no-reference metric would be awesome because it would enable a whole new class of things you could do with the optimizer. Of course it would depend on how good the metric actually is...

What's your opinion on WaDIQaM / DIQaM? The paper is here (https://arxiv.org/abs/1612.01697) and someone did a PyTorch implementation of WaDIQaM (https://github.com/lidq92/WaDIQaM). It can also work as full reference metric.

Anyway, you can set 'downsample=False' in SSIM to skip downsampling.
Ah yes, I noticed the downscaling when I tried to compare Avisynth's SSIM and your implementation. I couldn't make them give identical results even though I set the k1, k2 and downsample -parameters to match the Avisynth version. Perhaps the Avisynth version is non-standard.

I think those full-reference metrics are highly correlated.
Agreed, at least in this particular case. I compared SSIM and GMSD with denoising and there SSIM had more ringing artifacts.

[WolframRhodium]

What's your opinion on WaDIQaM / DIQaM? The paper is here (https://arxiv.org/abs/1612.01697) and someone did a PyTorch implementation of WaDIQaM (https://github.com/lidq92/WaDIQaM). It can also work as full reference metric.

It looks good, but a network with 10 conv layers and 2 fc layers seems to be too heavy for parameter optimization (upconv7 model from waifu2x consists of only 7 conv layers though it doesn't have any pooling layer). Based on my experience in image restoration, I'm also worry that those DL-based methods can't generalize well in the real-world settings.

Porting it to VapourSynth is not hard. I will check it later.

[zorr]

Based on my experience in image restoration, I'm also worry that those DL-based methods can't generalize well in the real-world settings.

I suspect that too, there was a cross-database evaluation and the no-reference models only do reasonably well when the distortions are of the same type as in the training set. SOM (Semantic obviousness metric for image quality assessment) looks pretty good at generalizing but I couldn't find any implementations of it.

Porting it to VapourSynth is not hard. I will check it later.

Thanks, it will be interesting even if it doesn't perform well in real-world tests.

[zorr]

It took almost 24 hours to create that picture, hopefully I don't have to do that again... but of course I have to because this was calculated without gamma correction.

Here's a comparison of Butteraugli without gamma correction and with it (in linear RGB):

https://i.postimg.cc/65ygH0R1/butteraugli-gamma-comparison.png

There are some changes, nothing radical but the gamma corrected version is slightly smoother. Best value has moved from c=2 to c=1.

[kriNon]

From what I can tell this plugin is still being actively developed. Is it in a finished enough state that it is usable? and if so, is there any documentation on how to use it?

My goal would be to use it to find settings for MVtools to interpolate frames for anime at the highest possible quality. Motion interpolation normally fails with animated content, and so I would like to see how well it works in a best case scenario with optimized settings. I'm working on trying to restore a field blended show, and if this works well enough, it would be very helpful for me.

[zorr]

Zopti version 1.0.1-beta released, see the first post.

#output definition is no longer needed when using zoptilib (reads output file and metrics from Zopti initialization line if #output not given)
accept zero time as valid output value (happens when sub 10ms times are rounded towards zero)
updated zoptilib to version 1.0.3
-new similarity metrics MDSI and Butteraugli
-new (semi)automatic YUV -> RGB conversion for metrics that need RGB
-new init parameter matrix to set the YUV color matrix for the RGB conversion
-added toRGB() function for manual RGB conversions
-no downsampling by default in any of the metrics


Here's an example on how to use the new features:
-no #output line
-using MDSI metric with (semi)automatic RGB conversion

# read input video
video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')

# initialize metrics, specify YUV color matrix for RGB conversion (MDSI and Butteraugli need RGB)
zopti = Zopti(r'results.txt', metrics=['mdsi', 'time'], matrix='601')

... process the video ...

# measure similarity of original and alternate videos, save results to output file
zopti.run(orig, alternate)

[zorr]

From what I can tell this plugin is still being actively developed. Is it in a finished enough state that it is usable? and if so, is there any documentation on how to use it?

Yes to all of the above. The documentation is a bit lacking though, there is a tutorial series at the Avisynth Developer section. My intention is to add similar tutorials here as well (obviously using Vapoursynth syntax). Those Avisynth tutorials are usable though, the syntax of the optimizer "augmentations" has not changed. So take a look at these:

Augmented Script (part 1/2) (https://forum.doom9.org/showthread.php?p=1851085#post1851085)
Augmented Script (part 2/2) (https://forum.doom9.org/showthread.php?p=1851086#post1851086)
Hands-on tutorial (part 1/2) (https://forum.doom9.org/showthread.php?p=1851670#post1851670)
Hands-on tutorial (part 2/2) (https://forum.doom9.org/showthread.php?p=1851672#post1851672)
Optimizer arguments (https://forum.doom9.org/showthread.php?p=1853726#post1853726)

The optimizer is fully functional, the remaining issues are mostly about how to use it most effectively. For example to figure out things like

how many frames are needed for the optimization
how to select the frames used in the optimization (and how to do that automatically)
which parameters should one select for optimization for each filter (ie where can you get the best bang for buck)
create optimization "templates" for different use cases (denoising, frame interpolation, color correction, etc) with optimal parameter ranges
which similarity metrics to use (and does it depend on the use case)
what kind of preprocessing works best for MVTools frame interpolation (and does it depend on the source video)


My goal would be to use it to find settings for MVtools to interpolate frames for anime at the highest possible quality. Motion interpolation normally fails with animated content, and so I would like to see how well it works in a best case scenario with optimized settings. I'm working on trying to restore a field blended show, and if this works well enough, it would be very helpful for me.

To get you started here's an MVTools script with the augmentations you need. I'm not sure if I used all the parameters possible, you can add/remove them as you wish.

import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 10 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

# read input video
video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')
orig = video

# initialize Zopti - starts measuring runtime
zopti = Zopti(r'results.txt', metrics=['ssim', 'time'])

# you could add preprocessing here to help MSuper - not used here
searchClip = orig

super_pel = 4 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2 # optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = core.mv.Super(searchClip, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter)
super_render = core.mv.Super(orig, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, levels=1)

blockSize = 8 # optimize blockSize = _n_ | 4,8,16,32,64 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 5 # optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 2 # optimize searchRange = _n_ | 1..10 | searchRange
searchRangeFinest = 2 # optimize searchRangeFinest = _n_ | 1..10 | searchRangeFinest
_lambda = 1000*(blockSize*blockSize)/(8*8) # optimize _lambda = _n_ | 0..20000 | lambda
lsad=1200 # optimize lsad=_n_ | 8..20000 | LSAD
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=1 # optimize plevel=_n_ | 0..2 | plevel
overlap=0 # optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=0 # optimize overlapv=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / | overlapv
divide=0 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = True # optimize globalMotion = _n_ | False,True | globalMotion
badSAD = 10000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 4..50 | badRange
meander = True # optimize meander = _n_ | False,True | meander
trymany = False # optimize trymany = _n_ | False,True | trymany

# delta 2 because we interpolate a frame that matches original frame
delta = 2

useChroma = True
bv = core.mv.Analyse(super_search, isb = True, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, _lambda=_lambda, lsad=lsad, pnew=pnew, plevel=plevel, _global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badsad=badSAD, \
badrange=badRange, meander=meander, trymany=trymany)
fv = core.mv.Analyse(super_search, isb = False, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, _lambda=_lambda, lsad=lsad, pnew=pnew, plevel=plevel, _global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badsad=badSAD, \
badrange=badRange, meander=meander, trymany=trymany)

# NOTE: we disable scene change detection by setting thSCD1 very high
blockChangeThreshold = 10000
maskScale = 70 # optimize maskScale = _n_ | 1..300 | maskScale
inter = core.mv.FlowInter(orig, super_render, bv, fv, time=50, ml=maskScale, thscd1=blockChangeThreshold, thscd2=100, blend=False)

# for comparison original must be forwarded one frame
orig = orig[1:]

# cut out the part used in quality / speed evaluation
inter = inter[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
orig = orig[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(orig, inter)

Note that this script interpolates using delta=2 (uses even frames to recreate odd frames and vice versa). You can use delta=1 (which should be an easier goal) if you do the interpolation twice - first create new inbetween frames, then throw away the original frames and finally interpolate new inbetween frames again which should match the original frames. I can give you an example script of that if you want.

[zorr]

@ChaosKing, I stole your improvements to zoptilib, added some of my own and packaged it to the latest Zopti installation zip. :D

Perhaps we could agree on how to do the updates in the future. We could use your Github repository (https://github.com/theChaosCoder/zoptilib) but I'd need to have write permissions there. Is that possible?

[ChaosKing]

Yes it is possible. I can add you as a Collaborator.
You could also make a new repo and/or clone mine and just accept Pull requests.

[zorr]

Yes it is possible. I can add you as a Collaborator.
You could also make a new repo and/or clone mine and just accept Pull requests.

Let's go with the Collaborator route for now. I can make my own repo later if/when I have the time. :)

[ChaosKing]

Ok, send me your github email via PM then so I can add you :)

[zorr]

I'm trying to track down a crash which happens very consistently with this script:

import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 2 # how many frames are tested
MIDDLE_FRAME = 10 # middle frame number

# read input video
video = core.raws.Source(r'flower_cif.yuv', 352, 288, src_fmt='I420') # YUV420, crash
#video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi') # YUV420, crash
#video = core.ffms2.Source(source=r'huffyuv_rgb.avi') # RGB, works
#video = core.ffms2.Source(source=r'24-1200.mkv') # YUV420, works

#zopti = Zopti(r'results.txt', metrics=['ssim', 'time'], matrix='601') # works
#zopti = Zopti(r'results.txt', metrics=['gmsd', 'time'], matrix='601') # works
#zopti = Zopti(r'results.txt', metrics=['vmaf', 'time'], matrix='601') # works
zopti = Zopti(r'results.txt', metrics=['mdsi', 'time'], matrix='601') # crash
#zopti = Zopti(r'results.txt', metrics=['butteraugli', 'time'], matrix='601') # crash

# input color range is PC (full)
video = video.std.SetFrameProp(prop="_ColorRange", intval=0)

#noisy = video
noisy = video.grain.Add(var=25)
#denoised = noisy
denoised = noisy.grain.Add(var=25)

# cut out the part used in quality / speed evaluation
video = video[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
denoised = denoised[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(video, denoised)

Can someone else verify the problem and run that script? You can use this batch file to run it repeatedly until it fails, just give the script file as an argument. On my machine it always fails on the first try though. Change the vspipe path if needed.

:Loop
"D:\VapourSynth64Portable\VapourSynth64\vspipe" %1 .
@if %errorlevel% equ 0 goto :Loop
@echo Exit Code is %errorlevel%


So far I have figured out that

Crash happens on first frame, so it doesn't have anything to do with writing the result file. This I verified by running the script in VapourSynth editor.
Only crashes on specific source videos, I tried four different ones. You can download the flower_cif.yuv which crashes from here (http://trace.eas.asu.edu/yuv/flower/flower_cif.7z).
Does not depend on source filter, crashed with core.raws.Source as well as core.ffms2.Source
Seems to need the RGB conversion for the crash to occur. If video is already RGB there's no crash, and when using metrics where RGB conversion is not needed there's no crash.
Adding noise to compared clips does not have any effect, the crash occurs even without it (for example when comparing identical clips)
Crashes less frequently when run with VapourSynth editor.
Cutting the video shorter has no effect.
Setting or not setting _ColorRange FrameProp has no effect.
Sometimes hangs for several seconds before it crashes, sometimes it happens much faster. CPU load stays low when it hangs.

[ChaosKing]

No crashes. Tried msdi and butteraugli. Source is YUV420 1080p.
You could also try lsmas.LWLibavSource or Avisource. Maybe there are seeking issues?

EDIT
ok... just after I posted this I got Error: Failed to retrieve frame 0 with error: Expr: Failed to convert 'inf' to float
Output 0 frames in 1.01 seconds (0.00 fps)
Exit Code is 1
I think it ran for about 5min, ffms2 + mdsi


EDIT2
Inded, ffms2 has seeking issues with my source file. Thats very odd since it is a "normal" h264.mkv

EDIT3
It crashed with lsmash+ mdsi after ~10min with the same err message. (now its frame 1 :P)

EDIT4
I removed both trim lines and get immediately a crash

Script exceeded memory limit. Consider raising cache size.
Error: Failed to retrieve frame 12 with error: Expr: Failed to convert 'inf' to float
Output 12 frames in 2.19 seconds (5.49 fps)
Exit Code is 1

EDIT5
running in vseditor:

Error on frame 35 request:
Expr: Failed to convert 'inf' to float
Error on frame 53 request:
Expr: Failed to convert 'inf' to float
Error on frame 77 request:
Expr: Failed to convert 'inf' to float
Error on frame 105 request:
Expr: Failed to convert 'inf' to float
Error on frame 120 request:
Expr: Failed to convert 'inf' to float
2019-02-21 00:34:51.879
Error on frame 164 request:
Expr: Failed to convert 'inf' to float
Error on frame 172 request:
Expr: Failed to convert 'inf' to float
Error on frame 196 request:
Expr: Failed to convert 'inf' to float

[Are_]

I'm not able to make it crash with the default script and the source file you provided.

[ChaosKing]

If I remove addgrain or set var to a low value (var=2) it runs fine. Try setting it much higher like 80 and the convert err is triggered much more frequent.

It also happens when I replace addgrain with noisegen.Generate(). So I'm still not sure what the cause is.

Edit
I replaced convertToRGB() in zoptilib with return core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709"), but nothing changed.
Edit2
I can't trigger this error with butteraugli, so it could be a bug in msdi.

Edit3
https://github.com/WolframRhodium/muvsfunc/issues/19

[ChaosKing]

An older version of muvsfunc caused this Expr error. Now with the latest version it runs perfectly for more than an hour now. No crashes.

[Are_]

That explains why it was not triggering for me, I was using latest version.

[zorr]

An older version of muvsfunc caused this Expr error. Now with the latest version it runs perfectly for more than an hour now. No crashes.

This didn't fix the problem for me and I already had a version of muvsfunc where this inf problem was fixed. Also I get the error with Butteraugli as well.

I don't get any error messages either, just the return code, which is usually a large negative number like -1073741819. When it crashes in VS Editor the whole program goes down.

So I will have to keep investigating, thanks for the tests anyway. Chaos and Are, can you tell me the OS you run the tests with?

[EDIT]

It works when doing the RGB conversion with core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709"). So looks like the error is in fmtconv. Is that plugin still maintained by someone (latest change is 3 years ago)?

I could of course just use core.resize instead. What's the closest to this:
clip = core.fmtc.resample(clip=clip, css="444")
clip = core.fmtc.matrix(clip=clip, mat=matrix, col_fam=vs.RGB)
clip = core.fmtc.bitdepth(clip=clip, bits=bits_per_sample)


and converting to linear RGB:

clip = core.fmtc.transfer(clip, transs=matrix, transd='linear')


core.Resize doesn't have "601" matrix which should be used for standard definition content. Is it called something else?

[ChaosKing]

My OS is a Win10 Pro (1809) x64, I used my VS portable fatpack for the tests.

bt470bg = PAL 601
smpte170m = NTSC 601
Source https://forum.doom9.org/showthread.php?p=1681454#post1681454