Zopti

[zorr]

Zopti is what was called AvisynthOptimizer. The name was changed because it now supports VapourSynth as well. This first post is an introduction similar to this one (https://forum.doom9.org/showpost.php?p=1850994&postcount=1) on the Avisynth Development forum, just changed a bit to reflect the latest features and what is possible with VapourSynth. I also cut out some unnecessary blabbering.

Let's say you want to motion compensation to double frame rate or replace a few corrupted frames. The right tool for that job is obviously MVTools. The only problem now is finding good parameters to get the best quality (because every video needs a little different settings). Ok no problem, I will just quickly adjust these... wait... there are about 60 parameters to adjust! Finding good settings would involve *a lot* of manual testing. This is where Zopti jumps in and says "Don't worry pal, I will find the best settings for you!".

But how? A VapourSynth script is first "augmented" by adding instructions on which parameters should be optimized. The instructions are written inside comments so the script will continue to work normally. The script also needs to measure the quality of the processed video using current parameter values or measure the running time (or preferably both the quality and runtime). The script needs to write these results to a specific file. There is a helper Python library "zoptilib" you can use to do the quality/runtime measurements and write the file.

Now you might be wondering how on earth can the script measure the quality. I can only think of one way: to compare the frames to reference frames and measure the similarity. The closer the similary value, the better the quality. VapourSynth has (at least) these similarity metrics available: SSIM (https://en.wikipedia.org/wiki/Structural_similarity) (the classic, most well known), GMSD (http://www4.comp.polyu.edu.hk/~cslzhang/IQA/GMSD/GMSD.pdf) (seems better than SSIM) and VMAF (https://en.wikipedia.org/wiki/Video_Multimethod_Assessment_Fusion) (by Netflix). It's also possible to get MS-SSIM (https://www.researchgate.net/publication/4071876_Multiscale_structural_similarity_for_image_quality_assessment) (multi-scale SSIM) from the VMAF plugin and I also implemented B-SSIM (https://www.researchgate.net/publication/265164680_B-SSIM_Structural_Similarity_Index_for_Blurred_Videos) for AviSynth (SSIM with blurring adjustment).

Ok, but where do we get the reference frames? In case of MVTools we can use the original frames as reference frames and the script will try to reconstruct them using motion compensation (but it's not allowed to use the reference frame in the reconstruction). We can do something similar if we want to use MVTools to double the framerate: we first create the double-rate video, then remove the original frames from it, then double the framerate again and finally compare these to the original frames. Or you could reconstruct every odd frame using the even frames. This idea is not limited to MVTools, you could for example do color grading using some other software and then try to recreate the same result using VapourSynth. I'm sure the smart people here will find use cases I couldn't even dream about. :D

MVTools is a complex plugin and the settings can change the runtime of the script dramatically. In addition to quality we can also measure the runtime to find settings that are fast enough (for example if you want to use the script in real-time).

So if the purpose is to find the best settings, what do we consider the "best" when both quality and time are involved? For example we can have settings A with quality 99 (larger is better), time 2300ms (larger is worse) and settings B with quality 95 and time 200ms. Which one of these is better? We can use the concept of pareto domination to answer this question. When one solution is at least as good as the other solution in every objective (in this case the objectives are quality and speed) and better in at least one objective, it dominates the other solution. In this example, neither dominates the other. But if we have settings C with quality 99 and time 200ms it dominates both A and B. In the end we want to know all the nondominated solutions, which is called the pareto front. So there's going to be a list of parameters with increasing quality and runtime. You can have more than two objectives if you want, the same pareto concept works. There's a good and free ebook called "Essentials of Metaheuristics (https://cs.gmu.edu/~sean/book/metaheuristics/)" which describes the pareto concept and much more.

Zopti reads the augmented script, verifies it and starts running a metaheuristic optimization algorithm. There are currently four algorithm choices: NSGA-II, SPEA2, mutation and exhaustive. The first two are some of the best metaheuristic algorithms available and also described in the ebook mentioned above. The mutation is my own simplistic algorithm (but it can be useful if you're in a hurry since it's the most efficient). Exhaustive will try all possible combinations, it's useful if you only have a small number of them (you could for example do an exhaustive search on one parameter value only). All the metaheuristic algorithms are working in a similar manner generating different solutions and testing them. The optimizer does these tests by creating scripts with certain parameter values and running them. The script then writes the measurements (quality/time) into a file which the optimizer reads. The metaheuristic then decides which parameters to try next based on the results. This continues until some end criteria is met.

There are three different ending criterias: number of iterations, time limit and "dynamic". Number of iterations is just that, the algorithm runs the script specific number of times. Setting a time limit can be pretty useful if you know how much time you can use for the optimization. You could for example let it run overnight for 8 hours and see the results in the morning. Dynamic variation is stopping only when it doesn't make any progress anymore. Making progress is defined by "no more pareto front members in last x iterations". This can be useful if you want to find the best possible results regardless of how long it takes.

During the optimization all tested parameters and their results are written to a log file. This log file can be "evaluated" during and after the optimization process. The evaluation basically means finding the pareto front and showing it. You can also create scripts from the pareto front in order to test them yourself. It's also possible to visualize the results of the log file in a two dimensional scatter chart. This chart highlights the pareto front and shows all the other results too. The chart can also be "autorefreshing": it loads the log file every few seconds and updates the visuals, which is a fun way to track how the optimization is progressing. Here's a gif what it looks like (obviously sped up):

https://s22.postimg.cc/lgnp0apsh/animation.gif (https://postimg.cc/image/q2jt8nbbh/)

The visualization has a few other bells and whistles but one I'd like to highlight here is the group by functionality: you can group the results by certain parameter's values and show a separate pareto front for each value. For example this what grouping by MVTools' blocksize looks like:

https://s22.postimg.cc/x5rooblwx/groupby-blocksize.png (https://postimg.cc/image/hkad4d9yl/)

Another visualization mode draws a heat map where two parameters are chosen for the x and y axis, the color of the cell at x,y is the best result found with that parameter combination, brighter color meaning better result.

https://i.postimg.cc/XJXVRkjQ/heatmap-example.png

Measuring the script's runtime is not very accurate, ie it has some variation. All the other processes running at the same time are using CPU cycles and messing with the cache so you should try to minimize other activity on the computer. In order to get more accurate results you can run a validation on the finished results log file. In validation the idea is to run the pareto front results multiple times and calculate the average, median, minimum or maximum of these multiple measurements (you can decide which one(s)).

So how good is the optimizer? Let's take a look at one example. A while back there was a thread (https://forum.doom9.org/showthread.php?t=175508) about best motion interpolation filters. There's a test video with a girl waving her hand. The best options that I know of are John Meyer's jm_fps script and FrameRateConverter. Here's a comparison gif with those two and Zopti. FramerateConverter was run with preset="slowest".

https://i.postimg.cc/Qd1wM4xH/ezgif-2-c921f9e1c33e-zopti.gif

Now obviously I'm showing a bit of a cherry-picked example here. The optimizer was instructed to search the best parameters for this short 10 frame sequence. I have run most of my optimization runs using only 10 frames because otherwise the optimization takes too long. Ideally the optimizer would automatically select the frames from a longer video, I have started working on such a feature but it's not finished yet (got sidetracked to implement a scene change detector...) so for now user has to make the selection.

At this point I envision that Zopti can be an useful tool for plugin authors so they can test plugin parameters and try to search for the optimal ones. Zopti can also be a bug hunter, it has found several bugs with the Avisynth MVTools by Pinterf which he has since fixed. The VapourSynth MVTools is currently not yet as robust, but jackoneill has already fixed one bug (thanks!). At some later point Zopti could be useful for normal users, when combined with a script with limited search space so that the search will not take excessively long time. Zopti (or rather the previous version AvisynthOptimizer) has already been used by some courageous people from these forums for scaling, motion compensation, denoising and HDR to SRD tone-mapping. I'd like to thank them all for providing important feedback, feature requests and bug reports.

You can download the latest Zopti version here (https://drive.google.com/file/d/1Ox_HfyvpV1bbL2_j-jx9hGOOYQAunXQp/view?usp=sharing). I will keep this link updated.

Version history:

1.2.3 (full details here (https://forum.doom9.org/showthread.php?p=1965241#post1965241))

new mode: convert
-converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
-supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
-output formats: CSV and zopti
-example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
-example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
improvements to visualization mode: line
-now supports option -types which controls which aggregate lines are displayed in the chart
-valid values are "best", "worst", "average", "median", "count" and "samples"
display of option -groupby improved, pareto fronts can no longer be cut off the screen
option -continue also supported with mutation algorithm
validation mode behavior change: it now validates all the results in the log file and not just the pareto front


1.2.2

supports a new way for an Avisynth script to write the result file
-first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
-then come the per frame result lines in any order, using the same format as before
-do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
-this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
-old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
-new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value


1.2.1

new option: retry
-tries to run the script again this many times if execution fails
-can be useful if the script uses plugins which are not 100% reliable and can crash
-default value is zero (do not retry)
-example: zopti script.avs -pop 24 -iters 10000 -retry 4
new visualization mode: line
-draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
-accepts parameter -range to limit the displayed values to certain range
-black line shows the best values, blue line indicates the number of results per value
-a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
-example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
support for custom output properties written by the script (only in VapourSynth scripts for now)
better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
option -continue now also supported by exhaustive algorithm (when using more than one thread)
order of runs in a visualization is now based on file name, not by modified date
all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
visualization mode seriespareto no longer contains the global pareto line
VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
bugfix: fixed a memory leak in XChart (at least partially)


Pre-1.2.1 history removed due to character count limitation.

The next post will be a hands-on denoising tutorial. Stay tuned.

[ChaosKing]

Nice work!

But I get a file not found error :-)
java.io.FileNotFoundException: D:\Download\Zopti-1.0-beta\work\result_1549155407307_1549155407426.txt
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileReader.<init>(Unknown Source)
at avisynthoptimizer.file.FileUtil.readLinesFromFile(FileUtil.java:58)
at avisynthoptimizer.AviSynthOptimizer.readVapoursynthResults(AviSynthOptimizer.java:6566)
at avisynthoptimizer.AviSynthOptimizer.waitAndReadResults(AviSynthOptimizer.java:6587)
at avisynthoptimizer.AviSynthOptimizer.runAVS(AviSynthOptimizer.java:6308)
at avisynthoptimizer.AviSynthOptimizer.evaluateFitness(AviSynthOptimizer.java:5588)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:190)
at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:35)
at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.lambda$evaluate$1(SequentialSolutionListEvaluator.java:24)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.evaluate(SequentialSolutionListEvaluator.java:24)
at avisynthoptimizer.spea2.DynamicSPEA2.evaluatePopulation(DynamicSPEA2.java:76)
at org.uma.jmetal.algorithm.impl.AbstractEvolutionaryAlgorithm.run(AbstractEvolutionaryAlgorithm.java:56)
at java.lang.Thread.run(Unknown Source)
java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(Unknown Source)
....
My script:
from zoptilib import Zopti
output_file = r'D:\results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['ssim', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 4 # optimize sigma = _n_ | 1,10 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)

alternate.set_output()

java -jar .\Zopti.jar D:\z.vpy
vspipe path is set in the ini file.

[zorr]

Nice work!

But I get a file not found error :-)


The # output line should probably look like this:

output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"


The file name must match.

Also this line
alternate.set_output()
is not necessary, Zopti sets the output. In some cases it is critical that the correct file is set as the output. The alternate file is the correct one in this case but for some reason it doesn't work if it's set first by Zopti and then again in the script.

[ChaosKing]

Thx, works now without alternate.set_output() :D

[ChaosKing]

I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13

Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)

VMAF just stops:
Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.

Maybe I do something wrong. Here is my script again

from zoptilib import Zopti
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 35 # optimize sigma = _n_ | 9..40 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)


result_*.txt:
<?xml version="1.0"?>
<VMAF version="1.3.11">
<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="100" aggregateVMAF="96.037" execFps="1.36185" timeTaken="73.4294" />
<frames>
<frame frameNum="0" adm2="0.983001" motion2="0" vif_scale0="0.709537" vif_scale1="0.953811" vif_scale2="0.974778" vif_scale3="0.983557" vmaf="91.1718" />
<frame frameNum="1" adm2="0.98031" motion2="14.7504" vif_scale0="0.760672" vif_scale1="0.952192" vif_scale2="0.97319" vif_scale3="0.982214" vmaf="100" />

<frame frameNum="98" adm2="0.992219" motion2="1.11641" vif_scale0="0.824504" vif_scale1="0.974561" vif_scale2="0.986421" vif_scale3="0.991392" vmaf="95.7912" />
<frame frameNum="99" adm2="0.990791" motion2="3.7195" vif_scale0="0.787278" vif_scale1="0.973289" vif_scale2="0.986163" vif_scale3="0.991446" vmaf="98.8478" />
</frames>
</VMAF>

EDIT
changed out1="SSIM: to out1="vmaf: and it works now

[zorr]

VMAF just stops:
Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.


I will look at the GMSD later, but the VMAF problem can be solved by renaming one of the #output parameters "vmaf" (not case sensitive).

[zorr]

I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13

Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)


In GMSD smaller number means better quality, so make sure you have GMSD: MIN(float) as the output. Zopti returns the sum of per frame values so otherwise it looks good to me.

[ChaosKing]

Ahh now I see it too, thx.

[ChaosKing]

My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.
This barely changed since the start (it's still running) https://i.imgur.com/4JszzyZ.png
There are (too?) many lines with height=1080

My log so far https://pastebin.com/UrKnMwWz

script
from zoptilib import Zopti
#output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"
#output_file = r'results.txt' # output out1="ssim: MAX(float)" file="results.txt"
output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"

zopti = Zopti(output_file, metrics=['gmsd', 'time']) #gmsd vmaf ssim
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[19157:19157+1]

# https://github.com/Infiziert90/getnative/blob/master/getnative.py#L112
# change format to GrayS with bitdepth 32 for descale
matrix_s = '709' if orig.format.color_family == vapoursynth.RGB else None
src_luma32 = core.resize.Point(orig, format=vapoursynth.YUV444PS, matrix_s=matrix_s)
src_luma32 = core.std.ShufflePlanes(src_luma32, 0, vapoursynth.GRAY)
orig = core.std.Cache(src_luma32)

width = 1400 # optimize width = _n_ | 1400..1920 | width
height = 800 # optimize height = _n_ | 800..1080 | height
x = ds.Debilinear(orig, width=width, height=height)
alternate = x.resize.Bilinear(orig.width, orig.height)

zopti.run(orig, alternate)

The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?

Is it possible that Zopti use only even numbers for width/height?

[zorr]

My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.


What optimization arguments were you using?


There are (too?) many lines with height=1080


Seems like the results are really good when height=1080 and that's why it's so common. Also when you only have two parameters to optimize and assuming you're using the default mutationCount "60% 1" then about halfway though it will only mutate one parameter. If the best results have height=1080 then at least half of the mutations have that value when it's mutating the width only. If you want it to explore a bit more you can set the mutationCount to 2 (or 100% which means the same in this case). Then it will mutate both parametes all the time.

The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?


When running the evalution on your log the pareto front is

1.5509998E-7 1190 width=1919 height=1080
3.7165887E-7 1120 width=1920 height=1079
4.1259636E-6 1110 width=1920 height=1077
8.270416E-6 1100 width=1920 height=1075
4.067268E-4 1090 width=1920 height=1034
0.0016862862 1080 width=1702 height=1080
0.0028691771 1070 width=1610 height=1080
0.0066714296 1060 width=1450 height=1080

so yes, it can understand the scientific notation. There's a problem though, the best results are really close to 1920x1080 and looking at the logs that got a perfect GMSD score 0.0. But the optimizer currently interprets 0.0 as invalid, that's why it's not in the pareto front. I will have to think about how to fix that.

It's a bit suspicious that you can get a perfect similarity score. Should it happen in your use case?

Is it possible that Zopti use only even numbers for width/height?

Yes, there are a couple of ways to do that. The simplest is probably:

width = 2*700 # optimize width = 2*_n_ | 700..960 | width
height = 2*400 # optimize height = 2*_n_ | 400..540 | height


Or you can specify a filter which only accepts numbers divisible by 2:

width = 1400 # optimize width = _n_ | 1400..1920 ; filter:x 2 % 0 == | width
height = 800 # optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height

[ChaosKing]

I don't know if a perfect score should be possible, but because the "resize error" is very small it is not easy to detect I guess.

The native res in my case should be 1600x900 (900p). Setting width & height to
width = 1400 # optimize width = _n_ | 1500..1680 | width
height = 800 # optimize height = _n_ | 880..910 | height
leads to this, which look almost to what I expected in the first place.
https://i.imgur.com/QNbTYMW.png
(And maybe I just need to test another frame)

There is already a native resolution "detector" here https://github.com/Infiziert90/getnative/blob/master/getnative.py
I just wanted to see how Zopti will perform. I think the differences need to be emphasized more, so the resizing error becomes more visible.

Zopti could come in handy if a bicubic resizer was used on the clip. It will be easy to find the b and c values like you have shown here https://forum.doom9.org/showthread.php?p=1859544#post1859544

[ChaosKing]

Breaking news: I'm an idot :D
I used my already encoded and "descaled" file instead of the source file. That also explains the perfect 1080p match -_-


height = 800 # optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height
I want to optimize the runs but I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3

[zorr]

Breaking news: I'm an idot :D
Welcome to the club. :)

I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3

I always visualize the rpn notation as a stack. Every operation can take values from the stack, do some computation with them and then put the result on the top of the stack. So for example this x 2 % 0 ==:

1) x - put x on the stack, stack is "x"
2) 2 - put 2 on the stack, stack is "x 2" (top of the stack on the right)
3) % - modulus operator, take two topmost values from the stack, calculate modulus and put the result back into stack - stack is (x % 2)
4) 0 - put 0 on the stack, stack is (x % 2) 0
5) == - equals operator, take two topmost values and put true on the top of the stack if they are equal, false otherwise - stack is ((x % 2) == 0)

But actually in this case the filter is not needed. You can optimize the height only and calculate width from it:

height = 800 # optimize height = _n_ | 800..1080 | height
width = height*(16/9)


or if you want to give it a small range around the calculated value you can do

height = 800 # optimize height = _n_ | 800..1080 | height
width_offset = 0 # optimize width_offset = _n_ | -3..3 | width_offset
width = height*(16/9) + width_offset

[Boulder]

I've been trying to run my resizing test with VMAF, the sample clip is 5 separate single frames from an episode of Black Sails. My first run ended up with the b value at -100, so I enlarged the area so that both b and c can go -300..300. However, VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?

e 68 / 1,00 : 100.0 200ms b=-33 c=236
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 (no samples) c 1,000
MUTATED GENERATION 69
Mutating 1 params by 1,0 % (phase 1,00)
e 69 / 1,00 : 100.0 200ms b=-52 c=236
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 (no samples) c 1,000
MUTATED GENERATION 70
Mutating 1 params by 1,0 % (phase 1,00)
70 / 1,00 : 100.0 210ms b=-50 c=236
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 (no samples) c 1,000
MUTATED GENERATION 71
Mutating 1 params by 1,0 % (phase 1,00)
e 71 / 1,00 : 100.0 200ms b=-42 c=246
Parameter sensitivity estimation with 4 result combinations
-> b 1,000 c 1,000
MUTATED GENERATION 72
Mutating 1 params by 1,0 % (phase 1,00)
e 72 / 1,00 : 100.0 200ms b=-42 c=225
0 iterations remaining is this generation
No improvement in 24 iterations - stopping
Parameter sensitivity estimation with 4 result combinations
-> (no samples) b 1,000 c 1,000

This is the script:
orig = core.ffms2.Source(source=r'blacksails.avi')

output_file = r'results.txt' # output out1="vmaf: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time']) # initialize output file and chosen metrics

b = 0/100.0 # optimize b = _n_/100.0 | -300..300 | b
c = 0/100.0 # optimize c = _n_/100.0 | -300..300 | c

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)

zopti.run(orig, alternate)

I've used this command line for these quick tests:
zopti test.vpy -alg mutation -iters dyn -dyniters 24 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

With gmsd, the run resulted in b=-77 c=5 as the optimal pair. In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.

[zorr]

VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?


I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.

With gmsd, the run resulted in b=-77 c=5 as the optimal pair.

Can you also try what SSIM gives as the optimal pair?

In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.

You can set the VMAF model with zopti.setVMAFModel(). Changing the pool won't have any effect since Zopti calculates the result from the individual frames and always just sums them up. The ci shouldn't change the scores either, it's just extra information.

[Boulder]

I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.
A very quick test shows this kind of behaviour, so it looks like VMAF cannot measure the difference between the clips accurately enough. That is quite odd because I would expect that downsizing to 720p before upsizing to 4K would have a big enough effect. Maybe I'll also have to try keeping the original at 1080p and use model 0 even though my TV is 4K.

e 178 / 20301: 100.0 200ms b=-123 c=0
e 179 / 20301: 100.0 200ms b=-122 c=0
e 180 / 20301: 100.0 200ms b=-121 c=0
e 181 / 20301: 100.0 200ms b=-120 c=0
e 182 / 20301: 100.0 200ms b=-119 c=0
e 183 / 20301: 100.0 200ms b=-118 c=0
e 184 / 20301: 100.0 200ms b=-117 c=0
e 185 / 20301: 100.0 200ms b=-116 c=0
186 / 20301: 99.98204 200ms b=-115 c=0
187 / 20301: 99.99383 200ms b=-114 c=0
188 / 20301: 99.9717 200ms b=-113 c=0
189 / 20301: 99.97675 200ms b=-112 c=0
190 / 20301: 99.96704 200ms b=-111 c=0
191 / 20301: 99.953285 200ms b=-110 c=0
192 / 20301: 99.944626 200ms b=-109 c=0

SSIM gives b=-59, c=31 as the optimal pair. That difference with gmsd would be expected if SSIM favours blurring more. Without testing, I assume that gmsd's result would be slightly sharper looking.

EDIT: The clip is here if you'd like to test it: https://drive.google.com/open?id=1rSc-XBrVSGB5VupxyLVQSg40vwtUZFNN

[zorr]

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)


Where is the resamplehq function from? I found resamplehq (http://vsdb.top/plugins/resamplehq) script but it doesn't have the same method name or arguments.

[ChaosKing]

It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122e359c37543fe/revisions

[WorBry]

.. so it looks like VMAF cannot measure the difference between the clips accurately enough.

There's also this issue where the VMAF result is skewed by a component 'motion2=0' score for the first frame in a clip:

https://forum.doom9.org/showthread.php?p=1864561#post1864561

The only information I can find about this parameter is:

"Motion. This is a simple measure of the temporal difference between adjacent frames. This is accomplished by calculating the average absolute pixel difference for the luminance component."

https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

and:

"motion2 score typically ranges from 0 (static) to 20 (high-motion)"

https://github.com/Netflix/vmaf/blob/master/resource/doc/VMAF_Python_library.md

Testing your Blacksails clip against itself, with VapourSynth VMAF v3 (Model=0) gives:

<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="8" aggregateVMAF="99.6711" aggregatePSNR="60" aggregateSSIM="0.99993" aggregateMS_SSIM="0.999936" execFps="2.27555" timeTaken="3.51563" />
<frames>
<frame frameNum="0" adm2="1" motion2="0" ms_ssim="0.999889" psnr="60" ssim="0.9999" vif_scale0="0.999999" vif_scale1="0.999997" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="97.4274" />
<frame frameNum="1" adm2="1" motion2="51.9751" ms_ssim="0.999997" psnr="60" ssim="0.999997" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999997" vif_scale3="0.999997" vmaf="100" />
<frame frameNum="2" adm2="1" motion2="45.4778" ms_ssim="0.999998" psnr="60" ssim="0.999999" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999996" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="3" adm2="1" motion2="45.4778" ms_ssim="0.999994" psnr="60" ssim="0.999995" vif_scale0="0.999997" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="4" adm2="1" motion2="48.0867" ms_ssim="0.99986" psnr="60" ssim="0.999837" vif_scale0="0.999999" vif_scale1="0.999996" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="100" />
<frame frameNum="5" adm2="1" motion2="71.4518" ms_ssim="0.999999" psnr="60" ssim="0.999999" vif_scale0="1" vif_scale1="0.999999" vif_scale2="0.999998" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="6" adm2="1" motion2="74.8847" ms_ssim="0.999751" psnr="60" ssim="0.999716" vif_scale0="0.999994" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="7" adm2="1" motion2="74.8847" ms_ssim="1" psnr="60" ssim="1" vif_scale0="0.999998" vif_scale1="0.999995" vif_scale2="0.999993" vif_scale3="0.999994" vmaf="100" />
FFMPEG SSIM reports lossless, 1.00000 (Inf)

[Boulder]

It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122e359c37543fe/revisions

I should have the latest version, but looking at the code, I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.

[zorr]

I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.

Thanks, got it working. Also had to set kernel='bicubic' since the default is spline36 in the latest version.

I ran some exhaustive tests with your clip, I set the divider to 10.0 instead of 100.0 to make it faster. The results were interesting:

https://i.postimg.cc/JzBccFr0/ssim-gmsd-vmaf2.png

SSIM and GMSD are roughly in agreement where the best settings are. SSIM had best at b=-6 c=3. Note that this is equivalent to b=-60 c=30 with your original script which is very close to your best result b=-59, c=31. GMSD's best is at b=-7 c=1, a bit further away from your result b=-77 c=5. GMSD is a bit more "focused", it seems to see the differences better than SSIM.

VMAF on the other hand gives many "perfect" results and when we look at only those the center is very close to the top left coordinates b=-30 c=30. I think VMAF is trying to say that the differences are so small that they're not perceptible to average human. Have you tried eyeballing the different settings? It is however a bit of a mystery why VMAF has the center of best results in a different place than SSIM and GMSD.

[Boulder]

To be honest, without zooming in, it's not easy to see the differences of the optimal parameters provided by the various metrics. My guess is that VMAF is not the best method for these single frame comparisons but has much better value with moving video. GMSD looks quite promising with these scaling comparisons.

[ChaosKing]

muvsfunc added a new similarity metric: MDSI(Mean Deviation Similarity Index) https://github.com/WolframRhodium/muvsfunc/commit/0e17b8ca44bde1844ce68749c70cf0cb6b1db304
MDSI is a full reference IQA model that utilize gradient similarity (GS), chromaticity similarity (CS), and deviation pooling (DP).
The lowerer the MDSI score, the higher the image perceptual quality.
Larger MDSI values indicate to the more severe distorted images, while an image with perfect quality is assessed by a quality score of zero.

@Boulder you shoud also try https://github.com/fdar0536/VapourSynth-butteraugli
It detects even the smallest change very reliably.

[ChaosKing]

I've added butteraugli and mdsi to zoptilib https://pastebin.com/511BcmNp

[Boulder]

Thanks, nice to test various options :)

[zorr]

I've added butteraugli and mdsi to zoptilib https://pastebin.com/511BcmNp

Thanks! I'm running some tests on those two. Some initial observations:

MDSI seems promising based on the authors' paper. It requires clips to be RGB. Boulder, is your source material in BT 709 colorspace? I'm not sure if Zopti should automatically do the conversation or just require RGB. MDSI by default downscales the clips by 2 which is kinda counterproductive when trying to measure the effects of scaling. I will add a method to set the downscaling value. The current MDSI implementation in muvsfunc returns "inf" for one of the frames and the script doesn't finish. Perhaps it's not ready for prime time (since it's not even released yet).

Butteraugli can indeed measure small differences, that's its main purpose. So it may not be accurate with large differences but should work well for Boulder's use case. It's also slow. I mean slooooow... one iteration (8 frames) takes 22 seconds on my machine. :scared: Butteraugli also needs RGB and it needs to be gamma corrected... if I'm interpreting it correctly clips should be in linear RGB space. How does one do that conversion with Vapoursynth?

// Value of pixels of images rgb0 and rgb1 need to be represented as raw
// intensity. Most image formats store gamma corrected intensity in pixel
// values. This gamma correction has to be removed, by applying the following
// function:
// butteraugli_val = 255.0 * pow(png_val / 255.0, gamma);
// A typical value of gamma is 2.2. It is usually stored in the image header.
// Take care not to confuse that value with its inverse. The gamma value should
// be always greater than one.
// Butteraugli does not work as intended if the caller does not perform
// gamma correction.

[WolframRhodium]

The current MDSI implementation in muvsfunc returns "inf" for one of the frames and the script doesn't finish.

Could you provide the material related to such error to me, so that I could fix it asap?

[zorr]

Could you provide the material related to such error to me, so that I could fix it asap?

Here's the script:

import vapoursynth as vs
import resamplehq as rhq
from zoptilib import Zopti

core = vs.core

orig = core.ffms2.Source(source=r'blacksails.avi')

# convert to RGB
orig = core.fmtc.resample(clip=orig, css="444")
orig = core.fmtc.matrix(clip=orig, mat="709", col_fam=vs.RGB)
orig = core.fmtc.bitdepth(clip=orig, bits=8)

zopti = Zopti(r'result.txt', metrics=['mdsi', 'time']) # initialize output file and chosen metrics

b = -30/10.0 # optimize b = _n_/10.0 | -30..30 | b
c = -30/10.0 # optimize c = _n_/10.0 | -30..30 | c

alternate = rhq.resample_hq(orig, width=1280, height=720, kernel='bicubic', filter_param_a=b, filter_param_b=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
zopti.run(orig, alternate)


You can download the source clip from here (https://drive.google.com/file/d/1rSc-XBrVSGB5VupxyLVQSg40vwtUZFNN/view).

You need the zoptilib version ChaosKing modded, here (https://pastebin.com/511BcmNp).
Oh and thanks for the muvsfunc library, the similarity metrics are perhaps the most important ingredient on making an optimizer like zopti work! :thanks:

[WolframRhodium]

You can download the source clip from here (https://drive.google.com/file/d/1rSc-XBrVSGB5VupxyLVQSg40vwtUZFNN/view).

You need the zoptilib version ChaosKing modded, here (https://pastebin.com/511BcmNp).


It may have been fixed after this commit (https://github.com/WolframRhodium/muvsfunc/commit/bbdac4695c99e686222f537ab201290e36f3788f). I also disable downsampling by default now.

Thank you for your support and recognition. Your work on Zopti is impressive.

[zorr]

It may have been fixed after this commit (https://github.com/WolframRhodium/muvsfunc/commit/bbdac4695c99e686222f537ab201290e36f3788f). I also disable downsampling by default now.

Thanks, it is indeed fixed. And the results are in:

https://i.postimg.cc/QXQvGWK0/ssim-gmsd-vmaf-butteraugli-mdsi.png

Butteraugli is not as smooth function as the others especially near the best value. It's more focused than SSIM and slightly more focused than GMSD near the best value. Best value at b=-8, c=2 which again quite nicely agrees with SSIM and GMSD. It took almost 24 hours to create that picture, hopefully I don't have to do that again... but of course I have to because this was calculated without gamma correction. But I think I now know how to do it:

# convert to linear RGB
orig = core.fmtc.resample(clip=orig, css="444")
orig = core.fmtc.matrix(clip=orig, mat="709", col_fam=vs.RGB)
orig = core.fmtc.transfer(orig, transs="709", transd='linear') # to linear RGB
orig = core.fmtc.bitdepth(clip=orig, bits=8)


MDSI is even more focused than GMSD. Best value is at b=-6, c=4, still very close to the others.

All except VMAF are giving best b within [-6 .. -8] and best c within [1 .. 4]. So for this use case it probably doesn't matter which one of those is chosen. More tests are needed to determine which one works best for other use cases.


Thank you for your support and recognition. Your work on Zopti is impressive.

Thanks. Are you planning on implementing more similarity metrics? Of course quality matters more than quantity but it's always possible to optimize for more than one similarity metric and there might be some useful combinations.

[WolframRhodium]

Are you planning on implementing more similarity metrics? Of course quality matters more than quantity but it's always possible to optimize for more than one similarity metric and there might be some useful combinations.

I will implement more if I find them practical. It's also interesting to see if those no-reference metrics work well, but I have not found good candidates.

Anyway, you can set 'downsample=False' in SSIM to skip downsampling. I think those full-reference metrics are highly correlated.

[zorr]

I will implement more if I find them practical. It's also interesting to see if those no-reference metrics work well, but I have not found good candidates.

A no-reference metric would be awesome because it would enable a whole new class of things you could do with the optimizer. Of course it would depend on how good the metric actually is...

What's your opinion on WaDIQaM / DIQaM? The paper is here (https://arxiv.org/abs/1612.01697) and someone did a PyTorch implementation of WaDIQaM (https://github.com/lidq92/WaDIQaM). It can also work as full reference metric.

Anyway, you can set 'downsample=False' in SSIM to skip downsampling.
Ah yes, I noticed the downscaling when I tried to compare Avisynth's SSIM and your implementation. I couldn't make them give identical results even though I set the k1, k2 and downsample -parameters to match the Avisynth version. Perhaps the Avisynth version is non-standard.

I think those full-reference metrics are highly correlated.
Agreed, at least in this particular case. I compared SSIM and GMSD with denoising and there SSIM had more ringing artifacts.

[WolframRhodium]

What's your opinion on WaDIQaM / DIQaM? The paper is here (https://arxiv.org/abs/1612.01697) and someone did a PyTorch implementation of WaDIQaM (https://github.com/lidq92/WaDIQaM). It can also work as full reference metric.

It looks good, but a network with 10 conv layers and 2 fc layers seems to be too heavy for parameter optimization (upconv7 model from waifu2x consists of only 7 conv layers though it doesn't have any pooling layer). Based on my experience in image restoration, I'm also worry that those DL-based methods can't generalize well in the real-world settings.

Porting it to VapourSynth is not hard. I will check it later.

[zorr]

Based on my experience in image restoration, I'm also worry that those DL-based methods can't generalize well in the real-world settings.

I suspect that too, there was a cross-database evaluation and the no-reference models only do reasonably well when the distortions are of the same type as in the training set. SOM (Semantic obviousness metric for image quality assessment) looks pretty good at generalizing but I couldn't find any implementations of it.

Porting it to VapourSynth is not hard. I will check it later.

Thanks, it will be interesting even if it doesn't perform well in real-world tests.

[zorr]

It took almost 24 hours to create that picture, hopefully I don't have to do that again... but of course I have to because this was calculated without gamma correction.

Here's a comparison of Butteraugli without gamma correction and with it (in linear RGB):

https://i.postimg.cc/65ygH0R1/butteraugli-gamma-comparison.png

There are some changes, nothing radical but the gamma corrected version is slightly smoother. Best value has moved from c=2 to c=1.

[kriNon]

From what I can tell this plugin is still being actively developed. Is it in a finished enough state that it is usable? and if so, is there any documentation on how to use it?

My goal would be to use it to find settings for MVtools to interpolate frames for anime at the highest possible quality. Motion interpolation normally fails with animated content, and so I would like to see how well it works in a best case scenario with optimized settings. I'm working on trying to restore a field blended show, and if this works well enough, it would be very helpful for me.

[zorr]

Zopti version 1.0.1-beta released, see the first post.

#output definition is no longer needed when using zoptilib (reads output file and metrics from Zopti initialization line if #output not given)
accept zero time as valid output value (happens when sub 10ms times are rounded towards zero)
updated zoptilib to version 1.0.3
-new similarity metrics MDSI and Butteraugli
-new (semi)automatic YUV -> RGB conversion for metrics that need RGB
-new init parameter matrix to set the YUV color matrix for the RGB conversion
-added toRGB() function for manual RGB conversions
-no downsampling by default in any of the metrics


Here's an example on how to use the new features:
-no #output line
-using MDSI metric with (semi)automatic RGB conversion

# read input video
video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')

# initialize metrics, specify YUV color matrix for RGB conversion (MDSI and Butteraugli need RGB)
zopti = Zopti(r'results.txt', metrics=['mdsi', 'time'], matrix='601')

... process the video ...

# measure similarity of original and alternate videos, save results to output file
zopti.run(orig, alternate)

[zorr]

From what I can tell this plugin is still being actively developed. Is it in a finished enough state that it is usable? and if so, is there any documentation on how to use it?

Yes to all of the above. The documentation is a bit lacking though, there is a tutorial series at the Avisynth Developer section. My intention is to add similar tutorials here as well (obviously using Vapoursynth syntax). Those Avisynth tutorials are usable though, the syntax of the optimizer "augmentations" has not changed. So take a look at these:

Augmented Script (part 1/2) (https://forum.doom9.org/showthread.php?p=1851085#post1851085)
Augmented Script (part 2/2) (https://forum.doom9.org/showthread.php?p=1851086#post1851086)
Hands-on tutorial (part 1/2) (https://forum.doom9.org/showthread.php?p=1851670#post1851670)
Hands-on tutorial (part 2/2) (https://forum.doom9.org/showthread.php?p=1851672#post1851672)
Optimizer arguments (https://forum.doom9.org/showthread.php?p=1853726#post1853726)

The optimizer is fully functional, the remaining issues are mostly about how to use it most effectively. For example to figure out things like

how many frames are needed for the optimization
how to select the frames used in the optimization (and how to do that automatically)
which parameters should one select for optimization for each filter (ie where can you get the best bang for buck)
create optimization "templates" for different use cases (denoising, frame interpolation, color correction, etc) with optimal parameter ranges
which similarity metrics to use (and does it depend on the use case)
what kind of preprocessing works best for MVTools frame interpolation (and does it depend on the source video)


My goal would be to use it to find settings for MVtools to interpolate frames for anime at the highest possible quality. Motion interpolation normally fails with animated content, and so I would like to see how well it works in a best case scenario with optimized settings. I'm working on trying to restore a field blended show, and if this works well enough, it would be very helpful for me.

To get you started here's an MVTools script with the augmentations you need. I'm not sure if I used all the parameters possible, you can add/remove them as you wish.

import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 10 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

# read input video
video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')
orig = video

# initialize Zopti - starts measuring runtime
zopti = Zopti(r'results.txt', metrics=['ssim', 'time'])

# you could add preprocessing here to help MSuper - not used here
searchClip = orig

super_pel = 4 # optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2 # optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = core.mv.Super(searchClip, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter)
super_render = core.mv.Super(orig, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, levels=1)

blockSize = 8 # optimize blockSize = _n_ | 4,8,16,32,64 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 5 # optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 2 # optimize searchRange = _n_ | 1..10 | searchRange
searchRangeFinest = 2 # optimize searchRangeFinest = _n_ | 1..10 | searchRangeFinest
_lambda = 1000*(blockSize*blockSize)/(8*8) # optimize _lambda = _n_ | 0..20000 | lambda
lsad=1200 # optimize lsad=_n_ | 8..20000 | LSAD
pnew=0 # optimize pnew=_n_ | 0..256 | pnew
plevel=1 # optimize plevel=_n_ | 0..2 | plevel
overlap=0 # optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=0 # optimize overlapv=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / | overlapv
divide=0 # optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = True # optimize globalMotion = _n_ | False,True | globalMotion
badSAD = 10000 # optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24 # optimize badRange = _n_ | 4..50 | badRange
meander = True # optimize meander = _n_ | False,True | meander
trymany = False # optimize trymany = _n_ | False,True | trymany

# delta 2 because we interpolate a frame that matches original frame
delta = 2

useChroma = True
bv = core.mv.Analyse(super_search, isb = True, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, _lambda=_lambda, lsad=lsad, pnew=pnew, plevel=plevel, _global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badsad=badSAD, \
badrange=badRange, meander=meander, trymany=trymany)
fv = core.mv.Analyse(super_search, isb = False, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, _lambda=_lambda, lsad=lsad, pnew=pnew, plevel=plevel, _global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badsad=badSAD, \
badrange=badRange, meander=meander, trymany=trymany)

# NOTE: we disable scene change detection by setting thSCD1 very high
blockChangeThreshold = 10000
maskScale = 70 # optimize maskScale = _n_ | 1..300 | maskScale
inter = core.mv.FlowInter(orig, super_render, bv, fv, time=50, ml=maskScale, thscd1=blockChangeThreshold, thscd2=100, blend=False)

# for comparison original must be forwarded one frame
orig = orig[1:]

# cut out the part used in quality / speed evaluation
inter = inter[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
orig = orig[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(orig, inter)

Note that this script interpolates using delta=2 (uses even frames to recreate odd frames and vice versa). You can use delta=1 (which should be an easier goal) if you do the interpolation twice - first create new inbetween frames, then throw away the original frames and finally interpolate new inbetween frames again which should match the original frames. I can give you an example script of that if you want.

[zorr]

@ChaosKing, I stole your improvements to zoptilib, added some of my own and packaged it to the latest Zopti installation zip. :D

Perhaps we could agree on how to do the updates in the future. We could use your Github repository (https://github.com/theChaosCoder/zoptilib) but I'd need to have write permissions there. Is that possible?

[ChaosKing]

Yes it is possible. I can add you as a Collaborator.
You could also make a new repo and/or clone mine and just accept Pull requests.

[zorr]

Yes it is possible. I can add you as a Collaborator.
You could also make a new repo and/or clone mine and just accept Pull requests.

Let's go with the Collaborator route for now. I can make my own repo later if/when I have the time. :)

[ChaosKing]

Ok, send me your github email via PM then so I can add you :)

[zorr]

I'm trying to track down a crash which happens very consistently with this script:

import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 2 # how many frames are tested
MIDDLE_FRAME = 10 # middle frame number

# read input video
video = core.raws.Source(r'flower_cif.yuv', 352, 288, src_fmt='I420') # YUV420, crash
#video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi') # YUV420, crash
#video = core.ffms2.Source(source=r'huffyuv_rgb.avi') # RGB, works
#video = core.ffms2.Source(source=r'24-1200.mkv') # YUV420, works

#zopti = Zopti(r'results.txt', metrics=['ssim', 'time'], matrix='601') # works
#zopti = Zopti(r'results.txt', metrics=['gmsd', 'time'], matrix='601') # works
#zopti = Zopti(r'results.txt', metrics=['vmaf', 'time'], matrix='601') # works
zopti = Zopti(r'results.txt', metrics=['mdsi', 'time'], matrix='601') # crash
#zopti = Zopti(r'results.txt', metrics=['butteraugli', 'time'], matrix='601') # crash

# input color range is PC (full)
video = video.std.SetFrameProp(prop="_ColorRange", intval=0)

#noisy = video
noisy = video.grain.Add(var=25)
#denoised = noisy
denoised = noisy.grain.Add(var=25)

# cut out the part used in quality / speed evaluation
video = video[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
denoised = denoised[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(video, denoised)

Can someone else verify the problem and run that script? You can use this batch file to run it repeatedly until it fails, just give the script file as an argument. On my machine it always fails on the first try though. Change the vspipe path if needed.

:Loop
"D:\VapourSynth64Portable\VapourSynth64\vspipe" %1 .
@if %errorlevel% equ 0 goto :Loop
@echo Exit Code is %errorlevel%


So far I have figured out that

Crash happens on first frame, so it doesn't have anything to do with writing the result file. This I verified by running the script in VapourSynth editor.
Only crashes on specific source videos, I tried four different ones. You can download the flower_cif.yuv which crashes from here (http://trace.eas.asu.edu/yuv/flower/flower_cif.7z).
Does not depend on source filter, crashed with core.raws.Source as well as core.ffms2.Source
Seems to need the RGB conversion for the crash to occur. If video is already RGB there's no crash, and when using metrics where RGB conversion is not needed there's no crash.
Adding noise to compared clips does not have any effect, the crash occurs even without it (for example when comparing identical clips)
Crashes less frequently when run with VapourSynth editor.
Cutting the video shorter has no effect.
Setting or not setting _ColorRange FrameProp has no effect.
Sometimes hangs for several seconds before it crashes, sometimes it happens much faster. CPU load stays low when it hangs.

[ChaosKing]

No crashes. Tried msdi and butteraugli. Source is YUV420 1080p.
You could also try lsmas.LWLibavSource or Avisource. Maybe there are seeking issues?

EDIT
ok... just after I posted this I got Error: Failed to retrieve frame 0 with error: Expr: Failed to convert 'inf' to float
Output 0 frames in 1.01 seconds (0.00 fps)
Exit Code is 1
I think it ran for about 5min, ffms2 + mdsi


EDIT2
Inded, ffms2 has seeking issues with my source file. Thats very odd since it is a "normal" h264.mkv

EDIT3
It crashed with lsmash+ mdsi after ~10min with the same err message. (now its frame 1 :P)

EDIT4
I removed both trim lines and get immediately a crash

Script exceeded memory limit. Consider raising cache size.
Error: Failed to retrieve frame 12 with error: Expr: Failed to convert 'inf' to float
Output 12 frames in 2.19 seconds (5.49 fps)
Exit Code is 1

EDIT5
running in vseditor:

Error on frame 35 request:
Expr: Failed to convert 'inf' to float
Error on frame 53 request:
Expr: Failed to convert 'inf' to float
Error on frame 77 request:
Expr: Failed to convert 'inf' to float
Error on frame 105 request:
Expr: Failed to convert 'inf' to float
Error on frame 120 request:
Expr: Failed to convert 'inf' to float
2019-02-21 00:34:51.879
Error on frame 164 request:
Expr: Failed to convert 'inf' to float
Error on frame 172 request:
Expr: Failed to convert 'inf' to float
Error on frame 196 request:
Expr: Failed to convert 'inf' to float

[Are_]

I'm not able to make it crash with the default script and the source file you provided.

[ChaosKing]

If I remove addgrain or set var to a low value (var=2) it runs fine. Try setting it much higher like 80 and the convert err is triggered much more frequent.

It also happens when I replace addgrain with noisegen.Generate(). So I'm still not sure what the cause is.

Edit
I replaced convertToRGB() in zoptilib with return core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709"), but nothing changed.
Edit2
I can't trigger this error with butteraugli, so it could be a bug in msdi.

Edit3
https://github.com/WolframRhodium/muvsfunc/issues/19

[ChaosKing]

An older version of muvsfunc caused this Expr error. Now with the latest version it runs perfectly for more than an hour now. No crashes.

[Are_]

That explains why it was not triggering for me, I was using latest version.

[zorr]

An older version of muvsfunc caused this Expr error. Now with the latest version it runs perfectly for more than an hour now. No crashes.

This didn't fix the problem for me and I already had a version of muvsfunc where this inf problem was fixed. Also I get the error with Butteraugli as well.

I don't get any error messages either, just the return code, which is usually a large negative number like -1073741819. When it crashes in VS Editor the whole program goes down.

So I will have to keep investigating, thanks for the tests anyway. Chaos and Are, can you tell me the OS you run the tests with?

[EDIT]

It works when doing the RGB conversion with core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709"). So looks like the error is in fmtconv. Is that plugin still maintained by someone (latest change is 3 years ago)?

I could of course just use core.resize instead. What's the closest to this:
clip = core.fmtc.resample(clip=clip, css="444")
clip = core.fmtc.matrix(clip=clip, mat=matrix, col_fam=vs.RGB)
clip = core.fmtc.bitdepth(clip=clip, bits=bits_per_sample)


and converting to linear RGB:

clip = core.fmtc.transfer(clip, transs=matrix, transd='linear')


core.Resize doesn't have "601" matrix which should be used for standard definition content. Is it called something else?

[ChaosKing]

My OS is a Win10 Pro (1809) x64, I used my VS portable fatpack for the tests.

bt470bg = PAL 601
smpte170m = NTSC 601
Source https://forum.doom9.org/showthread.php?p=1681454#post1681454

[Are_]

My OS is Gentoo Linux, I used self compiled with GCC builds of everything (via package manager).

[EDIT]
clip = core.resize.Bicubic(clip, format=vs.YUV444PS)
clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s=matrix)
clip = core.resize.Bicubic(clip, format=vs.RGBS, transfer_in_s=transfer, transfer_s='linear')
You will need to work around the fact that bitdepth is in the format constant.

[Are_]

I know you don't need, but the shortcuts sometimes produce a different result, that 50% is not free, last time I checked out it was not that different though.

[ChaosKing]

Yep, this shows definitely some differences.
clip = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix)
long = core.resize.Bicubic(clip, format=vs.RGB24, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix, transfer_in_s=transfer, transfer_s='linear')

clip = core.Butteraugli.butteraugli(short, long)

[poisondeathray]

Yep, this shows definitely some differences.
clip = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix)
long = core.resize.Bicubic(clip, format=vs.RGB24, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix, transfer_in_s=transfer, transfer_s='linear')

clip = core.Butteraugli.butteraugli(short, long)


It's different because the "long" method uses 2 discrete operations (more rounding errors at 8bit precision)

If you use RGBS for the 1st clip step, you get same results

So the short method is better, faster, more accurate

[ChaosKing]

ok thx.
A quick test with RGBS first and then convert to RGB24:

clip = core.resize.Bicubic(orig, format=vs.RGBS, matrix_in_s=matrix)
long = core.resize.Bicubic(clip, format=vs.RGBS, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(orig, format=vs.RGBS, matrix_in_s=matrix, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(short, format=vs.RGB24)
long = core.resize.Bicubic(long, format=vs.RGB24)

clip = core.Butteraugli.butteraugli(short, long)
Now there are 0 differences.
Then I tested the speed (RGBS-long vs RGBS-short) ... and wow ... a 100fps difference!!! 163 fps vs 260 fps

[zorr]

clip = core.resize.Bicubic(clip, format=vs.YUV444PS)
clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s=matrix)
clip = core.resize.Bicubic(clip, format=vs.RGBS, transfer_in_s=transfer, transfer_s='linear')
You will need to work around the fact that bitdepth is in the format constant.

Thanks, got it working and there are no crashes anymore.

If you use RGBS for the 1st clip step, you get same results

So the short method is better, faster, more accurate

I verified this myself, the short method is equivalent to the long method in quality and is faster so it's preferred.

I also wanted to see if there are differences between fmtconv and core.resize. Looks like there are some, core.resize gives slightly brighter result as can be seen in the levels histogram.

https://i.postimg.cc/L508gSjb/rgbtest-vpy-2.gif

It's much more difficult to see any differences in the actual video frame though (and especially in the example as the colors have been munged by gif).

The differences are smaller (although spread to wider area) when using dithering with core.resize so I will enable that, performance didn't change much with the different dithering options. [EDIT] The third frame is the difference heat map from Butteraugli.

https://i.postimg.cc/Rq5VLRb7/rgbtest-vpy-none.png

https://i.postimg.cc/yxS8R5n6/rgbtest-vpy-error-diffusion.png

[EDIT] I have updated Zoptilib (https://github.com/theChaosCoder/zoptilib) with the new RGB conversion implementation. You can still use matrix '601' even though core.resize doesn't have it, it's the same as '470bg' and '170m' (those two give identical results).

[zorr]

I made some comparisons of the five similarity metrics. I used the same video as in the above post. Perhaps not the best choice since it looks like it was recorded with a potato but the small size makes the optimization task much faster.

I added some noise with grain.Add(25, seed=3) and then denoised it with FFT3DFilter using only bt, blockSize (arguments bw and bh), overlap (arguments ow and oh) and sigma. FFT3DFilter has many more arguments but the goal here is to compare the similarity metrics and not to find the best possible quality.

I ran an optimization with the default settings except set the iterations to 10000 to make sure I find the best settings, five runs per metric, five frames used in the metric calculation. No downscaling in any of the metrics (which is the default in the latest Zoptilib).

The first image here shows the best found result for each metric and heatmaps of the argument pairs bt / sigma and blockSize / overlap. The reddish dot in the maps is located at where the best result was found.

https://i.postimg.cc/HdHLQjrQ/comparison-1.png

Let's first focus on what the metrics agree on: the best result has bt=5 which is the maximum supported (two previous, current and two next frames). Also the best overlap is always half the blockSize.

BlockSize varies from 10 to 22. VMAF result is an outlier in this group, the other metrics have best blockSize at 10, 12 or 14.

Sigma varies from 308 to 482. Butteraugli has the smallest sigma, SSIM the largest. It's interesting that Butteraugli really dislikes the larger sigmas, VMAF does that too but it's a bit hard to see in this image. I suspect that there are some artifacts with the larger sigmas but most of the metrics are not sensitive to it. I will do some manual image quality comparisons later to very this.

SSIM, GMSD and MDSI are very "smooth", there are no large jumps in results when the parameter's value changes a little (with the exception of bt where large jumps happen at the lowest values). Butteraugli and VMAF look rougher. The smoothness is a good feature when trying to find the optimal settings, makes the search easier.

The VMAF results are pretty weird. It really likes bt=-1 ("sharpen only") which the other metrics considered inferior. There was a really good set of parameters at sigma=170 (actually any sigma, it doesn't change the result) blockTemporal=-1 blockSize=50 overlap=0 with score 98.62038 whereas the best result at sigma=381 blockTemporal=5 blockSize=22 overlap=11 had score 98.62366. The difference was small enough that three out of five runs didn't find the optimal value but this secondary set instead.

To be continued...

[zorr]

Let's take a closer look at how the metrics behave near the optimal result. I run exhaustive searches where two of the four arguments were locked to the best result found with each metric. The other two arguments are tested with every combination.

The first pair is sigma and bt (blockTemporal):

https://i.postimg.cc/3K6mLyRw/comparison-2-ex-si-bt.png

I have limited the x resolution to 100 in the maps but the results were calculated for every sigma from 100 to 800 (divided by 100 so the actual range is 1.0 - 8.0) so each box represents the best found result of 8 sigmas.

SSIM and GMSD are very similar, GMSD doesn't like the high sigmas as much though (and it's also reflected in the sigma of the best result).

MDSI is a bit more focused, area of good results is smaller. While SSIM and GMSD didn't have a large improvement from bt=4 to bt=5 MDSI thinks there's a bigger difference.

Butteraugli is more uneven compared to the above three. The best sigma seems to get smaller with smaller bt. For some reason Butteraugli thinks bt=0 is much better than the above three metrics.

And finally VMAF... still thinking bt=-1 means high quality. It also strongly dislikes bt=1 and bt=2 with high sigmas, so much that it's hard to see any differences elsewhere.

Let's see the other pair blockSize / overlap:

https://i.postimg.cc/9W65cp0H/comparison-2-ex-bs-ol.png

Again SSIM and GMSD are very similar, but GMSD is more focused.

MDSI continues the trend and is even more focused. All three have better scores when overlap increases. Looks like the optimal value is always at overlap = blockSize/2 for every blockSize.

Butteraugli is kinda close to MDSI near the optimum result but there are some weird stripes elsewhere.

The VMAF on the other hand looks like it had a meltdown. There's no clear optimal range, values jump up and down even with small changes. The above mentioned rule overlap=blockSize/2 is also broken. Very hard for the optimizer to make sense of.

To be continued...

[ChaosKing]

Maybe it means that visually there is not that much difference? Have you checked some good and some bad candidates yourself? Or this source needs a different model?

[zorr]

Maybe it means that visually there is not that much difference? Have you checked some good and some bad candidates yourself? Or this source needs a different model?

No I haven't done that yet, but I intend to. It's possible that there isn't that much visual difference but it would be strange if doing mere sharpening (bt=-1) would look as good as 5D temporal denoising.

[zorr]

Before I start comparing the results visually I wanted to make one more exhaustive search using the knowledge gained earlier. Since we now know the best result always has bt=5 that can be left out of the search. Also the best result has overlap=blockSize/2 (except with VMAF, but it lost the vote on this one) so that can be used as well. The only things remaining are sigma and blockSize, so let's see what that looks like. I calculated results for even blockSizes only.

https://i.postimg.cc/q43GYPss/comparison-3-ex-si-bs.png

It's the same story again: GMSD is a bit more focused than SSIM, MDSI is more focused than GMSD. SSIM doesn't give almost any penalty for the higher sigmas but Butteraugli clearly dislikes them. GMSD and MDSI give smallish penalty for the higher sigmas. And again Butteraugli is a bit uneven and has a couple of stripes but the amount is quite moderate and I don't think it will be much of a problem for the optimizer.

The first VMAF map is almost white because the scores for the small blockSizes are so bad. In the second map I started from blockSize 10 up and we can see the details better. VMAF has more stripes and "noise" but it's easy to see that it also dislikes the higher sigmas.

I also wanted to see how VMAF sees the other highly rated parameter combination sigma=170 blockTemporal=-1 blockSize=50 overlap=0. So in this run the bt was set to -1 and overlap to 0. Looks like I was wrong when I said the sigma doesn't effect the score. It does, but only with specific blockSizes. The best blockSize 50 has red dots everywhere because the same top result was found in 699 out of 801 sigmas. For some reason the blockSizes divisible by 10 are usually good.

[ChaosKing]

Now it needs to be tested with some other sources to confirm the results xD I wonder how the results will change with a "HQ/clean" source or a clip with fine details.

Btw would it be possible to implement parallel execution to speed up things? Where the 'time' dimension is not important like in your test :)

[zorr]

Now it needs to be tested with some other sources to confirm the results xD

:scared:

I wonder how the results will change with a "HQ/clean" source or a clip with fine details.

Yes you're right, we can't make any deep conclusions from this one source. All right, I will do another test run. Perhaps Boulder's blacksails.avi would be a good one.

Btw would it be possible to implement parallel execution to speed up things? Where the 'time' dimension is not important like in your test :)

Yes. JMetal (the metaheuristics library used in Zopti) supports parallel evaluators, I just need to make sure that my own code is thread-safe before I enable that. For some of the algorithms (mutation, exhaustive) I have to make my own parallel implementation but that's not a problem.

There might not be that much of a speed improvement though. At least on my 4-core machine most of the cores are fully utilized even with one evaluation thread when using Vapoursynth.

[ChaosKing]

There might not be that much of a speed improvement though. At least on my 4-core machine most of the cores are fully utilized even with one evaluation thread when using Vapoursynth.

My 8 core 16 thread cpu is still hungry though :D
yeah the speed increase will be mainly in shorter clips or 1 frame clips.
But I think we can also scale better if we limit the thread count in VS and use multiple instances.

p.s. Does someone know there to find very clean animated (cartoon/anime) hd material. Rick and Morty for example looks very clean and doesn't have halos, noise, etc

[Mystery Keeper]

p.s. Does someone know there to find very clean animated (cartoon/anime) hd material. Rick and Morty for example looks very clean and doesn't have halos, noise, etcDisney's "Frozen" seems noiseless.

[WolframRhodium]

Here (https://gist.github.com/WolframRhodium/d4b117ccc98081e40a70946b884dbe36) is my port of WaDIQaM (https://github.com/lidq92/WaDIQaM), a CNN-based full-reference image quality evaluator.
(The paper also addresses the problem of no-reference evaluation, but the PyTorch re-implementation only contains a pre-trained FR model. Lets check it first)

One should install PyTorch and download the model at here (https://github.com/lidq92/WaDIQaM/blob/master/models/WaDIQaM-FR-TID2008) before testing.

I post the .py file on GitHub's gist since this is experimental.

[ChaosKing]

Another toy \o/

Mini tut how to install in portable fatpack
From here https://pytorch.org/get-started/locally/ I installed the packages.

Start powershell inside VapourSynth64 folder and run this command:
.\python.exe -m pip install https://download.pytorch.org/whl/cu100/torch-1.0.1-cp37-cp37m-win_amd64.whl <- (I used cuda 10)
.\python.exe -m pip install torchvision

Maybe torchvision is not necessary?

Quick benchmark 1080p
- cpu: 1.14 fps (Ryzen1700 @ 3.7ghz)
- cuda: 19.5fps (GTX 1070)


My script
clip = core.lsmas.LWLibavSource(r"E:\a.mkv").grain.Add(60)
clip2 = core.lsmas.LWLibavSource(r"E:\a.mkv")

clip = mvf.ToRGB(mvf.Depth(clip, 32))
clip2 = mvf.ToRGB(mvf.Depth(clip2, 32))

clip = mm.vs_wadiqam(clip, clip2, model_path=r"D:\Fansubs\WaDIQaM-FR-TID2008", seed=2019, use_cuda=False)

clip = core.text.FrameProps(clip, props=['Frame_WaDIQaM'])
clip.set_output()

Without addgrain I get a score of 6.x and with grain 4.x. vmaf would show 99-100 (with same clip for both inputs), but 6 seems so random + it stays between 6.0 - 6.4 depending on the current frame number. Is this correct?

[zorr]

Thanks Wolfram, it's an interesting metric.

I have updated Zoptilib (https://github.com/theChaosCoder/zoptilib) but it's not related to WaDIQaM, just some minor improvements I have done. I figured Chaos is about to add the WaDIQaM there soon so I better make my edits first. :)

[WolframRhodium]

Maybe torchvision is not necessary?

Yes, it has been removed. Thanks.


Without addgrain I get a score of 6.x and with grain 4.x. vmaf would show 99-100 (with same clip for both inputs), but 6 seems so random + it stays between 6.0 - 6.4 depending on the current frame number. Is this correct?


I'm not sure. I got similar results here. Maybe I should switch to official Chainer-based implementation later. :confused:`

Another consideration is that, while the official implementation uses every 32x32 sized patches in an image in quality evaluation, current PyTorch-based implementation only randomly samples 32 patches (adjustable) for acceleration.
It seems that it is acceptable to use every patches? The algorithm runs faster than I expected.

[WolframRhodium]

Official deepIQA (https://github.com/dmaniry/deepIQA) (WaDIQaM-FR and WaDIQaM-NR) has been ported in the same address (https://gist.github.com/WolframRhodium/d4b117ccc98081e40a70946b884dbe36).

[zorr]

This is what the best results look like for each metric. On the first row is the original frame and the frame after added noise. The rows after that show the best denoising result from each metric and an enhanced comparison view (I used muvsfunc Compare with power 1.4 and mode 2).

[EDIT] Seems like there are some weird patterns in that image when shown on the forum page. Those are not present if you open the image alone (at least in Firefox).
[EDIT2] Apparently that was because I had the page at 110% magnification which also scales images.

https://i.postimg.cc/FrcDTgnV/denoise2-analyze-vpy-0.png

It's really hard to see any major differences between the contenders. The task seems to have been too easy and using more noise would perhaps made the differences more clear. Oh well, I can always run the tests again...

The lower sigma preferred by Butteraugli has left some noise to the sky but on the other hand looking at the enhanced view the branches are no longer sticking out as clearly as in the other versions.

The last row shows VMAF with sigma=170 blockTemporal=-1 blockSize=50 overlap=0 which scored almost as good as the best settings. Clearly this mode is inferior and has hardly removed any noise. So at least we can conclude that sometimes VMAF gives bad results.

[VS_Fan]

... The last row shows VMAF with sigma=170 blockTemporal=-1 blockSize=50 overlap=0 which scored almost as good as the best settings. Clearly this mode is inferior and has hardly removed any noisebt= -1 is "Sharpen only". It won't reduce any noise but only sharpen the image.

May be bt=1 (2D) Spatial only is what you could use to get some interesting results. Also, you could try different noise values (sigma, sigma2, sigma3 and sigma4) for high, mid-high, mid-low and lower frequencies

[WorBry]

@ WolframRhodium,

Regarding MDSI. Is there provision in the original code for producing the Gradient Similarity (GS), Chromacity Similarity (CS) and pooled Gradient-Chromacity Similarity (GCS) maps shown in the original paper ?

http://i.imgur.com/tNSGgiHl.png (https://imgur.com/tNSGgiH)

https://arxiv.org/pdf/1608.07433.pdf

I've been using quality metrics to compare various 'visually lossless' intermediate codecs and I'm seeing some interesting results with MDSI that appear to be chroma-related - they don't occur when testing with greyscale images. The similarity maps would helpful in identifying the implicated image components.

[WolframRhodium]

Regarding MDSI. Is there provision in the original code for producing the Gradient Similarity (GS), Chromacity Similarity (CS) and pooled Gradient-Chromacity Similarity (GCS) maps shown in the original paper ?


I have updated the code of MDSI. Usage:

src, gs, cs, gcs = MDSI(src, ref, show_maps=True)


You may have to normalize these maps yourself for better visualization.

[ChaosKing]

Official deepIQA (https://github.com/dmaniry/deepIQA) (WaDIQaM-FR and WaDIQaM-NR) has been ported in the same address (https://gist.github.com/WolframRhodium/d4b117ccc98081e40a70946b884dbe36).

So, your "old" WaDIQaM is now vs_wadiqam_pytorch.py?
And vs_wadiqam_chainer is new with 2 "new" implementations , wadiqam_fr & wadiqam_nr (Full-reference WaDIQaM & No-reference WaDIQaM)?
Correct? Is it ready to test?

[WolframRhodium]

So, your "old" WaDIQaM is now vs_wadiqam_pytorch.py?
And vs_wadiqam_chainer is new with 2 "new" implementations , wadiqam_fr & wadiqam_nr (Full-reference WaDIQaM & No-reference WaDIQaM)?
Correct? Is it ready to test?

Yes, and I think it is ready to test. I have verified scores on some images.

Note I have make a modification to the code to make the result deterministic. Issue (https://github.com/dmaniry/deepIQA/issues/9#issuecomment-469211408)

[WorBry]

I have updated the code of MDSI. Usage:

src, gs, cs, gcs = MDSI(src, ref, show_maps=True)


You may have to normalize these maps yourself for better visualization.

Thanks a lot. ChaosKing/zorr, any chance you could update zoptilib accordingly ? Cheers.

[zorr]

Thanks a lot. ChaosKing/zorr, any chance you could update zoptilib accordingly ? Cheers.

You can set extra arguments using addParams.

For example in this case:

zopti.addParams('mdsi', dict(show_maps=True))

[WorBry]

Thanks, but I'm getting this error:

final = alt_clip.std.FrameEval(functools.partial(calc, clip=alt_clip, data=data), prop_src=prop_src)
AttributeError: 'tuple' object has no attribute 'std'

[ChaosKing]

Yeah show_map returns not a clip but a clip tuple https://github.com/WolframRhodium/muvsfunc/blob/116aa4a8db0403d44450e9115504e585453f7e3d/muvsfunc.py#L4584
I think FrameEval awlays expects a clip as return!?

[ChaosKing]

When I try to show the clip in vsedit I get this error msg:
GREY color family cannot have RGB matrix coefficients
orig = mvf.ToRGB(orig)
muv.MDSI(clip, orig, show_maps=True)[1]

and with mvf.ToYUV( muv.MDSI(clip, orig, show_maps=True)[1] )
Resize error 1026: YUV color family cannot have RGB matrix coefficients.

How can I remove RGB matrix coefficients? Or what should I do?

[WorBry]

Yeah show_map returns not a clip but a clip tuple https://github.com/WolframRhodium/muvsfunc/blob/116aa4a8db0403d44450e9115504e585453f7e3d/muvsfunc.py#L4584
I think FrameEval awlays expects a clip as return!?

So what to do ?

[WorBry]

When I try to show the clip in vsedit I get this error msg:
GREY color family cannot have RGB matrix coefficients
orig = mvf.ToRGB(orig)
muv.MDSI(clip, orig, show_maps=True)[1]

and with mvf.ToYUV( muv.MDSI(clip, orig, show_maps=True)[1] )
Resize error 1026: YUV color family cannot have RGB matrix coefficients.

How can I remove RGB matrix coefficients? Or what should I do?


core.std.SetFrameProp(clip, prop='_Matrix', delete=True)


I'm confused now. Is this a way to output the maps ? What does the [1] at the end of the MDSI call line do ?

[ChaosKing]

I'm confused now. Is this a way to output the maps ? What does the [1] at the end of the MDSI call line do ?

show_maps returns multiple clips
if show_maps:
return clip1, gs_hvs, cs, gcs

muv.MDSI(clipA, clipB, show_maps=True)[0] <-- 0 means clip1
muv.MDSI(clipA, clipB, show_maps=True)[1] <-- 1 = gs_hvs, 2 = cs, 3 = gcs

OR

clips = muv.MDSI(clipA, clipB, show_maps=True)
clip = clips[0]
gs_hvs = clips[1]
...

This works now, thx HolyWu
orig = muv.MDSI(orig, orig.grain.Add(42), show_maps=True)[2]
orig = core.std.SetFrameProp(orig, prop='_Matrix', delete=True)

[WorBry]

Yep, that works. Thanks.

[Iron_Mike]

Hi guys,

trying to run some metrics via Zopti... got a couple of questions...

(1) what is the easiest way to pass an image sequence (specifically EXR/DPX) to Zopti ?

(2) @WolframRhodium: why does muvsfunc only allow clips of the exact same format for the SSIM / GMSD metric ?

Thanks.

[poisondeathray]

(1) what is the easiest way to pass an image sequence (specifically EXR/DPX) to Zopti ?


ImageMagick .

The vapoursynth version supports many pixel formats, 10bit, 16bit, float .... all in their native format (e.g. DPX will be 10bit RGB, instead of 16bit)
http://www.vapoursynth.com/doc/plugins/imwri.html

(The avs version is very outdated, 8bit, limited format support)

[WolframRhodium]

(2) @WolframRhodium: why does muvsfunc only allow clips of the exact same format for the SSIM / GMSD metric ?

I expect people to convert format of clips themselves. I believe that allows the program to be correct, stable and concise.

[Iron_Mike]

ImageMagick .

The vapoursynth version supports many pixel formats, 10bit, 16bit, float .... all in their native format (e.g. DPX will be 10bit RGB, instead of 16bit)
http://www.vapoursynth.com/doc/plugins/imwri.html

(The avs version is very outdated, 8bit, limited format support)

thank you, that worked.

[Iron_Mike]

I expect people to convert format of clips themselves. I believe that allows the program to be correct, stable and concise.

trying to understand the approach here... your implementation can only compare the exact same video format ?

so if we want to compare a 16bit full RGB master with a 12/10/8 bit encode in various chroma sub sampling formats, we would convert the master to the format of the encode... ?

or how is this done correctly ?

Thanks.

[WorBry]

I have updated the code of MDSI. Usage:

src, gs, cs, gcs = MDSI(src, ref, show_maps=True)


You may have to normalize these maps yourself for better visualization.

The traces are very faint, but an Unsharp Mask (two passes if needs be) brings them up nicely.

[WolframRhodium]

your implementation can only compare the exact same video format ?

Yes.


so if we want to compare a 16bit full RGB master with a 12/10/8 bit encode in various chroma sub sampling formats, we would convert the master to the format of the encode... ?

Normally it would be:

yuv420p8 = core.resize.Bicubic(rgb48, format=vs.YUV420P8, matrix_s='709')

for Blu-ray material.

But it won't work for every case, like chroma locations other than left, or different matrix specification.

[Iron_Mike]

yuv420p8 = core.resize.Bicubic(rgb48, format=vs.YUV420P8, matrix_s='709')

thanks for that, ran some test with the resize cmd you posted and with core.fmtc... results seems to be almost identical...

now, why do you downscale the master and not upscale the encode to the masters format before comparison ?

Thanks.

[WolframRhodium]

now, why do you downscale the master and not upscale the encode to the masters format before comparison ?

It's just a trade-off. Downscaling is fast but inaccurate, while upscaling is accurate but slow. I'm just make a demonstration here instead of providing a high-quality conversion. (The quality of upscaling by nnedi3 is even higher, for example)

[WorBry]

@ WolframRhodium,

Regarding MDSI. Is there provision in the original code for producing the Gradient Similarity (GS), Chromacity Similarity (CS) and pooled Gradient-Chromacity Similarity (GCS) maps shown in the original paper ?

http://i.imgur.com/tNSGgiHl.png (https://imgur.com/tNSGgiH)

https://arxiv.org/pdf/1608.07433.pdf

I've been using quality metrics to compare various 'visually lossless' intermediate codecs and I'm seeing some interesting results with MDSI that appear to be chroma-related - they don't occur when testing with greyscale images. The similarity maps would helpful in identifying the implicated image components.

Thought the results might be of interest:

https://forum.doom9.org/showthread.php?p=1868159#post1868159

[ChaosKing]

https://github.com/theChaosCoder/zoptilib
New parameter: showstats - show metric stats per frame (vmaf and time are not supported) - Zopti(r'results.txt', metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli', 'time'], showstats=True, matrix='709')
Updated WaDIQaM (pytorch) and added WaDIQaM nr/fr (chainer), untested bcs it needs cupy and other stuff. Lots of gigabytes und modules needs to be installed first.
https://i.imgur.com/RuoSMxc.png

Now I don't like the forced heatmap in butteraugli :/

[WorBry]

https://github.com/theChaosCoder/zoptilib
New parameter: showstats - show metric stats per frame (vmaf and time are not supported) - Zopti(r'results.txt', metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli', 'time'], showstats=True, matrix='709')

Very useful. Thanks for that.

Now I don't like the forced heatmap in butteraugli :/

Me neither.

[zorr]

https://github.com/theChaosCoder/zoptilib
New parameter: showstats - show metric stats per frame

Great addition. It would look cleaner if Butteraugli stats were lined with the others. So can we shorten it to... butt?

[ChaosKing]

done :)

[ChaosKing]

https://github.com/theChaosCoder/zoptilib
Mdsi and butteraugli no longer contaminate the clip. That means metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] is the same as metrics=['mdsi', 'butteraugli', 'ssim', 'gmsd'] now.
This also means that you won't see the heatmap...

[Iron_Mike]

https://github.com/theChaosCoder/zoptilib
Mdsi and butteraugli no longer contaminate the clip. That means metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] is the same as metrics=['mdsi', 'butteraugli', 'ssim', 'gmsd'] now.
This also means that you won't see the heatmap...

could you elaborate on that ?

is this in regards to the RGB conversion for MDSI/Butteraugly ?

Also, running Butteraugly via Zopti takes quite a while on my system... way longer than VMAF, GMSD, SSIm etc... is this normal ?

Any settings to speed this up ?

Thanks.

[ChaosKing]

Yes, it doesn't pass the rgb converted clip but instead it copies the clip props to the old clip now.
Butteraugli is sloooow, can't do much about it.

[Iron_Mike]

Yes, it doesn't pass the rgb converted clip but instead it copies the clip props to the old clip now.
Butteraugli is sloooow, can't do much about it.

okay, so if I run metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] in the same Zopti call, it will convert the clips to RGB for MDSI/Butter but will do GMSD/SSIM calc in YUV ?

[ChaosKing]

okay, so if I run metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] in the same Zopti call, it will convert the clips to RGB for MDSI/Butter but will do GMSD/SSIM calc in YUV ?

Yes and yes and the "auto-rgb-convert" is still present for mdsi and butter.

[Iron_Mike]

does SSIM/GMSD via Zopti/muvsfunc upscale the chroma planes to the src resolution of the clip if we specifiy planes=1|2 (e..g is src is 444 but dist is 420) ?

[WorBry]

No it does not. That was the whole point of my last post in the 'SSIM/GMSD metrics thread':

https://forum.doom9.org/showthread.php?p=1870863#post1870863


You posted the same question there and before I could answer, deleted it and are now asking it here. What gives ?

No resize or weights are applied to the 'processed-as-luma' chroma planes.

e..g is src is 444 but dist is 420) ?

That combination would throw an error irrespective of the plane being processed.

"clip1" and "clip2" must be of the same format!

[Iron_Mike]

No it does not. That was the whole point of my last post in the 'SSIM/GMSD metrics thread':

https://forum.doom9.org/showthread.php?p=1870863#post1870863

You posted the same question there and before I could answer, deleted it and are now asking it here. What gives ?

what gives ?

well, Brian... take a really hard guess why I opted to move this to the ZOPTI thread instead of leaving it in the GMSD thread.... ? considering we're using Zopti as a wrapper... :cool:


in any case - back to the point - so in case of GMSD/SSIM scoring planes 1/2 of a 444 src vs 420 dist, is Zopti/muvsfunc downscaling the src to the dist resolution or does the GMSD/SSIM metric not require both clips to be in the same resolution ?

[WorBry]

I repeat...


That combination would throw an error irrespective of the plane being processed.

"clip1" and "clip2" must be of the same format!

[Iron_Mike]

I repeat...

yeah, that's why I ask here in the Zopti thread...

I just ran a 444 ref against a 420 dist, scored GMSD/SSIM in planes 1|2 and it did not throw an error (meaning: chroma resolution of src/dist was NOT the same)... also (as far as I understand it), vs.format does not include src/Y resolution...

so, question remains...

[ChaosKing]

yeah, that's why I ask here in the Zopti thread...

I just ran a 444 ref against a 420 dist, scored GMSD/SSIM in planes 1|2 and it did not throw an error (meaning: chroma resolution of src/dist was NOT the same)... also (as far as I understand it), vs.format does not include src/Y resolution...

so, question remains...

Could you provide your script? I always get "clip1" and "clip2" must be of the same format!

My test

#c is 420
a = mvf.GetPlane(c, 1)
c = core.fmtc.resample(c, css="444")
b = mvf.GetPlane(c, 1)#.fmtc.resample(css="444")

c = muf.SSIM(a,b)

[WorBry]

Could you provide your script? I always get "clip1" and "clip2" must be of the same format!


Quite !

[Iron_Mike]

Could you provide your script? I always get "clip1" and "clip2" must be of the same format!

my bad, u're right - I have a block of code there for SSIM/GMSD (via Zopti) that auto-converts dist to src format in case format differs - to avoid that exact error... ;-)

so what are opinions about how to weigh the GMSD/SSIM results if the css of the dist differs ?

[WorBry]

so what are opinions about how to weigh the GMSD/SSIM results if the css of the dist differs ?

You don't. You either convert the css of the 'distorted' clip to that of the reference or vice versa. They must be the same format - period. It's the fundamental requirement of Full Reference Image Quality Assessment. You seem to be laboring under this notion that it should somehow be different. 'Full Reference' - think about it.

[Iron_Mike]

You don't. You either convert the css of the 'distorted' clip to that of the reference or vice versa. They must have the same format - period. It's the fundamental requirement of Full Reference Image Quality Assessment. You seem to be laboring under this notion that it should somehow be different.

well Brian, a "fundamental requirement" is to read posts properly... or to put 1+1 together :D :eek:

I was referring to weighing the chroma plane results (if dist had different css than ref) when combining them with the Y plane result for a total GMSD/SSIM score based on all three planes...............

in addition... chroma plane results can also be (optionally) weighed (in a combined score) even if both ref/dist have the exact same css considering the fact that color drift has less perceived impact in human vision...


whatever it is that u're mumbling in regards to "format" has nothing to do with any of it... just because u're matching a format so u can run the metric does not mean the data is (or should be considered) equal........

[WorBry]

Someone else deal with this please. I'm done with this nonsense.

Also Iron_Mike, please desist in referring to me by anything other than my forum user name. I don't know what you are trying to convey, but it's rather creepy and smacks of trollism.

[ChaosKing]

well Brian, a "fundamental requirement" is to read posts properly... or to put 1+1 together :D :eek:

I was referring to weighing the chroma plane results (if dist had different css than ref) when combining them with the Y plane result for a total GMSD/SSIM score based on all three planes...............



But to compare different css you have to resize the chroma plane to match the same clip format for ssim/gmsd. It's similar to a rgb and yuv clip, one has to be converted wich alters your final score a little bit.

hmm a score based on all three planes... For that I tried to see how the video looks like if you extremely blur the chroma plane. Surprisingly only the colors are a bit off.
I guess the final score could be as simple as Plane1 80% + P2 10% + P3 10% ... I don't know what a good weighted total score should look like.

c2_1 = mvf.GetPlane(c2, 1).std.BoxBlur(hradius=5, vradius=5)
c2_2 = mvf.GetPlane(c2, 2).std.BoxBlur(hradius=5, vradius=5)
c2 = core.std.ShufflePlanes(clips=[mvf.GetPlane(c2, 0), c2_1, c2_2], planes=[0, 0, 0], colorfamily=vs.YUV)

a = mvf.GetPlane(c, 1)
b = mvf.GetPlane(c2, 1)
#a=c
#b=c2
c = muf.SSIM(b,a).text.FrameProps()
c.set_output()

p.s the ssim score for the blurred chroma plane is 0.8

https://i.imgur.com/TEgxbwD.jpg
https://i.imgur.com/MCDuF5f.png

[WorBry]

I don't know what a good weighted total score should look like.

I'm not sure. As mentioned in the other 'SSIM: GMSD metric thread', Zorr looked at the code for the AVISynth and FFMPEG SSIM metrics to see how they derive an overall score for the Y, U and V channels:

Zorr checked out the code for ffmpeg SSIM (AVISynth SSIM also) earlier in the thread. In essence, it's akin to the 'fast' SSIM metric in the MSU Quality Tool:

https://forum.doom9.org/showthread.php?p=1866030#post1866030

https://forum.doom9.org/showthread.php?p=1866162#post1866162



The Avisynth version is not doing the gaussian kernel at all - it's implemented using summed area tables. That's a faster but lower quality way to calculate it, the MSU Quality measurement tool page (http://www.compression.ru/video/quality_measure/info.html#ssim) has an example of the difference.

.....Avisynth SSIM has a plane argument which defaults to 0 and then it returns a weighted sum of the luma and chroma channels:
(0.8 * Y) + (0.1*(U+V))


But the AVISynth version is YV12 only.

I looked at the ffmpeg source. It's doing the fast version too - no gaussian kernels there. The total SSIM takes into account all the planes but the weighting is different, each plane is scaled by the resolution is has. So for example with YUV420 the color planes have 4 times smaller weight. The constants 0.01 and 0.03 appear in the code and I assume that means the weights k1 and k2 are the same as in other implementations.


And:

To improve speed, this implementation uses the standard approximation of overlapped 8x8 block sums, rather than the original gaussian weights

https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_ssim.c

So the ffmpeg variant applies different weights according to pixel format of the clips being tested, but they must have the same pixel format and resolution

https://ffmpeg.org/ffmpeg-filters.html#ssim

Of course the FFMPEG metric can also be run on RGB clips, generating scores for the individual R,G,B channels and an overall aggregate score. But it's still using the 'fast' and less precise 'standard approximation of overlapped 8x8 block sums'.

Is there provision in the original Matlab SSIM code for measuring and weighting the chroma or some other reference where this is defined - I don't know ?

I checked out the demo 'Pro' version of the MSU-QMT, btw - the demo version intentionally generates incorrect results as a kind of 'metric watermark' but it shows the options that are available for the different measures. Looks like the 'precise' (gauss blur) and 'fast' SSIM variants are measuring Y luma only.

And then there are these other 'SSIM algorithm-based' variants - DSSIM and SSIMULACRA - intended (like Butteraugli) primarily for measuring differences in still images, which process in CIE Lab* color space and apply weights to the L, a and b channels to derive an overall score:

https://github.com/kornelski/dssim

https://github.com/cloudinary/ssimulacra

https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra

Personally, I'd be more interested in deriving a composite SSIM score for the combined chroma channels, rather than (or in addition to) an 'overall' score, but that would also require consideration of the weightings.

[VS_Fan]

... Personally, I'd be more interested in deriving a composite SSIM score for the combined chroma channels, rather than (or in addition to) an 'overall' scoreHow about stacking (horizontally or vertically) the u and v planes in vapoursynth, then generate YUV/Y4M files so you can obtain SSIM metrics by using any of the tools mentioned?

I would expect the result to be very close to averaging the results of the individual u and v planes.

[WorBry]

Now that's a novel idea. I was thinking of maybe merging the luma-converted U and V planes.

[VS_Fan]

With the same general idea, one could obtain an “an 'overall' score” by stacking all the planes:


ret = some_video_source
#This should work to preserve vertical resolution, and stacking all Y,U and V, no matter the css (chroma subsampling: 411, 420, 422, 444)
retc = core.std.StackHorizontal(clips=[retU,retV]) if ret.format.subsampling_h == 0 else core.std.StackVertical(clips=[retU,retV])
ret = core.std.StackHorizontal(clips=[retY,retc])


I would also expect this ‘overall’ result to be close to that of applying the weights as you exposed they do in ffmpeg: “each plane is scaled by the resolution it has”, and I guess they are something like:


sub- Y U V
sampling ratio ratio ratio

4:1:1 2/3 1/6 1/6
4:2:0 2/3 1/6 1/6
4:2:2 1/2 1/4 1/4
4:4:4 1/3 1/3 1/3

[Iron_Mike]

@WorBry: I'm glad u finally woke up and entertain the idea of "weights" - a concept used in scientific formulas and equations since the dawn of time, and that includes the field of color science.

This is good, this could indicate progress !

Maybe next time before u have ur temper tantrum, u can put the bottle aside, and read posts properly. Zorr did directly mention weights to u in regards to ffmpeg's implementation but u seem to have chosen to ignore that... until u remembered.

well, to quote u: what gives ? :cool:


I don't know what a good weighted total score should look like.


yeah, hence me throwing it out there and asking for opinions... ;-)

[Iron_Mike]

sub- Y U V
sampling ratio ratio ratio

4:1:1 2/3 1/6 1/6
4:2:0 2/3 1/6 1/6
4:2:2 1/2 1/4 1/4
4:4:4 1/3 1/3 1/3

good stuff getting the ball rolling...

this was my pretty much my first approach.

so what about the idea - in case of src/ref and dist/enc having different css - of flipping that table and the more css the distorted clip has (compared to src/ref), to give chroma score more weight than luma ?

that would bring the total score down, but with a 420 dist clip, the chroma drift/delta will have a larger perceived impact than with a 444 dist clip...

for example:
source is 444 and the dist is 420... we could consider giving the chroma score more weight than luma, b/c that color drift will have a larger impact


for WorBry: no, just b/c u put the 420 dist in a 444 clip to run the metric does not make the 420 data inside that new clip a "444" encode... it's resampled 420 data

[poisondeathray]

I think the dist. should always be converted to the ref. pixel format

The metric should take into account the results of resizing the data - that's the whole idea of metrics measuring the effect of chroma subsampling . It should be a measureable phenomenon (and it is with standard metrics)

eg
a) 444 source reference
b) a is converted to "b" 4:2:0 by algorithm "z" (this means the U,V planes are resized by 1/2 width, 1/2 height)

You need to convert b back to 444 to compare against "a" . ie. resize back the U,V planes *2 width/height .

- you expect 422 to do better than 420 - and you do with ssim, psnr
- you expect to get slightly different results with different resize algorithms (e.g. sharper isn't always "better" ; sharper actually tends to get "penalized" by some metrics like ssim/psnr)
- serial conversions get blurrier colors, and worse results, as expected (because really you're up/down scaling the chroma planes)

Or if you're starting with a 420 reference ; and you're testing against something at got converted to 444 . That up/down conversion should be reflected in the scores as a larger deviation from the original reference.



Don't mix in "weights" ; it's a separate concept dealing with human perception of color vs. black/white . Your perception doesn't change , based on different pixel format of videos - it's the source that is changing . Weighting should be the same whatever you decide the formula is . You don't become less sensitive to color with 444 vs 420 . The effect of chroma subsampling should be measured and reflected by the metric

[Iron_Mike]

I think the dist. should always be converted to the ref. pixel format

The metric should take into account the results of resizing the data - that's the whole idea of metrics measuring the effect of chroma subsampling . It should be a measureable phenomenon (and it is with standard metrics)

eg
a) 444 source reference
b) a is converted to "b" 4:2:0 by algorithm "z" (this means the U,V planes are resized by 1/2 width, 1/2 height)

You need to convert b back to 444 to compare against "a" . ie. resize back the U,V planes *2 width/height .

- you expect 422 to do better than 420 - and you do with ssim, psnr
- you expect to get slightly different results with different resize algorithms (e.g. sharper isn't always "better" ; sharper actually tends to get "penalized" by some metrics like ssim/psnr)
- serial conversions get blurrier colors, and worse results, as expected (because really you're up/down scaling the chroma planes)

Or if you're starting with a 420 reference ; and you're testing against something at got converted to 444 . That up/down conversion should be reflected in the scores as a larger deviation from the original reference.

Don't mix in "weights" ; it's a separate concept dealing with human perception of color vs. black/white . Your perception doesn't change , based on different pixel format of videos - it's the source that is changing . Weighting should be the same whatever you decide the formula is . You don't become less sensitive to color with 444 vs 420 . The effect of chroma subsampling should be measured and reflected by the metric

@pd: all agreed, but we were elaborating combining scores of various planes into a combined metric score for all planes... and in a 420 dist (from a 444 ref), the chroma gets a different treatment than the luma in the encoding process...

so when u combine metric score for luma and chroma planes, one could consider that... or are u saying the chroma metric scores already state that and u would just pull the avg ?

Btw, weights weren't mixed in (in my posts) - weights are constants - luma vs. chroma perception is one, but u can have additional weights for other effects... for example, the luma vs. chroma weight also only kicks in if you're actually dealing w/ color footage (so it is a conditional factor)...

[WorBry]

@WorBry: I'm glad u finally woke up and entertain the idea of "weights" - a concept used in scientific formulas and equations since the dawn of time, and that includes the field of color science.

This is good, this could indicate progress !

Maybe next time before u have ur temper tantrum, u can put the bottle aside, and read posts properly. Zorr did directly mention weights to u in regards to ffmpeg's implementation but u seem to have chosen to ignore that... until u remembered.

well, to quote u: what gives ? :cool:

yeah, hence me throwing it out there and asking for opinions... ;-)

I have never not had an 'idea of weights'. That's what my last series of tests in the 'SSIM and GMSD metrics' thread was about - to see if any weighting or resizing is being applied to the luma-converted U and V planes in the muvsfunc SSIM and GMSD implementations:

https://forum.doom9.org/showthread.php?p=1870863#post1870863

And your original question about 'weights' here was:


...so what are opinions about how to weigh the GMSD/SSIM results if the css of the dist differs ?

Which both I and ChaosKing answered:

You don't. You either convert the css of the 'distorted' clip to that of the reference or vice versa. They must be the same format - period. It's the fundamental requirement of Full Reference Image Quality Assessment. You seem to be laboring under this notion that it should somehow be different. 'Full Reference' - think about it.

(You still seem to have that notion)

But to compare different css you have to resize the chroma plane to match the same clip format for ssim/gmsd. It's similar to a rgb and yuv clip, one has to be converted wich alters your final score a little bit.


Now enough of your insults and overt gas-lighting.

@VS_Fan and Poisondeathray. Thanks. Alot to think about there.

[poisondeathray]

@pd: all agreed, but we were elaborating combining scores of various planes into a combined metric score for all planes... and in a 420 dist (from a 444 ref), the chroma gets a different treatment than the luma in the encoding process...

so when u combine metric score for luma and chroma planes, one could consider that... or are u saying the chroma metric scores already state that and u would just pull the avg ?

Btw, weights weren't mixed in (in my posts) - weights are constants - luma vs. chroma perception is one, but u can have additional weights for other effects... for example, the luma vs. chroma weight also only kicks in if you're actually dealing w/ color footage (so it is a conditional factor)...




The idea of "weights" for a combined Y/U/V aggregate metric score, should reflect that the Y plane should recieve a proportionally higher weighting due to human perceptions - I think everyone will agree on the general idea, but might disagree on the actual formula for the weighting

All I'm saying is the relative weighting shouldn't change because of subsampling . Your perception of the proportion of black/white vs. color importance doesn't suddenly change if you watch a 4:4:4 video or a 4:2:0 video.

That lower quality of 4:2:0 should already be reflected in the lower U, V scores for 4:2:0 . That up/down converison is the "penalty" already incurred . The "different treatment" of chroma is exactly what you're measuring in the first place when you measure U-SSIM, V-SSIM, or U-PSNR, V-PSNR or whatever metric

Yes you can have other weights, other categories, combine /mix/match in any way you want , analyze it in whatever way you want, call it whatever you want ; but eitherway you're measureing Y,U,V separately - so you should have the "raw" scores

[Iron_Mike]

The idea of "weights" for a combined Y/U/V aggregate metric score, should reflect that the Y plane should recieve a proportionally higher weighting due to human perceptions - I think everyone will agree on the general idea, but might disagree on the actual formula for the weighting

All I'm saying is the relative weighting shouldn't change because of subsampling . Your perception of the proportion of black/white vs. color importance doesn't suddenly change if you watch a 4:4:4 video or a 4:2:0 video.

That lower quality of 4:2:0 should already be reflected in the lower U, V scores for 4:2:0 . That up/down converison is the "penalty" already incurred . The "different treatment" of chroma is exactly what you're measuring in the first place when you measure U-SSIM, V-SSIM, or U-PSNR, V-PSNR or whatever metric

Yes you can have other weights, other categories, combine /mix/match in any way you want , analyze it in whatever way you want, call it whatever you want ; but eitherway you're measureing Y,U,V separately - so you should have the "raw" scores

@pd: yup, all agreed.

now stop dancing around the subject, what would be your weights/formula for a combined metric score of all planes ? :D

[poisondeathray]

I don't think there is a commonly accepted formula/value for all planes. There are no scientific studies that conclusively measure this value to be "x/y/z"

Just pick one and as long as you state your assumptions, criteria, and reasoning- it's ok :) . e.g if you decide 0.8/0.1/0.1 , say so

What's not ok - is if you post results, but no details - ie. lack of transparency .

e.g. if you measure a black/white movie - you might still decide to look at U/V . There are old movies that have discoloration and chroma noise / rainbows due to the transfer process for example .

Everyone knows (or should know) , that all metrics have various pros/cons and are context dependent. But the trends can still be useful to look at

[Wilbert]

@Iron_Mike, refrain your language. There is no need to be disrespectful to others.

[Boulder]

I have a small feature request regarding cases where colorspace conversion to RGB is required. It would be nice if Zopti as an option could do the conversion just once and save the resulting file as an intermediate file to use in all the subsequent iterations. I don't know how much time consuming it actually is, but I'd expect that the number of iterations is usually so big that over a long test run, it would same a nice amount of time.

[zorr]

I have a small feature request regarding cases where colorspace conversion to RGB is required. It would be nice if Zopti as an option could do the conversion just once and save the resulting file as an intermediate file to use in all the subsequent iterations.

That's a bit tricky, currently Zopti doesn't know what the input file is so it's unable to change it. I think it could save the converted file using clip.output() (someone can correct me here if I'm wrong) and then read that one instead, but I'm not sure if it's any faster if the file is in uncompressed format which the output() supports. Does someone have experience on using output() and reading the resulting file?

[ChaosKing]

Btw Asd-g ported some metrics like MDSI, GMSD or vsSSIM to avs https://github.com/Asd-g/AviSynthPlus-Scripts

[Boulder]

I was trying to run Zopti with these parameters (rest of the script is not there):

zopti = Zopti(r'results.txt', metrics=['mdsi'], matrix='709')

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

And I get this error when trying to evaluate the results. The process itself goes fine. If I add 'time' to the metrics list, it works.
java.lang.NumberFormatException: For input string: "b=-75"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at sun.misc.FloatingDecimal.parseFloat(Unknown Source)
at java.lang.Float.parseFloat(Unknown Source)
at java.lang.Float.<init>(Unknown Source)
at avisynthoptimizer.Result.parseValues(Result.java:93)
at avisynthoptimizer.Result.<init>(Result.java:240)
at avisynthoptimizer.LogFile.read(LogFile.java:276)
at avisynthoptimizer.AviSynthOptimizer.evaluateResults(AviSynthOptimizer.java:3564)
at avisynthoptimizer.AviSynthOptimizer.main(AviSynthOptimizer.java:1872)
ERROR: For input string: "b=-75"

The reason why I wanted to skip measuring the time is that I don't want the time to reset "the counter" after which it's clear that the result doesn't get any better. I'm only looking for the best value for the actual measurement.

This is the command line I've used:
for %f in (*.vpy) do zopti "%f" -alg mutation -iters dyn -dyniters 15 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

[zorr]

I was trying to run Zopti with these parameters (rest of the script is not there):

zopti = Zopti(r'results.txt', metrics=['mdsi'], matrix='709')

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

And I get this error when trying to evaluate the results.

I tried to reproduce this but it's working on my end. I think there was a version which didn't handle the missing time measurement but I believe it was fixed at some point. However, the version I tested with was the latest which I haven't released so there's a small chance that it's only fixed in my version. Which version of Zopti are you running?

[Boulder]

I tried to reproduce this but it's working on my end. I think there was a version which didn't handle the missing time measurement but I believe it was fixed at some point. However, the version I tested with was the latest which I haven't released so there's a small chance that it's only fixed in my version. Which version of Zopti are you running?

Mine is version 1.0-beta, should be the latest released one.

[zorr]

Mine is version 1.0-beta, should be the latest released one.

The latest is 1.0.1-beta, link is available in the first post. :)

[Boulder]

The latest is 1.0.1-beta, link is available in the first post. :)

That's what I have, it looks like the application just reports 1.0-beta. I verified that the files in the latest archive are the same as what I have in my Zopti folder.

[zorr]

That's what I have, it looks like the application just reports 1.0-beta. I verified that the files in the latest archive are the same as what I have in my Zopti folder.

All right, can you provide a few lines from your log file? Does it look something like this (I used SSIM here)?

# output out1="SSIM: MAX(float)" file="results.txt"
7.86225 b=0 c=0
7.862938 b=1 c=1
7.854728 b=1 c=0 # phase=0,00
7.889888 b=-4 c=1
7.8532143 b=-18 c=1
7.8917427 b=-4 c=6
7.8901386 b=-4 c=7

[Boulder]

Yes, looks similar:
# output out1="MDSI: MIN(float)" file="results.txt"
30.7859 b=-60 c=30
31.428144 b=-47 c=12
30.756887 b=-63 c=30 # phase=0,00
30.797626 b=-59 c=30

[WorBry]

Btw Asd-g ported some metrics like MDSI, GMSD or vsSSIM to avs https://github.com/Asd-g/AviSynthPlus-Scripts

Didn't see this before. Excellent. Love the color-enhanced maps. How would one print a log file that compiles the per-frame test results in this case?

[zorr]

How would one print a log file that compiles the per-frame test results in this case?

Looks like there is no way to read the similarity metric back into a script variable which is how the current SSIM metric is handled. Perhaps this could be implemented by passing a (global?) variable name and GMSD / MDSI functions would assign that variable?

[WolframRhodium]

Looks like there is no way to read the similarity metric back into a script variable which is how the current SSIM metric is handled. Perhaps this could be implemented by passing a (global?) variable name and GMSD / MDSI functions would assign that variable?

It is possible to write the data to a file via FrameEval or get_frame.

For example,

def output_frameprop(
clip: vs.VideoNode, filename: str, prop_name: str
) -> vs.VideoNode:

records = []
frames = clip.num_frames

def output(n, f, clip=clip):
records.append((n, f.props[prop_name]))

nonlocal frames
frames -= 1

if frames == 0:
with open(filename, "w") as f:
f.write(f"frameIdx {prop_name}\n")
f.writelines(
f"{idx:8} {value:{len(prop_name)}}\n"
for idx, value in sorted(records))

return clip

return core.std.FrameEval(clip, output, clip)


clip = calculate_some_metric(clip)
clip = output_frameprop(clip, "stats.txt", metric_name)


And evaluate the script from beginning to end via vspipe / benchmark utility from vsedit. The result is then in file "stats.txt".

[ChaosKing]

@WolframRhodium I think he means the avisynth version. For VS we already have this scrpt here https://github.com/theChaosCoder/zoptilib/blob/master/zoptilib.py

[WorBry]

Yes, I mean the ported AVISynth version. It appears that Asd-g is not a forum member.

[WolframRhodium]

Sorry about that, I did not read the thread carefully. I thint it is possible through WriteFile, but I am not very familiar with it.

[ChaosKing]

Looks like there is no way to read the similarity metric back into a script variable which is how the current SSIM metric is handled. Perhaps this could be implemented by passing a (global?) variable name and GMSD / MDSI functions would assign that variable?

I think it should work like in Vapoursynth now.

https://github.com/Asd-g/AviSynthPlus-Scripts/blob/6604ddaadb84697c0c718a9fefd085e7e60c46a2/MDSI.avsi#L103

"""propSet("_FrameMDSI",

Avisynth+ 3.X supports clip props just like vapoursynth.

They are read for show=true at the script bottom ScriptClip("""Subtitle("FrameMDSI: " + String(propGetFloat("_FrameMDSI")))"""

So maybe zopti needs a "zoptilib.py" version for avisynth now?

[zorr]

Thanks Chaos, I didn't notice that the metric can indeed be read from the frame properties. So here's an example script using GMSD:

TEST_FRAMES = 5 # how many frames are tested
MIDDLE_FRAME = 50 # middle frame number

RawSourcePlus("D:/optimizer/test/flower/flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)
#return noisy

#denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

sigma = 400/100.0 # optimize sigma = _n_/100.0 | 100..800 | sigma
bt = 4 # optimize bt = _n_ | -1..5 | blockTemporal
blockSize = 32 # optimize blockSize = _n_ | 2..64 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 0..32 ; max:blockSize 2 / | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

last = GMSD(source, denoised)

global total = 0.0
global gmsd_total = 0.0
FrameEvaluate(last, """
global gmsd = propGetFloat("_PlaneGMSD")
global gmsd = (gmsd == 0.0 ? 1.0 : gmsd)
global gmsd_total = gmsd_total + gmsd
""")

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (gmsd, time)
delimiter = "; "
resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "gmsd", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "gmsd_total", append=true)

return last

This is just a slightly modified version of the script I used in the hands-on tutorial (https://forum.doom9.org/showthread.php?p=1851670#post1851670). Other than renaming the variables the only relevant changes are highlighted in red.

In this script GMSD gave indentical optimal values compared to SSIM except for the sigma which is 415 instead of 479. I didn't try MDSI yet.

[Boulder]

Is there a way to utilize Avisynth+'s Prefetch to speed up calculation? I tested it briefly but there were problems probably related to writing to the file asynchronously.

[zorr]

Is there a way to utilize Avisynth+'s Prefetch to speed up calculation? I tested it briefly but there were problems probably related to writing to the file asynchronously.

I haven't tried it myself but it's worth an investigation. I think there's going to be problems if multiple threads are writing to the same file.

Is it possible to force WriteFile to execute on one thread while the filters before it are multithreaded? The documentation states that you cannot set MT mode on script function calls, but would placing a MT_SERIALIZED plugin call just before WriteFile have that side effect? Or can you just say SetFilterMTMode("WriteFile", MT_SERIALIZED)?

On the other hand, perhaps the WriteFile is already MT_SERIALIZED but the threads are writing into it in arbitrary order and the last line is not always last as it should be. In that case the per frame results should be written into a temporary thread-safe variable which would be written to file in correct order when all the frames are finished. It works like that in Vapoursynth using a map, but Avisynth doesn't have such a data structure so I'm not sure how to implement that. Perhaps a simple array would suffice as long as it's thread safe.

[Boulder]

I'm not sure if WriteFile is internally marked as MT_SERIALIZED, but in general, it should already have a "correct" mode set. It is very much likely that the order of frames is very random since they finish at different times.

Setting both Prefetch(threads=1, frames=24) and Prefetch(threads=24, frames=1) give this error: java.lang.NumberFormatException: For input string: "1I don't know what 'delimiter' means.0.179144I don't know what 'delimiter' means.3150.284912" and the log file is full of these errors.

[zorr]

Setting both Prefetch(threads=1, frames=24) and Prefetch(threads=24, frames=1) give this error: java.lang.NumberFormatException: For input string: "1I don't know what 'delimiter' means.0.179144I don't know what 'delimiter' means.3150.284912" and the log file is full of these errors.

I got the same result, but I was able to make it work by defining both delimiter and frame_count as global. I'm still not sure if the result file will always be correct but it seems to work. However I didn't seem to gain any performance with the prefetch. Perhaps the serialized file writing escalates upstream and makes everything run serially.

[Boulder]

I got the same result, but I was able to make it work by defining both delimiter and frame_count as global. I'm still not sure if the result file will always be correct but it seems to work. However I didn't seem to gain any performance with the prefetch. Perhaps the serialized file writing escalates upstream and makes everything run serially.

Yes, I think that's how it goes. Perhaps the only way to make it faster and threadsafe in Avisynth would be to use an array as you suggested. Or is it possible to make all the frames create their own log file and combine them when it's time to calculate the total score?

The VS version is also not perfect in multithreading sense but it is somewhat faster.

[zorr]

A new major release is here! Zopti now supports executing multiple scripts simultaneously which can lead to very nice performance improvement. This can be very useful when the script itself is not able to utilize all of your cores.

Here's the full release info:


added support for multithreaded execution of scripts with argument -threads
-default value of -threads is 1 meaning one script is executed at a time just like before
-all optimization algorithms (nsga-ii, spea2, mutation, exhaustive) support the the -threads option
-due to the nature of the heuristic algorithms (nsga-ii, spea2, mutation) there can be at most P threads running the scripts at the same time, where P is the population size
-exhaustive algorithm has no thread limits
-validate mode also supports -threads. there the maximum number of threads is the number of validated results (size of the pareto front)
-note: using more than one thread will make time measurements less accurate but will not change the quality measurements
avsr version update (v0.2.2)
added jMetal.log.ini to disable multithreading related JMetal logging
updated Zoptilib to version 1.0.9m
bugfix: evaluate could not parse log files when Zopti was used


Download link is at the first post. I will shortly post some data on the multithreading performance.

[zorr]

I tested the new -threads argument using the denoising tutorial script, this time using GMSD as the similarity metric and using 50 frames instead of 5. Here's the script:


import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 50 # how many frames are tested
MIDDLE_FRAME = 100 # middle frame number

# read input video
video = core.raws.Source(r'D:\optimizer\test\flower\flower_cif.yuv', 352, 288, src_fmt='I420')

zopti = Zopti(r'results.txt', metrics=['gmsd', 'time'])

# input color range is PC (full)
video = video.std.SetFrameProp(prop="_ColorRange", intval=0)

# add noise
noisy = video.grain.Add(var=25, seed=3)
#noisy.set_output()

# remove noise
sigma = 400/100 # optimize sigma = _n_/100 | 400..600 ; filter:x 5 % 0 == | sigma
bt = 5 # optimize bt = _n_ | 3,5 | blockTemporal
blockSize = 14 # optimize blockSize = _n_ | 6,12 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 4..6 ; max:blockSize 2 / | overlap
denoised = noisy.fft3dfilter.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
video = video[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
denoised = denoised[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(video, denoised)

There's also an AviSynth version of the script which I posted to the Avisynth Zopti thread (https://forum.doom9.org/showthread.php?p=1940863#post1940863).

The script only has 246 valid parameter combinations to test so we can try all of them using
zopti denoise_ex.vpy -alg exhaustive -threads 1

It takes 140 seconds to test all of the combinations using one thread. Let's see the performance when using more threads:

https://i.postimg.cc/QdZrh7St/zopti-performance-scaling.png

The best result is about 20 seconds using all 24 threads (on a Ryzen 3900X which has 12 cores / 24 threads).

Another way to look at the scaling is to calculate how much faster we get the results when using -threads:

https://i.postimg.cc/RFDj8JNv/zopti-performance-factor.png

Using 24 threads is 7,1 times as fast as using only one thread. Of course this is just one data point and I don't mean to imply that you can always get such a performance improvement. I would have liked to include more tests using more real-world usage scenarios (HD source etc) but my stock cooled processor cannot handle those for more than a few seconds, it becomes so hot that my PC shuts down. :eek:

Comparing AviSynth and VapourSynth it looks like VS has the edge when using one or just a few threads, but the differences almost vanish when using 12 or more threads. The VS is also able to utilize more than 16 cores while with AVS the performance starts to degrade. The point where more threads are just slowing down is probably dependent on the specific script, I will have to run more tests on that.

[Boulder]

With the new multithreading support, is it beneficial to increase the population size even with this kind of approach to optimizing the resizer params? A word of warning: VS uses a lot of memory for caching by default so it may be wise to lower the max_cache_size if you launch multiple instances :)

zopti "test.vpy" -alg mutation -iters dyn -dyniters 18 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

[zorr]

With the new multithreading support, is it beneficial to increase the population size even with this kind of approach to optimizing the resizer params?

zopti "test.vpy" -alg mutation -iters dyn -dyniters 18 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

In this case you have to increase the population size from 1, otherwise there's no use for more than one thread. It will for sure make the search faster but the quality (or the probability that you will find the optimal values) might go down a little bit. You can compensate for that with increasing -dyniters slightly. The net result should be as good (or better) results faster. You may have to do some testing to find how many threads give you the best bang for buck though.

A word of warning: VS uses a lot of memory for caching by default so it may be wise to lower the max_cache_size if you launch multiple instances :)

Yes that's one downside of using -threads. It's process level multithreading and every VSPipe process will use as much memory as a single one would. So the amount of free memory might limit how many threads you can actually use even though CPU is still not maxed out.

[zorr]

Time for a new update! This version has some major additions so I bumped the version to 1.2.0.


new option: -timeout (default is disabled). if the execution of Avisynth script takes longer than timeout seconds
-the script execution process (and subprocesses) will be terminated
-the result is set to worst possible as a penalty so it will not be considered as a valid candidate
-useful with scripts where certain parameter combinations can result in execution times several hundred times slower than usual
-NOTE: only implemented for AviSynth scripts (for now)
-example: zopti script.avs -pop 24 -iters 1000 -timeout 60
new option: -continue
-aborted optimization can be continued from the last complete generation (one generation is N iterations, where N is the population size)
-give the log file name of the aborted optimization as the value of continue
-old results up to last complete generation are copied to a new log file and optimization will continue from there
-useful if optimization was unexpectedly aborted or the CPU resources are needed for something else for a while
-example: zopti script.avs -pop 24 -iters 10000 -continue "mvtools spea2 run-01.log"
-NOTE: it is possible to change the optimization parameters (population, mutation count etc.) from those used in the continued log
-NOTE: random seed is not restored so the results will not be identical to what they would have been if execution had not been aborted
-NOTE: only works with algorithm spea2 (for now)
new visualization mode: history
-displays the best found value and all tried values of certain optimized parameter (given with -param)
-invalid results (script execution has failed OR timed out) are highlighted in orange
-also shows the phase of dynamic iteration (if applicable)
-accepts parameter -range to limit the displayed history to certain iteration range
-example: zopti -mode evaluate -vismode history -param lambda (displays all history of parameter "lambda")
-example: zopti -mode evaluate -vismode history -param lambda -range 1000 5000 (displays history from iteration 1000 to iteration 5000)
-example: zopti -mode evaluate -vismode history -param lambda -range -10000 (displays history of the last 10000 iterations)
argument -dynphases can now be zero (previously minimum was 1)
pareto front and total runtime is displayed also after mutation or exhaustive algorithm is finished
if population is based on a log file and the pareto front is smaller than the population size, the population will be filled with mutations of the pareto front instead of completely random combinations
if a log file for the optimization already exists it will be cleared in the beginning of the optimization
XChart updated to version 3.8.0 with some customization


And here's what -vismode history looks like:

https://i.postimg.cc/nLjnb7Y8/vismode-history.png

The black thick line shows the best found value of the chosen parameter. The red diamonds indicate a point where a new best result has been found and this parameter's value was changed. The white diamonds indicate a new best result but this parameter's value remained the same.

In the example optimization was finished and then restarted with different settings at around 190 000 iterations.

The -timeout was needed when I started running MVTools2 tests with all the parameters and it turned out that some combinations took an enormous amount of time to finish (typical time was 5 seconds but some combinations took over 30 minutes...). Using a timeout of 60 seconds solved that problem.

-continue is useful if you want to keep optimizing a result that is finished, you can try different settings. Also very handy when the optimization has been aborted by hardware failure or something like Windows update... you can just restart it using the same settings. This also makes it possible to run huge iteration counts little by little even if you need/want to use the computer for other things once in a while. :D

Download link updated at first post.

[Boulder]

Thank you for the new version :) It's always nice to see development even though the old product is running well.

One request though: would it be possible to remove the need for keeping the script that was used when running the optimization? I sometimes do many seasons of some TV series sequentially, and reuse the scripts since the properties are usually almost the same between the episodes from various seasons. Zopti doesn't like it when it cannot find the original script as I've already renamed it. I think writing the best result on the first line (to be able to see it right after opening the files in Notepad++ :)) of the log file would do just fine.

[zorr]

Thank you for the new version :) It's always nice to see development even though the old product is running well.

You're welcome. :)

One request though: would it be possible to remove the need for keeping the script that was used when running the optimization?

What it's looking for in the script are the parameters and their valid values. Those are needed (for example) in deciding how to map the values to x,y coordinates in the heat maps.

Zopti doesn't like it when it cannot find the original script as I've already renamed it. I think writing the best result on the first line (to be able to see it right after opening the files in Notepad++ :)) of the log file would do just fine.

If the script still has the same parameters (and value ranges) it doesn't matter which script is loaded. You could change the first line of the log file (starts with # script) to point to some other existing script.

On the other hand if you just run -vismode none then the script should not be needed and the best results could be displayed. I'll have to get back to you on that.

[ChaosKing]

Info: The latest VMAF release added PSNR-HVS and CIEDE2000 metrics.

[zorr]

Info: The latest VMAF release added PSNR-HVS and CIEDE2000 metrics.

Thanks. Could you add them to Zoptilib?
I haven't gotten around to test even wadiqam yet so no hurry. :)

[ChaosKing]

Will do. I think vmaf is a special case with xml log output or something like that? I don't really remember :p

VMAF metrics
0: PSNR
1: PSNR-HVS
2: SSIM
3: MS-SSIM
4: CIEDE2000

[zorr]

Will do. I think vmaf is a special case with xml log output or something like that? I don't really remember :p

Oh yeah. That means I need to add support for the new metrics into Zopti as well.

[Boulder]

I just updated to the new version to run my next optimization, but I get a Java error.

Exception in thread "main" java.lang.UnsupportedClassVersionError: avisynthoptimizer/AviSynthOptimizer has been compiled by a more recent version of the Java Runtime (class file version 53.0),
this version of the Java Runtime only recognizes class file versions up to 52.0

I am currently running the latest JRE version, 1.8.0u291. I checked and there are no old versions hanging around, the path variable also points to the correct place.

[zorr]

I am currently running the latest JRE version, 1.8.0u291.

That's the latest Java 8 version but the latest version is... 16. :eek: You can download one from https://adoptopenjdk.net/ (I personally use these) or from Oracle at https://jdk.java.net/.

Make sure to update your PATH variable to point to the correct version, the installers are not guaranteed to update it automatically anymore (at least that was my experience).

[Boulder]

So much for trusting Java update to do the dirty work :devil:

EDIT: well, Oracle recommends JRE 8 so that's why it's the one that comes up when searching and the one it will keep up-to-date automatically.

[Boulder]

I just noticed that MDSI seems to produce a bit erratic results. I ran the same job 10 times with these wildly varying results:

Run 1 best: 4.1299334 b=-99 c=-19
Run 2 best: 1.27114432E8 b=-73 c=68
Run 3 best: 1.27114432E8 b=-74 c=63
Run 4 best: 1.5116536E8 b=-90 c=37
Run 5 best: 42.133507 b=-78 c=-18
Run 6 best: 1.5116536E8 b=-110 c=28
Run 7 best: 1.27114432E8 b=-126 c=10
Run 8 best: 1.5116536E8 b=-108 c=9
Run 9 best: 1.27114432E8 b=-73 c=68
Run 10 best: 1.5116536E8 b=-101 c=36

The last run shows this at the end -- weird that the scores are the exact same..

No improvement in 30 iterations - stopping
Pareto front:
1.5116536E8 b=-101 c=36
1.5116536E8 b=-99 c=36
1.5116536E8 b=-71 c=36
1.5116536E8 b=-68 c=36

CLI parameters: zopti "strangerthings_s02e01.vpy" -alg mutation -iters dyn -dyniters 30 -dynphases 3 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

from vapoursynth import core
from zoptilib import Zopti

orig = core.ffms2.Source(source=r"c:\zopti\strangerthings_s02e01.avi")

zopti = Zopti(r'results.txt', metrics=['mdsi'], matrix='709')

b = -90/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 45/100.0 # optimize c = _n_/100.0 | -60..80 | c
downscaled_width = 1280
downscaled_height = 640
upscaled_width = orig.width
upscaled_height = orig.height

alternate = core.resize.Bicubic(orig, width=downscaled_width, height=downscaled_height, filter_param_a=b, filter_param_b=c)
alternate = core.resize.Bicubic(alternate, width=upscaled_width, height=upscaled_height, filter_param_a=0, filter_param_b=0.5)
zopti.run(orig, alternate)


Test clip here: https://drive.google.com/file/d/11GtJ1gfT-hStUDl8UmY3-fVJjBDXPD1M/view?usp=sharing

GMSD seems to work fine and the heatmap looks good.

Any ideas what is happening here? I'm using the latest Zopti and this time remembered to update the zoptilib.py in the site-packages folder :o, running on Vapoursynth R55 RC3. I had to fix mvsfunc.py a little bit to make it work, but it shouldn't affect the calculations at all.

[Selur]

What am I doing wrong? :)
It's my first time playing with Zopti and I'm trying to get Zopti running.

What I did so far:

download Zopti-1.2.0 put on my Desktop
download a portable java (OpenJDKJRE64_16.0.1_Build_9.paf) and install it into the Zopti folder
adjust zopti.bat to:
OpenJDKJRE64\bin\java.exe -jar Zopti.jar %*
so that it will find the java binary
adjust zopti.ini to:
log=
architecture=x64
vspipe=I:\Hybrid\64bit\Vapoursynth\VSPipe.exe
to point to VSPipe.exe (using a portable Vapoursynth).
adjusting zoptilib.py, by changing the start to:
# Imports
import os
import sys
import vapoursynth as vs
import time
import functools

# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))

import muvsfunc as muv
so that my scripts folder is known.

created a input.py:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("I:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
core = vs.core

# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")

orig = core.lsmas.LWLibavSource(r"g:\TestClips&Co\test.avi")
alternate = core.grain.Add(clip=orig)

from zoptilib import Zopti
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['ssim', 'time'])
sigma = 4 # optimize sigma = _n_ | 1,10 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)



I then called:
zopti.bat input.py
which gave me:
C:\Users\Selur\Desktop\Zopti-1.2.0>OpenJDKJRE64\bin\java.exe -jar Zopti.jar input.py

+---------+
| |
+---+ | +---------+ +---------+ +---------+ +---+
/ / | | | | | | | |
/ / | +-+ | | +-+ | +--+ +--+ | |
/ / | | | | | | | | | | | |
| +---+ | +-+ | | +-+ | | | | |
| | | | | | | | | |
+---------+ +---------+ | +-----+ +---+ +---+
| |
+---+ version 1.2.0

Arguments

Running optimization for script input.py
Using these settings:
ARGUMENT DESCRIPTION VALUE
-runs runs 5
-alg algorithm spea2
-pop population 8
-initial initial population random
-iters iterations 2000
-mutamount mutation amount 0.3 0.01
-mutcount mutation count 60% 1
-crossprob crossover probability 0.1
-crossdist crossover distribution 20
-sensitivity sensitivity estimation true
-dynphases dynamic phases N/A
-dyniters iterations per phase N/A
-errors error handling stop
-priority process priority (default)
-threads number of threads to use 1
-timeout script execution timeout (seconds) (OFF)
-continue continue from aborted optimization (log file) (OFF)

parsing script output [out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"]
Output 1:
name SSIM
type FLOAT
goal MAX
unit
round 0
Output 2:
name time
type TIME
goal MIN
unit ms
round 10

Found following optimizable parameters:
# optimize sigma = _n_ | 1,10 | sigma
found 1 parameters to optimize

Number of parameter combinations is smaller than the population size - setting the population size to same (2)
Running SPEA2

Error in script execution:



Cannot load avisynth.dll:
Das System kann die angegebene Datei nicht finden.
(no clue why it's looking for avisynth.dll)

Then I called:
C:\Users\Selur\Desktop\Zopti-1.2.0>zopti.bat input.py -initial script
which resulted in:
C:\Users\Selur\Desktop\Zopti-1.2.0>OpenJDKJRE64\bin\java.exe -jar Zopti.jar input.py -initial script

+---------+
| |
+---+ | +---------+ +---------+ +---------+ +---+
/ / | | | | | | | |
/ / | +-+ | | +-+ | +--+ +--+ | |
/ / | | | | | | | | | | | |
| +---+ | +-+ | | +-+ | | | | |
| | | | | | | | | |
+---------+ +---------+ | +-----+ +---+ +---+
| |
+---+ version 1.2.0

Arguments
initial = script

Running optimization for script input.py
Using these settings:
ARGUMENT DESCRIPTION VALUE
-runs runs 5
-alg algorithm spea2
-pop population 8
-initial initial population script
-iters iterations 2000
-mutamount mutation amount 0.3 0.01
-mutcount mutation count 60% 1
-crossprob crossover probability 0.1
-crossdist crossover distribution 20
-sensitivity sensitivity estimation true
-dynphases dynamic phases N/A
-dyniters iterations per phase N/A
-errors error handling stop
-priority process priority (default)
-threads number of threads to use 1
-timeout script execution timeout (seconds) (OFF)
-continue continue from aborted optimization (log file) (OFF)

parsing script output [out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"]
Output 1:
name SSIM
type FLOAT
goal MAX
unit
round 0
Output 2:
name time
type TIME
goal MIN
unit ms
round 10

Found following optimizable parameters:
# optimize sigma = _n_ | 1,10 | sigma
found 1 parameters to optimize

Number of parameter combinations is smaller than the population size - setting the population size to same (2)
Running SPEA2
Taking script's values into initial population
4 is not a valid value for parameter sigma


-> What am I missing / doing wrong?

Cu Selur

[Boulder]

At least this line needs a small correction:
sigma = 4 # optimize sigma = _n_ | 1,10 | sigma
should be
sigma = 4 # optimize sigma = _n_ | 1..10 | sigma

Then it will use range 1-10 with a step of 1. If you need a smaller step, you need to use a divide operator like in my previous post with Bicubic's b and c.

[Selur]

Thanks, using:
_n_ | 1..10 | sigma
I now get:
Found following optimizable parameters:
# optimize sigma = _n_ | 1..10 | sigma
found 1 parameters to optimize

Running SPEA2

Error in script execution:



Cannot load avisynth.dll:
Das System kann die angegebene Datei nicht finden.
in both cases. :/

Does Zopti need avisynth even when this is the Vapoursynth version?

Cu Selur

[Boulder]

I wonder if it internally uses avsr and needs Avisynth? It was originally an Avisynth application.

[zorr]

Does Zopti need avisynth even when this is the Vapoursynth version?

It's incorrectly determining that you're running an AviSynth script because the filename doesn't end with '.vpy'. I'll add '.py' for the next version. :)

[zorr]

I just noticed that MDSI seems to produce a bit erratic results.

I wasn't able to reproduce this, results from two runs were

39.561905 b=-84 c=64
39.569714 b=-90 c=59

and there are no wild results such as 1.27114432E8 (that's over 127 million) in the logs.

I'm still on Vapoursynth portable R53. Not sure what's going on, but could be related to the latest VapourSynth API revision.

[Selur]

It's incorrectly determining that you're running an AviSynth script because the filename doesn't end with '.vpy'. I'll add '.py' for the next version.
Thanks.

Next problem.
Now I get:
Error in script execution:
Script evaluation failed:
Python exception: No module named 'zoptilib'[/com]
My guess is that I would need to add:
[code]
import os
import sys
scriptPath = 'C:/Users/Selur/Desktop/Zopti-1.2.0'
sys.path.insert(0, os.path.abspath(scriptPath))

somewhere, since it seems like Zopti assumes the current dir is inside the Python discovery folder,...
-> okay, adding the above to my script does help,..

Now it's complaining that it needs tcanny.
-> adding that to my input script too helped
Now it's again about missing fmtc.
-> adding that to my input script too any I can finally run the script :)


Cu Selur

[Boulder]

I wasn't able to reproduce this, results from two runs were

39.561905 b=-84 c=64
39.569714 b=-90 c=59

and there are no wild results such as 1.27114432E8 (that's over 127 million) in the logs.

I'm still on Vapoursynth portable R53. Not sure what's going on, but could be related to the latest VapourSynth API revision.

Looks like it starts to happen at R54. I just tested R53, where it works fine but R54 has these strange values for results. I'll report it to the Vapoursynth thread.

[Myrsloik]

Looks like it starts to happen at R54. I just tested R53, where it works fine but R54 has these strange values for results. I'll report it to the Vapoursynth thread.

Can you make a simple test case. Like one R53 and one R54 portable vs folder zipped up with everything included and idiot proof instructions? I'm not sure I understand exactly how to reproduce.

[Boulder]

Can you make a simple test case. Like one R53 and one R54 portable vs folder zipped up with everything included and idiot proof instructions? I'm not sure I understand exactly how to reproduce.
First of all, you need Java 16, a portable version here if you don't want to install it:
https://portableapps.com/apps/utilities/OpenJDKJRE

This zip file contains those two Vapoursynth portable versions with Zopti related files extracted inside the folders:
https://drive.google.com/file/d/1YFtkr612IdvauiRBe57Ii_DE-qxA7ebJ/view?usp=sharing

The clip to run Zopti on is here:
https://drive.google.com/file/d/11GtJ1gfT-hStUDl8UmY3-fVJjBDXPD1M/view?usp=sharing

And the command line:
zopti "strangerthings_s02e01.vpy" -alg mutation -iters dyn -dyniters 30 -dynphases 3 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

You need to edit zopti.ini to point to the vspipe.exe path.

Hope the packages work as intended :)

If the problem lies in muvsfunc, it could probably be replicated by checking the MDSI score it returns for the same frame in different VS versions. Didn't do that yet.

EDIT: just posted in the VS thread since this can be reproduced in an easier way.

[zorr]

Now it's complaining that it needs tcanny.
-> adding that to my input script too helped
Now it's again about missing fmtc.
-> adding that to my input script too any I can finally run the script :)

Zoptilib.py has a helpful comment about the requirements: Requirements: muvsfunc (for SSIM, GMSD and MDSI), VapourSynth-VMAF (for VMAF), VapourSynth-butteraugli (for Butteraugli)

I wasn't aware that it needs tcanny though. :confused:

[Selur]

I wasn't aware that it needs tcanny though.
Probably came with an updated muvsfunc version.

[poisondeathray]

Can zopti help with selection on a per frame basis ?

eg. 4 versions of interpolate video, RIFE 2.4, 3.1, 3.4, 3.8 . On some frames maybe 2.4 "solves" it better, maybe 3.8 on others, you get the idea...

You can stackvertical/horizontal and manually choose with trim or remapframes, but can zopti help automatic this task ?

[zorr]

Can zopti help with selection on a per frame basis ?

eg. 4 versions of interpolate video, RIFE 2.4, 3.1, 3.4, 3.8 . On some frames maybe 2.4 "solves" it better, maybe 3.8 on others, you get the idea...

You can stackvertical/horizontal and manually choose with trim or remapframes, but can zopti help automatic this task ?

Almost.

0 1 2 3 time (frames)
----------------------------------
[A ] [B ] [C ] [D ] original frames
[A ] [AB] [B ] [BC] [C ] [CD] [D ] 1) interpolate new frames (RIFE)
[AB] [BC] [CD] 2) remove original frames (SelectOdd)
[AB] [BB] [BC] [CC] [CD] 3) interpolate new frames again (RIFE)
[BB] [CC] 4) leave only latest interpolated frames (SelectOdd)

This is the basic structure for evaluating interpolated frames. In the end you can compare [BB] with [B] and [CC] with [C] etc. Let's say you calculate [AB], [BC] and [BB] with 4 different RIFE versions. You can now find out which one created the best match between [BB] and [B]. The only problem here is that [BB] was calculated from two interpolated frames [AB] and [BC], and [BC] will also have effect on the next frame [CC]. So if RIFE v1 was the best choice for [BB] and RIFE v2 was the best choice for [CC] then we don't know which one should be chosen for the interpolated frame [BC]. But it could be good enough to use the best choice from [BB] to select frame [AB] and the best of [CC] to select frame [BC] etc. Did I make any sense?

EDIT:
Now that I think about it, we could also use both [BB] vs [B] and [CC] vs [C] to select interpolated frame [BC]. The contribution of [BC] to [BB] and [CC] is 50% while [AB] and [CD] only contribute 25% each. Selecting the best RIFE version would be done with formula (assuming GMSD similarity metric) GMSD(BB, B) + GMSD(CC,C).

[zorr]

Here's a new release, it contains the following goodies (including the .py support for Selur :))

new option: retry
-tries to run the script again this many times if execution fails
-can be useful if the script uses plugins which are not 100% reliable and can crash
-default value is zero (do not retry)
-example: zopti script.avs -pop 24 -iters 10000 -retry 4
new visualization mode: line
-draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
-accepts parameter -range to limit the displayed values to certain range
-black line shows the best values, blue line indicates the number of results per value
-a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
-example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
support for custom output properties written by the script (only in VapourSynth scripts for now)
better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
option -continue now also supported by exhaustive algorithm (when using more than one thread)
order of runs in a visualization is now based on file name, not by modified date
all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
visualization mode seriespareto no longer contains the global pareto line
VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
bugfix: fixed a memory leak in XChart (at least partially)


Here's what -vismode line looks like:

https://i.postimg.cc/mDb5mW9t/lambda.png

Download link updated at first post.

[zorr]

This release introduces a better way for Avisynth scripts to write the results (VS version doesn't need this change as it's already very reliable) and fixes a couple of bugs. I recommend the upgrade especially if you're using more than one thread, there was a rare but possible bug which could result in flawed results.


supports a new way for an Avisynth script to write the result file
-first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
-then come the per frame result lines in any order, using the same format as before
-do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
-this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
-old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
-new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value

Download link updated at first post.

[zorr]

Zopti v1.2.3 is ready. This one has a new conversion feature to help importing Zopti data into other software. Also improvements to -vismode line and some other minor things. Details below:

new mode: convert
-converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
-writes a new file and doesn't remove the original
-does not overwrite an existing file (error message given in such situation)
-supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
-sort direction can be specified by adding a + or - after the field name, for example "-sort time-" sorts by time, descending
-ascending direction (+) is the default if no + or - is given
-multiple sort criterias supported, for example "-sort dct GMSD-" sorts by dct (ascending) and the rows with the same dct value by GMSD (descending)
-output format: CSV
-specified with -format csv or by not giving format option
-target file name is the same as the original .log file but file ending changes from .log to .csv
-csv file includes title row with the names of result values and optimized parameters
-can be opened directly (for example) in Excel
-uses the default decimal format of the OS with floating point values
-output format: zopti
-specified with -format zopti
-keeps the original Zopti format and header rows, only sorts the results when option -sort is given
-filename is "converted " + original file name
-example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
-example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
improvements to visualization mode: line
-now supports option -types which controls which aggregate lines are displayed in the chart
-valid values are "best", "worst", "average", "median", "count" and "samples"
-best = best result per parameter value
-worst = worst result per parameter value
-average = average result per parameter value
-median = median result per parameter value
-count = number of samples per parameter value
-samples = display all results as small dots
-separate multiple values with space, for example "-types best average"
-default value is "best count" which will display the best value per parameter value and the number of samples per parameter value
-a legend is shown if more than one type is selected
display of option -groupby improved, pareto fronts can no longer be cut off the screen
option -continue also supported with mutation algorithm
-but only for the generation of the initial random population
-useful if you only want to test independent random samples with a large population
validation mode behavior change: it now validates all the results in the log file and not just the pareto front

[ChaosKing]

Not sure if Zopti supports PSNR yet, but there's a plugin for that now :) https://github.com/AmusementClub/vs-ComparePlane