Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 3rd February 2019, 00:33   #1  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Zopti

Zopti is what was called AvisynthOptimizer. The name was changed because it now supports VapourSynth as well. This first post is an introduction similar to this one on the Avisynth Development forum, just changed a bit to reflect the latest features and what is possible with VapourSynth. I also cut out some unnecessary blabbering.

Let's say you want to motion compensation to double frame rate or replace a few corrupted frames. The right tool for that job is obviously MVTools. The only problem now is finding good parameters to get the best quality (because every video needs a little different settings). Ok no problem, I will just quickly adjust these... wait... there are about 60 parameters to adjust! Finding good settings would involve *a lot* of manual testing. This is where Zopti jumps in and says "Don't worry pal, I will find the best settings for you!".

But how? A VapourSynth script is first "augmented" by adding instructions on which parameters should be optimized. The instructions are written inside comments so the script will continue to work normally. The script also needs to measure the quality of the processed video using current parameter values or measure the running time (or preferably both the quality and runtime). The script needs to write these results to a specific file. There is a helper Python library "zoptilib" you can use to do the quality/runtime measurements and write the file.

Now you might be wondering how on earth can the script measure the quality. I can only think of one way: to compare the frames to reference frames and measure the similarity. The closer the similary value, the better the quality. VapourSynth has (at least) these similarity metrics available: SSIM (the classic, most well known), GMSD (seems better than SSIM) and VMAF (by Netflix). It's also possible to get MS-SSIM (multi-scale SSIM) from the VMAF plugin and I also implemented B-SSIM for AviSynth (SSIM with blurring adjustment).

Ok, but where do we get the reference frames? In case of MVTools we can use the original frames as reference frames and the script will try to reconstruct them using motion compensation (but it's not allowed to use the reference frame in the reconstruction). We can do something similar if we want to use MVTools to double the framerate: we first create the double-rate video, then remove the original frames from it, then double the framerate again and finally compare these to the original frames. Or you could reconstruct every odd frame using the even frames. This idea is not limited to MVTools, you could for example do color grading using some other software and then try to recreate the same result using VapourSynth. I'm sure the smart people here will find use cases I couldn't even dream about.

MVTools is a complex plugin and the settings can change the runtime of the script dramatically. In addition to quality we can also measure the runtime to find settings that are fast enough (for example if you want to use the script in real-time).

So if the purpose is to find the best settings, what do we consider the "best" when both quality and time are involved? For example we can have settings A with quality 99 (larger is better), time 2300ms (larger is worse) and settings B with quality 95 and time 200ms. Which one of these is better? We can use the concept of pareto domination to answer this question. When one solution is at least as good as the other solution in every objective (in this case the objectives are quality and speed) and better in at least one objective, it dominates the other solution. In this example, neither dominates the other. But if we have settings C with quality 99 and time 200ms it dominates both A and B. In the end we want to know all the nondominated solutions, which is called the pareto front. So there's going to be a list of parameters with increasing quality and runtime. You can have more than two objectives if you want, the same pareto concept works. There's a good and free ebook called "Essentials of Metaheuristics" which describes the pareto concept and much more.

Zopti reads the augmented script, verifies it and starts running a metaheuristic optimization algorithm. There are currently four algorithm choices: NSGA-II, SPEA2, mutation and exhaustive. The first two are some of the best metaheuristic algorithms available and also described in the ebook mentioned above. The mutation is my own simplistic algorithm (but it can be useful if you're in a hurry since it's the most efficient). Exhaustive will try all possible combinations, it's useful if you only have a small number of them (you could for example do an exhaustive search on one parameter value only). All the metaheuristic algorithms are working in a similar manner generating different solutions and testing them. The optimizer does these tests by creating scripts with certain parameter values and running them. The script then writes the measurements (quality/time) into a file which the optimizer reads. The metaheuristic then decides which parameters to try next based on the results. This continues until some end criteria is met.

There are three different ending criterias: number of iterations, time limit and "dynamic". Number of iterations is just that, the algorithm runs the script specific number of times. Setting a time limit can be pretty useful if you know how much time you can use for the optimization. You could for example let it run overnight for 8 hours and see the results in the morning. Dynamic variation is stopping only when it doesn't make any progress anymore. Making progress is defined by "no more pareto front members in last x iterations". This can be useful if you want to find the best possible results regardless of how long it takes.

During the optimization all tested parameters and their results are written to a log file. This log file can be "evaluated" during and after the optimization process. The evaluation basically means finding the pareto front and showing it. You can also create scripts from the pareto front in order to test them yourself. It's also possible to visualize the results of the log file in a two dimensional scatter chart. This chart highlights the pareto front and shows all the other results too. The chart can also be "autorefreshing": it loads the log file every few seconds and updates the visuals, which is a fun way to track how the optimization is progressing. Here's a gif what it looks like (obviously sped up):



The visualization has a few other bells and whistles but one I'd like to highlight here is the group by functionality: you can group the results by certain parameter's values and show a separate pareto front for each value. For example this what grouping by MVTools' blocksize looks like:



Another visualization mode draws a heat map where two parameters are chosen for the x and y axis, the color of the cell at x,y is the best result found with that parameter combination, brighter color meaning better result.



Measuring the script's runtime is not very accurate, ie it has some variation. All the other processes running at the same time are using CPU cycles and messing with the cache so you should try to minimize other activity on the computer. In order to get more accurate results you can run a validation on the finished results log file. In validation the idea is to run the pareto front results multiple times and calculate the average, median, minimum or maximum of these multiple measurements (you can decide which one(s)).

So how good is the optimizer? Let's take a look at one example. A while back there was a thread about best motion interpolation filters. There's a test video with a girl waving her hand. The best options that I know of are John Meyer's jm_fps script and FrameRateConverter. Here's a comparison gif with those two and Zopti. FramerateConverter was run with preset="slowest".



Now obviously I'm showing a bit of a cherry-picked example here. The optimizer was instructed to search the best parameters for this short 10 frame sequence. I have run most of my optimization runs using only 10 frames because otherwise the optimization takes too long. Ideally the optimizer would automatically select the frames from a longer video, I have started working on such a feature but it's not finished yet (got sidetracked to implement a scene change detector...) so for now user has to make the selection.

At this point I envision that Zopti can be an useful tool for plugin authors so they can test plugin parameters and try to search for the optimal ones. Zopti can also be a bug hunter, it has found several bugs with the Avisynth MVTools by Pinterf which he has since fixed. The VapourSynth MVTools is currently not yet as robust, but jackoneill has already fixed one bug (thanks!). At some later point Zopti could be useful for normal users, when combined with a script with limited search space so that the search will not take excessively long time. Zopti (or rather the previous version AvisynthOptimizer) has already been used by some courageous people from these forums for scaling, motion compensation, denoising and HDR to SRD tone-mapping. I'd like to thank them all for providing important feedback, feature requests and bug reports.

You can download the latest Zopti version here. I will keep this link updated.

Version history:

1.0.1-beta
  • #output definition is no longer needed when using zoptilib (reads output file and metrics from Zopti initialization line if #output not given)
  • accept zero time as valid output value (happens when sub 10ms times are rounded towards zero)
  • updated zoptilib to version 1.0.3
    -new similarity metrics MDSI and Butteraugli
    -new (semi)automatic YUV -> RGB conversion for metrics that need RGB
    -new init parameter matrix to set the YUV color matrix for the RGB conversion
    -added toRGB() function for manual RGB conversions
    -no downsampling by default in any of the metrics

1.0-beta
  • added support for VapourSynth scripts
  • Linux support (with VapourSynth)
  • added zoptilib python module to ease script writing
  • added support for VMAF output logs
  • improved heat map visualization:
    -updates optimization progress to title bar when using autorefresh
    -shows correct visuals also when smallest value is best
    -added latest results visualization
    -heat maps can be saved to image sequence with the -animation argument
    -initial window size scaled to full screen
  • improved scatter chart visualization:
    -improved zooming (-top)
    -better legend position when smallest value is best
  • uses resolve for initial population when using randomized arguments fails 100 times
  • more error handling options when script execution fails: new argument -errors with options:
    -stop (default, stops execution)
    -ignore (continues optimization)
    -log (continues optimization and logs errors into a separate error log)
  • checks that input arguments are all valid for the chosen -mode
  • improved error handling and error reporting in reverse polish notation parser
  • new option -priority for setting the optimizer process priority (on Windows only), options are:
    -"": same as /NORMAL, default
    -lowest: same as /LOW
    -lower: same as /BELOWNORMAL
    -higher: same as /ABOVENORMAL
    -highest: same as /HIGH
  • removed bias from the random number generation

The next post will be a hands-on denoising tutorial. Stay tuned.

Last edited by zorr; 19th February 2019 at 00:09.
zorr is online now   Reply With Quote
Old 3rd February 2019, 02:04   #2  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
Nice work!

But I get a file not found error :-)
Code:
java.io.FileNotFoundException: D:\Download\Zopti-1.0-beta\work\result_1549155407307_1549155407426.txt
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(Unknown Source)
        at java.io.FileInputStream.<init>(Unknown Source)
        at java.io.FileReader.<init>(Unknown Source)
        at avisynthoptimizer.file.FileUtil.readLinesFromFile(FileUtil.java:58)
        at avisynthoptimizer.AviSynthOptimizer.readVapoursynthResults(AviSynthOptimizer.java:6566)
        at avisynthoptimizer.AviSynthOptimizer.waitAndReadResults(AviSynthOptimizer.java:6587)
        at avisynthoptimizer.AviSynthOptimizer.runAVS(AviSynthOptimizer.java:6308)
        at avisynthoptimizer.AviSynthOptimizer.evaluateFitness(AviSynthOptimizer.java:5588)
        at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:190)
        at avisynthoptimizer.nsga_ii.AviSynthProblem.evaluate(AviSynthProblem.java:35)
        at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.lambda$evaluate$1(SequentialSolutionListEvaluator.java:24)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
        at org.uma.jmetal.util.evaluator.impl.SequentialSolutionListEvaluator.evaluate(SequentialSolutionListEvaluator.java:24)
        at avisynthoptimizer.spea2.DynamicSPEA2.evaluatePopulation(DynamicSPEA2.java:76)
        at org.uma.jmetal.algorithm.impl.AbstractEvolutionaryAlgorithm.run(AbstractEvolutionaryAlgorithm.java:56)
        at java.lang.Thread.run(Unknown Source)
java.lang.ArrayIndexOutOfBoundsException: -1
        at java.util.ArrayList.elementData(Unknown Source)
....
My script:
Code:
from zoptilib import Zopti
output_file = r'D:\results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['ssim', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 4					# optimize sigma = _n_ | 1,10 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)
	
alternate.set_output()
java -jar .\Zopti.jar D:\z.vpy
vspipe path is set in the ini file.
__________________
Search and denoise
ChaosKing is offline   Reply With Quote
Old 3rd February 2019, 09:48   #3  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by ChaosKing View Post
Nice work!

But I get a file not found error :-)
The # output line should probably look like this:

Code:
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
The file name must match.

Also this line
Code:
alternate.set_output()
is not necessary, Zopti sets the output. In some cases it is critical that the correct file is set as the output. The alternate file is the correct one in this case but for some reason it doesn't work if it's set first by Zopti and then again in the script.
zorr is online now   Reply With Quote
Old 3rd February 2019, 10:08   #4  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
Thx, works now without alternate.set_output()
__________________
Search and denoise

Last edited by ChaosKing; 3rd February 2019 at 11:08.
ChaosKing is offline   Reply With Quote
Old 3rd February 2019, 11:08   #5  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

Code:
* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13
Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)

VMAF just stops:
Code:
Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.
Maybe I do something wrong. Here is my script again

Code:
from zoptilib import Zopti
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time'])
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[5500:5600]
alternate = orig.grain.Add(24)
sigma = 35					# optimize sigma = _n_ | 9..40 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)

result_*.txt:
Code:
<?xml version="1.0"?>
<VMAF version="1.3.11">
	<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
	<fyi numOfFrames="100" aggregateVMAF="96.037" execFps="1.36185" timeTaken="73.4294" />
	<frames>
		<frame frameNum="0" adm2="0.983001" motion2="0" vif_scale0="0.709537" vif_scale1="0.953811" vif_scale2="0.974778" vif_scale3="0.983557" vmaf="91.1718" />
		<frame frameNum="1" adm2="0.98031" motion2="14.7504" vif_scale0="0.760672" vif_scale1="0.952192" vif_scale2="0.97319" vif_scale3="0.982214" vmaf="100" />

		<frame frameNum="98" adm2="0.992219" motion2="1.11641" vif_scale0="0.824504" vif_scale1="0.974561" vif_scale2="0.986421" vif_scale3="0.991392" vmaf="95.7912" />
		<frame frameNum="99" adm2="0.990791" motion2="3.7195" vif_scale0="0.787278" vif_scale1="0.973289" vif_scale2="0.986163" vif_scale3="0.991446" vmaf="98.8478" />
	</frames>
</VMAF>
EDIT
changed out1="SSIM: to out1="vmaf: and it works now
__________________
Search and denoise

Last edited by ChaosKing; 3rd February 2019 at 11:50.
ChaosKing is offline   Reply With Quote
Old 3rd February 2019, 11:30   #6  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by ChaosKing View Post
VMAF just stops:
Code:
Running SPEA2
Error: The last line of the result file did not start with "stop"
This looks like VMAF xml file but none of the outputs defined in the script is VMAF.
I will look at the GMSD later, but the VMAF problem can be solved by renaming one of the #output parameters "vmaf" (not case sensitive).
zorr is online now   Reply With Quote
Old 3rd February 2019, 12:49   #7  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by ChaosKing View Post
I made some tests now and it seems that it works with ssim, but gmsd produces numbers like this

Code:
* 1 / 2000 : 6.144252 12480ms sigma=23
* 2 / 2000 : 9.186479 12160ms sigma=14
+ 3 / 2000 : 8.801965 12070ms sigma=15
+ 4 / 2000 : 7.343317 11990ms sigma=19
* 5 / 2000 : 10.544009 12300ms sigma=11
+ 6 / 2000 : 4.6340036 11610ms sigma=30
+ 7 / 2000 : 8.400525 11740ms sigma=16
+ 8 / 2000 : 9.643034 12150ms sigma=13
Are these correct numbers? Shouldn't these be between 0-1? (I use the latest muvsfunc script and vmaf plugin)
In GMSD smaller number means better quality, so make sure you have GMSD: MIN(float) as the output. Zopti returns the sum of per frame values so otherwise it looks good to me.
zorr is online now   Reply With Quote
Old 3rd February 2019, 13:16   #8  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
Ahh now I see it too, thx.
__________________
Search and denoise
ChaosKing is offline   Reply With Quote
Old 3rd February 2019, 15:35   #9  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.
This barely changed since the start (it's still running) https://i.imgur.com/4JszzyZ.png
There are (too?) many lines with height=1080

My log so far https://pastebin.com/UrKnMwWz

script
Code:
from zoptilib import Zopti
#output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"
#output_file = r'results.txt' # output out1="ssim: MAX(float)" file="results.txt"
output_file = r'results.txt' # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="results.txt"

zopti = Zopti(output_file, metrics=['gmsd', 'time']) #gmsd vmaf ssim
orig = core.lsmas.LWLibavSource(r"E:\test.mkv")[19157:19157+1]

# https://github.com/Infiziert90/getnative/blob/master/getnative.py#L112
# change format to GrayS with bitdepth 32 for descale
matrix_s = '709' if orig.format.color_family == vapoursynth.RGB else None
src_luma32 = core.resize.Point(orig, format=vapoursynth.YUV444PS, matrix_s=matrix_s)
src_luma32 = core.std.ShufflePlanes(src_luma32, 0, vapoursynth.GRAY)
orig = core.std.Cache(src_luma32)

width = 1400		# optimize width = _n_ | 1400..1920 | width
height = 800		# optimize height = _n_ | 800..1080 | height
x = ds.Debilinear(orig, width=width, height=height)
alternate = x.resize.Bilinear(orig.width, orig.height)

zopti.run(orig, alternate)
The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?

Is it possible that Zopti use only even numbers for width/height?
__________________
Search and denoise

Last edited by ChaosKing; 3rd February 2019 at 15:40.
ChaosKing is offline   Reply With Quote
Old 3rd February 2019, 22:31   #10  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by ChaosKing View Post
My scripts runs for over 1 hour now and it seems it's stuck in some kind of loop. There are not many changes between a mutation.
What optimization arguments were you using?

Quote:
Originally Posted by ChaosKing View Post
There are (too?) many lines with height=1080
Seems like the results are really good when height=1080 and that's why it's so common. Also when you only have two parameters to optimize and assuming you're using the default mutationCount "60% 1" then about halfway though it will only mutate one parameter. If the best results have height=1080 then at least half of the mutations have that value when it's mutating the width only. If you want it to explore a bit more you can set the mutationCount to 2 (or 100% which means the same in this case). Then it will mutate both parametes all the time.

Quote:
Originally Posted by ChaosKing View Post
The goal is here to find the native resolution in a anime video.
The only thing what stands out are values like this: 5.625693E-6
Are they processed correctly in Zopti?
When running the evalution on your log the pareto front is

Code:
1.5509998E-7 1190 width=1919 height=1080
  3.7165887E-7 1120 width=1920 height=1079
  4.1259636E-6 1110 width=1920 height=1077
  8.270416E-6 1100 width=1920 height=1075
  4.067268E-4 1090 width=1920 height=1034
  0.0016862862 1080 width=1702 height=1080
  0.0028691771 1070 width=1610 height=1080
  0.0066714296 1060 width=1450 height=1080
so yes, it can understand the scientific notation. There's a problem though, the best results are really close to 1920x1080 and looking at the logs that got a perfect GMSD score 0.0. But the optimizer currently interprets 0.0 as invalid, that's why it's not in the pareto front. I will have to think about how to fix that.

It's a bit suspicious that you can get a perfect similarity score. Should it happen in your use case?

Quote:
Originally Posted by ChaosKing View Post
Is it possible that Zopti use only even numbers for width/height?
Yes, there are a couple of ways to do that. The simplest is probably:

Code:
width = 2*700	# optimize width = 2*_n_ | 700..960 | width
height = 2*400	# optimize height = 2*_n_ | 400..540 | height
Or you can specify a filter which only accepts numbers divisible by 2:

Code:
width = 1400	# optimize width = _n_ | 1400..1920 ; filter:x 2 % 0 == | width
height = 800	# optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height
zorr is online now   Reply With Quote
Old 3rd February 2019, 22:59   #11  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
I don't know if a perfect score should be possible, but because the "resize error" is very small it is not easy to detect I guess.

The native res in my case should be 1600x900 (900p). Setting width & height to
width = 1400 # optimize width = _n_ | 1500..1680 | width
height = 800 # optimize height = _n_ | 880..910 | height
leads to this, which look almost to what I expected in the first place.
https://i.imgur.com/QNbTYMW.png
(And maybe I just need to test another frame)

There is already a native resolution "detector" here https://github.com/Infiziert90/getna...r/getnative.py
I just wanted to see how Zopti will perform. I think the differences need to be emphasized more, so the resizing error becomes more visible.

Zopti could come in handy if a bicubic resizer was used on the clip. It will be easy to find the b and c values like you have shown here https://forum.doom9.org/showthread.p...44#post1859544
__________________
Search and denoise

Last edited by ChaosKing; 3rd February 2019 at 23:04.
ChaosKing is offline   Reply With Quote
Old 4th February 2019, 12:09   #12  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
Breaking news: I'm an idot
I used my already encoded and "descaled" file instead of the source file. That also explains the perfect 1080p match -_-


height = 800 # optimize height = _n_ | 800..1080 ; filter:x 2 % 0 == | height
I want to optimize the runs but I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3
__________________
Search and denoise
ChaosKing is offline   Reply With Quote
Old 4th February 2019, 21:32   #13  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by ChaosKing View Post
Breaking news: I'm an idot
Welcome to the club.

Quote:
Originally Posted by ChaosKing View Post
I don't quite understand the reverse polish notation yet. Maybe you could help me with the width:

width = height * (16/9)
or if possible
width = height * (16/9) - 3 .. height * (16/9) + 3
I always visualize the rpn notation as a stack. Every operation can take values from the stack, do some computation with them and then put the result on the top of the stack. So for example this x 2 % 0 ==:

1) x - put x on the stack, stack is "x"
2) 2 - put 2 on the stack, stack is "x 2" (top of the stack on the right)
3) % - modulus operator, take two topmost values from the stack, calculate modulus and put the result back into stack - stack is (x % 2)
4) 0 - put 0 on the stack, stack is (x % 2) 0
5) == - equals operator, take two topmost values and put true on the top of the stack if they are equal, false otherwise - stack is ((x % 2) == 0)

But actually in this case the filter is not needed. You can optimize the height only and calculate width from it:

Code:
height = 800 # optimize height = _n_ | 800..1080 | height
width = height*(16/9)
or if you want to give it a small range around the calculated value you can do

Code:
height = 800 # optimize height = _n_ | 800..1080 | height
width_offset = 0 # optimize width_offset = _n_ | -3..3 | width_offset
width = height*(16/9) + width_offset
zorr is online now   Reply With Quote
Old 4th February 2019, 21:41   #14  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,513
I've been trying to run my resizing test with VMAF, the sample clip is 5 separate single frames from an episode of Black Sails. My first run ended up with the b value at -100, so I enlarged the area so that both b and c can go -300..300. However, VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?

Code:
e 68 / 1,00 : 100.0 200ms b=-33 c=236
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 (no samples) c 1,000
MUTATED GENERATION 69
Mutating 1 params by 1,0 % (phase 1,00)
e 69 / 1,00 : 100.0 200ms b=-52 c=236
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 (no samples) c 1,000
MUTATED GENERATION 70
Mutating 1 params by 1,0 % (phase 1,00)
  70 / 1,00 : 100.0 210ms b=-50 c=236
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 (no samples) c 1,000
MUTATED GENERATION 71
Mutating 1 params by 1,0 % (phase 1,00)
e 71 / 1,00 : 100.0 200ms b=-42 c=246
Parameter sensitivity estimation with 4 result combinations
  -> b 1,000 c 1,000
MUTATED GENERATION 72
Mutating 1 params by 1,0 % (phase 1,00)
e 72 / 1,00 : 100.0 200ms b=-42 c=225
0 iterations remaining is this generation
No improvement in 24 iterations - stopping
Parameter sensitivity estimation with 4 result combinations
  -> (no samples) b 1,000 c 1,000
This is the script:
Code:
orig = core.ffms2.Source(source=r'blacksails.avi')
   
output_file = r'results.txt' # output out1="vmaf: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['vmaf', 'time'])	# initialize output file and chosen metrics 

b = 0/100.0					# optimize b = _n_/100.0 | -300..300 | b
c = 0/100.0					# optimize c = _n_/100.0 | -300..300 | c

alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)

zopti.run(orig, alternate)
I've used this command line for these quick tests:
zopti test.vpy -alg mutation -iters dyn -dyniters 24 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

With gmsd, the run resulted in b=-77 c=5 as the optimal pair. In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 5th February 2019, 00:04   #15  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by Boulder View Post
VMAF seems to hit 100.0 almost if not every time. Is the result rounded somewhere or is the plugin just a bit too inaccurate?
I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.

Quote:
Originally Posted by Boulder View Post
With gmsd, the run resulted in b=-77 c=5 as the optimal pair.
Can you also try what SSIM gives as the optimal pair?

Quote:
Originally Posted by Boulder View Post
In the zoptilib.py, I've set the VMAF model to 1, pool to 1 and ci to True.
You can set the VMAF model with zopti.setVMAFModel(). Changing the pool won't have any effect since Zopti calculates the result from the individual frames and always just sums them up. The ci shouldn't change the scores either, it's just extra information.

Last edited by zorr; 5th February 2019 at 00:08. Reason: missed a word
zorr is online now   Reply With Quote
Old 5th February 2019, 05:06   #16  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,513
Quote:
Originally Posted by zorr View Post
I haven't done enough tests with VMAF to say but I'd say it's possible.
I will so some tests on my own tomorrow, but in the meantime if you have the time can you do an exhaustive run and see what kind of heat map you get? That would tell for sure if VMAF is inaccurate.
A very quick test shows this kind of behaviour, so it looks like VMAF cannot measure the difference between the clips accurately enough. That is quite odd because I would expect that downsizing to 720p before upsizing to 4K would have a big enough effect. Maybe I'll also have to try keeping the original at 1080p and use model 0 even though my TV is 4K.

Code:
e 178 / 20301: 100.0 200ms b=-123 c=0
e 179 / 20301: 100.0 200ms b=-122 c=0
e 180 / 20301: 100.0 200ms b=-121 c=0
e 181 / 20301: 100.0 200ms b=-120 c=0
e 182 / 20301: 100.0 200ms b=-119 c=0
e 183 / 20301: 100.0 200ms b=-118 c=0
e 184 / 20301: 100.0 200ms b=-117 c=0
e 185 / 20301: 100.0 200ms b=-116 c=0
  186 / 20301: 99.98204 200ms b=-115 c=0
  187 / 20301: 99.99383 200ms b=-114 c=0
  188 / 20301: 99.9717 200ms b=-113 c=0
  189 / 20301: 99.97675 200ms b=-112 c=0
  190 / 20301: 99.96704 200ms b=-111 c=0
  191 / 20301: 99.953285 200ms b=-110 c=0
  192 / 20301: 99.944626 200ms b=-109 c=0
SSIM gives b=-59, c=31 as the optimal pair. That difference with gmsd would be expected if SSIM favours blurring more. Without testing, I assume that gmsd's result would be slightly sharper looking.

EDIT: The clip is here if you'd like to test it: https://drive.google.com/open?id=1rS...VQSg40vwtUZFNN
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...

Last edited by Boulder; 5th February 2019 at 17:42.
Boulder is offline   Reply With Quote
Old 5th February 2019, 22:12   #17  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 209
Quote:
Originally Posted by Boulder View Post
alternate = rhq.resamplehq(orig, width=1280, height=720, a1=b, a2=c)
Where is the resamplehq function from? I found resamplehq script but it doesn't have the same method name or arguments.
zorr is online now   Reply With Quote
Old 5th February 2019, 23:25   #18  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 686
It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122...43fe/revisions
__________________
Search and denoise
ChaosKing is offline   Reply With Quote
Old 6th February 2019, 03:17   #19  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,132
Quote:
Originally Posted by Boulder View Post
.. so it looks like VMAF cannot measure the difference between the clips accurately enough.
There's also this issue where the VMAF result is skewed by a component 'motion2=0' score for the first frame in a clip:

https://forum.doom9.org/showthread.p...61#post1864561

The only information I can find about this parameter is:

"Motion. This is a simple measure of the temporal difference between adjacent frames. This is accomplished by calculating the average absolute pixel difference for the luminance component."

https://medium.com/netflix-techblog/...c-653f208b9652

and:

"motion2 score typically ranges from 0 (static) to 20 (high-motion)"

https://github.com/Netflix/vmaf/blob...hon_library.md

Testing your Blacksails clip against itself, with VapourSynth VMAF v3 (Model=0) gives:

Code:
<params model="" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="8" aggregateVMAF="99.6711" aggregatePSNR="60" aggregateSSIM="0.99993" aggregateMS_SSIM="0.999936" execFps="2.27555" timeTaken="3.51563" />
<frames>
<frame frameNum="0" adm2="1" motion2="0" ms_ssim="0.999889" psnr="60" ssim="0.9999" vif_scale0="0.999999" vif_scale1="0.999997" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="97.4274" />
<frame frameNum="1" adm2="1" motion2="51.9751" ms_ssim="0.999997" psnr="60" ssim="0.999997" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999997" vif_scale3="0.999997" vmaf="100" />
<frame frameNum="2" adm2="1" motion2="45.4778" ms_ssim="0.999998" psnr="60" ssim="0.999999" vif_scale0="0.999999" vif_scale1="0.999998" vif_scale2="0.999996" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="3" adm2="1" motion2="45.4778" ms_ssim="0.999994" psnr="60" ssim="0.999995" vif_scale0="0.999997" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="4" adm2="1" motion2="48.0867" ms_ssim="0.99986" psnr="60" ssim="0.999837" vif_scale0="0.999999" vif_scale1="0.999996" vif_scale2="0.999995" vif_scale3="0.999995" vmaf="100" />
<frame frameNum="5" adm2="1" motion2="71.4518" ms_ssim="0.999999" psnr="60" ssim="0.999999" vif_scale0="1" vif_scale1="0.999999" vif_scale2="0.999998" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="6" adm2="1" motion2="74.8847" ms_ssim="0.999751" psnr="60" ssim="0.999716" vif_scale0="0.999994" vif_scale1="0.999994" vif_scale2="0.999994" vif_scale3="0.999994" vmaf="100" />
<frame frameNum="7" adm2="1" motion2="74.8847" ms_ssim="1" psnr="60" ssim="1" vif_scale0="0.999998" vif_scale1="0.999995" vif_scale2="0.999993" vif_scale3="0.999994" vmaf="100" />

FFMPEG SSIM reports lossless, 1.00000 (Inf)
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 6th February 2019 at 03:28.
WorBry is offline   Reply With Quote
Old 6th February 2019, 04:52   #20  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,513
Quote:
Originally Posted by ChaosKing View Post
It should be the correct one, but it's an older version. Search for a2 in here https://gist.github.com/4re/64642122...43fe/revisions
I should have the latest version, but looking at the code, I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:57.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.