Zopti

[zorr]

Setting both Prefetch(threads=1, frames=24) and Prefetch(threads=24, frames=1) give this error: java.lang.NumberFormatException: For input string: "1I don't know what 'delimiter' means.0.179144I don't know what 'delimiter' means.3150.284912" and the log file is full of these errors.

I got the same result, but I was able to make it work by defining both delimiter and frame_count as global. I'm still not sure if the result file will always be correct but it seems to work. However I didn't seem to gain any performance with the prefetch. Perhaps the serialized file writing escalates upstream and makes everything run serially.

[Boulder]

I got the same result, but I was able to make it work by defining both delimiter and frame_count as global. I'm still not sure if the result file will always be correct but it seems to work. However I didn't seem to gain any performance with the prefetch. Perhaps the serialized file writing escalates upstream and makes everything run serially.

Yes, I think that's how it goes. Perhaps the only way to make it faster and threadsafe in Avisynth would be to use an array as you suggested. Or is it possible to make all the frames create their own log file and combine them when it's time to calculate the total score?

The VS version is also not perfect in multithreading sense but it is somewhat faster.

[zorr]

A new major release is here! Zopti now supports executing multiple scripts simultaneously which can lead to very nice performance improvement. This can be very useful when the script itself is not able to utilize all of your cores.

Here's the full release info:


added support for multithreaded execution of scripts with argument -threads
-default value of -threads is 1 meaning one script is executed at a time just like before
-all optimization algorithms (nsga-ii, spea2, mutation, exhaustive) support the the -threads option
-due to the nature of the heuristic algorithms (nsga-ii, spea2, mutation) there can be at most P threads running the scripts at the same time, where P is the population size
-exhaustive algorithm has no thread limits
-validate mode also supports -threads. there the maximum number of threads is the number of validated results (size of the pareto front)
-note: using more than one thread will make time measurements less accurate but will not change the quality measurements
avsr version update (v0.2.2)
added jMetal.log.ini to disable multithreading related JMetal logging
updated Zoptilib to version 1.0.9m
bugfix: evaluate could not parse log files when Zopti was used


Download link is at the first post. I will shortly post some data on the multithreading performance.

[zorr]

I tested the new -threads argument using the denoising tutorial script, this time using GMSD as the similarity metric and using 50 frames instead of 5. Here's the script:


import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 50 # how many frames are tested
MIDDLE_FRAME = 100 # middle frame number

# read input video
video = core.raws.Source(r'D:\optimizer\test\flower\flower_cif.yuv', 352, 288, src_fmt='I420')

zopti = Zopti(r'results.txt', metrics=['gmsd', 'time'])

# input color range is PC (full)
video = video.std.SetFrameProp(prop="_ColorRange", intval=0)

# add noise
noisy = video.grain.Add(var=25, seed=3)
#noisy.set_output()

# remove noise
sigma = 400/100 # optimize sigma = _n_/100 | 400..600 ; filter:x 5 % 0 == | sigma
bt = 5 # optimize bt = _n_ | 3,5 | blockTemporal
blockSize = 14 # optimize blockSize = _n_ | 6,12 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 4..6 ; max:blockSize 2 / | overlap
denoised = noisy.fft3dfilter.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
video = video[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
denoised = denoised[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(video, denoised)

There's also an AviSynth version of the script which I posted to the Avisynth Zopti thread (https://forum.doom9.org/showthread.php?p=1940863#post1940863).

The script only has 246 valid parameter combinations to test so we can try all of them using
zopti denoise_ex.vpy -alg exhaustive -threads 1

It takes 140 seconds to test all of the combinations using one thread. Let's see the performance when using more threads:

https://i.postimg.cc/QdZrh7St/zopti-performance-scaling.png

The best result is about 20 seconds using all 24 threads (on a Ryzen 3900X which has 12 cores / 24 threads).

Another way to look at the scaling is to calculate how much faster we get the results when using -threads:

https://i.postimg.cc/RFDj8JNv/zopti-performance-factor.png

Using 24 threads is 7,1 times as fast as using only one thread. Of course this is just one data point and I don't mean to imply that you can always get such a performance improvement. I would have liked to include more tests using more real-world usage scenarios (HD source etc) but my stock cooled processor cannot handle those for more than a few seconds, it becomes so hot that my PC shuts down. :eek:

Comparing AviSynth and VapourSynth it looks like VS has the edge when using one or just a few threads, but the differences almost vanish when using 12 or more threads. The VS is also able to utilize more than 16 cores while with AVS the performance starts to degrade. The point where more threads are just slowing down is probably dependent on the specific script, I will have to run more tests on that.

[Boulder]

With the new multithreading support, is it beneficial to increase the population size even with this kind of approach to optimizing the resizer params? A word of warning: VS uses a lot of memory for caching by default so it may be wise to lower the max_cache_size if you launch multiple instances :)

zopti "test.vpy" -alg mutation -iters dyn -dyniters 18 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

[zorr]

With the new multithreading support, is it beneficial to increase the population size even with this kind of approach to optimizing the resizer params?

zopti "test.vpy" -alg mutation -iters dyn -dyniters 18 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

In this case you have to increase the population size from 1, otherwise there's no use for more than one thread. It will for sure make the search faster but the quality (or the probability that you will find the optimal values) might go down a little bit. You can compensate for that with increasing -dyniters slightly. The net result should be as good (or better) results faster. You may have to do some testing to find how many threads give you the best bang for buck though.

A word of warning: VS uses a lot of memory for caching by default so it may be wise to lower the max_cache_size if you launch multiple instances :)

Yes that's one downside of using -threads. It's process level multithreading and every VSPipe process will use as much memory as a single one would. So the amount of free memory might limit how many threads you can actually use even though CPU is still not maxed out.

[zorr]

Time for a new update! This version has some major additions so I bumped the version to 1.2.0.


new option: -timeout (default is disabled). if the execution of Avisynth script takes longer than timeout seconds
-the script execution process (and subprocesses) will be terminated
-the result is set to worst possible as a penalty so it will not be considered as a valid candidate
-useful with scripts where certain parameter combinations can result in execution times several hundred times slower than usual
-NOTE: only implemented for AviSynth scripts (for now)
-example: zopti script.avs -pop 24 -iters 1000 -timeout 60
new option: -continue
-aborted optimization can be continued from the last complete generation (one generation is N iterations, where N is the population size)
-give the log file name of the aborted optimization as the value of continue
-old results up to last complete generation are copied to a new log file and optimization will continue from there
-useful if optimization was unexpectedly aborted or the CPU resources are needed for something else for a while
-example: zopti script.avs -pop 24 -iters 10000 -continue "mvtools spea2 run-01.log"
-NOTE: it is possible to change the optimization parameters (population, mutation count etc.) from those used in the continued log
-NOTE: random seed is not restored so the results will not be identical to what they would have been if execution had not been aborted
-NOTE: only works with algorithm spea2 (for now)
new visualization mode: history
-displays the best found value and all tried values of certain optimized parameter (given with -param)
-invalid results (script execution has failed OR timed out) are highlighted in orange
-also shows the phase of dynamic iteration (if applicable)
-accepts parameter -range to limit the displayed history to certain iteration range
-example: zopti -mode evaluate -vismode history -param lambda (displays all history of parameter "lambda")
-example: zopti -mode evaluate -vismode history -param lambda -range 1000 5000 (displays history from iteration 1000 to iteration 5000)
-example: zopti -mode evaluate -vismode history -param lambda -range -10000 (displays history of the last 10000 iterations)
argument -dynphases can now be zero (previously minimum was 1)
pareto front and total runtime is displayed also after mutation or exhaustive algorithm is finished
if population is based on a log file and the pareto front is smaller than the population size, the population will be filled with mutations of the pareto front instead of completely random combinations
if a log file for the optimization already exists it will be cleared in the beginning of the optimization
XChart updated to version 3.8.0 with some customization


And here's what -vismode history looks like:

https://i.postimg.cc/nLjnb7Y8/vismode-history.png

The black thick line shows the best found value of the chosen parameter. The red diamonds indicate a point where a new best result has been found and this parameter's value was changed. The white diamonds indicate a new best result but this parameter's value remained the same.

In the example optimization was finished and then restarted with different settings at around 190 000 iterations.

The -timeout was needed when I started running MVTools2 tests with all the parameters and it turned out that some combinations took an enormous amount of time to finish (typical time was 5 seconds but some combinations took over 30 minutes...). Using a timeout of 60 seconds solved that problem.

-continue is useful if you want to keep optimizing a result that is finished, you can try different settings. Also very handy when the optimization has been aborted by hardware failure or something like Windows update... you can just restart it using the same settings. This also makes it possible to run huge iteration counts little by little even if you need/want to use the computer for other things once in a while. :D

Download link updated at first post.

[Boulder]

Thank you for the new version :) It's always nice to see development even though the old product is running well.

One request though: would it be possible to remove the need for keeping the script that was used when running the optimization? I sometimes do many seasons of some TV series sequentially, and reuse the scripts since the properties are usually almost the same between the episodes from various seasons. Zopti doesn't like it when it cannot find the original script as I've already renamed it. I think writing the best result on the first line (to be able to see it right after opening the files in Notepad++ :)) of the log file would do just fine.

[zorr]

Thank you for the new version :) It's always nice to see development even though the old product is running well.

You're welcome. :)

One request though: would it be possible to remove the need for keeping the script that was used when running the optimization?

What it's looking for in the script are the parameters and their valid values. Those are needed (for example) in deciding how to map the values to x,y coordinates in the heat maps.

Zopti doesn't like it when it cannot find the original script as I've already renamed it. I think writing the best result on the first line (to be able to see it right after opening the files in Notepad++ :)) of the log file would do just fine.

If the script still has the same parameters (and value ranges) it doesn't matter which script is loaded. You could change the first line of the log file (starts with # script) to point to some other existing script.

On the other hand if you just run -vismode none then the script should not be needed and the best results could be displayed. I'll have to get back to you on that.

[ChaosKing]

Info: The latest VMAF release added PSNR-HVS and CIEDE2000 metrics.

[zorr]

Info: The latest VMAF release added PSNR-HVS and CIEDE2000 metrics.

Thanks. Could you add them to Zoptilib?
I haven't gotten around to test even wadiqam yet so no hurry. :)

[ChaosKing]

Will do. I think vmaf is a special case with xml log output or something like that? I don't really remember :p

VMAF metrics
0: PSNR
1: PSNR-HVS
2: SSIM
3: MS-SSIM
4: CIEDE2000

[zorr]

Will do. I think vmaf is a special case with xml log output or something like that? I don't really remember :p

Oh yeah. That means I need to add support for the new metrics into Zopti as well.

[Boulder]

I just updated to the new version to run my next optimization, but I get a Java error.

Exception in thread "main" java.lang.UnsupportedClassVersionError: avisynthoptimizer/AviSynthOptimizer has been compiled by a more recent version of the Java Runtime (class file version 53.0),
this version of the Java Runtime only recognizes class file versions up to 52.0

I am currently running the latest JRE version, 1.8.0u291. I checked and there are no old versions hanging around, the path variable also points to the correct place.

[zorr]

I am currently running the latest JRE version, 1.8.0u291.

That's the latest Java 8 version but the latest version is... 16. :eek: You can download one from https://adoptopenjdk.net/ (I personally use these) or from Oracle at https://jdk.java.net/.

Make sure to update your PATH variable to point to the correct version, the installers are not guaranteed to update it automatically anymore (at least that was my experience).

[Boulder]

So much for trusting Java update to do the dirty work :devil:

EDIT: well, Oracle recommends JRE 8 so that's why it's the one that comes up when searching and the one it will keep up-to-date automatically.

[Boulder]

I just noticed that MDSI seems to produce a bit erratic results. I ran the same job 10 times with these wildly varying results:

Run 1 best: 4.1299334 b=-99 c=-19
Run 2 best: 1.27114432E8 b=-73 c=68
Run 3 best: 1.27114432E8 b=-74 c=63
Run 4 best: 1.5116536E8 b=-90 c=37
Run 5 best: 42.133507 b=-78 c=-18
Run 6 best: 1.5116536E8 b=-110 c=28
Run 7 best: 1.27114432E8 b=-126 c=10
Run 8 best: 1.5116536E8 b=-108 c=9
Run 9 best: 1.27114432E8 b=-73 c=68
Run 10 best: 1.5116536E8 b=-101 c=36

The last run shows this at the end -- weird that the scores are the exact same..

No improvement in 30 iterations - stopping
Pareto front:
1.5116536E8 b=-101 c=36
1.5116536E8 b=-99 c=36
1.5116536E8 b=-71 c=36
1.5116536E8 b=-68 c=36

CLI parameters: zopti "strangerthings_s02e01.vpy" -alg mutation -iters dyn -dyniters 30 -dynphases 3 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

from vapoursynth import core
from zoptilib import Zopti

orig = core.ffms2.Source(source=r"c:\zopti\strangerthings_s02e01.avi")

zopti = Zopti(r'results.txt', metrics=['mdsi'], matrix='709')

b = -90/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 45/100.0 # optimize c = _n_/100.0 | -60..80 | c
downscaled_width = 1280
downscaled_height = 640
upscaled_width = orig.width
upscaled_height = orig.height

alternate = core.resize.Bicubic(orig, width=downscaled_width, height=downscaled_height, filter_param_a=b, filter_param_b=c)
alternate = core.resize.Bicubic(alternate, width=upscaled_width, height=upscaled_height, filter_param_a=0, filter_param_b=0.5)
zopti.run(orig, alternate)


Test clip here: https://drive.google.com/file/d/11GtJ1gfT-hStUDl8UmY3-fVJjBDXPD1M/view?usp=sharing

GMSD seems to work fine and the heatmap looks good.

Any ideas what is happening here? I'm using the latest Zopti and this time remembered to update the zoptilib.py in the site-packages folder :o, running on Vapoursynth R55 RC3. I had to fix mvsfunc.py a little bit to make it work, but it shouldn't affect the calculations at all.

[Selur]

What am I doing wrong? :)
It's my first time playing with Zopti and I'm trying to get Zopti running.

What I did so far:

download Zopti-1.2.0 put on my Desktop
download a portable java (OpenJDKJRE64_16.0.1_Build_9.paf) and install it into the Zopti folder
adjust zopti.bat to:
OpenJDKJRE64\bin\java.exe -jar Zopti.jar %*
so that it will find the java binary
adjust zopti.ini to:
log=
architecture=x64
vspipe=I:\Hybrid\64bit\Vapoursynth\VSPipe.exe
to point to VSPipe.exe (using a portable Vapoursynth).
adjusting zoptilib.py, by changing the start to:
# Imports
import os
import sys
import vapoursynth as vs
import time
import functools

# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))

import muvsfunc as muv
so that my scripts folder is known.

created a input.py:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("I:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
core = vs.core

# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")

orig = core.lsmas.LWLibavSource(r"g:\TestClips&Co\test.avi")
alternate = core.grain.Add(clip=orig)

from zoptilib import Zopti
output_file = r'results.txt' # output out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"
zopti = Zopti(output_file, metrics=['ssim', 'time'])
sigma = 4 # optimize sigma = _n_ | 1,10 | sigma
alternate = core.dfttest.DFTTest(alternate, sigma=sigma)
zopti.run(orig, alternate)



I then called:
zopti.bat input.py
which gave me:
C:\Users\Selur\Desktop\Zopti-1.2.0>OpenJDKJRE64\bin\java.exe -jar Zopti.jar input.py

+---------+
| |
+---+ | +---------+ +---------+ +---------+ +---+
/ / | | | | | | | |
/ / | +-+ | | +-+ | +--+ +--+ | |
/ / | | | | | | | | | | | |
| +---+ | +-+ | | +-+ | | | | |
| | | | | | | | | |
+---------+ +---------+ | +-----+ +---+ +---+
| |
+---+ version 1.2.0

Arguments

Running optimization for script input.py
Using these settings:
ARGUMENT DESCRIPTION VALUE
-runs runs 5
-alg algorithm spea2
-pop population 8
-initial initial population random
-iters iterations 2000
-mutamount mutation amount 0.3 0.01
-mutcount mutation count 60% 1
-crossprob crossover probability 0.1
-crossdist crossover distribution 20
-sensitivity sensitivity estimation true
-dynphases dynamic phases N/A
-dyniters iterations per phase N/A
-errors error handling stop
-priority process priority (default)
-threads number of threads to use 1
-timeout script execution timeout (seconds) (OFF)
-continue continue from aborted optimization (log file) (OFF)

parsing script output [out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"]
Output 1:
name SSIM
type FLOAT
goal MAX
unit
round 0
Output 2:
name time
type TIME
goal MIN
unit ms
round 10

Found following optimizable parameters:
# optimize sigma = _n_ | 1,10 | sigma
found 1 parameters to optimize

Number of parameter combinations is smaller than the population size - setting the population size to same (2)
Running SPEA2

Error in script execution:



Cannot load avisynth.dll:
Das System kann die angegebene Datei nicht finden.
(no clue why it's looking for avisynth.dll)

Then I called:
C:\Users\Selur\Desktop\Zopti-1.2.0>zopti.bat input.py -initial script
which resulted in:
C:\Users\Selur\Desktop\Zopti-1.2.0>OpenJDKJRE64\bin\java.exe -jar Zopti.jar input.py -initial script

+---------+
| |
+---+ | +---------+ +---------+ +---------+ +---+
/ / | | | | | | | |
/ / | +-+ | | +-+ | +--+ +--+ | |
/ / | | | | | | | | | | | |
| +---+ | +-+ | | +-+ | | | | |
| | | | | | | | | |
+---------+ +---------+ | +-----+ +---+ +---+
| |
+---+ version 1.2.0

Arguments
initial = script

Running optimization for script input.py
Using these settings:
ARGUMENT DESCRIPTION VALUE
-runs runs 5
-alg algorithm spea2
-pop population 8
-initial initial population script
-iters iterations 2000
-mutamount mutation amount 0.3 0.01
-mutcount mutation count 60% 1
-crossprob crossover probability 0.1
-crossdist crossover distribution 20
-sensitivity sensitivity estimation true
-dynphases dynamic phases N/A
-dyniters iterations per phase N/A
-errors error handling stop
-priority process priority (default)
-threads number of threads to use 1
-timeout script execution timeout (seconds) (OFF)
-continue continue from aborted optimization (log file) (OFF)

parsing script output [out1="SSIM: MAX(float)" out2="time: MIN(time) ms" file="results.txt"]
Output 1:
name SSIM
type FLOAT
goal MAX
unit
round 0
Output 2:
name time
type TIME
goal MIN
unit ms
round 10

Found following optimizable parameters:
# optimize sigma = _n_ | 1,10 | sigma
found 1 parameters to optimize

Number of parameter combinations is smaller than the population size - setting the population size to same (2)
Running SPEA2
Taking script's values into initial population
4 is not a valid value for parameter sigma


-> What am I missing / doing wrong?

Cu Selur

[Boulder]

At least this line needs a small correction:
sigma = 4 # optimize sigma = _n_ | 1,10 | sigma
should be
sigma = 4 # optimize sigma = _n_ | 1..10 | sigma

Then it will use range 1-10 with a step of 1. If you need a smaller step, you need to use a divide operator like in my previous post with Bicubic's b and c.

[Selur]

Thanks, using:
_n_ | 1..10 | sigma
I now get:
Found following optimizable parameters:
# optimize sigma = _n_ | 1..10 | sigma
found 1 parameters to optimize

Running SPEA2

Error in script execution:



Cannot load avisynth.dll:
Das System kann die angegebene Datei nicht finden.
in both cases. :/

Does Zopti need avisynth even when this is the Vapoursynth version?

Cu Selur

[Boulder]

I wonder if it internally uses avsr and needs Avisynth? It was originally an Avisynth application.

[zorr]

Does Zopti need avisynth even when this is the Vapoursynth version?

It's incorrectly determining that you're running an AviSynth script because the filename doesn't end with '.vpy'. I'll add '.py' for the next version. :)

[zorr]

I just noticed that MDSI seems to produce a bit erratic results.

I wasn't able to reproduce this, results from two runs were

39.561905 b=-84 c=64
39.569714 b=-90 c=59

and there are no wild results such as 1.27114432E8 (that's over 127 million) in the logs.

I'm still on Vapoursynth portable R53. Not sure what's going on, but could be related to the latest VapourSynth API revision.

[Selur]

It's incorrectly determining that you're running an AviSynth script because the filename doesn't end with '.vpy'. I'll add '.py' for the next version.
Thanks.

Next problem.
Now I get:
Error in script execution:
Script evaluation failed:
Python exception: No module named 'zoptilib'[/com]
My guess is that I would need to add:
[code]
import os
import sys
scriptPath = 'C:/Users/Selur/Desktop/Zopti-1.2.0'
sys.path.insert(0, os.path.abspath(scriptPath))

somewhere, since it seems like Zopti assumes the current dir is inside the Python discovery folder,...
-> okay, adding the above to my script does help,..

Now it's complaining that it needs tcanny.
-> adding that to my input script too helped
Now it's again about missing fmtc.
-> adding that to my input script too any I can finally run the script :)


Cu Selur

[Boulder]

I wasn't able to reproduce this, results from two runs were

39.561905 b=-84 c=64
39.569714 b=-90 c=59

and there are no wild results such as 1.27114432E8 (that's over 127 million) in the logs.

I'm still on Vapoursynth portable R53. Not sure what's going on, but could be related to the latest VapourSynth API revision.

Looks like it starts to happen at R54. I just tested R53, where it works fine but R54 has these strange values for results. I'll report it to the Vapoursynth thread.

[Myrsloik]

Looks like it starts to happen at R54. I just tested R53, where it works fine but R54 has these strange values for results. I'll report it to the Vapoursynth thread.

Can you make a simple test case. Like one R53 and one R54 portable vs folder zipped up with everything included and idiot proof instructions? I'm not sure I understand exactly how to reproduce.

[Boulder]

Can you make a simple test case. Like one R53 and one R54 portable vs folder zipped up with everything included and idiot proof instructions? I'm not sure I understand exactly how to reproduce.
First of all, you need Java 16, a portable version here if you don't want to install it:
https://portableapps.com/apps/utilities/OpenJDKJRE

This zip file contains those two Vapoursynth portable versions with Zopti related files extracted inside the folders:
https://drive.google.com/file/d/1YFtkr612IdvauiRBe57Ii_DE-qxA7ebJ/view?usp=sharing

The clip to run Zopti on is here:
https://drive.google.com/file/d/11GtJ1gfT-hStUDl8UmY3-fVJjBDXPD1M/view?usp=sharing

And the command line:
zopti "strangerthings_s02e01.vpy" -alg mutation -iters dyn -dyniters 30 -dynphases 3 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01 -initial script

You need to edit zopti.ini to point to the vspipe.exe path.

Hope the packages work as intended :)

If the problem lies in muvsfunc, it could probably be replicated by checking the MDSI score it returns for the same frame in different VS versions. Didn't do that yet.

EDIT: just posted in the VS thread since this can be reproduced in an easier way.

[zorr]

Now it's complaining that it needs tcanny.
-> adding that to my input script too helped
Now it's again about missing fmtc.
-> adding that to my input script too any I can finally run the script :)

Zoptilib.py has a helpful comment about the requirements: Requirements: muvsfunc (for SSIM, GMSD and MDSI), VapourSynth-VMAF (for VMAF), VapourSynth-butteraugli (for Butteraugli)

I wasn't aware that it needs tcanny though. :confused:

[Selur]

I wasn't aware that it needs tcanny though.
Probably came with an updated muvsfunc version.

[poisondeathray]

Can zopti help with selection on a per frame basis ?

eg. 4 versions of interpolate video, RIFE 2.4, 3.1, 3.4, 3.8 . On some frames maybe 2.4 "solves" it better, maybe 3.8 on others, you get the idea...

You can stackvertical/horizontal and manually choose with trim or remapframes, but can zopti help automatic this task ?

[zorr]

Can zopti help with selection on a per frame basis ?

eg. 4 versions of interpolate video, RIFE 2.4, 3.1, 3.4, 3.8 . On some frames maybe 2.4 "solves" it better, maybe 3.8 on others, you get the idea...

You can stackvertical/horizontal and manually choose with trim or remapframes, but can zopti help automatic this task ?

Almost.

0 1 2 3 time (frames)
----------------------------------
[A ] [B ] [C ] [D ] original frames
[A ] [AB] [B ] [BC] [C ] [CD] [D ] 1) interpolate new frames (RIFE)
[AB] [BC] [CD] 2) remove original frames (SelectOdd)
[AB] [BB] [BC] [CC] [CD] 3) interpolate new frames again (RIFE)
[BB] [CC] 4) leave only latest interpolated frames (SelectOdd)

This is the basic structure for evaluating interpolated frames. In the end you can compare [BB] with [B] and [CC] with [C] etc. Let's say you calculate [AB], [BC] and [BB] with 4 different RIFE versions. You can now find out which one created the best match between [BB] and [B]. The only problem here is that [BB] was calculated from two interpolated frames [AB] and [BC], and [BC] will also have effect on the next frame [CC]. So if RIFE v1 was the best choice for [BB] and RIFE v2 was the best choice for [CC] then we don't know which one should be chosen for the interpolated frame [BC]. But it could be good enough to use the best choice from [BB] to select frame [AB] and the best of [CC] to select frame [BC] etc. Did I make any sense?

EDIT:
Now that I think about it, we could also use both [BB] vs [B] and [CC] vs [C] to select interpolated frame [BC]. The contribution of [BC] to [BB] and [CC] is 50% while [AB] and [CD] only contribute 25% each. Selecting the best RIFE version would be done with formula (assuming GMSD similarity metric) GMSD(BB, B) + GMSD(CC,C).

[zorr]

Here's a new release, it contains the following goodies (including the .py support for Selur :))

new option: retry
-tries to run the script again this many times if execution fails
-can be useful if the script uses plugins which are not 100% reliable and can crash
-default value is zero (do not retry)
-example: zopti script.avs -pop 24 -iters 10000 -retry 4
new visualization mode: line
-draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
-accepts parameter -range to limit the displayed values to certain range
-black line shows the best values, blue line indicates the number of results per value
-a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
-example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
support for custom output properties written by the script (only in VapourSynth scripts for now)
better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
option -continue now also supported by exhaustive algorithm (when using more than one thread)
order of runs in a visualization is now based on file name, not by modified date
all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
visualization mode seriespareto no longer contains the global pareto line
VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
bugfix: fixed a memory leak in XChart (at least partially)


Here's what -vismode line looks like:

https://i.postimg.cc/mDb5mW9t/lambda.png

Download link updated at first post.

[zorr]

This release introduces a better way for Avisynth scripts to write the results (VS version doesn't need this change as it's already very reliable) and fixes a couple of bugs. I recommend the upgrade especially if you're using more than one thread, there was a rare but possible bug which could result in flawed results.


supports a new way for an Avisynth script to write the result file
-first line is the number of frames (and also the number of actual result lines to be written), use for example WriteFileStart(resultFile, "FrameCount()")
-then come the per frame result lines in any order, using the same format as before
-do not write the last line which starts 'stop', it's no longer necessary and might cause trouble if used with the new way
-this also means that calculating the sum of similarity metric or time is not needed, Zopti calculates those instead
-old method still works as well, Zopti differentiates between them by looking at the first line (if it's a positive integer -> new way)
-new way is safer as the last 'stop' line is no longer needed (it has the be the last line if used and that is hard to guarantee especially when the script is executed multithreaded)
bugfix: when using -threads > 1 it was possible for two threads to use the same result file which may have resulted in reporting invalid result for one thread or the same result for both threads
bugfix: -vismode line always drew the minimum result per parameter value even when the goal was to maximize the result value

Download link updated at first post.

[zorr]

Zopti v1.2.3 is ready. This one has a new conversion feature to help importing Zopti data into other software. Also improvements to -vismode line and some other minor things. Details below:

new mode: convert
-converts Zopti log file into other formats for easier importing into other software such as Excel or Google Docs
-writes a new file and doesn't remove the original
-does not overwrite an existing file (error message given in such situation)
-supports sorting of the rows by result or parameter value using the option -sort, for example "-sort GMSD" sorts by GMSD
-sort direction can be specified by adding a + or - after the field name, for example "-sort time-" sorts by time, descending
-ascending direction (+) is the default if no + or - is given
-multiple sort criterias supported, for example "-sort dct GMSD-" sorts by dct (ascending) and the rows with the same dct value by GMSD (descending)
-output format: CSV
-specified with -format csv or by not giving format option
-target file name is the same as the original .log file but file ending changes from .log to .csv
-csv file includes title row with the names of result values and optimized parameters
-can be opened directly (for example) in Excel
-uses the default decimal format of the OS with floating point values
-output format: zopti
-specified with -format zopti
-keeps the original Zopti format and header rows, only sorts the results when option -sort is given
-filename is "converted " + original file name
-example: zopti -mode convert -format csv -sort dct time (sorts the latest log file by parameter dct and result time and writes a new file in csv format)
-example: zopti -mode convert -log "./path/abc run-01.log" (converts file "./path/abc run-01.log" into a new csv file)
improvements to visualization mode: line
-now supports option -types which controls which aggregate lines are displayed in the chart
-valid values are "best", "worst", "average", "median", "count" and "samples"
-best = best result per parameter value
-worst = worst result per parameter value
-average = average result per parameter value
-median = median result per parameter value
-count = number of samples per parameter value
-samples = display all results as small dots
-separate multiple values with space, for example "-types best average"
-default value is "best count" which will display the best value per parameter value and the number of samples per parameter value
-a legend is shown if more than one type is selected
display of option -groupby improved, pareto fronts can no longer be cut off the screen
option -continue also supported with mutation algorithm
-but only for the generation of the initial random population
-useful if you only want to test independent random samples with a large population
validation mode behavior change: it now validates all the results in the log file and not just the pareto front

[ChaosKing]

Not sure if Zopti supports PSNR yet, but there's a plugin for that now :) https://github.com/AmusementClub/vs-ComparePlane