Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 15th April 2021, 23:55   #221  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Zopti v1.1.0 released

A new major release is here! Zopti now supports executing multiple scripts simultaneously which can lead to very nice performance improvement. This can be very useful when the script itself is not able to utilize all of your cores.

Here's the full release info:
  • added support for multithreaded execution of scripts with argument -threads
    -default value of -threads is 1 meaning one script is executed at a time just like before
    -all optimization algorithms (nsga-ii, spea2, mutation, exhaustive) support the the -threads option
    -due to the nature of the heuristic algorithms (nsga-ii, spea2, mutation) there can be at most P threads running the scripts at the same time, where P is the population size
    -exhaustive algorithm has no thread limits
    -validate mode also supports -threads. there the maximum number of threads is the number of validated results (size of the pareto front)
    -note: using more than one thread will make time measurements less accurate but will not change the quality measurements
  • avsr version update (v0.2.2)
  • added jMetal.log.ini to disable multithreading related JMetal logging
  • updated Zoptilib to version 1.0.9m
  • bugfix: evaluate could not parse log files when Zopti was used

Download link is at the first post. I will shortly post some data on the multithreading performance.
zorr is offline   Reply With Quote
Old 16th April 2021, 00:32   #222  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
I tested the new -threads argument using the denoising tutorial script, this time using GMSD as the similarity metric and using 50 frames instead of 5. Here's the script:

Code:
SetCacheMode(0)

TEST_FRAMES = 50		# how many frames are tested
MIDDLE_FRAME = 100		# middle frame number

RawSourcePlus("D:/optimizer/test/flower/flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)

#denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8)		# best settings by Fizick

sigma = 400/100.0 	# optimize sigma = _n_/100.0 | 400..600 ; filter:x 5 % 0 == | sigma
bt = 5			# optimize bt = _n_ | 3,5 | blockTemporal
blockSize = 14		# optimize blockSize = _n_ | 6,12 ; min:overlap 2 * | blockSize
overlap = 16		# optimize overlap = _n_ | 4..6 ; max:blockSize 2 / | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

last = GMSD(source, denoised, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt"	# output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
    current_frame == frame_count-1
}, function() {
	gmsd = 0.0
	str = ""
	for (i = 0, frame_count-1) {
		value = propGetFloat("_PlaneGMSD", offset = -i)
		gmsd = gmsd + value
		if (i>0) { str = str + e"\n" }
		str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
	}
	return str + e"\nstop " + string(gmsd)
}, append=false)

return last
There's also a VapourSynth version of the script which I posted to the VapourSynth Zopti thread.

The script only has 246 valid parameter combinations to test so we can try all of them using
Code:
zopti denoise_ex.avs -alg exhaustive -threads 1
It takes 179 seconds to test all of the combinations using one thread. Let's see the performance when using more threads:



The best result is about 20 seconds using 16 threads (on a Ryzen 3900X which has 12 cores / 24 threads).

Another way to look at the scaling is to calculate how much faster we get the results when using -threads:



Using 16 threads is 8,6 times as fast as using only one thread. Of course this is just one data point and I don't mean to imply that you can always get such a performance improvement. I would have liked to include more tests using more real-world usage scenarios (HD source etc) but my stock cooled processor cannot handle those for more than a few seconds, it becomes so hot that my PC shuts down.

Comparing AviSynth and VapourSynth it looks like VS has the edge when using one or just a few threads, but the differences almost vanish when using 12 or more threads. The VS is also able to utilize more than 16 cores while with AVS the performance starts to degrade. The point where more threads are just slowing down is probably dependent on the specific script, I will have to run more tests on that.

Last edited by zorr; 16th April 2021 at 00:58.
zorr is offline   Reply With Quote
Old 16th April 2021, 05:14   #223  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Thanks zorr, looks like a very handy approach. I'll try testing it over the weekend.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 4th May 2021, 10:32   #224  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Let's say I want to do compression/quality tests on 1-3% of a whole movie, changing the parameters on SMDegrain in a fixed range, mostly tr and thSAD.

Is there a way to batch that process? Usually I have to manually set a frame range and have multiple avs scripts.

Can you help me?

My usual script, with the parts I want to have permutations in red:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource(something)

SMDegrain (tr=4, thSAD=400, refinemotion=false, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Prefetch(6)
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 4th May 2021, 21:42   #225  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Quote:
Originally Posted by tormento View Post
Let's say I want to do compression/quality tests on 1-3% of a whole movie, changing the parameters on SMDegrain in a fixed range, mostly tr and thSAD.

Is there a way to batch that process? Usually I have to manually set a frame range and have multiple avs scripts.
How do you evaluate the quality? Do you have a "reference" video you could compare the different settings with? Zopti needs some way to tell the quality and usually that means comparing two videos.

If you use your own eyes to measure the quality and just want to generate the different versions... well, Zopti doesn't output any video so it can't help you there. But it could generate you the different script variations if you provide a file with the parameter combinations you want, like this
Code:
1 tr=4 thSAD=40
1 tr=5 thSAD=40
1 tr=4 thSAD=50
etc.
zorr is offline   Reply With Quote
Old 5th May 2021, 11:02   #226  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by zorr View Post
How do you evaluate the quality?
I don't need automatic quality comparison, just a way to automate the changes in the script.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 28th May 2021, 21:35   #227  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Zopti v1.2.0 released

Time for a new update! This version has some major additions so I bumped the version to 1.2.0.
  • new option: -timeout (default is disabled). if the execution of Avisynth script takes longer than timeout seconds
    -the script execution process (and subprocesses) will be terminated
    -the result is set to worst possible as a penalty so it will not be considered as a valid candidate
    -useful with scripts where certain parameter combinations can result in execution times several hundred times slower than usual
    -example: zopti script.avs -pop 24 -iters 1000 -timeout 60
  • new option: -continue
    -aborted optimization can be continued from the last complete generation (one generation is N iterations, where N is the population size)
    -give the log file name of the aborted optimization as the value of continue
    -old results up to last complete generation are copied to a new log file and optimization will continue from there
    -useful if optimization was unexpectedly aborted or the CPU resources are needed for something else for a while
    -example: zopti script.avs -pop 24 -iters 10000 -continue "mvtools spea2 run-01.log"
    -NOTE: it is possible to change the optimization parameters (population, mutation count etc.) from those used in the continued log
    -NOTE: random seed is not restored so the results will not be identical to what they would have been if execution had not been aborted
    -NOTE: only works with algorithm spea2 (for now)
  • new visualization mode: history
    -displays the best found value and all tried values of certain optimized parameter (given with -param)
    -invalid results (script execution has failed OR timed out) are highlighted in orange
    -also shows the phase of dynamic iteration (if applicable)
    -accepts parameter -range to limit the displayed history to certain iteration range
    -example: zopti -mode evaluate -vismode history -param lambda (displays all history of parameter "lambda")
    -example: zopti -mode evaluate -vismode history -param lambda -range 1000 5000 (displays history from iteration 1000 to iteration 5000)
    -example: zopti -mode evaluate -vismode history -param lambda -range -10000 (displays history of the last 10000 iterations)
  • argument -dynphases can now be zero (previously minimum was 1)
  • pareto front and total runtime is displayed also after mutation or exhaustive algorithm is finished
  • if population is based on a log file and the pareto front is smaller than the population size, the population will be filled with mutations of the pareto front instead of completely random combinations
  • if a log file for the optimization already exists it will be cleared in the beginning of the optimization
  • XChart updated to version 3.8.0 with some customization

And here's what -vismode history looks like:



The black thick line shows the best found value of the chosen parameter. The red diamonds indicate a point where a new best result has been found and this parameter's value was changed. The white diamonds indicate a new best result but this parameter's value remained the same.

In the example optimization was finished and then restarted with different settings at around 190 000 iterations.

The -timeout was needed when I started running MVTools2 tests with all the parameters and it turned out that some combinations took an enormous amount of time to finish (typical time was 5 seconds but some combinations took over 30 minutes...). Using a timeout of 60 seconds solved that problem.

-continue is useful if you want to keep optimizing a result that is finished, you can try different settings. Also very handy when the optimization has been aborted by hardware failure or something like Windows update... you can just restart it using the same settings. This also makes it possible to run huge iteration counts little by little even if you need/want to use the computer for other things once in a while.

Download link updated at first post.
zorr is offline   Reply With Quote
Old 13th September 2021, 13:04   #228  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Hey thanks a lot for the tool. I plan to use it now heavily, is it possible to combine it with VMAF? I tweaked your GMSD client script but VMAF doesn't seem to write frameProps, not at least as _PlaneVMAF.
By the way, if you need to combine/merge arrays (didn't read what for) I made this function (you need to explicitly declare the args in the call):

Code:
function ArrayAdd( val_array "a", val_array "b") {

    as = ArraySize(a)-1
    bs = ArraySize(b)-1
    na = ""
    for (i = 0, as+bs+1, 1) {

        o = i - as - 1
        cm = i != as+bs+1 ? "," : ""
        na = na + ( i > as ? String(Eval(Format("b[{o}]"))) : \
                             String(Eval(Format("a[{i}]")))) + cm
      }

    return Eval("["+na+"]") }
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 13th September 2021, 23:25   #229  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Quote:
Originally Posted by Dogway View Post
Hey thanks a lot for the tool. I plan to use it now heavily, is it possible to combine it with VMAF? I tweaked your GMSD client script but VMAF doesn't seem to write frameProps, not at least as _PlaneVMAF.
VMAF writes its own result file (xml) and Zopti can read it directly so there's no need to read any frame props and save them to file. I haven't actually tested the Avisynth VMAF version but it should work if you set the log_path same as the Zopti file= parameter and log_fmt=0. Also set pool=0 to be consistent with the pooling method of other similarity metrics.

But before you start using it you might want to check these tests I conducted (there's more tests in that thread) about the different similarity metrics, seems like VMAF is not very consistent and sometimes gives downright bad results.

Quote:
Originally Posted by Dogway View Post
By the way, if you need to combine/merge arrays (didn't read what for) I made this function (you need to explicitly declare the args in the call):
Thanks, might be useful some day. Arrays were considered as a solution to synchronization problem when using Prefetch() and the script tries to write the results of each frame to the same file. Reading the frame props in the end solved that problem, but I think it's not the optimal way (I think sometimes the frame props are no longer in cache when they are requested so some frames could be calculated twice).

Last edited by zorr; 13th September 2021 at 23:57. Reason: Added VMAF parameter instructions
zorr is offline   Reply With Quote
Old 14th October 2021, 23:33   #230  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
zorr, I'm trying to come up with a self resolving BBSIM function, I have troubles with SSIM_FRAME(), not only it requires 8-bit but it's output value isn't affected by the filtered clip. I tried to implement reading _PlaneSSIM property if created with vsSSIM but its output value is very different from SSIM_FRAME().

This is my attempt at normalizing PlaneSSIM to SSIM_FRAME (by /335 and then 2*bssim).
Code:
function BSSIM(clip source, clip filtered, bool "PlaneSSIM", bool "show", bool "fulls") {

    show = Default(show,      false)
    PS   = Default(PlaneSSIM, false)
    fs   = Default(fulls,     false)

    blurred        = source.ex_blur(1.5*2,bifit=true,UV=1)

    source8        = PS ? source  :  source.ConvertBits(8, dither=-1, fulls=fs)
    blurred8       = PS ? blurred : blurred.ConvertBits(8, dither=-1, fulls=fs)

    scharr_orig    = ex_edge(source8,  "pscharr",0,255)
    scharr_blurred = ex_edge(blurred8, "pscharr",0,255)

    source         = PS ?         source.ConvertBits(32, fulls=fs) : source8
    blurred        = PS ?        blurred.ConvertBits(32, fulls=fs) : blurred8
    scharr_orig    = PS ?    scharr_orig.ConvertBits(32, fulls=fs) : scharr_orig
    scharr_blurred = PS ? scharr_blurred.ConvertBits(32, fulls=fs) : scharr_blurred

    ScriptClip(source, function [source, blurred, scharr_orig, scharr_blurred, PS, show] () {

        SSIM        = PS ? propGetFloat(source,"_PlaneSSIM")/335 : \
                           SSIM_FRAME(source, blurred)
        stddev_orig = RT_YPlaneStdev(scharr_orig)
        stddev_alt  = RT_YPlaneStdev(scharr_blurred)

        bssim = SSIM * (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)
        bssim = PS ? 2*bssim : bssim

        propSet("_PlaneBSSIM", bssim, 0)

        return show ? Subtitle("PlaneBSSIM: " + String(bssim)) : last } ) }
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 15th October 2021, 00:05   #231  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Quote:
Originally Posted by Dogway View Post
I have troubles with SSIM_FRAME(), not only it requires 8-bit but it's output value isn't affected by the filtered clip.
If you mean the input argument clip filtered then that's not a mystery, it's not used in the function at all.

Quote:
Originally Posted by Dogway View Post
I tried to implement reading _PlaneSSIM property if created with vsSSIM but its output value is very different from SSIM_FRAME().
There are some differences in SSIM implementations. I don't think it's that important to try to match the different implementations. The important aspect of B-SSIM is just the multiplier which scales the result based on how blurry the clip is. Although there might be a small problem with the presented algorithm as SSIM (at least in some implementations) has a range of -1 to 1 and it would scale the negative results in the wrong direction.
zorr is offline   Reply With Quote
Old 15th October 2021, 10:02   #232  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
haha, yes didn't notice. I thought 'blurred' was a preprocessing.
Well, I think this will do it. (EDIT: I need to fix vsSSIM as I borked it when optimizing it)

Code:
function BSSIM(clip source, clip blurred, bool "PlaneSSIM", bool "show", bool "fulls") {

    show = Default(show,      false)
    PS   = Default(PlaneSSIM, false)  # Set to True if the clip has the _PlaneSSIM frame property
    fs   = Default(fulls,     false)

    source8        =  source.ConvertBits(8, dither=-1, fulls=fs)
    blurred8       = blurred.ConvertBits(8, dither=-1, fulls=fs)

    scharr_orig    = ex_edge(source8,  "pscharr",0,255)
    scharr_blurred = ex_edge(blurred8, "pscharr",0,255)

    ScriptClip(source, function [source, source8, blurred8, scharr_orig, scharr_blurred, PS, show] () {

        SSIM        = PS ? propGetFloat(source,"_PlaneSSIM") : \
                           SSIM_FRAME(source8, blurred8) # only works on 8-bit
        stddev_orig = RT_YPlaneStdev(scharr_orig)        # only works on 8-bit (output is also 8-bit)
        stddev_alt  = RT_YPlaneStdev(scharr_blurred)     # only works on 8-bit (output is also 8-bit)

        bssim = SSIM * (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)

        propSet("_PlaneBSSIM", bssim, 0)

        return show ? Subtitle("PlaneBSSIM: " + String(bssim)) : last } ) }
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 15th October 2021 at 10:19.
Dogway is offline   Reply With Quote
Old 17th January 2022, 14:49   #233  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
I'm running some tests for motion vectors and I have a few questions.
-How do you terminate the process? Normally I 'ctrl+c' on the console but then I have to kill each avsr64.exe process
-What is the best approach to simply run a bunch of random iterations without any refinement? I simply used low iterations with high number of runs, but 'runs' doesn't allow multithreading, and 'timeout' finishes the whole thing. The problem here is that some settings have more weight than others so it can bias the refinement so I want to give equal probability to all of them to draw some conclusions (ie. in Excel), at least for the initial tests.
Code:
SET        RUNS= 4
SET         ALG= mutation
SET  POPULATION= 4
SET     INITIAL= random
SET       ITERS= dyn
SET   MUTAMOUNT= 0.5 0.01
SET    MUTCOUNT= 60% 1
SET   CROSSPROB= 0.1
SET   CROSSDIST= 20
SET SENSITIVITY= TRUE
SET   DYNPHASES= 1
SET    DYNITERS= 8
SET      ERRORS= STOP
SET    PRIORITY=
SET     THREADS= 4
SET     TIMEOUT= 410
SET    CONTINUE=

java -jar Zopti.jar "..\Zopti_test.avs" ^
-alg %ALG% -initial %INITIAL% -iters %ITERS% -dyniters %DYNITERS% -dynphases %DYNPHASES% -pop %POPULATION% -runs %RUNS% -mutcount 1 -mutamount %MUTAMOUNT% -timeout %TIMEOUT% -threads %THREADS%
I don't quite understand the settings marked in red though. Maybe they are related to what I want to do.

-Also I tried to kill some iterations as soon as some frames are rendered longer than necessary so I played a bit with WriteFileIf() but didn't find what keyword is for timeout. Tested with:
Code:
WriteFileIf(resultFile, "avstimer > 4500.0", """ 0.0 9999999"""+Chr(10)+""" "stop " """, append=true)
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 17th January 2022, 23:09   #234  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Quote:
Originally Posted by Dogway View Post
-How do you terminate the process? Normally I 'ctrl+c' on the console but then I have to kill each avsr64.exe process
Currently you cannot easily terminate the process (and all the subprocesses that were started by Zopti). I'll have to look if I can implement a way to detect the termination and shut down the subprocesses as well.

Quote:
Originally Posted by Dogway View Post
-What is the best approach to simply run a bunch of random iterations without any refinement? I simply used low iterations with high number of runs, but 'runs' doesn't allow multithreading, and 'timeout' finishes the whole thing.
The best approach (with no bias) is to run with a high population count and the same number of iterations, one run only. That way all the iterations are random and not refined in any way. Also there will be no duplicate parameter combinations, Zopti takes care of that.

Quote:
Originally Posted by Dogway View Post
The problem here is that some settings have more weight than others so it can bias the refinement so I want to give equal probability to all of them to draw some conclusions (ie. in Excel), at least for the initial tests.
I see what you mean. But it's also difficult to make any conclusions in that early phase of the optimization, unless you only have a couple of parameters you want to vary. With one or two parameters your best bet is to run exhaustive search, you can then make nice visualizations from those. Zopti has a heatmap visualization for two parameters, for example. And exhaustive search can also be "sparse" in a way that you test for example only every 10th value.

Quote:
Originally Posted by Dogway View Post
Code:
SET        RUNS= 4
SET         ALG= mutation
SET  POPULATION= 4
SET     INITIAL= random
SET       ITERS= dyn
SET   MUTAMOUNT= 0.5 0.01
SET    MUTCOUNT= 60% 1
SET   CROSSPROB= 0.1
SET   CROSSDIST= 20
SET SENSITIVITY= TRUE
SET   DYNPHASES= 1
SET    DYNITERS= 8
SET      ERRORS= STOP
SET    PRIORITY=
SET     THREADS= 4
SET     TIMEOUT= 410
SET    CONTINUE=
...
I don't quite understand the settings marked in red though. Maybe they are related to what I want to do.
This is the official "reference post" of the optimizer arguments. It doesn't have the later additions though but covers the ones you needed clarification about.

Quote:
Originally Posted by Dogway View Post
-Also I tried to kill some iterations as soon as some frames are rendered longer than necessary so I played a bit with WriteFileIf() but didn't find what keyword is for timeout. Tested with:
Code:
WriteFileIf(resultFile, "avstimer > 4500.0", """ 0.0 9999999"""+Chr(10)+""" "stop " """, append=true)
If you're playing with MVTools there are some parameter combinations that can take a very, very long time before they even return the first frame. The -timeout argument was implemented just for that purpose, to stop the execution of the AviSynth script if it takes longer than the specified time. The termination is immediate as it simply kills the whole process and can start executing another one instead.
zorr is offline   Reply With Quote
Old 19th January 2022, 22:56   #235  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Zopti v1.2.1 released

I implemented a shutdown hook to kill all the running subprocesses if Zopti is terminated (thanks for the idea Dogway). I had some other changes in the pipeline as well, so here's a new release:
  • new option: retry
    -tries to run the script again this many times if execution fails
    -can be useful if the script uses plugins which are not 100% reliable and can crash
    -default value is zero (do not retry)
    -example: zopti script.avs -pop 24 -iters 10000 -retry 4
  • new visualization mode: line
    -draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
    -accepts parameter -range to limit the displayed values to certain range
    -black line shows the best values, blue line indicates the number of results per value
    -a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
    -example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
  • a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
  • mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
  • reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
  • scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
  • support for custom output properties written by the script (only in VapourSynth scripts for now)
  • better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
  • option -continue now also supported by exhaustive algorithm (when using more than one thread)
  • order of runs in a visualization is now based on file name, not by modified date
  • all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
  • visualization mode seriespareto no longer contains the global pareto line
  • VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
  • bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
  • bugfix: fixed a memory leak in XChart (at least partially)

Here's what -vismode line looks like:



Download link updated at first post.
zorr is offline   Reply With Quote
Old 20th January 2022, 13:36   #236  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Thanks for the update!!

I got some issues with sensitivity option. Probably my optimization range in the script wasn't too broad or something and I was getting this error message after DYNITERS iterations using "mutation" as algo, and "dyn" as iterations.

Code:
Parameter sensitivity estimation with 256 result combinations
java.lang.ArrayIndexOutOfBoundsException: Index -700 out of bounds for length 23
00
        at avisynthoptimizer.Parameter.updateAverages(Parameter.java:954)
        at avisynthoptimizer.Parameter.updateSensitivityEstimation(Parameter.jav
a:970)
        at avisynthoptimizer.SensitivityEstimation.estimateParameterSensitivity(
SensitivityEstimation.java:102)
        at avisynthoptimizer.AviSynthOptimizer.optimize_mutation_multithreaded(A
viSynthOptimizer.java:4726)
        at avisynthoptimizer.AviSynthOptimizer.main(AviSynthOptimizer.java:620)
ERROR: Index -700 out of bounds for length 2300

C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0>rem ERROR: Series name
 >phase 1,00< has already been used. Use unique names for each series!!!

By the way on another note I was trying to evaluate motion vectors as a general case without the intervention of client functions like MFlowFPS, or MDegrain by using the SAD mask in MMask (kind=1) and using the average value as the frame metric. Well this didn't yield good results, it tends to optimize to a large searchRange while for example MDegrain with a searchRange over 2 smears the content. So first I'm locking searchRange to 2 and optimizing for MDegrain with the MDegrain client.
Also most of the hardest settings to optimize are those of truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global... they show bad correlation overall.


Here's my Excel chart for the first randomized data samples. You can easily see that any other than DCT=0 is slow and bad quality, so you can remove those entries and continue guessing settings that can be locked down. Ideally one would run some multivariate analysis but I'm still studying statistics.

Google Spreadsheets
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 21st January 2022 at 15:40.
Dogway is offline   Reply With Quote
Old 22nd January 2022, 00:12   #237  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Quote:
Originally Posted by Dogway View Post
I got some issues with sensitivity option. Probably my optimization range in the script wasn't too broad or something and I was getting this error message
That looks intriguing, can I take a look at your script so that I can try to replicate this error?

Quote:
Originally Posted by Dogway View Post
By the way on another note I was trying to evaluate motion vectors as a general case without the intervention of client functions like MFlowFPS, or MDegrain by using the SAD mask in MMask (kind=1) and using the average value as the frame metric. Well this didn't yield good results
Yeah you have to be careful what you optimize for. If there's a way to cheat a good result Zopti most likely finds a way to do that. So it's always a good idea to only optimize for the final result and not some intermediate metric.

Quote:
Originally Posted by Dogway View Post
Also most of the hardest settings to optimize are those of truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global... they show bad correlation overall.
Most of the settings are only relevant after the most important ones are found and the search starts to focus.

Note that you don't need truemotion if you set lambda, lsad, pnew, plevel and global in your script as those values will overwrite whatever truemotion sets them to.

Quote:
Originally Posted by Dogway View Post
Here's my Excel chart for the first randomized data samples. You can easily see that any other than DCT=0 is slow and bad quality
I wouldn't be so sure. Maybe you've already seen this but I did some pretty extensive tests with MVTools frame interpolation and found that dct=1 was consistently best, it was the winner in three test runs with at least 100000 iterations each. It could very well be that dct=0 is the best for MDegrain and also it may have more to do with the specific video, as even I haven't done tests with multiple videos to find out which parameters are consistently best with most source material (or perhaps none are, or perhaps it depends on the resolution or content type).

In any case 100 tests is not enough to say anything sure. I try to make conclusions on the best value only after Zopti has consistently selected it as the best one, after which it can be locked down and another tests can focus on finding the best values for the remaining unlocked parameters. MVTools has so many parameters that it's a real challenge to optimize them due to enormous search space, that's why it has to be done meticuously and using large iteration counts.

Quote:
Originally Posted by Dogway View Post
Ideally one would run some multivariate analysis but I'm still studying statistics.
That sounds interesting, let me know if you find a good method to analyze the results.
zorr is offline   Reply With Quote
Old 22nd January 2022, 10:07   #238  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
The error didn't happen to me in my last edit, I think it was related to the parameter range AND the "filter:" used. It will happen again I guess so I will post the offending script here.

Currently I'm benchmarking MDegrain since MFlowFPS was a bit too complex and as you say DCT=1 was the best. DCT=1 and searchAlgo=3 are the slowest settings in MAnalyse aside of a high searchRange, so it would be difficult to find a pareto front with them. In the case of MDegrain I don't feel they are critical (did some runs) so I exclude them.

Quote:
Note that you don't need truemotion if you set lambda, lsad, pnew, plevel and global in your script as those values will overwrite whatever truemotion sets them to.
Oh I though it was the other way around, truemotion overriding those settings.

I don't know how you can run so many iterations, I'm testing with 150 frames and as you can see it takes about 2 to 5 minutes per iteration (taking out DCT=1 and search=3). I run 8 scripts at the same time each for a thread on my CPU.
What I did, instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain.

Here's my reference clip (a sample from @tormento).
Code:
setmemorymax(2048)
DGSource("brazil.dgi",cl=0,ct=40,cr=0,cb=40)

ConvertBits(16,fulls=true)

pre=smdegrain(tr=1,mode="temporalsoften",blksize=16,thSAD=900,LFR=400,prefilter=3,DCTFlicker=true,contrasharp=false,refinemotion=true).ex_unsharp(0.5, 1920.0/8, 0.0)
smdegrain(tr=2,mode="MDegrain",blksize=16,prefilter=pre,thSAD=400,LFR=200,contrasharp=true,refinemotion=true)
ConvertBits(8,dither=1)
And this is my Zopti script with refined ranges after about 400 random iterations, and a few locked ones after a week of tests:
Code:
setmemorymax(16384/8)
DGSource("brazil-ref.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=last
DGSource("brazil.dgi",cl=0,ct=40,cr=0,cb=40)

AssumeFPS(24000,1001)
RequestLinear(30)

thSAD = 285          # optimize thSAD = _n_ | 230..290 ; filter:x 5 % 0 == | thSAD
thSADC = 200         # optimize thSADC = _n_ | 140..250 ; filter:x 10 % 0 == | thSADC
thSADR = 280         # optimize thSADR = _n_ | 150..300 ; filter:x 10 % 0 == | thSADR
BlkSize = 16
BlkSizeR = 8
overlap = 4
overlapR = 4
pel = 1
sharp = 2            # optimize sharp = _n_ | 2 | sharp
scaleCSAD = 2
trymany = false
truemotion  = false  # optimize truemotion  = _n_ | true,false | truemotion
truemotionR = true   # optimize truemotionR = _n_ | false,true | truemotionR
temporal = true      # optimize temporal = _n_ | true,false | temporal


# Settings that depend on source noise
# lsad  the noisier the lower
# sigma the noisier the higher
# badSAD

# TRUEMOTION SETTINGS
lambda = 440         # optimize lambda = _n_ | 400..900 ; filter:x 10 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 1320       # optimize lambdaR = _n_ | 960..2000 ; min:lambda ; max:lambda 3 * ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114           # optimize pnew = _n_ | 100..200 ; filter:x 2 % 0 == | pnew
pnewR = 136          # optimize pnewR = _n_ | 100..250 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 100          # optimize pzero = _n_ | 28..100 ; filter:x 2 % 0 == | pzero
lsad = 6000          # optimize lsad = _n_ | 1000..8000 ; filter:x 100 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79          # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1              # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 8          # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2         # optimize badrange = _n_ | 0..50 ; filter:x 2 % 0 == | badrange
badSAD = 1350        # optimize badSAD = _n_ | 1100..2200 ; filter:x 50 % 0 == | badSAD
dct = 0              # optimize dct = _n_ | 0 | dct
dctre = 9            # optimize dctre = _n_ | 0,2,3,6,7,9 | dctre
rfilter = 3
searchAlgo = 1       # optimize searchAlgo = _n_ | 1 | searchAlgo
searchAlgoR = 4      # optimize searchAlgoR = _n_ | 0,1,2,4,5 | searchAlgoR
searchRange = 15     # optimize searchRange = _n_ | 2..17 | searchRange
searchRangeR = 13     # optimize searchRangeR = _n_ | 2..18 | searchRangeR
searchRangeFinest = 17# optimize searchRangeFinest = _n_ | 9..20 | searchRangeFinest
sglobal = true       # optimize sglobal = _n_ | true,false | sglobal

trim(1,12)+\
trim(60,96)+\
trim(173,285-12)

C=ConvertBits(16,fulls=false)
pre=C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true


superfilt = MSuper(pre, hpad=16, vpad=16,             sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
superR    = MSuper(C,   hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false, chroma=false)
superRe   = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
bak2 =            MAnalyse(superfilt, isb=true,  delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 =            MAnalyse(superfilt, isb=true,  delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 =            MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 =            MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe,  bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD,  pnew=pnewR, thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : bak
bak1 = Recalculate ? MRecalculate(superRe,  bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD,  pnew=pnewR, thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : bak
fwd1 = Recalculate ? MRecalculate(superRe,  fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR , thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe,  fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR , thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=true)

global frame_count = FrameCount()

GMSD(src.ConvertBits(32, fulls=false, fulld=true), ConvertBits(32, fulls=false, fulld=true), show=0)

global total = 0.0
global GMSD = 0.0
global GMSD_total = 0.0
FrameEvaluate(last, """
           GMSD = 1.0-propGetFloat("_PlaneGMSD")
    global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
    global GMSD_total = GMSD_total + GMSD
""",local=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (GMSD, time)
global delimiter = "; "
resultFile = "perFrameResults.txt"  # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "GMSD_total", append=true)

#Prefetch(4)
return last
My last benchmark will be to test with a different clip and check what parameters depend on grain/thSAD, and what parameters are independent and coincide.
In any case IMO degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc. They divide the film in scenes and collect metrics for each one (they even encode by scenes). I tried without success to modify StainlessS' ScSelect_HBD() to know beforehand the last and next scene changes so filters can use accumulated per-scene metrics.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 22nd January 2022, 10:14   #239  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
It's often mentioned that dct=5 would be the most optimal choice, so it would be interesting to see how it fares. Also a bigger value for pel seems to change things a lot, at least when viewing things with MShow.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 22nd January 2022, 12:14   #240  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Quote:
Originally Posted by zorr View Post
That sounds interesting, let me know if you find a good method to analyze the results.
In lack of MANOVA we can try Excel Data Analysis tool, my Excel version is too old though. Will try to do in python or something.
A correlation matrix builds a variable x variable sized matrix showing the correlations between them. Simply go to Data -> Data Analysis -> Correlation. I don't think Google Spreadsheets has analysis tools.

@Boulder., in my tests results differ a bit whether I'm optimizing for MFlowFPS or MDegrain. For MFlowFPS DCT=1 and pel=2 looks like a no brainer even for 1080p. For MDegrain I found that DCT=0 and DCTR=9 is more optimal, and pel=2 doesn't have a big impact. DCT=5 if any would be more useful for RefineMotion.

EDIT: Correlation matrix for 210 random iterations (also in Google Spreadsheets). This (correlation calculation) can work to tune the Sensitivity internals.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 22nd January 2022 at 15:18.
Dogway is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:49.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.