Introducing Zopti (ex AvisynthOptimizer)

[Seedmanc]

I don't think I understand how to use the new method. I tried incorporating it into my script and thing went wrong I think, the pareto graphs look weird (http://puu.sh/CG9K0/35bddd7660.png) and the "best" results now are terribly slow and very different from anything I've gotten before.
Where should I get a x64 version of GaussianBlur? How do you determine the radius that should be used there?

Here's the script https://pastebin.com/c65dhju2, I launch it like F:\avsoptim\optimizer.bat F:\avsoptim\genBC.avs -iters 7h -runs 2 -pop 24

[StainlessS]

Where should I get a x64 version of GaussianBlur?
I think its in the VariableBlur whatsit:- http://avisynth.nl/index.php/AviSynth%2B_x64_plugins

[Reel.Deel]

I think its in the VariableBlur whatsit:- http://avisynth.nl/index.php/AviSynth%2B_x64_plugins

Actually at the moment is not on there, the download link points to the 32-bit version. I did add it on there but someone edited a few things and changed the link :mad:

Also I don't recall a 64-bit version of the latest VariableBlur (v0.7), only v.05 is available:


Binary: http://www.mediafire.com/download/0z0hl43za6bwlb4/VariableBlur05_x64.7z

Source: http://www.mediafire.com/download/hzwtctzyu5vw9vc/variableblur05_x64src.7z

[StainlessS]

Hi RD, glad that you are still lurking about :)

Just checked, I got v0.5 only on x64.

OK, had a bit more of a look, here is v0.7 (x86 only) with source, but its got lots of asm.
http://www.mediafire.com/file/1ugcb886dxq3tdo/variableblur_0.7.zip/file

[zorr]

I don't think I understand how to use the new method. I tried incorporating it into my script and thing went wrong I think, the pareto graphs look weird (http://puu.sh/CG9K0/35bddd7660.png) and the "best" results now are terribly slow and very different from anything I've gotten before.

Sorry, I should have made a better example. You don't actually need to blur anything, it was there just to demonstrate how B-SSIM score gets lower if you increase the blurring.

Replace the lines 135-146 with this and it should work.

scharr_orig = scharr(orig_yv12)
scharr_inter = scharr(inter_yv12)
FrameEvaluate(last, """
#global ssim = SSIM_FRAME(scaled_orig, scaled_inter)
global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
global bssim = BSSIM_MOD(scharr_orig, scharr_inter, ssim)
global ssim_total = ssim_total + (bssim == 1.0 ? 0.0 : bssim)
""")

[zorr]

New release, new name! AvisynthOptimizer is called "Zopti" from now on. Why? 'Cause it supports VapourSynth as well, that's why. Download the version 1.0-beta here (https://drive.google.com/open?id=1ZJ-4ETdp2fFlL-YXZcesEikMi98TTB84).

There's some other improvements as well, here's a complete list:


added support for VapourSynth scripts
Linux support (with VapourSynth)
added zoptilib python module to ease script writing
added support for VMAF output logs
improved heat map visualization:
-updates optimization progress to title bar when using autorefresh
-shows correct visuals also when smallest value is best
-added latest results visualization
-heat maps can be saved to image sequence with the -animation argument
-initial window size scaled to full screen
improved scatter chart visualization:
-improved zooming (-top)
-better legend position when smallest value is best
uses resolve for initial population when using randomized arguments fails 100 times
more error handling options when script execution fails: new argument -errors with options:
-stop (default, stops execution)
-ignore (continues optimization)
-log (continues optimization and logs errors into a separate error log)
checks that input arguments are all valid for the chosen -mode
improved error handling and error reporting in reverse polish notation parser
new option -priority for setting the optimizer process priority (on Windows only), options are:
-"": same as /NORMAL, default
-lowest: same as /LOW
-lower: same as /BELOWNORMAL
-higher: same as /ABOVENORMAL
-highest: same as /HIGH
removed bias from the random number generation


A few notes about upgrading: I renamed the optimizer.ini to zopti.ini and the optimizer.bat to zopti.bat. If you already have the .ini you can just rename it to zopti.ini and it will work.

I have used the -priority option to set the thread priority lower because when VapourSynth/MVTools is running a heavy task the other programs can become sluggish with the default priority. Priority option is Windows only for now, Linux probably doesn't even need it.

The error handling options are useful when running the VapourSynth MVTools which is currently not as robust as Pinterf's version. With -errors ignore or -errors log you can keep running the optimization even when the plugin crashes. I still recommend running with the default error handling option (which stops the execution) unless you are expecting errors.

I will make another thread to the VapourSynth section to tell more about the VapourSynth-specific stuff.

[Seedmanc]

Is this version abandoned and will any development only happen for the VS fork? I don't feel like learning a whole new language just to continue processing video as I did before just fine with AVS.

For some reason I was unable to find this thread in the thread list for Avisynth Development section, it said "Displaying 12 of 12 threads". Now that I bumped it, it's displayed there and says "13 of 13". How do I see all threads at once?

[wonkey_monkey]

In your forum user Control Panel, go to Edit Options, then find and change "Default Thread Age Cut Off."

[zorr]

Is this version abandoned and will any development only happen for the VS fork? I don't feel like learning a whole new language just to continue processing video as I did before just fine with AVS.

No, the same version works for both AviSynth and VapourSynth. So most of the improvements will affect both versions (new algorithms etc). The only VapourSynth-specific thing is the Zoptilib helper library which is not really needed. I wish I was able to write a similar helper for AviSynth but it would take a guru like StainlessS to accomplish that. :)

I haven't made much progress lately, so it's quiet on both fronts.

[StainlessS]

Oh do stop it. https://www.cosgan.de/images/smilie/konfus/s030.gif

[zorr]

Perhaps the only way to make it faster and threadsafe in Avisynth would be to use an array as you suggested.

After some tests I've found that Avisynth doesn't have array variables, only array constants. You cannot store values into an array after it has been created so currently arrays are not an option.

Or is it possible to make all the frames create their own log file and combine them when it's time to calculate the total score?

It would be easy to create a separate log file for each frame but that wouldn't make anything faster.

I tried some other ways. You can create a new variable for each frame (using Eval) and store the metric there. It doesn't solve the synchronization problem though, there are sometimes missing variables when it is time to combine the results. So that method only works without Prefetch. It would be useful for Avisynth to have a callback function which would be triggered when all the frames have been processed.

I also thought that maybe there's no need for any extra variables at all, we can simply read the metric from all the frames at the last frame using the offset parameter of propGetFloat(). The good news is that it actually works and is faster than the default method which writes to a file at each frame. The bad news is that the speedup is simply due to writing to the file only once (saves about 1,6ms per frame) and there is no additional speedup when using Prefetch. Here's the code:

TEST_FRAMES = 50 # how many frames are tested
MIDDLE_FRAME = 100 # middle frame number

RawSourcePlus("D:/optimizer/test/flower/flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)
#return noisy

#denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

sigma = 400/100.0 # optimize sigma = _n_/100.0 | 100..800 | sigma
bt = 4 # optimize bt = _n_ | -1..5 | blockTemporal
blockSize = 32 # optimize blockSize = _n_ | 2..64 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 0..32 ; max:blockSize 2 / | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

last = GMSD(source, denoised, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

Prefetch(4)

return last


I should also note that the previous GMSD version is not deterministic when used with Prefetch, it's rare but sometimes the last frame is not finished last and one of the frames is left out of the total sum. This version doesn't have that problem.

By the way this is a bit off-topic on this Vapoursynth forum. If any mods happen to read this it might be a good idea to move the posts starting from Worby's (https://forum.doom9.org/showthread.php?p=1938947#post1938947) to the Avisynth Zopti thread (https://forum.doom9.org/showthread.php?t=175723).

[Boulder]

I'm unable to get the new version to work with this test script. It processes the same step for minutes and if you take a look at Task Manager, the I/O read bytes shows that the source file has been read multiple times. Tested with and without Prefetch at the end.

SetCacheMode(0)
orig = FFVideoSource("c:\zopti\lotr_fotr.avi")

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

downscaled_width = 1920
downscaled_height = 808

alternate = BicubicResize(orig, downscaled_width, downscaled_height, b=b, c=c).Lanczos4Resize(orig.width(),orig.height())

GMSD(alternate, orig, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

#Prefetch(threads=24, frames=1)

return last

I used the "Report thread" button to notify that some posts should be moved to the Avisynth section.

[zorr]

I'm unable to get the new version to work with this test script.

I tested it using your silverado.avi clip from the other thread as the source. It does work on my setup, albeit slowly (about 170 seconds per script). Are you using the Avisynth+ 3.7 branch?

[Boulder]

I tested it using your silverado.avi clip from the other thread as the source. It does work on my setup, albeit slowly (about 170 seconds per script). Are you using the Avisynth+ 3.7 branch?

I did this test with a 4K clip of 400 frames. The "old" GMSD script processed one step of it in about 39 seconds while the new one was still processing after waiting for over 5 minutes. I'll try testing the Silverado clip later to make sure the frame size is not an issue somehow. EDIT: the new version finished the first step in 1095 seconds :D

I'm currently using the latest Avs+.

I was wondering if AVSTimer is one reason for seemingly serialized processing despite the prefetch value set. The GRunT stuff like WriteFileIf is MT-friendly if I remember correctly.

[Boulder]

I was wondering if AVSTimer is one reason for seemingly serialized processing despite the prefetch value set. The GRunT stuff like WriteFileIf is MT-friendly if I remember correctly.

Hmm..not that either. It must then be the way Avisynth parses things as all that is left is the wait for all frames to finish to get the final result. Strange though that using 24 threads instead of disabling Prefetch doesn't seem to have any effect even when the metric calculation per frame is the slow part.

[zorr]

Hmm..not that either. It must then be the way Avisynth parses things as all that is left is the wait for all frames to finish to get the final result. Strange though that using 24 threads instead of disabling Prefetch doesn't seem to have any effect even when the metric calculation per frame is the slow part.

I also tried without AVSTimer, made no difference. And I couldn't see any improvement when using 2, 4, 8, 16 or 24 threads.

Then I took out the scaling and only left the GMSD calculation. The time is still the same and the Prefetch behaviour remains. So it's the GMSD (and probably MDSI) which is slow and doesn't get any benefit from multiple threads. I guess it's because the multithreading only accelerates plugins and GMSD is implemented as a script. I tried to include a Prefetch call into the GMSD.avsi but it doesn't allow it there. :(

[StainlessS]

I was wondering if AVSTimer is one reason for seemingly serialized processing despite the prefetch value set. The GRunT stuff like WriteFileIf is MT-friendly if I remember correctly.

Just a wild guess, but would not WriteFileIf [if done non serialized] write any log file in randon-ish order. [maybe same for avsTimer].
Something like AvsMeter may be able to look at it from 'outside' and on total throughput of frames delivered, and not necessarily in any kind of order.
[although there would also have to be some kind of serialzation when played at output - which may/will affect AvsMeter too]

[kedautinh12]

VariableBlur 0.7 (x86 & x64)
https://drive.google.com/file/d/1tiC6bzQ40JOiNkarWYDWcRmSzS-NRqX7/view?usp=sharing

[Boulder]

I also tried without AVSTimer, made no difference. And I couldn't see any improvement when using 2, 4, 8, 16 or 24 threads.

Then I took out the scaling and only left the GMSD calculation. The time is still the same and the Prefetch behaviour remains. So it's the GMSD (and probably MDSI) which is slow and doesn't get any benefit from multiple threads. I guess it's because the multithreading only accelerates plugins and GMSD is implemented as a script. I tried to include a Prefetch call into the GMSD.avsi but it doesn't allow it there. :(

Multithreading definitely works with script functions as well, my normal method of processing contains both denoising and resizing from functions outside the script itself.

Both GMSD and MDSI use ScriptClip to put things in the frame properties, I don't see any other special thing in those functions. They're mostly Masktools stuff which runs multithreaded just fine.

[Boulder]

It's definitely something in those metrics functions, just posted to the Avisynth thread. I can only think the use of ScriptClip doing something which trips the multithreading.

[zorr]

A new major release is here! Zopti now supports executing multiple scripts simultaneously which can lead to very nice performance improvement. This can be very useful when the script itself is not able to utilize all of your cores.

Here's the full release info:


added support for multithreaded execution of scripts with argument -threads
-default value of -threads is 1 meaning one script is executed at a time just like before
-all optimization algorithms (nsga-ii, spea2, mutation, exhaustive) support the the -threads option
-due to the nature of the heuristic algorithms (nsga-ii, spea2, mutation) there can be at most P threads running the scripts at the same time, where P is the population size
-exhaustive algorithm has no thread limits
-validate mode also supports -threads. there the maximum number of threads is the number of validated results (size of the pareto front)
-note: using more than one thread will make time measurements less accurate but will not change the quality measurements
avsr version update (v0.2.2)
added jMetal.log.ini to disable multithreading related JMetal logging
updated Zoptilib to version 1.0.9m
bugfix: evaluate could not parse log files when Zopti was used


Download link is at the first post. I will shortly post some data on the multithreading performance.

[zorr]

I tested the new -threads argument using the denoising tutorial script, this time using GMSD as the similarity metric and using 50 frames instead of 5. Here's the script:


SetCacheMode(0)

TEST_FRAMES = 50 # how many frames are tested
MIDDLE_FRAME = 100 # middle frame number

RawSourcePlus("D:/optimizer/test/flower/flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)

#denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8) # best settings by Fizick

sigma = 400/100.0 # optimize sigma = _n_/100.0 | 400..600 ; filter:x 5 % 0 == | sigma
bt = 5 # optimize bt = _n_ | 3,5 | blockTemporal
blockSize = 14 # optimize blockSize = _n_ | 6,12 ; min:overlap 2 * | blockSize
overlap = 16 # optimize overlap = _n_ | 4..6 ; max:blockSize 2 / | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

last = GMSD(source, denoised, show=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

return last


There's also a VapourSynth version of the script which I posted to the VapourSynth Zopti thread (https://forum.doom9.org/showthread.php?p=1940864#post1940864).

The script only has 246 valid parameter combinations to test so we can try all of them using
zopti denoise_ex.avs -alg exhaustive -threads 1

It takes 179 seconds to test all of the combinations using one thread. Let's see the performance when using more threads:

https://i.postimg.cc/QdZrh7St/zopti-performance-scaling.png

The best result is about 20 seconds using 16 threads (on a Ryzen 3900X which has 12 cores / 24 threads).

Another way to look at the scaling is to calculate how much faster we get the results when using -threads:

https://i.postimg.cc/RFDj8JNv/zopti-performance-factor.png

Using 16 threads is 8,6 times as fast as using only one thread. Of course this is just one data point and I don't mean to imply that you can always get such a performance improvement. I would have liked to include more tests using more real-world usage scenarios (HD source etc) but my stock cooled processor cannot handle those for more than a few seconds, it becomes so hot that my PC shuts down. :eek:

Comparing AviSynth and VapourSynth it looks like VS has the edge when using one or just a few threads, but the differences almost vanish when using 12 or more threads. The VS is also able to utilize more than 16 cores while with AVS the performance starts to degrade. The point where more threads are just slowing down is probably dependent on the specific script, I will have to run more tests on that.

[Boulder]

Thanks zorr, looks like a very handy approach. I'll try testing it over the weekend.

[tormento]

Let's say I want to do compression/quality tests on 1-3% of a whole movie, changing the parameters on SMDegrain in a fixed range, mostly tr and thSAD.

Is there a way to batch that process? Usually I have to manually set a frame range and have multiple avs scripts.

Can you help me?

My usual script, with the parts I want to have permutations in red:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource(something)

SMDegrain (tr=4, thSAD=400, refinemotion=false, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Prefetch(6)

[zorr]

Let's say I want to do compression/quality tests on 1-3% of a whole movie, changing the parameters on SMDegrain in a fixed range, mostly tr and thSAD.

Is there a way to batch that process? Usually I have to manually set a frame range and have multiple avs scripts.

How do you evaluate the quality? Do you have a "reference" video you could compare the different settings with? Zopti needs some way to tell the quality and usually that means comparing two videos.

If you use your own eyes to measure the quality and just want to generate the different versions... well, Zopti doesn't output any video so it can't help you there. But it could generate you the different script variations if you provide a file with the parameter combinations you want, like this
1 tr=4 thSAD=40
1 tr=5 thSAD=40
1 tr=4 thSAD=50
etc.

[tormento]

How do you evaluate the quality?
I don't need automatic quality comparison, just a way to automate the changes in the script.

[zorr]

Time for a new update! This version has some major additions so I bumped the version to 1.2.0.


new option: -timeout (default is disabled). if the execution of Avisynth script takes longer than timeout seconds
-the script execution process (and subprocesses) will be terminated
-the result is set to worst possible as a penalty so it will not be considered as a valid candidate
-useful with scripts where certain parameter combinations can result in execution times several hundred times slower than usual
-example: zopti script.avs -pop 24 -iters 1000 -timeout 60
new option: -continue
-aborted optimization can be continued from the last complete generation (one generation is N iterations, where N is the population size)
-give the log file name of the aborted optimization as the value of continue
-old results up to last complete generation are copied to a new log file and optimization will continue from there
-useful if optimization was unexpectedly aborted or the CPU resources are needed for something else for a while
-example: zopti script.avs -pop 24 -iters 10000 -continue "mvtools spea2 run-01.log"
-NOTE: it is possible to change the optimization parameters (population, mutation count etc.) from those used in the continued log
-NOTE: random seed is not restored so the results will not be identical to what they would have been if execution had not been aborted
-NOTE: only works with algorithm spea2 (for now)
new visualization mode: history
-displays the best found value and all tried values of certain optimized parameter (given with -param)
-invalid results (script execution has failed OR timed out) are highlighted in orange
-also shows the phase of dynamic iteration (if applicable)
-accepts parameter -range to limit the displayed history to certain iteration range
-example: zopti -mode evaluate -vismode history -param lambda (displays all history of parameter "lambda")
-example: zopti -mode evaluate -vismode history -param lambda -range 1000 5000 (displays history from iteration 1000 to iteration 5000)
-example: zopti -mode evaluate -vismode history -param lambda -range -10000 (displays history of the last 10000 iterations)
argument -dynphases can now be zero (previously minimum was 1)
pareto front and total runtime is displayed also after mutation or exhaustive algorithm is finished
if population is based on a log file and the pareto front is smaller than the population size, the population will be filled with mutations of the pareto front instead of completely random combinations
if a log file for the optimization already exists it will be cleared in the beginning of the optimization
XChart updated to version 3.8.0 with some customization


And here's what -vismode history looks like:

https://i.postimg.cc/nLjnb7Y8/vismode-history.png

The black thick line shows the best found value of the chosen parameter. The red diamonds indicate a point where a new best result has been found and this parameter's value was changed. The white diamonds indicate a new best result but this parameter's value remained the same.

In the example optimization was finished and then restarted with different settings at around 190 000 iterations.

The -timeout was needed when I started running MVTools2 tests with all the parameters and it turned out that some combinations took an enormous amount of time to finish (typical time was 5 seconds but some combinations took over 30 minutes...). Using a timeout of 60 seconds solved that problem.

-continue is useful if you want to keep optimizing a result that is finished, you can try different settings. Also very handy when the optimization has been aborted by hardware failure or something like Windows update... you can just restart it using the same settings. This also makes it possible to run huge iteration counts little by little even if you need/want to use the computer for other things once in a while. :D

Download link updated at first post.

[Dogway]

Hey thanks a lot for the tool. I plan to use it now heavily, is it possible to combine it with VMAF? I tweaked your GMSD client script but VMAF doesn't seem to write frameProps, not at least as _PlaneVMAF.
By the way, if you need to combine/merge arrays (didn't read what for) I made this function (you need to explicitly declare the args in the call):

function ArrayAdd( val_array "a", val_array "b") {

as = ArraySize(a)-1
bs = ArraySize(b)-1
na = ""
for (i = 0, as+bs+1, 1) {

o = i - as - 1
cm = i != as+bs+1 ? "," : ""
na = na + ( i > as ? String(Eval(Format("b[{o}]"))) : \
String(Eval(Format("a[{i}]")))) + cm
}

return Eval("["+na+"]") }

[zorr]

Hey thanks a lot for the tool. I plan to use it now heavily, is it possible to combine it with VMAF? I tweaked your GMSD client script but VMAF doesn't seem to write frameProps, not at least as _PlaneVMAF.

VMAF writes its own result file (xml) and Zopti can read it directly so there's no need to read any frame props and save them to file. I haven't actually tested the Avisynth VMAF version but it should work if you set the log_path same as the Zopti file= parameter and log_fmt=0. Also set pool=0 to be consistent with the pooling method of other similarity metrics.

But before you start using it you might want to check these tests (https://forum.doom9.org/showthread.php?p=1867043#post1867043) I conducted (there's more tests in that thread) about the different similarity metrics, seems like VMAF is not very consistent and sometimes gives downright bad results.

By the way, if you need to combine/merge arrays (didn't read what for) I made this function (you need to explicitly declare the args in the call):

Thanks, might be useful some day. Arrays were considered as a solution to synchronization problem when using Prefetch() and the script tries to write the results of each frame to the same file. Reading the frame props in the end solved that problem, but I think it's not the optimal way (I think sometimes the frame props are no longer in cache when they are requested so some frames could be calculated twice).

[Dogway]

zorr, I'm trying to come up with a self resolving BBSIM function, I have troubles with SSIM_FRAME(), not only it requires 8-bit but it's output value isn't affected by the filtered clip. I tried to implement reading _PlaneSSIM property if created with vsSSIM but its output value is very different from SSIM_FRAME().

This is my attempt at normalizing PlaneSSIM to SSIM_FRAME (by /335 and then 2*bssim).
function BSSIM(clip source, clip filtered, bool "PlaneSSIM", bool "show", bool "fulls") {

show = Default(show, false)
PS = Default(PlaneSSIM, false)
fs = Default(fulls, false)

blurred = source.ex_blur(1.5*2,bifit=true,UV=1)

source8 = PS ? source : source.ConvertBits(8, dither=-1, fulls=fs)
blurred8 = PS ? blurred : blurred.ConvertBits(8, dither=-1, fulls=fs)

scharr_orig = ex_edge(source8, "pscharr",0,255)
scharr_blurred = ex_edge(blurred8, "pscharr",0,255)

source = PS ? source.ConvertBits(32, fulls=fs) : source8
blurred = PS ? blurred.ConvertBits(32, fulls=fs) : blurred8
scharr_orig = PS ? scharr_orig.ConvertBits(32, fulls=fs) : scharr_orig
scharr_blurred = PS ? scharr_blurred.ConvertBits(32, fulls=fs) : scharr_blurred

ScriptClip(source, function [source, blurred, scharr_orig, scharr_blurred, PS, show] () {

SSIM = PS ? propGetFloat(source,"_PlaneSSIM")/335 : \
SSIM_FRAME(source, blurred)
stddev_orig = RT_YPlaneStdev(scharr_orig)
stddev_alt = RT_YPlaneStdev(scharr_blurred)

bssim = SSIM * (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)
bssim = PS ? 2*bssim : bssim

propSet("_PlaneBSSIM", bssim, 0)

return show ? Subtitle("PlaneBSSIM: " + String(bssim)) : last } ) }

[zorr]

I have troubles with SSIM_FRAME(), not only it requires 8-bit but it's output value isn't affected by the filtered clip.

If you mean the input argument clip filtered then that's not a mystery, it's not used in the function at all. :)

I tried to implement reading _PlaneSSIM property if created with vsSSIM but its output value is very different from SSIM_FRAME().

There are some differences in SSIM implementations. I don't think it's that important to try to match the different implementations. The important aspect of B-SSIM is just the multiplier which scales the result based on how blurry the clip is. Although there might be a small problem with the presented algorithm as SSIM (at least in some implementations) has a range of -1 to 1 and it would scale the negative results in the wrong direction.

[Dogway]

haha, yes didn't notice. I thought 'blurred' was a preprocessing.
Well, I think this will do it. (EDIT: I need to fix vsSSIM as I borked it when optimizing it)

function BSSIM(clip source, clip blurred, bool "PlaneSSIM", bool "show", bool "fulls") {

show = Default(show, false)
PS = Default(PlaneSSIM, false) # Set to True if the clip has the _PlaneSSIM frame property
fs = Default(fulls, false)

source8 = source.ConvertBits(8, dither=-1, fulls=fs)
blurred8 = blurred.ConvertBits(8, dither=-1, fulls=fs)

scharr_orig = ex_edge(source8, "pscharr",0,255)
scharr_blurred = ex_edge(blurred8, "pscharr",0,255)

ScriptClip(source, function [source, source8, blurred8, scharr_orig, scharr_blurred, PS, show] () {

SSIM = PS ? propGetFloat(source,"_PlaneSSIM") : \
SSIM_FRAME(source8, blurred8) # only works on 8-bit
stddev_orig = RT_YPlaneStdev(scharr_orig) # only works on 8-bit (output is also 8-bit)
stddev_alt = RT_YPlaneStdev(scharr_blurred) # only works on 8-bit (output is also 8-bit)

bssim = SSIM * (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)

propSet("_PlaneBSSIM", bssim, 0)

return show ? Subtitle("PlaneBSSIM: " + String(bssim)) : last } ) }

[Dogway]

I'm running some tests for motion vectors and I have a few questions.
-How do you terminate the process? Normally I 'ctrl+c' on the console but then I have to kill each avsr64.exe process
-What is the best approach to simply run a bunch of random iterations without any refinement? I simply used low iterations with high number of runs, but 'runs' doesn't allow multithreading, and 'timeout' finishes the whole thing. The problem here is that some settings have more weight than others so it can bias the refinement so I want to give equal probability to all of them to draw some conclusions (ie. in Excel), at least for the initial tests.
SET RUNS= 4
SET ALG= mutation
SET POPULATION= 4
SET INITIAL= random
SET ITERS= dyn
SET MUTAMOUNT= 0.5 0.01
SET MUTCOUNT= 60% 1
SET CROSSPROB= 0.1
SET CROSSDIST= 20
SET SENSITIVITY= TRUE
SET DYNPHASES= 1
SET DYNITERS= 8
SET ERRORS= STOP
SET PRIORITY=
SET THREADS= 4
SET TIMEOUT= 410
SET CONTINUE=

java -jar Zopti.jar "..\Zopti_test.avs" ^
-alg %ALG% -initial %INITIAL% -iters %ITERS% -dyniters %DYNITERS% -dynphases %DYNPHASES% -pop %POPULATION% -runs %RUNS% -mutcount 1 -mutamount %MUTAMOUNT% -timeout %TIMEOUT% -threads %THREADS%
I don't quite understand the settings marked in red though. Maybe they are related to what I want to do.

-Also I tried to kill some iterations as soon as some frames are rendered longer than necessary so I played a bit with WriteFileIf() but didn't find what keyword is for timeout. Tested with:
WriteFileIf(resultFile, "avstimer > 4500.0", """ 0.0 9999999"""+Chr(10)+""" "stop " """, append=true)

[zorr]

-How do you terminate the process? Normally I 'ctrl+c' on the console but then I have to kill each avsr64.exe process

Currently you cannot easily terminate the process (and all the subprocesses that were started by Zopti). I'll have to look if I can implement a way to detect the termination and shut down the subprocesses as well.

-What is the best approach to simply run a bunch of random iterations without any refinement? I simply used low iterations with high number of runs, but 'runs' doesn't allow multithreading, and 'timeout' finishes the whole thing.

The best approach (with no bias) is to run with a high population count and the same number of iterations, one run only. That way all the iterations are random and not refined in any way. Also there will be no duplicate parameter combinations, Zopti takes care of that.

The problem here is that some settings have more weight than others so it can bias the refinement so I want to give equal probability to all of them to draw some conclusions (ie. in Excel), at least for the initial tests.

I see what you mean. But it's also difficult to make any conclusions in that early phase of the optimization, unless you only have a couple of parameters you want to vary. With one or two parameters your best bet is to run exhaustive search, you can then make nice visualizations from those. Zopti has a heatmap visualization for two parameters, for example. And exhaustive search can also be "sparse" in a way that you test for example only every 10th value.

SET RUNS= 4
SET ALG= mutation
SET POPULATION= 4
SET INITIAL= random
SET ITERS= dyn
SET MUTAMOUNT= 0.5 0.01
SET MUTCOUNT= 60% 1
SET CROSSPROB= 0.1
SET CROSSDIST= 20
SET SENSITIVITY= TRUE
SET DYNPHASES= 1
SET DYNITERS= 8
SET ERRORS= STOP
SET PRIORITY=
SET THREADS= 4
SET TIMEOUT= 410
SET CONTINUE=
...

I don't quite understand the settings marked in red though. Maybe they are related to what I want to do.

This (https://forum.doom9.org/showthread.php?p=1853726#post1853726) is the official "reference post" of the optimizer arguments. It doesn't have the later additions though but covers the ones you needed clarification about.

-Also I tried to kill some iterations as soon as some frames are rendered longer than necessary so I played a bit with WriteFileIf() but didn't find what keyword is for timeout. Tested with:
WriteFileIf(resultFile, "avstimer > 4500.0", """ 0.0 9999999"""+Chr(10)+""" "stop " """, append=true)

If you're playing with MVTools there are some parameter combinations that can take a very, very long time before they even return the first frame. The -timeout argument was implemented just for that purpose, to stop the execution of the AviSynth script if it takes longer than the specified time. The termination is immediate as it simply kills the whole process and can start executing another one instead.

[zorr]

I implemented a shutdown hook to kill all the running subprocesses if Zopti is terminated (thanks for the idea Dogway). I had some other changes in the pipeline as well, so here's a new release:

new option: retry
-tries to run the script again this many times if execution fails
-can be useful if the script uses plugins which are not 100% reliable and can crash
-default value is zero (do not retry)
-example: zopti script.avs -pop 24 -iters 10000 -retry 4
new visualization mode: line
-draws a line chart of the best found result per all tested values of certain optimized parameter (given with option -param)
-accepts parameter -range to limit the displayed values to certain range
-black line shows the best values, blue line indicates the number of results per value
-a red dot is displayed at the best found value (multiple dots possible if there are multiple values with the same best result)
-example: zopti -mode evaluate -vismode line -param lambda -range 1000 5000 (displays the best result of lambda values between 1000 and 5000)
a shutdown hook will terminate all the running subprocesses if Zopti is aborted (for example with CTRL + C)
mode -validate also tests that the first result value (usually quality) is the same as before, gives a warning message if they differ
reading and parsing the log file with -autorefresh true is now MUCH faster by reading the file backwards and only adding the new results (previously up to several seconds, now < 1 ms)
scripts ending .py also correctly detected as VapourSynth scripts (previously only .vpy was detected)
support for custom output properties written by the script (only in VapourSynth scripts for now)
better support for displaying results from output files with different parameters (scripts can have different parameters and still be compared in the same chart)
option -continue now also supported by exhaustive algorithm (when using more than one thread)
order of runs in a visualization is now based on file name, not by modified date
all chart types now support the window size in option -shot (example: zopti -mode evaluate -shot 1200x800)
visualization mode seriespareto no longer contains the global pareto line
VapourSynth script output files will be interpreted even when VapourSynth reports failed execution (sometimes file still has complete data)
bugfix: using -alg mutation and threads < population size would stall progress after initial generation due to parameter queue being too small
bugfix: fixed a memory leak in XChart (at least partially)


Here's what -vismode line looks like:

https://i.postimg.cc/mDb5mW9t/lambda.png

Download link updated at first post.

[Dogway]

Thanks for the update!!

I got some issues with sensitivity option. Probably my optimization range in the script wasn't too broad or something and I was getting this error message after DYNITERS iterations using "mutation" as algo, and "dyn" as iterations.

Parameter sensitivity estimation with 256 result combinations
java.lang.ArrayIndexOutOfBoundsException: Index -700 out of bounds for length 23
00
at avisynthoptimizer.Parameter.updateAverages(Parameter.java:954)
at avisynthoptimizer.Parameter.updateSensitivityEstimation(Parameter.jav
a:970)
at avisynthoptimizer.SensitivityEstimation.estimateParameterSensitivity(
SensitivityEstimation.java:102)
at avisynthoptimizer.AviSynthOptimizer.optimize_mutation_multithreaded(A
viSynthOptimizer.java:4726)
at avisynthoptimizer.AviSynthOptimizer.main(AviSynthOptimizer.java:620)
ERROR: Index -700 out of bounds for length 2300

C:\Program Files (x86)\AviSynth+\Authors\zorr\Zopti-1.2.0>rem ERROR: Series name
>phase 1,00< has already been used. Use unique names for each series!!!


By the way on another note I was trying to evaluate motion vectors as a general case without the intervention of client functions like MFlowFPS, or MDegrain by using the SAD mask in MMask (kind=1) and using the average value as the frame metric. Well this didn't yield good results, it tends to optimize to a large searchRange while for example MDegrain with a searchRange over 2 smears the content. So first I'm locking searchRange to 2 and optimizing for MDegrain with the MDegrain client.
Also most of the hardest settings to optimize are those of truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global... they show bad correlation overall.


Here's my Excel chart for the first randomized data samples. You can easily see that any other than DCT=0 is slow and bad quality, so you can remove those entries and continue guessing settings that can be locked down. Ideally one would run some multivariate analysis but I'm still studying statistics.

Google Spreadsheets (https://docs.google.com/spreadsheets/d/1MNrZe-cjzZGuOucvK7U-LpxKr_yIxoRIUKHep_mbxn0)

[zorr]

I got some issues with sensitivity option. Probably my optimization range in the script wasn't too broad or something and I was getting this error message

That looks intriguing, can I take a look at your script so that I can try to replicate this error?

By the way on another note I was trying to evaluate motion vectors as a general case without the intervention of client functions like MFlowFPS, or MDegrain by using the SAD mask in MMask (kind=1) and using the average value as the frame metric. Well this didn't yield good results

Yeah you have to be careful what you optimize for. If there's a way to cheat a good result Zopti most likely finds a way to do that. So it's always a good idea to only optimize for the final result and not some intermediate metric.

Also most of the hardest settings to optimize are those of truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global... they show bad correlation overall.

Most of the settings are only relevant after the most important ones are found and the search starts to focus.

Note that you don't need truemotion if you set lambda, lsad, pnew, plevel and global in your script as those values will overwrite whatever truemotion sets them to.

Here's my Excel chart for the first randomized data samples. You can easily see that any other than DCT=0 is slow and bad quality

I wouldn't be so sure. :) Maybe you've already seen this (https://forum.doom9.org/showthread.php?p=1942410#post1942410) but I did some pretty extensive tests with MVTools frame interpolation and found that dct=1 was consistently best, it was the winner in three test runs with at least 100000 iterations each. It could very well be that dct=0 is the best for MDegrain and also it may have more to do with the specific video, as even I haven't done tests with multiple videos to find out which parameters are consistently best with most source material (or perhaps none are, or perhaps it depends on the resolution or content type).

In any case 100 tests is not enough to say anything sure. I try to make conclusions on the best value only after Zopti has consistently selected it as the best one, after which it can be locked down and another tests can focus on finding the best values for the remaining unlocked parameters. MVTools has so many parameters that it's a real challenge to optimize them due to enormous search space, that's why it has to be done meticuously and using large iteration counts.

Ideally one would run some multivariate analysis but I'm still studying statistics.

That sounds interesting, let me know if you find a good method to analyze the results.

[Dogway]

The error didn't happen to me in my last edit, I think it was related to the parameter range AND the "filter:" used. It will happen again I guess so I will post the offending script here.

Currently I'm benchmarking MDegrain since MFlowFPS was a bit too complex and as you say DCT=1 was the best. DCT=1 and searchAlgo=3 are the slowest settings in MAnalyse aside of a high searchRange, so it would be difficult to find a pareto front with them. In the case of MDegrain I don't feel they are critical (did some runs) so I exclude them.

Note that you don't need truemotion if you set lambda, lsad, pnew, plevel and global in your script as those values will overwrite whatever truemotion sets them to.
Oh I though it was the other way around, truemotion overriding those settings.

I don't know how you can run so many iterations, I'm testing with 150 frames and as you can see it takes about 2 to 5 minutes per iteration (taking out DCT=1 and search=3). I run 8 scripts at the same time each for a thread on my CPU.
What I did, instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain.

Here's my reference clip (https://www.sendspace.com/file/jn69kp) (a sample (https://www.sendspace.com/file/se2g3g)from @tormento).
setmemorymax(2048)
DGSource("brazil.dgi",cl=0,ct=40,cr=0,cb=40)

ConvertBits(16,fulls=true)

pre=smdegrain(tr=1,mode="temporalsoften",blksize=16,thSAD=900,LFR=400,prefilter=3,DCTFlicker=true,contrasharp=false,refinemotion=true).ex_unsharp(0.5, 1920.0/8, 0.0)
smdegrain(tr=2,mode="MDegrain",blksize=16,prefilter=pre,thSAD=400,LFR=200,contrasharp=true,refinemotion=true)
ConvertBits(8,dither=1)

And this is my Zopti script with refined ranges after about 400 random iterations, and a few locked ones after a week of tests:
setmemorymax(16384/8)
DGSource("brazil-ref.dgi",cl=0,ct=0,cr=0,cb=0)
trim(1,12)+\
trim(60,96)+\
trim(173,285-12)
src=last
DGSource("brazil.dgi",cl=0,ct=40,cr=0,cb=40)

AssumeFPS(24000,1001)
RequestLinear(30)

thSAD = 285 # optimize thSAD = _n_ | 230..290 ; filter:x 5 % 0 == | thSAD
thSADC = 200 # optimize thSADC = _n_ | 140..250 ; filter:x 10 % 0 == | thSADC
thSADR = 280 # optimize thSADR = _n_ | 150..300 ; filter:x 10 % 0 == | thSADR
BlkSize = 16
BlkSizeR = 8
overlap = 4
overlapR = 4
pel = 1
sharp = 2 # optimize sharp = _n_ | 2 | sharp
scaleCSAD = 2
trymany = false
truemotion = false # optimize truemotion = _n_ | true,false | truemotion
truemotionR = true # optimize truemotionR = _n_ | false,true | truemotionR
temporal = true # optimize temporal = _n_ | true,false | temporal


# Settings that depend on source noise
# lsad the noisier the lower
# sigma the noisier the higher
# badSAD

# TRUEMOTION SETTINGS
lambda = 440 # optimize lambda = _n_ | 400..900 ; filter:x 10 % 0 == | lambda
# lambdaR normally optimizes between 1.6 and 2.0 times lambda
lambdaR = 1320 # optimize lambdaR = _n_ | 960..2000 ; min:lambda ; max:lambda 3 * ; filter:x 20 % 0 == | lambdaR
# pnew: Default is 0 for truemotion = false and 50 for truemotion = true.
pnew = 114 # optimize pnew = _n_ | 100..200 ; filter:x 2 % 0 == | pnew
pnewR = 136 # optimize pnewR = _n_ | 100..250 ; filter:x 2 % 0 == | pnewR
# lambda is not used when pzero is 0 (zero vector)
# there's a relationship between pzero and searchRangeR (and searchRangeR with searchRangeFinest)
pzero = 100 # optimize pzero = _n_ | 28..100 ; filter:x 2 % 0 == | pzero
lsad = 6000 # optimize lsad = _n_ | 1000..8000 ; filter:x 100 % 0 == | lsad
# plevel: Default is 0 for truemotion = false and 50 for truemotion = true
plevel = 79 # optimize plevel = _n_ | 1..99 ; filter:x 2 % 0 != | plevel
lvl = 1 # typically plevel is set same as level

# lambda is not used for global predictor
pglobal = 8 # optimize pglobal = _n_ | 0..20 | pglobal

badrange = 2 # optimize badrange = _n_ | 0..50 ; filter:x 2 % 0 == | badrange
badSAD = 1350 # optimize badSAD = _n_ | 1100..2200 ; filter:x 50 % 0 == | badSAD
dct = 0 # optimize dct = _n_ | 0 | dct
dctre = 9 # optimize dctre = _n_ | 0,2,3,6,7,9 | dctre
rfilter = 3
searchAlgo = 1 # optimize searchAlgo = _n_ | 1 | searchAlgo
searchAlgoR = 4 # optimize searchAlgoR = _n_ | 0,1,2,4,5 | searchAlgoR
searchRange = 15 # optimize searchRange = _n_ | 2..17 | searchRange
searchRangeR = 13 # optimize searchRangeR = _n_ | 2..18 | searchRangeR
searchRangeFinest = 17# optimize searchRangeFinest = _n_ | 9..20 | searchRangeFinest
sglobal = true # optimize sglobal = _n_ | true,false | sglobal

trim(1,12)+\
trim(60,96)+\
trim(173,285-12)

C=ConvertBits(16,fulls=false)
pre=C.ex_FluxSmoothST(2,2,255,0,false,UV=3).ex_Luma_Rebuild(s0=3,tv_range=true).ConvertBits(8,dither=-1,fulls=true)

Recalculate=true


superfilt = MSuper(pre, hpad=16, vpad=16, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
superR = MSuper(C, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false, chroma=false)
superRe = MSuper(pre, hpad=16, vpad=16, levels=lvl, sharp=sharp, rfilter=rfilter, pel=pel, mt=false)
bak2 = MAnalyse(superfilt, isb=true, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak1 = MAnalyse(superfilt, isb=true, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd1 = MAnalyse(superfilt, isb=false, delta=1, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
fwd2 = MAnalyse(superfilt, isb=false, delta=2, blksize=BlkSize, overlap = overlap, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, dct=dct, mt=false, scaleCSAD=scaleCSAD, pnew=pnew, pzero=pzero, truemotion=truemotion, badSAD=badSAD, badrange=badrange, temporal=temporal,lsad=lsad, lambda=lambda, pglobal=pglobal, plevel=plevel, trymany=trymany, global=sglobal)
bak2 = Recalculate ? MRecalculate(superRe, bak2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : bak
bak1 = Recalculate ? MRecalculate(superRe, bak1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR, thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : bak
fwd1 = Recalculate ? MRecalculate(superRe, fwd1, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR , thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : fwd
fwd2 = Recalculate ? MRecalculate(superRe, fwd2, blksize=BlkSizeR, overlap = overlapR, search=searchAlgoR, searchparam=searchRangeR, dct=dctre, mt=false, scaleCSAD=scaleCSAD, pnew=pnewR , thSAD=thSADR, truemotion=truemotionR, lambda=lambdaR) : fwd
C.MDegrain2(superR, bak1, fwd1, bak2, fwd2, thSAD=thSAD, thSADC=thSADC, plane=0, mt=true)

global frame_count = FrameCount()

GMSD(src.ConvertBits(32, fulls=false, fulld=true), ConvertBits(32, fulls=false, fulld=true), show=0)

global total = 0.0
global GMSD = 0.0
global GMSD_total = 0.0
FrameEvaluate(last, """
GMSD = 1.0-propGetFloat("_PlaneGMSD")
global GMSD = (GMSD == 1.0 ? 0.0 : GMSD)
global GMSD_total = GMSD_total + GMSD
""",local=false)

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (GMSD, time)
global delimiter = "; "
resultFile = "perFrameResults.txt" # output out1="GMSD: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "GMSD", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "GMSD_total", append=true)

#Prefetch(4)
return last

My last benchmark will be to test with a different clip and check what parameters depend on grain/thSAD, and what parameters are independent and coincide.
In any case IMO degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc. They divide the film in scenes and collect metrics for each one (they even encode by scenes). I tried without success to modify StainlessS' ScSelect_HBD() to know beforehand the last and next scene changes so filters can use accumulated per-scene metrics.

[Boulder]

It's often mentioned that dct=5 would be the most optimal choice, so it would be interesting to see how it fares. Also a bigger value for pel seems to change things a lot, at least when viewing things with MShow.

[Dogway]

That sounds interesting, let me know if you find a good method to analyze the results.

In lack of MANOVA we can try Excel Data Analysis tool, my Excel version is too old though. Will try to do in python or something.
A correlation matrix builds a variable x variable sized matrix showing the correlations between them. Simply go to Data -> Data Analysis -> Correlation. I don't think Google Spreadsheets has analysis tools.

@Boulder., in my tests results differ a bit whether I'm optimizing for MFlowFPS or MDegrain. For MFlowFPS DCT=1 and pel=2 looks like a no brainer even for 1080p. For MDegrain I found that DCT=0 and DCTR=9 is more optimal, and pel=2 doesn't have a big impact. DCT=5 if any would be more useful for RefineMotion.

EDIT: Correlation matrix for 210 random iterations (also in Google Spreadsheets (https://docs.google.com/spreadsheets/d/1pJwM6opK6BX7GKT2qzbVqrUyHL1-fZEuUQ_3GTrJkes/edit?usp=sharing)). This (correlation calculation) can work to tune the Sensitivity internals.
https://i.imgur.com/wOECyat.png

[DTL]

" truemotion, lambda, pnew, pzero, badSAD, badRange, plevel, temporal, global."

I can recommend to look also at scaleCSAD adjustment. It work only if chroma=true. It is too coarse now in 2.7.45 like +-2 integer including 0 (only 5 valid positions total) but may significantly change out MPEG encoder speed (like greatly influence SAD values). It looks even not 'auto-normalizing' after changing scaleCSAD value so you mostly probably need to re-adjust thSAD in MDegrain (and may be other thresholds).
Depending on relative amount of luma/chroma noise and its spectrum it looks may give visibly different results at different footages.

I even thinking of making it float param because difference between 0 and -1 may be too large. -1 may give less out MPEG speed but looks like smooth many. Though float calculation will visibly lower the speed on CPU-processing because it is frequent enough calculation.

"degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc."

In best case - yes. Because each scene may use different film ISO or camera gain and have different noise levels. Current approach is to set more tr and more thSAD cause slower processing and also may degrade more details on low noise scenes.

[Dogway]

Currently it's locked at a value of 2, this has shown increased metrics in most tests unless the clip is crap of course. It might be also useful to check its correlation to thSAD.

The problem with chroma motion vectors is that you might not want them to impact SAD much. I once did a test with normalized chroma planes (saturating bitdepth range) and yes it protected more and thus it denoised less in areas where to the eye is more noticeable, specially for anime. So what is this? YUV is YCbCr, Cb plane is blue opposing while Cr is red opposing. Human visual system is much less sensitive to the blue color than to red, so I wouldn't want to give much importance to the U plane.

I have even thought on adding a hue plane to process motion vectors from. It makes more sense as 2 totally different colors side to side will be very far in values and raise SAD, but this also requires scaling thSADC and other settings as well.

[DTL]

"Currently it's locked at a value of 2"

It is really very great deviation from default 0. It either mean default 0 is wrong or something else. Also 2 is currently max allowed value and not allow any more adjustment if it continue to increase quality (may be 3 is even better ?).

Default 0 also mean it have nominal adjustment range of +-2 around 'best recommended value'. So if new 'best default is 2' (for YV12 typical inputs ?) so it mean for good adjustment range we need 2(+-2) ?

As for documentation:
Fine tune chroma part weight in SAD calculation (since 2.7.18.22)


Possible values for luma:chroma SAD ratio
-2: 4:0.5
-1: 4:1
0: 4:2 (default, same as the native ratio for YV12)
1: 4:4
2: 4:8

So 'practical default 2' mean the chroma SAD is very important and its weight need to be increased more (higher 4:8) ?

Actual internal output SAD calculation from MAnalyse:
outSAD = SAD_Y + ScaleSADChroma(SAD_UV, eff_scale)

Where (from file def.h)

static MV_FORCEINLINE sad_t ScaleSadChroma(sad_t sad, int effective_scale) {
// effective scale: 1 -> div 2
// 2 -> div 4 (YV24 default)
// -2 -> *4
// -1 -> *2
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);
}


and effective_scale is calculated from scaleCSAD as

// New experiment from 2.7.18.22: keep LumaSAD:chromaSAD ratio to 4:2

// luma SAD : chroma SAD
// YV12 4:(1+1) = 4:2 (this 4:2 is the new standard from 2.7.18.22 even for YV24)
// YV16 4:(2+2) = 4:4
// YV24 4:(4+4) = 4:8

// that means that nSCD1 should be normalize not by subsampling but with user's chromaSADscale

// YV12 YV16 YV24
// nLogXRatioUV 1 1 0
// nLogYRatioUV 1 0 0
// effective_chromaSADscales: (shift right chromaSAD)
// chromaSADscale=0 -> 0 1 2 // default. YV12:no change. YV24: chroma SAD is divided by 4 (shift right 2)
// =1 -> -1 0 1 // YV12: shift right -1 (=left 1, =*2) YV24: divide by 2 (shift right 1)
// =2 -> -2 -1 0 // YV12: shift right -2 (=left 2, =*4) YV24: no change
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV));
effective_chromaSADscale -= chromaSADscale; // user parameter to have larger magnitude for chroma SAD
// effective effective_chromaSADscale can be -2..2.
// when chromaSADscale is zero (default), effective_chromaSADscale is 0..2


user input 2 -> eff_scale = -2 (for YV12) and ScaleSADChroma return saduv << 2 that is 4 * saduv, so total output SAD = SAD_Y + 4*SAD_UV. If is not enough so may be need SAD = SAD_Y + (5,6,7, ..?)*SAD_UV ?

Unfortunately current 'bitshift' scale method not allow fine adjustment so if even simply enable value +3 and add <<3 processing it will make only SAD = SAD_Y + 8*SAD_UV next adjustment step that may be too high. So for compatibility reason with old scripts may be only a way is to add new param like 'scaleCSADfine' that is true float multiplier and can be adjusted in good range like from 0.1 to 10.0. With default to 1.0.
So total new formula will be

static MV_FORCEINLINE sad_t ScaleSadChroma(sad_t sad, int effective_scale) {
if (chroma_finetune == 1.0f)
{ // high speed calculation for compatibility with old scripts
if (effective_scale == 0) return sad;
if (effective_scale > 0) return sad >> effective_scale;
return sad << (-effective_scale);
}
else
{ // new fine tuning in big range lower speed calculation
sad_t imm_uvsad;
if (effective_scale == 0) imm_uvsad = sad;
else if (effective_scale > 0) imm_uvsad = sad >> effective_scale;
else
imm_uvsad = sad << (-effective_scale);

return (sad_t)(imm_uvsad * chroma_finetune);
}
}


Unfortunately additional internal 'adaptation' of UV sad scale factor to pixel format makes things a bit more complex. Also it looks current docs do not mention it is really internally 'adapted' to input pixel format. So no simple one formula available for all possible input formats. Also it looks even in 2022 the practical 'lumaSAD/chromaSAD' ratio is still in 'experimental' state and still may changes in the very high range: like from +4*SAD_UV and more to +0.25*SAD_UV and may be less (for YV12 input).
Current format of output data from MAnalyse do not allow to output luma and chroma SAD separately - it may help to script writers for more complex processing. So we can only adjust internal processing of luma/chroma SAD values ratio with external adjustment param and with simple linear mixing function. But it not only output SAD but also total SAD using at search (with 'onCPU' mode - only available for 2.7.45) of MVs at each step. So it affect both MVs search and MDegrain client degrain operation.

Addition: I now find a bug with 'reversing direction' of scaleCSAD adjustment in new development version so my 'better denoise' also moves to scaleCSAD > 0. Though with some not very detailed compare it looks it more blur the most of frame and sharp textures. Also it was tested with not internal MAnalyse MVs search engine but with NVIDIA's encoder hardware ME.

[zorr]

The error didn't happen to me in my last edit, I think it was related to the parameter range AND the "filter:" used. It will happen again I guess so I will post the offending script here.

I took a look at your current script, didn't manage to run it yet because I think it needs the latest test version of AviSynth in order to run propNumElements (in ex_MinBlur) outside ScriptClip. But I had some strange errors earlier when I used the same way to write the frame props to a file, there may be rare errors when the info about the last frame is not written as the last line of the file. I implemented another way to write all the props only at the last frame of the clip, you can see an example of that here (https://forum.doom9.org/showthread.php?p=1940863#post1940863).

If you encounter the bug again look for the offending script in the zopti work folder and see if the result file has a wrong order of result lines (the others don't matter but the last line should start with "stop").

Oh I though it was the other way around, truemotion overriding those settings.

That's easy to test, run a script twice and only change the truemotion true/false, you should get the exact GMSD score either way.

I don't know how you can run so many iterations, I'm testing with 150 frames and as you can see it takes about 2 to 5 minutes per iteration (taking out DCT=1 and search=3). I run 8 scripts at the same time each for a thread on my CPU.

I try to make the script runtimes fast. In the thread I mentioned earlier I cropped the video into a smaller one (528x580) where the interesting and challenging motion was located at. I also cut a segment out of it, it was only 29 frames in total. If the script can handle the difficult part, it should be able to handle the rest of the scene as well. The runtimes were from 2 seconds upto around 25 seconds in the later parts of the optimization (it mostly depends on what kind of speed the scripts in the pareto front have). In the beginning of the optimization there can be parameter combinations which run for 30 minutes or longer so I used a timeout of 60 seconds to stop the scripts which take too long.

I have a 12 core / 24 thread CPU and 64GB RAM to be able to run many scripts at the same time. I do some tests to find out how many threads saturates the cores nicely, also important to check that the script are not limited by having too little RAM as that will definitely cause slowdowns.

Oh, and you need patience. :) For large tests I use a dynamic iteration count as that's a good way to guarantee that the optimization process is not limited by the iteration count too much. So sometimes the runs take a week, or more. Sometimes I stop the run in order to use the CPU for something else and use the -continue option to proceed with the same test run. The largest iteration count so far was almost 700k.

What I did, instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain.

That's a good way to do it, that way you can remove real noise and not some synthetic one which may look very different. There are many kinds of noise reduction algorithms and it's because different noise needs a different algorithm (or at least the optimal settings are different). The synthetic noise tests may still be valuable to try to find new and perhaps surprising parameter combinations, those can then be tested with real noise.

In any case IMO degraining (and most filters by extension) should be a scene based procedure, that's how it's done in big studios like Netflix, etc. They divide the film in scenes and collect metrics for each one (they even encode by scenes). I tried without success to modify StainlessS' ScSelect_HBD() to know beforehand the last and next scene changes so filters can use accumulated per-scene metrics.

That would indeed be awesome, to drive the filter settings with frame props. Determining those settings from some kind of noise profile could also be optimized with Zopti but someone would have to do the hard work of optimizing all the scenes of the reference clip by hand first.

[DTL]

"instead of adding grain to a clean clip, I did my best to degrain a clip employing all sorts of tricks, then try to mimic that with basic MDegrain."

May be also good to try with real video camera to make 2 shots with low and high gain settings (with disabled in-camera noise reduction) and use low-gain recording as reference. Though with moving picture it is not easy make 2 identical shots for PSNR-like metric to work. But it makes natural noise distribution in both luma and chroma channels. I not sure if many or any 'noise addition' plugins may generate real natural photon-shot noise with it complex enough params (like moving from Poisson distribution at low photons number to 'Natural-Gaussian' with high photons number) + some 'read-noise' of sensors.
Though for static scene it may be much simple like make interval shooting with digital camera from good tripod with RAW files writing.

[zorr]

It's often mentioned that dct=5 would be the most optimal choice, so it would be interesting to see how it fares. Also a bigger value for pel seems to change things a lot, at least when viewing things with MShow.

Before I locked DCT=1 I did three runs with all the choices and DCT=5 was the second best. The comparison is not completely fair though, as you can see the DCT=1 was tested with almost 70000 combinations and DCT=5 with "only" about 35000. That's the nature of the optimization as it tries to focus on the most promising results it has found so far.

There were some DCT=5 results in the pareto front as it was able to beat DCT=1 in speed.

https://i.postimg.cc/xd36gdpF/MFlow-FPS-DCT.png

[zorr]

In lack of MANOVA we can try Excel Data Analysis tool, my Excel version is too old though. Will try to do in python or something.
A correlation matrix builds a variable x variable sized matrix showing the correlations between them. Simply go to Data -> Data Analysis -> Correlation. I don't think Google Spreadsheets has analysis tools.

Thanks for the tip, I have Excel so I can try that. But first we have to stop and think what this correlation matrix is actually telling us. So we're talking about the Pearson correlation coefficient (https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) which is a number between -1 and 1. If the value is -1 or 1 then the two populations can be perfectly described by a linear equation. If the value is zero the populations definitely cannot be described that way. So a large absolute correlation implies that the two variables have a very simple relation with one another. So for example if we calculate correlation between time(ms) and DCT (which in your chart is very large, 0.81) what would mean that by increasing the DCT the script runs slower. DCT correlates negatively with GMSD so by increasing the DCT the GMSD should decrease (and smaller GMSD means better quality in this context). I think that is true to some extent but we cannot just assume that we can get the best quality by using the maximum DCT value. Also since DCT=1 is very slow the relation of slower script and larger DCT does not hold very well either.

What about correlations with two parameters, what do they mean? So for example correlation between blockSize and DCT. Well, that only tells what values you have tested and how randomized they really were. If there was a strong correlation between any two parameters it would mean the population was not random.

The correlations between parameters might be useful if they're only calculated from the results in the pareto front after the optimization is finished. We might see interesting relations on how two change one variable in relation to another in order to stay at the pareto front.

There are of course other ways to visualize and analyze the pareto front, Zopti can create a heatmap of two parameters and display only the values in the pareto front, brightness reflecting the GMSD score for example.

If you really want to get into the relations between two parameters, you should lock down all the parameters except for two and do an exhaustive search and see the heatmap, the result can be very complex like this one below:

https://i.postimg.cc/ZqJ5XfjK/halos-ghosts-judder-ex4-2021-05-09-22-57-38-optimize-exhaustive-run-01-1.png

This (correlation calculation) can work to tune the Sensitivity internals.

The sensitivity estimation is such a dumb idea that it may actually be genious. You take the last N results from the log and do a NxN comparison (ie you compare every result with all the others). You calculate the average change in the primary result (in this case GMSD) and how much the parameter's value was changed. You calculate this for every parameter. The value tells how sensitive the GMSD is to changes in this parameter and the mutation algorithm can utilize this information to decide how much each parameter should be mutated on average. I think because the number of comparisons is so large (for example using 100 last results you get almost 10000 comparisons) the estimated sensitivity is useful even though we're disregarding which other parameters also changed.

I guess the idea could be extended to consider how much (and in which direction!) each parameter should mutate assuming these other parameters are also mutated, but I have to think about that more.

[zorr]

"Currently it's locked at a value of 2"

It is really very great deviation from default 0. It either mean default 0 is wrong or something else. Also 2 is currently max allowed value and not allow any more adjustment if it continue to increase quality (may be 3 is even better ?).


I also noticed that ScaleCSAD=2 is the best option in my MFlowFPS tests. I went further and visualized the best result of every ScaleCSAD value, trymany is also included because it was another head scratcher.

https://i.postimg.cc/4yTrJJZf/allparams-scale-CSAD-trymany.png

So there seems to be a strong correlation with larger ScaleCSAD -> better result.

DTL, since you know how to interpret the MVTools source could you take a look at trymany, what does it do and why trymany=true gives worse results than trymany=false?

My notes about these two parameters from the other thread were


trymany=false is surprising, or perhaps I don't understand what it means. The docs say "try to start searches around many predictors". So it sounds like trymany is doing more work to try to find the best possible block matches but somehow this ruins quality. Perhaps it's the same phenomenon we saw with the large search range (which also destroys quality).
scaleCSAD=2 means the luma:chroma ratio used in SAD calculations is 4:8 so chroma is twice as important as luma. That's a bit of a suprise. In my script the video is converted to YV24 before going into MVTools so chroma already has the same resolution as luma.

[DTL]

"what does it do"

It is long to type - better next day. It is night here and better to try to sleep somehow.

"scaleCSAD=2 means the luma:chroma ratio used in SAD calculations is 4:8 so chroma is twice as important as luma. That's a bit of a suprise. In my script the video is converted to YV24 before going into MVTools so chroma already has the same resolution as luma."

It is shorter to type I hope:

4:8 for YV12 not mean UV is twice more weighted.
Initially at imaging device that is RGB naturally (film or 3chip/3tube camera) we have 3 RGB full-band channels and they all have equal noise level (about equal because red and blue photons may be different in number ?).
When YV12 that is YUV 4:2:0 created from full-band RGB we have different band channels:
Y channel is full-band and 'initial noise level' that create SAD_Y of max absolute value.
U and V channels are half banded (as bt.601 require without anti-gibbs additional high frequency attetuation) that is about half of noise too. If you can read SAD for level 1 (half sized) you will see it is about half smaller in compare with level 0 (full size - output level). Or you can resize input frame 2 time smaller and look at the SAD value in MShow().
Also SAD values are absoluted and each UV block in 4:2:0 is 1/2 linear size and 1/4 of samples count. So if it even have same relative 'noise level' it will produce only 1/4 of SAD addition to full output sum.

So with simple YV12 luma + chroma addition (and equal noise, that is not true).
outSAD = SAD_Y + 1/4 (SAD_Y as SAD_U) + 1/4 (SAD_Y as SAD_V) . Because noise level in 1/4 sized U and V blocks also lower in compare with Y full-band channel - the real proportion is even worse: may be close to
outSAD = SAD_Y + 1/8 (SAD_Y as SAD_U) + 1/8 (SAD_Y as SAD_V).

Now MAnalyse have build-in ScaleSADchroma() function to tweak relative weight of U+V SADs in output sum. With 4:8 you get only x4 multiplier to SAD_U + SAD_V:

outSAD = SAD_Y + (1/4..1/8 (SAD_Y as SAD_U) + 1/4..1/8 (SAD_Y as SAD_V))x4. result is about SAD_Y + (1..1/2)SAD_U + (1..1/2) SAD_V - it is only close to 1:1 proportion of luma to chroma SADs in output sum.
Also it may be not correct to count U and V channels 'equal' to Y channel in this summing - may be it need to be something about 'medium geometric'. So we again have a filed for experiments:

Example of possible 'common SAD' calculation:
outSAD= SAD_Y + Chroma_weight * SQRT(SAD_U^2 + SAD_V^2));

Also the output SAD may be calculated not in YUV colour space but in something like HSV with different weigthing of H and S channels.

But it looks still no good science exist what is best proportion required. May be because with some footages the chroma noise may be too large in compare with luma noise - so it require to lower chroma weight. So current default 4:2 for YV12 still make chroma weight about 1/4 of luma or less. And increasing of scaleCSAD to 2 only make weights of SAD_Y and sum of SAD_U + SAD_V about equal.

[Dogway]

I knew your latest script version but it wasn't working correctly on my end, some error on non existant frame property in last frame (let me check tomorrow and I report). Also I wanted to track per frame times to debug or tune my script. I tried to modify it without success maybe you can sorta add the array trick you had in mind, ArrayAdd() is now internal. The good news is that your script gave some insight into accumulated stats for per scene metrics, I wanted to avoid globals because I'm very stubborn, but maybe I should give in lol

I was outside so I couldn't add a note on the correlation matrix. This is a random data set so it doesn't imply a coherent relationship for example between pnew and pzero, it just compares random sequences except for the first two columns GMSD and ms. Those are the ones that matter until I find other ways, or maybe you can find out since you have more math background. In my script GMSD values are inverted so 1 is reference quality, therefore higher DCT implies lower quality in the chart, same for other settings.
What I aim to find is a real correlation between settings, one way would be to limit the dataset to the first quantile or maybe pick only the pareto front. But I'm not sure if comparing by pairs is enough, raising thSAD might imply a more dimensional behaviour where thSADC needs to be lowered, maybe same for thSADR and who knows what else (lambda, badSAD, etc), everything is on multivariate analysis. In that regard sensitivity works because it's as simple as it gets, straight forward, but having those interrelations can also be useful.

index 0 frameprops was implemented in 3.7.1 test 22, or 24 or so, can't remember. It's a game changer in my opinion. Either way you can use other similar prefilter, I wanted to be fair with the comparison since you would never run such a clip without a prefilter.

I have 32Gb of RAM yet I'm using max memory of 2048 for each script, I'm not sure if more is necessary, but my CPU is old, I know things get a bit slower in MT but still faster in the big picture.

The problem I see is that a simple AddGrainC() can't mimic real grain. You would need to use something more like GrainFactory3mod, and then encode it so DCT takes the grain to lower frequencies (DCTFlicker as Didée called it). It's an option sure. But I have reserves with synthetic grain, in filmed content grain is the actual image not a transparent layer over it, grain models the detail contours so I like to think of grain as a convolution layer. I half made (https://forum.doom9.org/showthread.php?p=1959080#post1959080) a SAD analysis tool I need to finish it but it's a manual task. When knowing scene bounds beforehand it can run just once and save some time.

DTL, an idea, render a CGI animation with high samples and low samples. With low samples you get film like grain (not sure if it respects grain distribution though), then encode to h264.
By the way in my random tests trymany gave better scores but was too slow so I defaulted to false.