Introducing Zopti (ex AvisynthOptimizer)

[Seedmanc]

How do I filter/limit the boolean or string values? Suppose I want to have a variable always be false when another var is below 1. Usually I'd write it as filter:Clevel 1 > NULL true ? x != so that when Clevel is above 1, x is compared to something that it is guaranteed not to be, thus resulting in an always true statement, allowing both true and false. But AvsOptim doesn't know what NULL is or whatever else I try to put instead of that.

[zorr]

How do I filter/limit the boolean or string values? Suppose I want to have a variable always be false when another var is below 1. Usually I'd write it as filter:Clevel 1 > NULL true ? x != so that when Clevel is above 1, x is compared to something that it is guaranteed not to be, thus resulting in an always true statement, allowing both true and false. But AvsOptim doesn't know what NULL is or whatever else I try to put instead of that.

There are a couple of ways to approach this. The simplest way is probably returning true whenever Clevel > 1 (all values of x are valid in that case) and also return true when x == false (no matter what Clevel is, x is allowed to be false). In infix form it's

(Clevel > 1) or (x == false)

and translated to reverse polish notation

Clevel 1 > x false == or

Note that in your original filter and this one false is the only allowed value when Clevel is below 2. If you want it false when Clevel is below 1 then use Clevel > 0 or Clevel >= 1.

[Boulder]

I came up with an idea to reduce the amount of needed iterations, works at least for the resizer test that I run in exhaustive mode.

My results seem to be a bell-shaped curve, so it would be safe to skip the rest of the iterations of b against constant c. Thus, when the SSIM result of a pair is lower than the previous result, start the next round of varying b against c.

[zorr]

I came up with an idea to reduce the amount of needed iterations, works at least for the resizer test that I run in exhaustive mode.

My results seem to be a bell-shaped curve, so it would be safe to skip the rest of the iterations of b against constant c. Thus, when the SSIM result of a pair is lower than the previous result, start the next round of varying b against c.

Yes that would work. Your problem is the kind of problem that can be solved with a hill climbing (https://en.wikipedia.org/wiki/Hill_climbing) algorithm. The mutation algorithm works as hill climbing with population 1 and a small mutation amount. Try these parameters:

-alg mutation -pop 1 -runs 1 -mutcount 1 -mutamount 0.01

The search will start at some random location and gradually move towards the optimum. The heat map looks like this:

https://i.postimg.cc/mkjVCgRV/boulder-hillclimbing.png

The heat map will show a "cross" when the optimum has been reached (the optimum being located at the center of the cross).

[Boulder]

Try these parameters:

-alg mutation -pop 1 -runs 1 -mutcount 1 -mutamount 0.01

The heat map will show a "cross" when the optimum has been reached (the optimum being located at the center of the cross).

Thanks, worked fine at least with my test of 5 episodes. The first five were run with those settings and the last five with the exhaustive algorithm.


Run 1 best: 4.952373 570 b=-54 c=52
Run 2 best: 4.934916 600 b=-56 c=58
Run 3 best: 4.9637823 590 b=-46 c=34
Run 4 best: 4.97074 600 b=-41 c=60
Run 5 best: 4.8731537 570 b=-67 c=57
Run 6 best: 4.952373 570 b=-54 c=52
Run 7 best: 4.934916 600 b=-56 c=58
Run 8 best: 4.9637823 600 b=-47 c=34
Run 9 best: 4.97074 580 b=-41 c=60
Run 10 best: 4.8731537 590 b=-67 c=57


The results of a hillclimb series looks like this, so there's probably room for improvement. There were 2000 iterations (default?) there but the best result would probably have been found with less. The exhaustive test needs 2856 iterations at the moment - b is -75..25 and c is 15..70. If the best result hits any of those borders, I run a short optimization run with min-max limits close to the border in question.

https://i.postimg.cc/ZK9x76Sp/hillclimb.png

[zorr]

The results of a hillclimb series looks like this, so there's probably room for improvement. There were 2000 iterations (default?) there but the best result would probably have been found with less. The exhaustive test needs 2856 iterations at the moment - b is -75..25 and c is 15..70. If the best result hits any of those borders, I run a short optimization run with min-max limits close to the border in question.

Yes it looks like the optimal result can be found with less iterations. In my example it was found within 300 iterations but it could vary (it's a randomized process after all). You can set the number of iterations with the argument -iters.

I think this is where the dynamic iteration count could be useful because it can stop the optimizing process when no better results are found. I ran some tests and found settings where the optimum was found in 10 out of 10 runs (using b and c range -100..100). Try these:

-alg mutation -iters dyn -dyniters 12 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

The mutation amount is large (0.1) in the beginning and small (0.01) in the end, and looks like this:

https://i.postimg.cc/9QwCfFVs/boulder-dynamic.png

It takes on average about 100 iterations per run. That's 28 times faster than the exhaustive algorithm, not bad! :)

[Boulder]

Try these:

-alg mutation -iters dyn -dyniters 12 -dynphases 2 -pop 1 -runs 1 -mutcount 1 -mutamount 0.1 0.01

It takes on average about 100 iterations per run. That's 28 times faster than the exhaustive algorithm, not bad! :)

Thanks, I'll definitely test that with the next batch of encodes I have. A hundred iterations sounds tremendous, as I can then use more frames for analysis to make sure I get a good allround result.

[Dogway]

Is it possible to resume a test that finished too early and take logs into consideration to refine future mutations?
I'm using the last dynamic mode posted above but it finished too early.

I might have understood it wrong but in my tests the best result is always the one with least change, naturally we want a change since we are denoising. The following always ends with a smdegrain(tr=2,thSAD=100,contrasharp=true,blksize=8,overlap=4,divide=0,refinemotion=true,lsb=true)
sigma = 20*20 # optimize sigma = _n_*20 | 100..600 | sigma
blockSize = 8 # optimize blockSize = _n_ | 4,8,16,32 ; min:divide 0 > 8 2 ? ; filter:overlap 2 * x <= | blockSize
overlap = 4 # optimize overlap = _n_ | 4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 8 2 ? % 0 == | overlap
tr = 3 # optimize tr = _n_ | 2..4 | tr
divide = 0 # optimize divide = _n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
denoised=smdegrain(tr=tr,thSAD=sigma,contrasharp=true,prefilter=2,blksize=blockSize,divide=divide,overlap=overlap,refinemotion=true,lsb=true)

[zorr]

Is it possible to resume a test that finished too early and take logs into consideration to refine future mutations?

There's no real resume support, but you can take the best results from the logs as the initial population with the -initial argument. Take a look here (https://forum.doom9.org/showthread.php?p=1856877#post1856877). I can see the value in resuming the optimization, for example when Windows suddenly decides that it's mandatory to install some updates and reboot the computer while I'm running an optimization with 50000 iterations... So I will probably implement that at some point.

I'm using the last dynamic mode posted above but it finished too early.

Those settings are not really suitable for the general case, they were handcrafted to a specific kind of optimization which is easier than most. You probably need a lot more iterations than those settings provide. You should perhaps try the default optimization settings first and see if you get better results with those.

I might have understood it wrong but in my tests the best result is always the one with least change, naturally we want a change since we are denoising.

When doing denoising you need to make a clip with added noise, then remove the noise and finally compare the denoised clip with the original. See here (https://forum.doom9.org/showthread.php?p=1851670#post1851670) for an example. I have thought about other ways (https://forum.doom9.org/showthread.php?p=1856031#post1856031) to do it but this is currently the most reliable way.

Can you show what kind of values you are returning from the script?

[ChaosKing]

When doing denoising you need to make a clip with added noise, then remove the noise and finally compare the denoised clip with the original.

I would be nice if we would have also halo and "other crap" simulators to easily test (optimize) other kinds of filters.

[Dogway]

When doing denoising you need to make a clip with added noise, then remove the noise and finally compare the denoised clip with the original. See here (https://forum.doom9.org/showthread.php?p=1851670#post1851670) for an example. I have thought about other ways (https://forum.doom9.org/showthread.php?p=1856031#post1856031) to do it but this is currently the most reliable way.

I see, that makes more sense although it kind of becomes a synthetic test since normally we don't have a ground truth to compare to. I'll be doing grain modeling with GrainFactory instead since it resembles more film grain.

I thought on using the dynamic mode because if I only use iterations I felt it was being stucked into some ranges.
Leaving it at low iterations and 5 run cycles it would get out of said ranges but now it would totally ignore past cycle logs which is undesired. I will try to do some other tests with low cycles and high iterations.

[Dogway]

Following your indications and still get the least-change variables.

Best result:
smdegrain(tr=1,thSAD=100,contrasharp=0,prefilter=0,blksize=32,overlap=12,divide=0)

Not sure if I'm doing something wrong. If you need my scripts please tell me and I upload them.

[zorr]

Not sure if I'm doing something wrong. If you need my scripts please tell me and I upload them.

Yes that's a good idea. I will take a look.

[EDIT] I have a suspicion of what the problem is. If your video is very grainy then the degraining process cannot remove a lot of noise in order to look like the original. This add noise -> remove noise -> compare only works well when the original video doesn't have much (or any) noise. You could try finding a part of the video where there is very little noise or perhaps even use a completely different video (with very little noise but similar content and movement) as the source in the optimization.

[Dogway]

Here's the link with the 3 clips. A slightly modified version of smdegrain to test contrasharp and divide. Prefilter has a few dependencies so you might want to comment it.

[zorr]

Here's the link with the 3 clips.

Thanks. I don't know why but my 7zip says it doesn't support the compression method. I downloaded the latest version but apparently it needs a reboot in order to install and I'm in the middle of optimization process so I don't want to do it now. Can you provide the files in another zip format?

[StainlessS]

Zorr, My version 7zip from April 2018 opens Dogway zip no problem, yours must be reasonably old version.

Latest update 2 days ago, 2018 Dec 30:- https://www.7-zip.org/download.html

No reboot required here (x64 version).
I think about 12 -> 18 months ago, some fancy new algo was added to 7zip.

For 7zip using zip archive, should perhaps set to Normal compression,
and whatever you wish for 7z archives.

[zorr]

Here's the link with the 3 clips. A slightly modified version of smdegrain to test contrasharp and divide.

Is your modified version uploaded somewhere? Looks like the latest released SMDegrain version (v3.1.2.100s) doesn't have the divide argument. And it would be a good idea to use the exact same function anyway so we can compare results.

The source video doesn't have any grain so it should work well...

I'm going to have to delay the testing a bit because I also need to update MVTools2 and the optimization task I'm running uses an older version. I don't want to do anything radical while it's running... should be done tomorrow.

[zorr]

I would be nice if we would have also halo and "other crap" simulators to easily test (optimize) other kinds of filters.

The halo removal is something I have thought about. Using a synthetic test is probably not useful there because the effect is unique in every video so it needs customized parameters.

The way I would do it: take a single frame (or even a small part of the frame) where the halo effect is clearly visible and use Photoshop or whatever to remove it manually (using the clone stamp tool for example). Then use a halo removal filter in AviSynth and optimize its parameters so that it replicates the result. The same parameters should then work well for the whole video.

[Dogway]

Here (https://www.sendspace.com/file/3orm7i) it is, in the rush I forgot to include it. This is not a release or anything, I modified it a bit to test divide and integer type data for contrasharpening (the generic "val" data type wasn't suited for the variable). I skipped prefilter=3 because I can't make dfttest work on my end. The denoised version was denoised with: smdegrain(tr=3,thSAD=250,contrasharp=40,prefilter=2,blksize=16,overlap=8,divide=0,refinemotion=true,lsb=true). Which is (maybe?) a good starting point for grainy 1080p, in lack of better denoisers for avs+ like BM3D.

Whenever you have time is ok, I use the mvtools2 from pinterf github.

[zorr]

I see, that makes more sense although it kind of becomes a synthetic test since normally we don't have a ground truth to compare to.

I agree that it's a synthetic test. But there's something you can do to improve SMDegrain that doesn't rely on such a synthetic test. Since SMDegrain relies on MVTools you can optimize its arguments using motion compensation (MAnalyze + MCompensate). If the motion compensation is high quality then SMDegrain should benefit from that also.

Does anyone know whether MDeGrain uses pixel-based motion compensation (like MFlowInter) or the block based (like MCompensate)? I think it's the latter but I'm not sure.

Another open question: if you find good parameters for MCompensate (or MFlowInter) do they work equally well with MDeGrain? Even if they're not similar there's the MAnalyze (and MRecalculate) part which should work in identical way for all use cases.

[ChaosKing]

The way I would do it: take a single frame (or even a small part of the frame) where the halo effect is clearly visible and use Photoshop or whatever to remove it manually (using the clone stamp tool for example). Then use a halo removal filter in AviSynth and optimize its parameters so that it replicates the result. The same parameters should then work well for the whole video.

Ahhh why didn't I think of what! And testing one-two frame(s) should be enough :)

[Dogway]

Yes I dabbled with MCompensate back then, we used it for all kinds of filtering I guess I wanted to keep things simple and fast at that time but I had some beta version with it included. I want to run some tests with avsoptimizer and see where it takes me.
I think it uses MCompensate internally, you can see the vectors with MMask. In my opinion MFlowInter artifacts are less forgivable but I have been long out of the loop so things might have changed.

[Boulder]

Regarding denoising, do you have any plans for the Vapoursynth support that was briefly discussed when the optimization method was published?

[zorr]

This is not a release or anything, I modified it a bit to test divide and integer type data for contrasharpening (the generic "val" data type wasn't suited for the variable).
...
Whenever you have time is ok, I use the mvtools2 from pinterf github.

All right, figured out the problem. In the script you're comparing the denoised clip to the noisy clip when you should compare it to the original (without added noise). It was trying hard to come up with parameters that don't remove any noise. :)

So just add a line to load the original clip:
orig=ffvideosource("source.avi")

and then calculate SSIM between the original and denoised. Remove
global ssim = SSIM_FRAME(source, denoised)
and replace with
global ssim = SSIM_FRAME(orig, denoised)

I'm running a small test with such modifications and seems like it prefers tr=4 (as it should).

[zorr]

Zorr, My version 7zip from April 2018 opens Dogway zip no problem, yours must be reasonably old version.

Still couldn't open the 7zip after the update. Perhaps the package was corrupted during download. The .zip opened without problems.

[zorr]

Regarding denoising, do you have any plans for the Vapoursynth support that was briefly discussed when the optimization method was published?

I haven't looked at Vapoursynth yet. I know almost nothing about it so it's hard to say how much effort it would take. Can you recommend a tutorial / site to get one started with VapourSynth?

[ChaosKing]

The official site (http://www.vapoursynth.com/doc/) and for a quickstart my portable pack https://forum.doom9.org/showthread.php?t=175529 <- there are some bat ecnode and VS editor examples
It's very similar to avisynth.

[Dogway]

yes, that makes more sense. I guess I got confused since adding synthetic grain seems counterintuitive... off to some tests.

[StainlessS]

EDIT:
Can you recommend a tutorial / site

Perhaps a succinct Python3 intro would be easiest way into VS,
here "Learn X in Y Minutes" for Python3:- https://learnxinyminutes.com/docs/python3/
[already linked on Vapoursynth home site]

EDIT: Python 2 is somewhat different to Python 3, so give tutorials on that a wide berth.

[ChaosKing]

Here are some basics:

from vapoursynth import core
import somescript as xy


clip = core.d2v.Source(r'E:\test.d2v') # r means raw string, so u can copy paste a windows path. Alternatively use (without r) 'E:\\test.d2v'
video = clip

# call a function from our imported script
clip = xy.Filter(clip)
clip = somescript.Filter(clip) #alternative

# call a plugin function. All plugins have a namespace.
clip = core.dfttest.DFTTest(clip, sigma=5)
clip = clip.dfttest.DFTTest(sigma=5) #alternative

#compare
clip = core.std.StackHorizontal([video.text.Text("original"), clip])
clip.set_output()



EDIT
Oh and it would be also very easy to run Avisynth plugins/scripts within VS

clip = core.d2v.Source(r'E:\test.d2v')
clip_distorted = core.avsw.Eval('FineDehalo(darkstr =0)', clips=[clip], clip_names=["last"], slave_log=r"D:\debug.txt", avisynth =r"D:\AVSPLUS_x86\Avisynth.dll")

vmaf.VMAF(clip_reference, clip_distorted) # https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF

[Wolfberry]

From vapoursynth.com/doc/pythonreference.html#get_core (http://www.vapoursynth.com/doc/pythonreference.html#get_core)
core
Gets the singleton Core object. If it is the first time the function is called, the Core will be instantiated with the default options. This is the preferred way to reference the core.

get_core([threads = 0, add_cache = True])
Deprecated, use the core attribute instead.

Get the singleton Core object. If it is the first time the function is called, the Core will be instantiated with the given options. If the Core has already been instantiated, all options are ignored. Setting threads to a value greater than zero overrides the autodetection.
It's not recommended to use get_core as it is deprecated.
Just use vs.core instead of vs.get_core() (also simpler).
Alternatively, you can use "from vapoursynth import core".

[ChaosKing]

Thx, I updated it in my post. One line less now :-)

[Boulder]

I haven't looked at Vapoursynth yet. I know almost nothing about it so it's hard to say how much effort it would take. Can you recommend a tutorial / site to get one started with VapourSynth?

As others have posted, it's quite easy to grasp once you get the basics of Python code structure.

This one is something that could provide a nice enhancement for VS-based optimization compared to SSIM:
http://forum.doom9.org/showthread.php?t=175862.

[Dogway]

Based on some tests I guess prefiltering is a gimmick lol, aside so many filtering conventions sticked to our brains for so long (like blocksize 16 for 1080p). Thanks for the tool it takes some of the voodoo-guessing out of the equation. I think implementing VMAF is worth here, after reading the Netflix VMAF article a few weeks ago it seemed that SSIM did some bad assumptions, in my case contrasharpening was defaulting to 0 or false which in my opinion wasn't making justice to what I observed. That or if your port it I might start using VapourSynth occasionally for VMAF and BM3D, I just don't like how it requires you to write more, and script readability. Also something for x264 like SSIM Plus would be cool if someone knows about a free open source alternative.

[zorr]

Thanks @ChaosKing and @StainlessS, I have now read the Python tutorial, the official VapourSynth site and ChaosKing's FATPACK thread. I found a SSIM plugin for VapourSynth as well.

Is there a plugin for measuring runtime (preferably per frame)?

I suppose Python's standard I/O routines can be used to write stuff to a file during the script evaluation?

[StainlessS]

Maybe Groucho has better code for timing stuff.

But some stuff from RT_Stats below. [EDIT: Use as you will]


AVSValue __cdecl RT_LocalTimeString(AVSValue args, void* user_data, IScriptEnvironment* env) {
const bool file = args[0].AsBool(true);
SYSTEMTIME st = { 0 };
if(file) {
DWORD tick=GetTickCount();
while(GetTickCount()==tick) // Wait until system clock goes TICK (prevent two separate calls returning same time)
Sleep(0);
}
GetLocalTime(&st);
char bf[64];
if(file) {
sprintf(bf,"%4d%02d%02d_%02d%02d%02d_%03d",st.wYear,st.wMonth,st.wDay,st.wHour,st.wMinute,st.wSecond,st.wMilliseconds);
} else {
sprintf(bf,"%4d-%02d-%02d %02d:%02d:%02d.%03d",st.wYear,st.wMonth,st.wDay,st.wHour,st.wMinute,st.wSecond,st.wMilliseconds);
}
return env->SaveString(bf);
}


RT_LocalTimeString(Bool "file"=True)
Returns current local time as a string.
Where digits Y=Year, M=Month, D=Day, H=Hour, M=Minute, S=Second, m=millisecond.
When bool file==False, then string in format "YYYY-MM-DD HH:MM:SS.mmm"
When bool file==True (Default) string is in format "YYYYMMDD_HHMMSS_mmm"
Also when file==True, function first waits until the system tick count is incremented (about every 10ms)
before inquiring system time. This is to prevent 2 consecutive calls returning the same time string.
Perhaps useful for temporary filename generation.



AVSValue __cdecl RT_Timer(AVSValue args, void* user_data, IScriptEnvironment* env) {
return double(clock()) / double(CLOCKS_PER_SEC);
}


double __cdecl RT_TimerHP_Lo(void) {
LARGE_INTEGER liPerfCounter = {0,0};
LARGE_INTEGER liPerfFreq = {0,0};
bool bStat = true;
DWORD_PTR dwpOldMask = SetThreadAffinityMask(GetCurrentThread(), 0x01);
Sleep(0);
if ((QueryPerformanceFrequency(&liPerfFreq) == 0) || (QueryPerformanceCounter(&liPerfCounter) == 0))
bStat = false;
double tim;
if(!bStat) { // High precision NOT available.
static clock_t first_time32=0;
static bool done32=false;
const clock_t t = clock();
if(!done32) {
first_time32 = t;
done32=true;
}
tim = double(t-first_time32) / double(CLOCKS_PER_SEC); // Fallback, low rez timer
} else { // High precision IS available.
static __int64 first_time64=0;
static bool done64 = false;
if(!done64) {
first_time64 = liPerfCounter.QuadPart;
done64=true;
}
tim = (double)(liPerfCounter.QuadPart-first_time64) / (double)liPerfFreq.QuadPart;
}
SetThreadAffinityMask(GetCurrentThread(), dwpOldMask);
Sleep(0);
return tim; // return double for C client
}


AVSValue __cdecl RT_TimerHP(AVSValue args, void* user_data, IScriptEnvironment* env) {
return RT_TimerHP_Lo(); // Implicit type conversion to AVSValue float
}


RT_Timer()
Returns time in seconds since start of process.
Usage:
s=RT_Timer()
Test()
e=RT_Timer()
Str=RT_String("Test Start=%.2f End=%.2f Time=%.2f secs",s,e,e-s)
SubTitle(str)
RT_Debug(str)

***
***
***

RT_TimerHP()
Returns a higher precision time than RT_Timer (If available on your system, otherwise uses same routine as RT_Timer).
Note, where RT_Timer returns time elapsed in seconds since start of process, RT_TimerHP is not defined to return anything
in particular, the only thing that is meaningful is the difference between returns from 2 calls to RT_TimerHP.
DO NOT mix times from RT_Timer with RT_TimerHP, results may be meaningless.
Usage:
s=RT_TimerHP() Test() e=RT_TimerHP()
Str=RT_String("Test Start=%.3f End=%.3f Time=%.6f secs",s,e,e-s)
SubTitle(str)



High precision timer quite often called HPET in bios setup, usually defaults to OFF. [EDIT: Presumably, High Precision Event Timer]

EDIT:

env->AddFunction("RT_LocalTimeString", "[file]b",RT_LocalTimeString, 0);
env->AddFunction("RT_Timer", "",RT_Timer, 0);
env->AddFunction("RT_TimerHP", "",RT_TimerHP, 0);

[Groucho2004]

Maybe Groucho has better code for timing stuff.
Your 'RT_TimerHP_Lo' function looks pretty good in terms of precision and resolution (and somewhat familiar :D).

Edit: Déjà vu...

[StainlessS]

Your 'RT_TimerHP_Lo' function looks pretty good in terms of precision and resolution (and somewhat familiar :D).

Edit: Déjà vu...

Yep, If's I recalls correctly, me robbed some of that from your PM (I was wantin' to keep that a secret :D )

[Groucho2004]

Yep, If's I recalls correctly, me robbed some of that from your PM (I was wantin' to keep that a secret :D )Ah, yes. Alzheimer slowly kicking in.

[zorr]

But some stuff from RT_Stats below. [EDIT: Use as you will]

Thanks, I forgot to mention I need it in a VapourSynth script. :) I guess this VapourSynth discussion is a bit off topic here so I made a new thread (https://forum.doom9.org/showthread.php?t=176002) in the VapourSynth section.

[Dogway]

I wasn't convinced with leaving sharpness at 0, I run some comparisons with VMAF and other algorithms in MSU VQMT and indeed SSIM (and PSNR) are the only ones that favor blurry filters.

[Seedmanc]

So my assumptions during MFlow testing were correct, the metricts favor blur, hence the sharp=0 (and possibly pel=4) bias. Maybe you could try downscaling before calculation as well.


zorr, is there a length limit for the combination of filters and limits for a single parameter?
Whenever I add more params to dependencies than I have here

Clevel = 8/10.0 # optimize Clevel = _n_/10.0 | 0..10 ; min:super_pel2 1 > 2 0 ? super_sharp2 0 > 2 0 ? super_rfilter2 0 > 2 0 ? blockSize2 4 > 2 0 ? searchAlgo2 0 > 2 0 ? searchRange2 1 > 2 0 ? searchRangeFinest2 1 > 2 0 ? divide2 0 > 2 0 ? overlap2 0 > 2 0 ? CdoRecalc true == 2 0 ? truemotion2 true == 2 0 ? max | Clevel
I get "Unknown element badSA2" (I was adding badSAD2 after truemotion2, strange how it's skipping a letter while keeping the digit instead of just cutting the name).
Same here, add CRscaleCSAD2 to
CdoRecalc=true # optimize CdoRecalc=_n_ | false,true ; filter:Clevel 1 > Csmooth 0 > CRsearchAlgo 0 > CRsearchRange 1 > CRdivide 0 > CRoverlap 0 > Cthsad 0 > CRtruemotion true == CRmeander true == x false == or | CdoRecalc
and get "Unknown element CRscaleCSA"

[zorr]

I wasn't convinced with leaving sharpness at 0, I run some comparisons with VMAF and other algorithms in MSU VQMT and indeed SSIM (and PSNR) are the only ones that favor blurry filters.

That's a great find and good to know. This screenshot from the MSU page shows per pixel SSIM values and the resulting image is clearly blurry.

https://i.postimg.cc/Z5hXXbVK/ssim-precise.jpg

Also the SSIM index for the pixel is calculated using a 8x8 neighbourhood (fast method) or using a gaussian weighting function (high quality method). That explains why it's not able to "see" fine details.

I'm running some Vapoursynth tests now and there we also have GMSD (http://www4.comp.polyu.edu.hk/~cslzhang/IQA/GMSD/GMSD.pdf) which gave much better results than SSIM in my denoising test (best SSIM result had ringing artifacts). I haven't tried VMAF yet.

It would be great to have these quality metrics as Avisynth plugins too.

[zorr]

So my assumptions during MFlow testing were correct, the metricts favor blur, hence the sharp=0 (and possibly pel=4) bias. Maybe you could try downscaling before calculation as well.

Yes you were correct. But I think that to counter a blurry quality measurement you should upscale and not downscale.

zorr, is there a length limit for the combination of filters and limits for a single parameter?

There shouldn't be, I will take a look. Just curious though, what is that filter doing? :)

[EDIT] I tried the Clevel = ... line and it worked. But there are some small fixes in the parser code which are not released yet, so maybe only my development version accepts it. You should check that there are no line breaks anywhere, the optimizer doesn't support continuing lines with \ like Avisynth does.

Oh and if I'm interpreting it correctly you're trying to take the maximum of many numbers. That's not going to work with only one max function since it only takes the top two numbers from the stack.

[Seedmanc]

It's just a part of my ever-increasing MFlow script that now does the initial upsample, follows by the MCompensate trick and then tries to add recalculation to it (over 40 params now). However it always chooses not to do the latter part.
Since I have all the params of MCompensate tied to the weight at which I overlay the MCompensated version over the original one (so as not to try combination when the weight is under 0.2) I tried to follow the no-bias rule and tie the weight itself back to them. Same with the doRecalc switch later.

But the fact that max doesn't work with more than 2 arguments is a disappointing discovery, that means I won't be able to do the many-to-one parameter tying. Is the situation the same for the OR operator which I'm using to combine multiple boolean values?

[zub35]

I wasn't convinced with leaving sharpness at 0, I run some comparisons with VMAF and other algorithms in MSU VQMT and indeed SSIM (and PSNR) are the only ones that favor blurry filters.

Try MSU Blurring. It will show not only blur but also sharp.

Using MSU_Blurring can improve SSIM:
(SSIM+SSIM*BLUR)/(SSIM+1)

BLUR - if encode_blur < source_blur then (encode_blur / source_blur)
BLUR - if encode_blur > source_blur then (source_blur / encode_blur)

UPD: Fix calculation of the value, relative to SSIM. The lower the SSIM, the lower the effect of the blur on the final value. Example:
SSIM=0.9 BLUR=0.5 - Result=0.7105
SSIM=0.5 BLUR=0.9 - Result=0.6333
Previously, in both situations it was 0.7

p.s. There is also a B-SSIM metric that fixes SSIM problems for blurry images.
http://images2.imagebam.com/13/9d/d9/b736951096302774.png
Fig.b SSIM = 0.8241, B-SSIM = 0.6434
Fig.c SSIM = 0.7276, B-SSIM = 0.4526

[zorr]

It's just a part of my ever-increasing MFlow script that now does the initial upsample, follows by the MCompensate trick and then tries to add recalculation to it (over 40 params now). However it always chooses not to do the latter part.

It's possible that you've hit the curse of dimensionality. Every new parameter multiplies the number of possible combinations. At some point the search space gets so ridiculously large that no algorithm can find good solutions. Or it could be something else. :)

I tried doing MFlow + MRecalculate at the same time and there MRecalculate brought some extra quality. I haven't tried all three at the same time (and my method of combining MCompensate is different than yours).

I also tried MFlow + MCompensate in phases, I first optimized MFlow only and once I had optimal values I "freezed" them. I then added the MCompensate and only optimized its parameters. That way the number parameters stays lower and the search is easier. It might however result in little less optimal result than what could be possible doing them both at the same time.

I tried to follow the no-bias rule and tie the weight itself back to them.

I'm not actually sure if it's less biased to set some default values to MCompensate when it's not used because then those default values appear more often than others. You could try running with and without "no bias" and see which one gives better results.

But the fact that max doesn't work with more than 2 arguments is a disappointing discovery, that means I won't be able to do the many-to-one parameter tying. Is the situation the same for the OR operator which I'm using to combine multiple boolean values?

Yes it's the same with OR and all the other functions except "?" which takes three arguments. That's the way this reverse polish notation works. But it's not a limitation of reverse polish notation, it can do anything the infix form can and do it without parenthesis (which is the reason it's used, it's so easy to parse).

So if you need the max of say, three values, you can do it this way:
max(a,b,c) == max(a, max(b,c))

In rpn it's even simpler: a b c max max. So you just add (n-1) max functions to max n numbers. The same works with OR as well.

[zorr]

Try MSU Blurring. It will show not only blur but also sharp.

Using MSU_Blurring can improve SSIM:
(SSIM + MSU_Blurring*) / 2
* if encode_blur < source_blur then [encode_blur / source_blur]
* if encode_blur > source_blur then [source_blur / encode_blur]

That's an interesting approach, do you have link for more detailed explanation?
You could also calculate the blurriness and optimize for both SSIM and blurriness. The pareto front will then have results with increasing SSIM and blurriness and you can choose which one looks best.

Of course we'd need a blurriness detection in Avisynth to do that. I found a promising method which is simple to implement and should give good results: variance of the laplacian (https://www.pyimagesearch.com/2015/09/07/blur-detection-with-opencv/). The laplacian filter is easy to calculate with MaskTools and StainlessS's RT_YPlaneStdev can be used to calculate the variance.

p.s. There is also a B-SSIM metric that fixes SSIM problems for blurry images.

I took a look at B-SSIM (https://www.researchgate.net/publication/265164680_B-SSIM_Structural_Similarity_Index_for_Blurred_Videos) and looks like it's using almost the same method as the "variance of the laplacian", it just replaces laplacian with sobel and takes the standard deviation instead of variance. The SSIM value is then multiplied by (2*SIf*SIh)/(SIf^2 + SIh^2) where SIf is the stddev(sobel) of original frame and SIh is the same for the modified frame. Looks like that can also be implemented in a simple script.

[EDIT] Just remembered that SSIM value can be negative, the valid range is -1 .. 1. So by multiplying a really bad SSIM with < 1 you can actually make the metric better.

[zorr]

B-SSIM seemed too intriguing to pass so I went ahead and implemented it. Or at least a variation of it. The original uses the Sobel filter for edge detection, I'm using the Scharr operator (https://en.wikipedia.org/wiki/Sobel_operator#Alternative_operators) instead because it's more accurate. And by the way the "sobel" in MaskTools is not actually sobel, it detects edges but the mask is incorrect and gives sub-par results.

Other difference is that I'm not taking the maximum of standard deviations from the whole clip sample, I just take one from each frame. Taking the maximum would need a two pass algorithm and I wanted to avoid that. The difference to the original shouldn't be that large.

In my tests this seems to work as it should giving lower scores to blurrier frames.

Anyway, here's a sample code to compare a blurred clip to the original, showing both SSIM and B-SSIM(mod).

# source clip
AVISource("d:\process2\1 deinterlaced.avi")

ConvertToYV16() # for MaskTools
orig = last

blurred = orig.GaussianBlur(1.5)

scharr_orig = scharr(orig)
scharr_blurred = scharr(blurred)

final = ScriptClip(last, """
global ssim = SSIM_FRAME(orig, blurred)
bssim = BSSIM_MOD(scharr_orig, scharr_blurred, ssim)
return orig.SubTitle("SSIM "+String(ssim)+" BSSIM "+String(bssim))
""")

return final

function scharr(clip c) {
scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), u=1, v=1)
return scharr
}

function BSSIM_MOD(clip scharr_orig, clip scharr_alt, float ssim) {
stddev_orig = RT_YPlaneStdev(scharr_orig)
stddev_alt = RT_YPlaneStdev(scharr_alt)

mul = (2*stddev_orig*stddev_alt) / (stddev_orig*stddev_orig + stddev_alt*stddev_alt)
return mul*ssim
}


The implementation needs StainlessS's RT_Stats plugin (https://forum.doom9.org/showthread.php?t=165479) for the standard deviation.

Seedmanc and Dogway, can you try the B-SSIM variation and see if it fixes the blurring issue?

[ChaosKing]

An additional "toy" for comparing video qualitiy: https://github.com/fdar0536/vapoursynth-butteraugli