Zopti

[zorr]

Originally Posted by Boulder

I think I just renamed filter_param_a and filter_param_b to a1 and a2 to keep things backwards compatible with my scripts and templates.

Thanks, got it working. Also had to set kernel='bicubic' since the default is spline36 in the latest version.

I ran some exhaustive tests with your clip, I set the divider to 10.0 instead of 100.0 to make it faster. The results were interesting:



SSIM and GMSD are roughly in agreement where the best settings are. SSIM had best at b=-6 c=3. Note that this is equivalent to b=-60 c=30 with your original script which is very close to your best result b=-59, c=31. GMSD's best is at b=-7 c=1, a bit further away from your result b=-77 c=5. GMSD is a bit more "focused", it seems to see the differences better than SSIM.

VMAF on the other hand gives many "perfect" results and when we look at only those the center is very close to the top left coordinates b=-30 c=30. I think VMAF is trying to say that the differences are so small that they're not perceptible to average human. Have you tried eyeballing the different settings? It is however a bit of a mystery why VMAF has the center of best results in a different place than SSIM and GMSD.

[Boulder]

To be honest, without zooming in, it's not easy to see the differences of the optimal parameters provided by the various metrics. My guess is that VMAF is not the best method for these single frame comparisons but has much better value with moving video. GMSD looks quite promising with these scaling comparisons.

[ChaosKing]

muvsfunc added a new similarity metric: MDSI(Mean Deviation Similarity Index) https://github.com/WolframRhodium/mu...0cf0cb6b1db304

MDSI is a full reference IQA model that utilize gradient similarity (GS), chromaticity similarity (CS), and deviation pooling (DP).
The lowerer the MDSI score, the higher the image perceptual quality.
Larger MDSI values indicate to the more severe distorted images, while an image with perfect quality is assessed by a quality score of zero.

@Boulder you shoud also try https://github.com/fdar0536/VapourSynth-butteraugli
It detects even the smallest change very reliably.

[ChaosKing]

I've added butteraugli and mdsi to zoptilib https://pastebin.com/511BcmNp

[Boulder]

Thanks, nice to test various options

[zorr]

Originally Posted by ChaosKing

I've added butteraugli and mdsi to zoptilib https://pastebin.com/511BcmNp

Thanks! I'm running some tests on those two. Some initial observations:

MDSI seems promising based on the authors' paper. It requires clips to be RGB. Boulder, is your source material in BT 709 colorspace? I'm not sure if Zopti should automatically do the conversation or just require RGB. MDSI by default downscales the clips by 2 which is kinda counterproductive when trying to measure the effects of scaling. I will add a method to set the downscaling value. The current MDSI implementation in muvsfunc returns "inf" for one of the frames and the script doesn't finish. Perhaps it's not ready for prime time (since it's not even released yet).

Butteraugli can indeed measure small differences, that's its main purpose. So it may not be accurate with large differences but should work well for Boulder's use case. It's also slow. I mean slooooow... one iteration (8 frames) takes 22 seconds on my machine. Butteraugli also needs RGB and it needs to be gamma corrected... if I'm interpreting it correctly clips should be in linear RGB space. How does one do that conversion with Vapoursynth?

// Value of pixels of images rgb0 and rgb1 need to be represented as raw
// intensity. Most image formats store gamma corrected intensity in pixel
// values. This gamma correction has to be removed, by applying the following
// function:
// butteraugli_val = 255.0 * pow(png_val / 255.0, gamma);
// A typical value of gamma is 2.2. It is usually stored in the image header.
// Take care not to confuse that value with its inverse. The gamma value should
// be always greater than one.
// Butteraugli does not work as intended if the caller does not perform
// gamma correction.

[WolframRhodium]

Originally Posted by zorr

The current MDSI implementation in muvsfunc returns "inf" for one of the frames and the script doesn't finish.

Could you provide the material related to such error to me, so that I could fix it asap?

[zorr]

Originally Posted by WolframRhodium

Could you provide the material related to such error to me, so that I could fix it asap?

Here's the script:

import vapoursynth as vs
import resamplehq as rhq
from zoptilib import Zopti

core = vs.core

orig = core.ffms2.Source(source=r'blacksails.avi')

# convert to RGB
orig = core.fmtc.resample(clip=orig, css="444")
orig = core.fmtc.matrix(clip=orig, mat="709", col_fam=vs.RGB)
orig = core.fmtc.bitdepth(clip=orig, bits=8)

zopti = Zopti(r'result.txt', metrics=['mdsi', 'time'])	# initialize output file and chosen metrics 

b = -30/10.0					# optimize b = _n_/10.0 | -30..30 | b
c = -30/10.0					# optimize c = _n_/10.0 | -30..30 | c

alternate = rhq.resample_hq(orig, width=1280, height=720, kernel='bicubic', filter_param_a=b, filter_param_b=c)
alternate = core.resize.Bicubic(alternate, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
zopti.run(orig, alternate)

You can download the source clip from here.

You need the zoptilib version ChaosKing modded, here.
Oh and thanks for the muvsfunc library, the similarity metrics are perhaps the most important ingredient on making an optimizer like zopti work!

[WolframRhodium]

Originally Posted by zorr

You can download the source clip from here.

You need the zoptilib version ChaosKing modded, here.

It may have been fixed after this commit. I also disable downsampling by default now.

Thank you for your support and recognition. Your work on Zopti is impressive.

[zorr]

Originally Posted by WolframRhodium

It may have been fixed after this commit. I also disable downsampling by default now.

Thanks, it is indeed fixed. And the results are in:



Butteraugli is not as smooth function as the others especially near the best value. It's more focused than SSIM and slightly more focused than GMSD near the best value. Best value at b=-8, c=2 which again quite nicely agrees with SSIM and GMSD. It took almost 24 hours to create that picture, hopefully I don't have to do that again... but of course I have to because this was calculated without gamma correction. But I think I now know how to do it:

# convert to linear RGB
orig = core.fmtc.resample(clip=orig, css="444")
orig = core.fmtc.matrix(clip=orig, mat="709", col_fam=vs.RGB)
orig = core.fmtc.transfer(orig, transs="709", transd='linear')  # to linear RGB
orig = core.fmtc.bitdepth(clip=orig, bits=8)

MDSI is even more focused than GMSD. Best value is at b=-6, c=4, still very close to the others.

All except VMAF are giving best b within [-6 .. -8] and best c within [1 .. 4]. So for this use case it probably doesn't matter which one of those is chosen. More tests are needed to determine which one works best for other use cases.

Originally Posted by WolframRhodium

Thank you for your support and recognition. Your work on Zopti is impressive.

Thanks. Are you planning on implementing more similarity metrics? Of course quality matters more than quantity but it's always possible to optimize for more than one similarity metric and there might be some useful combinations.

[WolframRhodium]

Originally Posted by zorr

Are you planning on implementing more similarity metrics? Of course quality matters more than quantity but it's always possible to optimize for more than one similarity metric and there might be some useful combinations.

I will implement more if I find them practical. It's also interesting to see if those no-reference metrics work well, but I have not found good candidates.

Anyway, you can set 'downsample=False' in SSIM to skip downsampling. I think those full-reference metrics are highly correlated.

[zorr]

Originally Posted by WolframRhodium

I will implement more if I find them practical. It's also interesting to see if those no-reference metrics work well, but I have not found good candidates.

A no-reference metric would be awesome because it would enable a whole new class of things you could do with the optimizer. Of course it would depend on how good the metric actually is...

What's your opinion on WaDIQaM / DIQaM? The paper is here and someone did a PyTorch implementation of WaDIQaM. It can also work as full reference metric.

Originally Posted by WolframRhodium

Anyway, you can set 'downsample=False' in SSIM to skip downsampling.

Ah yes, I noticed the downscaling when I tried to compare Avisynth's SSIM and your implementation. I couldn't make them give identical results even though I set the k1, k2 and downsample -parameters to match the Avisynth version. Perhaps the Avisynth version is non-standard.

Originally Posted by WolframRhodium

I think those full-reference metrics are highly correlated.

Agreed, at least in this particular case. I compared SSIM and GMSD with denoising and there SSIM had more ringing artifacts.

[WolframRhodium]

Originally Posted by zorr

What's your opinion on WaDIQaM / DIQaM? The paper is here and someone did a PyTorch implementation of WaDIQaM. It can also work as full reference metric.

It looks good, but a network with 10 conv layers and 2 fc layers seems to be too heavy for parameter optimization (upconv7 model from waifu2x consists of only 7 conv layers though it doesn't have any pooling layer). Based on my experience in image restoration, I'm also worry that those DL-based methods can't generalize well in the real-world settings.

Porting it to VapourSynth is not hard. I will check it later.

[zorr]

Originally Posted by WolframRhodium

Based on my experience in image restoration, I'm also worry that those DL-based methods can't generalize well in the real-world settings.

I suspect that too, there was a cross-database evaluation and the no-reference models only do reasonably well when the distortions are of the same type as in the training set. SOM (Semantic obviousness metric for image quality assessment) looks pretty good at generalizing but I couldn't find any implementations of it.

Originally Posted by WolframRhodium

Porting it to VapourSynth is not hard. I will check it later.

Thanks, it will be interesting even if it doesn't perform well in real-world tests.

[zorr]

Originally Posted by zorr

It took almost 24 hours to create that picture, hopefully I don't have to do that again... but of course I have to because this was calculated without gamma correction.

Here's a comparison of Butteraugli without gamma correction and with it (in linear RGB):



There are some changes, nothing radical but the gamma corrected version is slightly smoother. Best value has moved from c=2 to c=1.

[kriNon]

From what I can tell this plugin is still being actively developed. Is it in a finished enough state that it is usable? and if so, is there any documentation on how to use it?

My goal would be to use it to find settings for MVtools to interpolate frames for anime at the highest possible quality. Motion interpolation normally fails with animated content, and so I would like to see how well it works in a best case scenario with optimized settings. I'm working on trying to restore a field blended show, and if this works well enough, it would be very helpful for me.

[zorr]

Zopti version 1.0.1-beta released, see the first post.

• #output definition is no longer needed when using zoptilib (reads output file and metrics from Zopti initialization line if #output not given)
• accept zero time as valid output value (happens when sub 10ms times are rounded towards zero)
• updated zoptilib to version 1.0.3
  -new similarity metrics MDSI and Butteraugli
  -new (semi)automatic YUV -> RGB conversion for metrics that need RGB
  -new init parameter matrix to set the YUV color matrix for the RGB conversion
  -added toRGB() function for manual RGB conversions
  -no downsampling by default in any of the metrics

Here's an example on how to use the new features:
-no #output line
-using MDSI metric with (semi)automatic RGB conversion

# read input video
video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')

# initialize metrics, specify YUV color matrix for RGB conversion (MDSI and Butteraugli need RGB)
zopti = Zopti(r'results.txt', metrics=['mdsi', 'time'], matrix='601')

... process the video ...

# measure similarity of original and alternate videos, save results to output file
zopti.run(orig, alternate)

[zorr]

Originally Posted by kriNon

From what I can tell this plugin is still being actively developed. Is it in a finished enough state that it is usable? and if so, is there any documentation on how to use it?

Yes to all of the above. The documentation is a bit lacking though, there is a tutorial series at the Avisynth Developer section. My intention is to add similar tutorials here as well (obviously using Vapoursynth syntax). Those Avisynth tutorials are usable though, the syntax of the optimizer "augmentations" has not changed. So take a look at these:

Augmented Script (part 1/2)
Augmented Script (part 2/2)
Hands-on tutorial (part 1/2)
Hands-on tutorial (part 2/2)
Optimizer arguments

The optimizer is fully functional, the remaining issues are mostly about how to use it most effectively. For example to figure out things like

• how many frames are needed for the optimization
• how to select the frames used in the optimization (and how to do that automatically)
• which parameters should one select for optimization for each filter (ie where can you get the best bang for buck)
• create optimization "templates" for different use cases (denoising, frame interpolation, color correction, etc) with optimal parameter ranges
• which similarity metrics to use (and does it depend on the use case)
• what kind of preprocessing works best for MVTools frame interpolation (and does it depend on the source video)

Originally Posted by kriNon

My goal would be to use it to find settings for MVtools to interpolate frames for anime at the highest possible quality. Motion interpolation normally fails with animated content, and so I would like to see how well it works in a best case scenario with optimized settings. I'm working on trying to restore a field blended show, and if this works well enough, it would be very helpful for me.

To get you started here's an MVTools script with the augmentations you need. I'm not sure if I used all the parameters possible, you can add/remove them as you wish.

import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 10		# how many frames are tested
MIDDLE_FRAME = 50		# middle frame number

# read input video
video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')
orig = video

# initialize Zopti - starts measuring runtime
zopti = Zopti(r'results.txt', metrics=['ssim', 'time'])

# you could add preprocessing here to help MSuper - not used here
searchClip = orig

super_pel = 4					# optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2					# optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2				# optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = core.mv.Super(searchClip, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter)
super_render = core.mv.Super(orig, pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, levels=1)

blockSize = 8					# optimize blockSize = _n_ | 4,8,16,32,64 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 5					# optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 2					# optimize searchRange = _n_ | 1..10 | searchRange
searchRangeFinest = 2				# optimize searchRangeFinest = _n_ | 1..10 | searchRangeFinest
_lambda = 1000*(blockSize*blockSize)/(8*8)	# optimize _lambda = _n_ | 0..20000 | lambda
lsad=1200					# optimize lsad=_n_ | 8..20000 | LSAD
pnew=0						# optimize pnew=_n_ | 0..256 | pnew
plevel=1					# optimize plevel=_n_ | 0..2 | plevel
overlap=0					# optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=0					# optimize overlapv=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / | overlapv
divide=0					# optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = True				# optimize globalMotion = _n_ | False,True | globalMotion
badSAD = 10000					# optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24					# optimize badRange = _n_ | 4..50 | badRange
meander = True					# optimize meander = _n_ | False,True | meander
trymany = False					# optimize trymany = _n_ | False,True | trymany

# delta 2 because we interpolate a frame that matches original frame
delta = 2

useChroma = True
bv = core.mv.Analyse(super_search, isb = True, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, _lambda=_lambda, lsad=lsad, pnew=pnew, plevel=plevel, _global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badsad=badSAD, \
 badrange=badRange, meander=meander, trymany=trymany)
fv = core.mv.Analyse(super_search, isb = False, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, _lambda=_lambda, lsad=lsad, pnew=pnew, plevel=plevel, _global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badsad=badSAD, \
 badrange=badRange, meander=meander, trymany=trymany)

# NOTE: we disable scene change detection by setting thSCD1 very high
blockChangeThreshold = 10000 
maskScale = 70								# optimize maskScale = _n_ | 1..300 | maskScale
inter = core.mv.FlowInter(orig, super_render, bv, fv, time=50, ml=maskScale, thscd1=blockChangeThreshold, thscd2=100, blend=False)

# for comparison original must be forwarded one frame
orig = orig[1:]

# cut out the part used in quality / speed evaluation
inter = inter[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
orig = orig[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(orig, inter)

Note that this script interpolates using delta=2 (uses even frames to recreate odd frames and vice versa). You can use delta=1 (which should be an easier goal) if you do the interpolation twice - first create new inbetween frames, then throw away the original frames and finally interpolate new inbetween frames again which should match the original frames. I can give you an example script of that if you want.

[zorr]

@ChaosKing, I stole your improvements to zoptilib, added some of my own and packaged it to the latest Zopti installation zip.

Perhaps we could agree on how to do the updates in the future. We could use your Github repository but I'd need to have write permissions there. Is that possible?

[ChaosKing]

Yes it is possible. I can add you as a Collaborator.
You could also make a new repo and/or clone mine and just accept Pull requests.