Zopti

[zorr]

Originally Posted by ChaosKing

Yes it is possible. I can add you as a Collaborator.
You could also make a new repo and/or clone mine and just accept Pull requests.

Let's go with the Collaborator route for now. I can make my own repo later if/when I have the time.

[ChaosKing]

Ok, send me your github email via PM then so I can add you

[zorr]

I'm trying to track down a crash which happens very consistently with this script:

import vapoursynth as vs
from zoptilib import Zopti
core = vs.core

TEST_FRAMES = 2			# how many frames are tested
MIDDLE_FRAME = 10		# middle frame number
	
# read input video
video = core.raws.Source(r'flower_cif.yuv', 352, 288, src_fmt='I420')			# YUV420, crash
#video = core.ffms2.Source(source=r'd:\process2\1 deinterlaced.avi')			# YUV420, crash
#video = core.ffms2.Source(source=r'huffyuv_rgb.avi')					# RGB, works
#video = core.ffms2.Source(source=r'24-1200.mkv')					# YUV420, works

#zopti = Zopti(r'results.txt', metrics=['ssim', 'time'], matrix='601')			# works
#zopti = Zopti(r'results.txt', metrics=['gmsd', 'time'], matrix='601')			# works
#zopti = Zopti(r'results.txt', metrics=['vmaf', 'time'], matrix='601')			# works
zopti = Zopti(r'results.txt', metrics=['mdsi', 'time'], matrix='601')			# crash
#zopti = Zopti(r'results.txt', metrics=['butteraugli', 'time'], matrix='601')		# crash

# input color range is PC (full)
video = video.std.SetFrameProp(prop="_ColorRange", intval=0)

#noisy = video
noisy = video.grain.Add(var=25)
#denoised = noisy
denoised = noisy.grain.Add(var=25)

# cut out the part used in quality / speed evaluation
video = video[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]
denoised = denoised[MIDDLE_FRAME - TEST_FRAMES//2 + (1 if (TEST_FRAMES%2==0) else 0) : MIDDLE_FRAME + TEST_FRAMES//2 + 1]

zopti.run(video, denoised)

Can someone else verify the problem and run that script? You can use this batch file to run it repeatedly until it fails, just give the script file as an argument. On my machine it always fails on the first try though. Change the vspipe path if needed.

:Loop
"D:\VapourSynth64Portable\VapourSynth64\vspipe" %1 .
@if %errorlevel% equ 0 goto :Loop
@echo Exit Code is %errorlevel%

So far I have figured out that

• Crash happens on first frame, so it doesn't have anything to do with writing the result file. This I verified by running the script in VapourSynth editor.
• Only crashes on specific source videos, I tried four different ones. You can download the flower_cif.yuv which crashes from here.
• Does not depend on source filter, crashed with core.raws.Source as well as core.ffms2.Source
• Seems to need the RGB conversion for the crash to occur. If video is already RGB there's no crash, and when using metrics where RGB conversion is not needed there's no crash.
• Adding noise to compared clips does not have any effect, the crash occurs even without it (for example when comparing identical clips)
• Crashes less frequently when run with VapourSynth editor.
• Cutting the video shorter has no effect.
• Setting or not setting _ColorRange FrameProp has no effect.
• Sometimes hangs for several seconds before it crashes, sometimes it happens much faster. CPU load stays low when it hangs.

[ChaosKing]

No crashes. Tried msdi and butteraugli. Source is YUV420 1080p.
You could also try lsmas.LWLibavSource or Avisource. Maybe there are seeking issues?

EDIT
ok... just after I posted this I got

Error: Failed to retrieve frame 0 with error: Expr: Failed to convert 'inf' to float
Output 0 frames in 1.01 seconds (0.00 fps)
Exit Code is 1

I think it ran for about 5min, ffms2 + mdsi


EDIT2
Inded, ffms2 has seeking issues with my source file. Thats very odd since it is a "normal" h264.mkv

EDIT3
It crashed with lsmash+ mdsi after ~10min with the same err message. (now its frame 1 :P)

EDIT4
I removed both trim lines and get immediately a crash

Script exceeded memory limit. Consider raising cache size.
Error: Failed to retrieve frame 12 with error: Expr: Failed to convert 'inf' to float
Output 12 frames in 2.19 seconds (5.49 fps)
Exit Code is 1

EDIT5
running in vseditor:

Error on frame 35 request:
Expr: Failed to convert 'inf' to float
Error on frame 53 request:
Expr: Failed to convert 'inf' to float
Error on frame 77 request:
Expr: Failed to convert 'inf' to float
Error on frame 105 request:
Expr: Failed to convert 'inf' to float
Error on frame 120 request:
Expr: Failed to convert 'inf' to float
2019-02-21 00:34:51.879
Error on frame 164 request:
Expr: Failed to convert 'inf' to float
Error on frame 172 request:
Expr: Failed to convert 'inf' to float
Error on frame 196 request:
Expr: Failed to convert 'inf' to float

[Are_]

I'm not able to make it crash with the default script and the source file you provided.

[ChaosKing]

If I remove addgrain or set var to a low value (var=2) it runs fine. Try setting it much higher like 80 and the convert err is triggered much more frequent.

It also happens when I replace addgrain with noisegen.Generate(). So I'm still not sure what the cause is.

Edit
I replaced convertToRGB() in zoptilib with return core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709"), but nothing changed.
Edit2
I can't trigger this error with butteraugli, so it could be a bug in msdi.

Edit3
https://github.com/WolframRhodium/muvsfunc/issues/19

[ChaosKing]

An older version of muvsfunc caused this Expr error. Now with the latest version it runs perfectly for more than an hour now. No crashes.

[Are_]

That explains why it was not triggering for me, I was using latest version.

[zorr]

Originally Posted by ChaosKing

An older version of muvsfunc caused this Expr error. Now with the latest version it runs perfectly for more than an hour now. No crashes.

This didn't fix the problem for me and I already had a version of muvsfunc where this inf problem was fixed. Also I get the error with Butteraugli as well.

I don't get any error messages either, just the return code, which is usually a large negative number like -1073741819. When it crashes in VS Editor the whole program goes down.

So I will have to keep investigating, thanks for the tests anyway. Chaos and Are, can you tell me the OS you run the tests with?

[EDIT]

It works when doing the RGB conversion with core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709"). So looks like the error is in fmtconv. Is that plugin still maintained by someone (latest change is 3 years ago)?

I could of course just use core.resize instead. What's the closest to this:

clip = core.fmtc.resample(clip=clip, css="444")
clip = core.fmtc.matrix(clip=clip, mat=matrix, col_fam=vs.RGB)
clip = core.fmtc.bitdepth(clip=clip, bits=bits_per_sample)

and converting to linear RGB:

clip = core.fmtc.transfer(clip, transs=matrix, transd='linear')

core.Resize doesn't have "601" matrix which should be used for standard definition content. Is it called something else?

[ChaosKing]

My OS is a Win10 Pro (1809) x64, I used my VS portable fatpack for the tests.

bt470bg = PAL 601
smpte170m = NTSC 601
Source https://forum.doom9.org/showthread.p...54#post1681454

[Are_]

My OS is Gentoo Linux, I used self compiled with GCC builds of everything (via package manager).

[EDIT]

clip = core.resize.Bicubic(clip, format=vs.YUV444PS)
clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s=matrix)
clip = core.resize.Bicubic(clip, format=vs.RGBS, transfer_in_s=transfer, transfer_s='linear')

You will need to work around the fact that bitdepth is in the format constant.

[Are_]

I know you don't need, but the shortcuts sometimes produce a different result, that 50% is not free, last time I checked out it was not that different though.

[ChaosKing]

Yep, this shows definitely some differences.

clip = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix)
long = core.resize.Bicubic(clip, format=vs.RGB24, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix, transfer_in_s=transfer, transfer_s='linear')

clip = core.Butteraugli.butteraugli(short, long)

[poisondeathray]

Originally Posted by ChaosKing

Yep, this shows definitely some differences.

clip = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix)
long = core.resize.Bicubic(clip, format=vs.RGB24, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(orig, format=vs.RGB24, matrix_in_s=matrix, transfer_in_s=transfer, transfer_s='linear')

clip = core.Butteraugli.butteraugli(short, long)

It's different because the "long" method uses 2 discrete operations (more rounding errors at 8bit precision)

If you use RGBS for the 1st clip step, you get same results

So the short method is better, faster, more accurate

[ChaosKing]

ok thx.
A quick test with RGBS first and then convert to RGB24:

clip = core.resize.Bicubic(orig, format=vs.RGBS, matrix_in_s=matrix)
long = core.resize.Bicubic(clip, format=vs.RGBS, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(orig, format=vs.RGBS, matrix_in_s=matrix, transfer_in_s=transfer, transfer_s='linear')

short = core.resize.Bicubic(short, format=vs.RGB24)
long  = core.resize.Bicubic(long, format=vs.RGB24)

clip = core.Butteraugli.butteraugli(short, long)

Now there are 0 differences.
Then I tested the speed (RGBS-long vs RGBS-short) ... and wow ... a 100fps difference!!! 163 fps vs 260 fps

[zorr]

Originally Posted by Are_

clip = core.resize.Bicubic(clip, format=vs.YUV444PS)
clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s=matrix)
clip = core.resize.Bicubic(clip, format=vs.RGBS, transfer_in_s=transfer, transfer_s='linear')

You will need to work around the fact that bitdepth is in the format constant.

Thanks, got it working and there are no crashes anymore.

Originally Posted by poisondeathray

If you use RGBS for the 1st clip step, you get same results

So the short method is better, faster, more accurate

I verified this myself, the short method is equivalent to the long method in quality and is faster so it's preferred.

I also wanted to see if there are differences between fmtconv and core.resize. Looks like there are some, core.resize gives slightly brighter result as can be seen in the levels histogram.



It's much more difficult to see any differences in the actual video frame though (and especially in the example as the colors have been munged by gif).

The differences are smaller (although spread to wider area) when using dithering with core.resize so I will enable that, performance didn't change much with the different dithering options. [EDIT] The third frame is the difference heat map from Butteraugli.





[EDIT] I have updated Zoptilib with the new RGB conversion implementation. You can still use matrix '601' even though core.resize doesn't have it, it's the same as '470bg' and '170m' (those two give identical results).

[zorr]

I made some comparisons of the five similarity metrics. I used the same video as in the above post. Perhaps not the best choice since it looks like it was recorded with a potato but the small size makes the optimization task much faster.

I added some noise with grain.Add(25, seed=3) and then denoised it with FFT3DFilter using only bt, blockSize (arguments bw and bh), overlap (arguments ow and oh) and sigma. FFT3DFilter has many more arguments but the goal here is to compare the similarity metrics and not to find the best possible quality.

I ran an optimization with the default settings except set the iterations to 10000 to make sure I find the best settings, five runs per metric, five frames used in the metric calculation. No downscaling in any of the metrics (which is the default in the latest Zoptilib).

The first image here shows the best found result for each metric and heatmaps of the argument pairs bt / sigma and blockSize / overlap. The reddish dot in the maps is located at where the best result was found.



Let's first focus on what the metrics agree on: the best result has bt=5 which is the maximum supported (two previous, current and two next frames). Also the best overlap is always half the blockSize.

BlockSize varies from 10 to 22. VMAF result is an outlier in this group, the other metrics have best blockSize at 10, 12 or 14.

Sigma varies from 308 to 482. Butteraugli has the smallest sigma, SSIM the largest. It's interesting that Butteraugli really dislikes the larger sigmas, VMAF does that too but it's a bit hard to see in this image. I suspect that there are some artifacts with the larger sigmas but most of the metrics are not sensitive to it. I will do some manual image quality comparisons later to very this.

SSIM, GMSD and MDSI are very "smooth", there are no large jumps in results when the parameter's value changes a little (with the exception of bt where large jumps happen at the lowest values). Butteraugli and VMAF look rougher. The smoothness is a good feature when trying to find the optimal settings, makes the search easier.

The VMAF results are pretty weird. It really likes bt=-1 ("sharpen only") which the other metrics considered inferior. There was a really good set of parameters at sigma=170 (actually any sigma, it doesn't change the result) blockTemporal=-1 blockSize=50 overlap=0 with score 98.62038 whereas the best result at sigma=381 blockTemporal=5 blockSize=22 overlap=11 had score 98.62366. The difference was small enough that three out of five runs didn't find the optimal value but this secondary set instead.

To be continued...

[zorr]

Let's take a closer look at how the metrics behave near the optimal result. I run exhaustive searches where two of the four arguments were locked to the best result found with each metric. The other two arguments are tested with every combination.

The first pair is sigma and bt (blockTemporal):



I have limited the x resolution to 100 in the maps but the results were calculated for every sigma from 100 to 800 (divided by 100 so the actual range is 1.0 - 8.0) so each box represents the best found result of 8 sigmas.

SSIM and GMSD are very similar, GMSD doesn't like the high sigmas as much though (and it's also reflected in the sigma of the best result).

MDSI is a bit more focused, area of good results is smaller. While SSIM and GMSD didn't have a large improvement from bt=4 to bt=5 MDSI thinks there's a bigger difference.

Butteraugli is more uneven compared to the above three. The best sigma seems to get smaller with smaller bt. For some reason Butteraugli thinks bt=0 is much better than the above three metrics.

And finally VMAF... still thinking bt=-1 means high quality. It also strongly dislikes bt=1 and bt=2 with high sigmas, so much that it's hard to see any differences elsewhere.

Let's see the other pair blockSize / overlap:



Again SSIM and GMSD are very similar, but GMSD is more focused.

MDSI continues the trend and is even more focused. All three have better scores when overlap increases. Looks like the optimal value is always at overlap = blockSize/2 for every blockSize.

Butteraugli is kinda close to MDSI near the optimum result but there are some weird stripes elsewhere.

The VMAF on the other hand looks like it had a meltdown. There's no clear optimal range, values jump up and down even with small changes. The above mentioned rule overlap=blockSize/2 is also broken. Very hard for the optimizer to make sense of.

To be continued...

[ChaosKing]

Maybe it means that visually there is not that much difference? Have you checked some good and some bad candidates yourself? Or this source needs a different model?

[zorr]

Originally Posted by ChaosKing

Maybe it means that visually there is not that much difference? Have you checked some good and some bad candidates yourself? Or this source needs a different model?

No I haven't done that yet, but I intend to. It's possible that there isn't that much visual difference but it would be strange if doing mere sharpening (bt=-1) would look as good as 5D temporal denoising.