Plum, blind deconvolution enhanced by pixel/block matching

[feisty2]

https://github.com/IFeelBloated/Plum/blob/master/Plum.py
Plum gives an alternative kind of sharpening, the kernel is an actual deconvolution filter instead of USM, which reverses blurring with circle shaped PSFs
Plum produces no ringing (but might enhance the existing ringing), almost no aliasing and does not enhance the noise much (will probably enhance static noise), and it's also not xylographing.
some previous discussion here:
http://forum.doom9.org/showthread.php?t=173756

the non-linear amplifying expression was copied from finesharp(credits to didee)

Plum could also do some "plastic surgeries" for DVD videos and make them look as close to the master tape as possible
it's impossible for Plum to restore the DVD video back to its exact mastertape state, that is, Plum tries to recover the general "mastertape texture" (image should "look" extremely delicate and fragile and fine, not "dumb" and coarse) instead of trying to recover every lost detail.

the underlying philosophy is similar to generative adversarial network: why even bother trying to reconstruct the ground truth (which is impossible anyways) when you could just fool everyone's eyes with the fake stuff as long as there's no reference to the ground truth?


some discussion: https://forum.doom9.org/showthread.php?p=1810234#post1810234

Master tape
http://i.imgur.com/0jcwUlr.png

DVD Release
http://i.imgur.com/LaXsqQj.png

DVD + Plum

ref = Plum.Basic(clip)
clip = Plum.Final([clip, ref], [Plum.Super(clip), Plum.Super(ref)])

http://i.imgur.com/GMTtrIG.png

[hydra3333]

Thanks !
Any dependencies ?

[feisty2]

Thanks !
Any dependencies ?

VCFreq and... must have stuff?

[sl1pkn07]

knlmeanscl
fmtconv
nnedi3
vsfreq (only windows :( )
dfttest
bm3d

[feisty2]

updated Plum and same to the pics at #1
@sl1pkn07
vcfreq is open source, maybe it will just compile without any weird shit on GCC...who knows

[sl1pkn07]

is not problem of GCC
vcfreq.cpp:45:21: fatal error: windows.h: No such file or directory


└───╼ locate windows.h
>snip
/usr/include/wine/windows/windows.h
>snip

[feisty2]

is not problem of GCC
vcfreq.cpp:45:21: fatal error: windows.h: No such file or directory


└───╼ locate windows.h
>snip
/usr/include/wine/windows/windows.h
>snip


very unlikely that vcfreq called some windows api or whatever like that
maybe you could just replace it with <cstdlib> or..

[Myrsloik]

It uses loadlibrary to load fftw. That's the main reason it won't compile in linux. But I fixed that (http://forum.doom9.org/showthread.php?p=1777315#post1777315).

[aculnaig]

Hi feisty2,

I tried your script, but it prompt me this error through vspipe




vspipe -y lo.gatto.vpy .


Script evaluation failed:
Python exception: DFTTest: invalid entry in sigma string
Traceback (most recent call last):
File "src/cython/vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src/cython/vapoursynth.c:26905)
File "lo.gatto.vpy", line 18, in <module>
v = Plum.Final([v, deconv, conv], [sup, supdeconv, supconv])
File "/usr/lib/python3.5/site-packages/Plum.py", line 315, in Final
clip = internal.final(src, super, radius, pel, sad, constants, attenuate_window, attenuate, cutoff)
File "/usr/lib/python3.5/site-packages/Plum.py", line 163, in final
dif = DFTTest(dif, sbsize=attenuate_window, sstring=attenuate, **dfttest_args)
File "src/cython/vapoursynth.pyx", line 1383, in vapoursynth.Function.__call__ (src/cython/vapoursynth.c:25212)
vapoursynth.Error: DFTTest: invalid entry in sigma string




sample clip: https://www.dropbox.com/s/w5plhz6ob7tea83/sample.mpeg?dl=0

Mediainfo output



General
Complete name : sample.mpeg
Format : MPEG-PS
File size : 3.04 MiB
Duration : 4 s 840 ms
Overall bit rate mode : Variable
Overall bit rate : 5 267 kb/s

Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=13
Format settings, picture structure : Frame
Duration : 4 s 840 ms
Bit rate mode : Variable
Bit rate : 5 162 kb/s
Maximum bit rate : 8 500 kb/s
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 2.40:1
Frame rate : 25.000 FPS
Standard : PAL
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.498
Time code of first frame : 11:01:25:07
Time code source : Group of pictures header
GOP, Open/Closed : Open
Stream size : 2.98 MiB (98%)
Color primaries : BT.601 PAL
Transfer characteristics : BT.470 System B, BT.470 System G
Matrix coefficients : BT.601




My VS script



import vapoursynth as vs
import Plum
core = vs.get_core()


v = core.d2v.Source('output.mpeg.d2v')
v = v[92173]

v = core.fmtc.bitdepth(v, flt = 1)
v = core.yadifmod.Yadifmod(v, edeint = core.nnedi3.nnedi3(v, 1), order = 1 )
v = core.fmtc.resample(v, w = 1024, h = 576, kernel = 'sinc', taps = 128)

deconv = Plum.Basic(v)
conv = Plum.Basic(v,mode="convolution")
sup = Plum.Super([v, None])
supdeconv = Plum.Super([v, deconv])
supconv = Plum.Super([conv, None])
v = Plum.Final([v, deconv, conv], [sup, supdeconv, supconv])

v = core.fmtc.bitdepth(v, bits = 8, dmode = 8, patsize = 32)

v.set_output()

[feisty2]

Hi feisty2,

I tried your script, but it prompt me this error through vspipe




vspipe -y lo.gatto.vpy .


Script evaluation failed:
Python exception: DFTTest: invalid entry in sigma string
Traceback (most recent call last):
File "src/cython/vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src/cython/vapoursynth.c:26905)
File "lo.gatto.vpy", line 18, in <module>
v = Plum.Final([v, deconv, conv], [sup, supdeconv, supconv])
File "/usr/lib/python3.5/site-packages/Plum.py", line 315, in Final
clip = internal.final(src, super, radius, pel, sad, constants, attenuate_window, attenuate, cutoff)
File "/usr/lib/python3.5/site-packages/Plum.py", line 163, in final
dif = DFTTest(dif, sbsize=attenuate_window, sstring=attenuate, **dfttest_args)
File "src/cython/vapoursynth.pyx", line 1383, in vapoursynth.Function.__call__ (src/cython/vapoursynth.c:25212)
vapoursynth.Error: DFTTest: invalid entry in sigma string




sample clip: https://www.dropbox.com/s/w5plhz6ob7tea83/sample.mpeg?dl=0

Mediainfo output



General
Complete name : sample.mpeg
Format : MPEG-PS
File size : 3.04 MiB
Duration : 4 s 840 ms
Overall bit rate mode : Variable
Overall bit rate : 5 267 kb/s

Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=13
Format settings, picture structure : Frame
Duration : 4 s 840 ms
Bit rate mode : Variable
Bit rate : 5 162 kb/s
Maximum bit rate : 8 500 kb/s
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 2.40:1
Frame rate : 25.000 FPS
Standard : PAL
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.498
Time code of first frame : 11:01:25:07
Time code source : Group of pictures header
GOP, Open/Closed : Open
Stream size : 2.98 MiB (98%)
Color primaries : BT.601 PAL
Transfer characteristics : BT.470 System B, BT.470 System G
Matrix coefficients : BT.601




My VS script



import vapoursynth as vs
import Plum
core = vs.get_core()


v = core.d2v.Source('output.mpeg.d2v')
v = v[92173]

v = core.fmtc.bitdepth(v, flt = 1)
v = core.yadifmod.Yadifmod(v, edeint = core.nnedi3.nnedi3(v, 1), order = 1 )
v = core.fmtc.resample(v, w = 1024, h = 576, kernel = 'sinc', taps = 128)

deconv = Plum.Basic(v)
conv = Plum.Basic(v,mode="convolution")
sup = Plum.Super([v, None])
supdeconv = Plum.Super([v, deconv])
supconv = Plum.Super([conv, None])
v = Plum.Final([v, deconv, conv], [sup, supdeconv, supconv])

v = core.fmtc.bitdepth(v, bits = 8, dmode = 8, patsize = 32)

v.set_output()



cannot reproduce the error, which version of DFTTest were you using, I assume not the latest one?

and your video is progressive, DO NOT deinterlace it

[aculnaig]

cannot reproduce the error, which version of DFTTest were you using, I assume not the latest one?



I cloned this repository



https://github.com/HomeOfVapourSynthEvolution/VapourSynth-DFTTest




is this the most recent updated branch, isn't it?




and your video is progressive, DO NOT deinterlace it


I thought it too, but I always let have the experts the final word.

Thanks.



I will edit this message if it turns with no errors.

Thanks again, feisty2.


EDIT1: Nothing, same erorr.

[feisty2]

I cloned this repository



https://github.com/HomeOfVapourSynthEvolution/VapourSynth-DFTTest




is this the most recent updated branch, isn't it?







I thought it too, but I always let have the experts the final word.

Thanks.



I will edit this message if it turns with no errors.

Thanks again, feisty2.

I did some tests on your sample anyways, and I think Plum is not the kind of sharpener for this type of vids,
the sample generally lacks textures and details and looks plain flat, and deconvolution does not work on stuff like this, Plum is designed for vids that are filled with rich and soft textures/details, and it then makes them sharp and delicate, and there's just no texture at all in that sample, so...

[aculnaig]

Ok, thanks.

I wanted to do some test with that movie but I always end up doing things that doesn't fit ! Sigh...


The film is this, btw...



https://en.wikipedia.org/wiki/Il_commissario_Lo_Gatto

[madshi]

I totally hate the USM "boom" look. So this algorithm looks interesting to me. However, I think FineSharp already does a pretty nice job on avoiding the USM look. So the first thing I did was look at the comparison images. But for some reason, the FineSharp image has a *much* stronger sharpening strength applied to it than the Plum image in the first post of this thread. Which makes it hard to judge which looks really better. Would you mind increasing the Plum sharpening strength for the screenshots in the first post to achieve the same subjective sharpening strength as FineSharp? That would make it *much* easier to see if plum really improves on FineSharp in quality.

Also, I prefer testing with high-quality sources instead of ultra-blurry SD sources. Maybe you could add comparison screenshots for this image?

https://s3.postimg.org/raes85wcf/unsharpened.png (https://postimg.org/image/raes85wcf/)

FWIW, when comparing sharpening algos I usually prefer using a rather high sharpening strength, higher than I would use in real life, because that makes it more obvious what the algorithms are really doing.

Thank you!! :)

[feisty2]

I totally hate the USM "boom" look. So this algorithm looks interesting to me. However, I think FineSharp already does a pretty nice job on avoiding the USM look. So the first thing I did was look at the comparison images. But for some reason, the FineSharp image has a *much* stronger sharpening strength applied to it than the Plum image in the first post of this thread. Which makes it hard to judge which looks really better. Would you mind increasing the Plum sharpening strength for the screenshots in the first post to achieve the same subjective sharpening strength as FineSharp? That would make it *much* easier to see if plum really improves on FineSharp in quality.

Also, I prefer testing with high-quality sources instead of ultra-blurry SD sources. Maybe you could add comparison screenshots for this image?

https://s3.postimg.org/raes85wcf/unsharpened.png (https://postimg.org/image/raes85wcf/)

FWIW, when comparing sharpening algos I usually prefer using a rather high sharpening strength, higher than I would use in real life, because that makes it more obvious what the algorithms are really doing.

Thank you!! :)

same sharpening strength for 3 algorithms, all 1.64

Plum looks much milder because it ONLY sharpens the most delicate parts of the image(makes high frequencies even higher), it does not amplify low/median frequencies at all while finesharp still amplifies median frequencies..
maybe I could lower down the strength of finesharp and make it easier to compare..

[madshi]

Well, the same sharpening "number" in different algos doesn't necessarily produce the same subjective sharpening strength. I'd like the images to produce the same *perceived* sharpness level, and then compare which I like best. This is how I usually compare sharpening algos, anyway. Usually when I do that, any USM like algo looks terribly bloated/artificial, while algos like FineSharp look much more natural.

I understand I could easily do this comparison myself, but I'm always having a hard time getting AviSynth/VapourSynth scripts with lots of dependencies to work... :o

Maybe you could just sharpen my preferred 1080p image with all dials turned up to "overload" with Plum, then I can compare your final result with the best I can achieve with my own preferred algos? That would be awesome!

[feisty2]

Well, the same sharpening "number" in different algos doesn't necessarily produce the same subjective sharpening strength. I'd like the images to produce the same *perceived* sharpness level, and then compare which I like best. This is how I usually compare sharpening algos, anyway. Usually when I do that, any USM like algo looks terribly bloated/artificial, while algos like FineSharp look much more natural.

I understand I could easily do this comparison myself, but I'm always having a hard time getting AviSynth/VapourSynth scripts with lots of dependencies to work... :o

Maybe you could just sharpen my preferred 1080p image with all dials turned up to "overload" with Plum, then I can compare your final result with the best I can achieve with my own preferred algos? That would be awesome!

Plum shares the exact same non-linear amplifying expression with finesharp... so same number gives the mathematically same amount of sharpening, anyways, I reduced the strength to 0.86 for finesharp to show the difference
Plum
http://i.imgur.com/C0WSTEb.png


clp = finesharp.sharpen(clp,sstr=0.86)

http://i.imgur.com/lXx3lBY.png

[feisty2]

and I cannot sharpen your image because Plum works in both spatial and temporal dimensions, gonna need a video sample and it should be no more than 720p
Plum is an advanced and very complex algorithm, I don't think my shitty i7-4790k could handle Plum on 1080p

and below is how Plum looks like with an insanely high strength

deconv = Plum.Basic(clp)
conv = Plum.Basic(clp,mode="convolution")
sup = Plum.Super([clp, None])
supdeconv = Plum.Super([clp, deconv])
supconv = Plum.Super([conv, None])
clp = Plum.Final([clp, deconv, conv], [sup, supdeconv, supconv], strength=3.2, cutoff=16)

http://i.imgur.com/93YTrDO.png

[madshi]

anyways, I reduced the strength to 0.86 for finesharp to show the difference
Thanks, Plum does look better (less bloated) than FineSharp in this comparison.

[Myrsloik]

I had a look at your script and saw some odd things...

You invoke several Expr several times here in series with max. You can have up to 25 inputs to Expr and combining several will be much faster.

Making the input clips something like [clip, SelectEvery(src, radius * 2 + 1, 1), SelectEvery(src, radius * 2 + 1, 2)...] and then the expr " x y max z max a max b max..." will be faster. And maybe 3% more annoying to generate the string for but definitely faster.

def extremum_multi(src, radius, mode):
core = vs.get_core()
SelectEvery = core.std.SelectEvery
Expr = core.std.Expr
clip = SelectEvery(src, radius * 2 + 1, 0)
for i in range(1, radius * 2 + 1):
clip = Expr([clip, SelectEvery(src, radius * 2 + 1, i)], "x y " + mode)
return clip

In here you can technically fold the makediff into the first expr and the mergediff into the second. I think.

Fom the shrink function:
DDD = Expr([DD, convDD], ["x y - x 0.5 - * 0 < 0.5 x y - abs x 0.5 - abs < x y - 0.5 + x ? ?"])
dif = MakeDiff(dif, DDD)
convD = Convolution(dif, **conv_args)
dif = Expr([dif, convD], ["y 0.5 - abs x 0.5 - abs > y 0.5 ?"])
clip = MergeDiff(src, dif)

May be possible to combine these 2 as well
clamped = helpers.clamp(averaged, bright_limit, dark_limit, 0.0, 0.0)
amplified = Expr([clamped, src[0]], expression)

[feisty2]

but that will limit the largest radius to 12... right?
and I won't be able to predict how many clips I should put in Expr input array cuz that's defined by "radius" at runtime, and the expression, also can only be defined at runtime
and that makes the programming extremely hard... I'm no professional programmer and that's just all too challenging to me..

about Expr folding, those 2 blocks of code were ported from earlier avisynth implementation, I think I should probably just leave it that way cuz that's the direct translation of how the algorithm came into being, it's more readable and makes future maintaining easier

[Myrsloik]

Yes, 12 would be the biggest unless you in turn put that in a similar loop again. But I'm just suggesting things. I've been meaning to look at havsfunc and see what else can be combined as well... I suspect there are several simplifications possible when lutxy was simply converted to expr.

[hydra3333]

Hello. Hints where to find mvmulti.py would be much appreciated.
T:\HDTV\WDTVlive\MP4-VS\1>"C:\SOFTWARE\Vapoursynth\VSPipe.exe" "T:\HDTV\WDTVlive\MP4-VS\1\test.mpg.2016.08.22.00.47.41.48.vpy" - --y4m | "C:\SOFTWARE\ffmpeg\0-homebuilt-x64\x264-mp4.exe" - --stdin y4m --thread-input --frames 45810 --profile high --level 4.1 --preset slow --interlaced --tff --no-cabac --crf 14 --sar 64:45 --colormatrix bt470bg -o "s:\HDTV\WDTVlive\MP4-VS\1\test-temp.2016.08.22.00.47.41.48.h264"
Script evaluation failed:
Python exception: No module named 'mvmulti'
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26897)
File "T:\HDTV\WDTVlive\MP4-VS\1\test.mpg.2016.08.22.00.47.41.48.vpy", line 7, in <module>
import Plum # http://forum.doom9.org/showthread.php?t=173775 https://github.com/IFeelBloated/Plum
File "C:\SOFTWARE\Vapoursynth\Plum.py", line 2, in <module>
import mvmulti
ImportError: No module named 'mvmulti'
(This install is python embedded and vapoursynth portable, if that makes any difference)

edit: I thought it may look to be a part of this http://forum.doom9.org/showthread.php?t=172525 single precision MVTools plugin (stable). R5 download bin x64 winnt libmvtools_sf_emt64.7z which suggests it's 64-bit only. I reckon I'm wrong :)

edit2: no, it is a part of it, it appears to be hidden down under the source folder https://github.com/IFeelBloated/vapoursynth-mvtools-sf/blob/master/src/mvmulti.py but is not included as a part of the Release .7z file ? It does appear to be x64 only which I can't use :(

[feisty2]

https://github.com/IFeelBloated/vapoursynth-mvtools-sf/blob/master/src/mvmulti.py

[hydra3333]

thank you.

[feisty2]

I kind of redesigned this thing, now it gives even more delicate sharpening results than the previous version did.
also the test scripts and pictures at #1 are updated, if anyone is still intrigued and would like to test it, feedbacks are still welcome.
I'll do the documentation for this thing in the next few days

[Jindadil007]

With new script...
video = core.d2v.Source(r'C:\DVD\Sample.d2v')
video = core.fmtc.bitdepth(video,bits=32,fulls=False,fulld=True)
deconv = Plum.Basic(video)
conv = Plum.Basic(video,mode="convolution")
sup = Plum.Super([video, None])
supdeconv = Plum.Super([video, deconv])
supconv = Plum.Super([conv, None])
video = Plum.Final([video, deconv, conv], [sup, supdeconv, supconv], strength=3.2, cutoff=16)
video.set_output()

I get following error :

Core freed but 441 filter instances still exist
Core freed but 441 filter instances still exist
Core freed but 1961764224 bytes still allocated in framebuffers
Core freed but 1961764224 bytes still allocated in framebuffers

[feisty2]

fine, post an unprocessed sample and I'll look into it if I have time

[dipje]

That 'core freed' message as you say it is not an error, but a (debugging) message from Vapoursynth (or even VapoursynthEditor I think) and nothing to be concerned about.

[Jindadil007]

That 'core freed' message as you say it is not an error, but a (debugging) message from Vapoursynth (or even VapoursynthEditor I think) and nothing to be concerned about.

OK Thanks...yes i tested on Vseditor...

[Mystery Keeper]

That 'core freed' message as you say it is not an error, but a (debugging) message from Vapoursynth (or even VapoursynthEditor I think) and nothing to be concerned about.Wrong! It is an indication that some plugins in the scripts are not managing the resources properly, leading to memory leaks. Such things should be reported.

[Jindadil007]

Wrong! It is an indication that some plugins in the scripts are not managing the resources properly, leading to memory leaks. Such things should be reported.

I just used the updated script...Now I get better quality video with this script :

ref = Plum.Basic(video)
video = Plum.Final([video, ref], [Plum.Super(video), Plum.Super(ref)], strength=3.2, cutoff=16)

but memory leaks are still there...

Core freed but 441 filter instances still exist
Core freed but 441 filter instances still exist
Core freed but 1961764224 bytes still allocated in framebuffers
Core freed but 1961764224 bytes still allocated in framebuffers

These should be looked into...

[feisty2]

and the documentation is DONE!
also I changed quite a few things in Plum.Basic

r1:
initial release

[feisty2]

I just used the updated script...Now I get better quality video with this script :

ref = Plum.Basic(video)
video = Plum.Final([video, ref], [Plum.Super(video), Plum.Super(ref)], strength=3.2, cutoff=16)

but memory leaks are still there...

Core freed but 441 filter instances still exist
Core freed but 441 filter instances still exist
Core freed but 1961764224 bytes still allocated in framebuffers
Core freed but 1961764224 bytes still allocated in framebuffers

These should be looked into...

that's an (outdated?) nnedi3 bug.

[Jindadil007]

that's an (outdated?) nnedi3 bug.

Plum and VBm3d also have dependencies on Nnedi3 but this error does not show up while using those except Plum...You need to check !!!

[feisty2]

Plum and VBm3d also have dependencies on Nnedi3 but this error does not show up while using those except Plum...You need to check !!!

Plum is a python module
that message was there cuz a plugin was having a memory leak.

it's not my business to check anything.

[feisty2]

r2:
small tweaks, here and there

[viccpa]

Hi everyone

I can't make Plum to work.

I tried this script:

import vapoursynth as vs
import Plum
core = vs.get_core(accept_lowercase=True)
video = core.ffms2.Source("intermediate.avi")
video = core.fmtc.bitdepth(video,bits=32,fulls=False,fulld=True)
ref = Plum.Basic(video)
video = Plum.Final([video, ref], [Plum.Super(video), Plum.Super(ref)])
video = core.fmtc.bitdepth(video,bits=8,fulls=True,fulld=False)
video.set_output()

Script work flawlessly without Plum. With Plum vseditor stop responding after several seconds and not displaying any kind of error. Vspipe behave similar. I left it about 30 min. After 20 min processor jump from 24 % to 100 % and stay there.

I have Core2Quad q6600, win7-x64, Python 3.6.0_x64, latest Vapoursynth. Plum.py is on site-packages.

Any help is appreciated
viccpa

[feisty2]

You might wanna upgrade your computer before using Plum..

[viccpa]

Ok, i will tested out on my work (i7, 6-gen)

Thanks

[feisty2]

Also, use KNLMeansCL v0.7.7 when you're doing tests..

[wonkey_monkey]

You might wanna upgrade your computer before using Plum..

Always remember to add --verbose when asking feisty a question.

[StainlessS]

Always remember to add --verbose when asking feisty a question.

Nah, no good without the --help switch. :)

[KingLir]

Thanks! This looks like an amazing plugin.

I am getting the following errors with R36, latest versions of the script and all dependencies. Anyone have an idea what I should try to workaround this ?

Script evaluation failed:
Python exception: knlm.KNLMeansCL: fatal error!
(clCreateImage(d.mem_P[memU2]): CL_OUT_OF_HOST_MEMORY)
Traceback (most recent call last):
File "src/cython/vapoursynth.pyx", line 1712, in vapoursynth.vpy_evaluateScript (src/cython/vapoursynth.c:34991)
File "VIVTC_with_TComb_script.vpy", line 35, in <module>
deconv = Plum.Basic(clip)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 234, in Basic
clip = internal.basic(core, src, strength, a, h, radius, wn, scale, cutoff)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 130, in basic
sharp = inline(sharp)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 118, in inline
sharp = core.NLErrors(ref, a, h[0], sharp)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 80, in NLErrors
nlm = self.KNLMeansCL(pad, d=0, a=a, s=0, h=h, rclip=rclip)
File "src/cython/vapoursynth.pyx", line 1604, in vapoursynth.Function.__call__ (src/cython/vapoursynth.c:33131)
vapoursynth.Error: knlm.KNLMeansCL: fatal error!
(clCreateImage(d.mem_P[memU2]): CL_OUT_OF_HOST_MEMORY)

[Are_]

Script evaluation failed:
Python exception: knlm.KNLMeansCL: fatal error!
(clCreateImage(d.mem_P[memU2]): CL_OUT_OF_HOST_MEMORY)
Traceback (most recent call last):
File "src/cython/vapoursynth.pyx", line 1712, in vapoursynth.vpy_evaluateScript (src/cython/vapoursynth.c:34991)
File "VIVTC_with_TComb_script.vpy", line 35, in <module>
deconv = Plum.Basic(clip)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 234, in Basic
clip = internal.basic(core, src, strength, a, h, radius, wn, scale, cutoff)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 130, in basic
sharp = inline(sharp)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 118, in inline
sharp = core.NLErrors(ref, a, h[0], sharp)
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/Plum.py", line 80, in NLErrors
nlm = self.KNLMeansCL(pad, d=0, a=a, s=0, h=h, rclip=rclip)
File "src/cython/vapoursynth.pyx", line 1604, in vapoursynth.Function.__call__ (src/cython/vapoursynth.c:33131)
vapoursynth.Error: knlm.KNLMeansCL: fatal error!
(clCreateImage(d.mem_P[memU2]): CL_OUT_OF_HOST_MEMORY)

Your computer specs are too low for this.

[KingLir]

Your computer specs are too low for this.

But I have a new computer.

OS:

macOS Sierra 10.12.3

CPU:

Processor Name: Intel Core i7
Processor Speed: 4 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 8 MB

Memory:

24 GB

GPU:

Chipset Model: AMD Radeon R9 M395X
Type: GPU
Bus: PCIe
PCIe Lane Width: x16
VRAM (Total): 4096 MB
Vendor: ATI (0x1002)
Device ID: 0x6920
Revision ID: 0x0000
ROM Revision: 113-C905AA-799
EFI Driver Version: 01.00.799
Metal: Supported

[Mystery Keeper]

But I have a new computer.

CPU:

Processor Name: Intel Core i7
Processor Speed: 4 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 8 MB

Memory:

24 GB

GPU:

Chipset Model: AMD Radeon R9 M395X
Type: GPU
Bus: PCIe
PCIe Lane Width: x16
VRAM (Total): 4096 MB
Vendor: ATI (0x1002)
Device ID: 0x6920
Revision ID: 0x0000
ROM Revision: 113-C905AA-799
EFI Driver Version: 01.00.799
Metal: Supported

Is your OS 32bit or are you using 32bit version of VS and plugins?

[KingLir]

Is your OS 32bit or are you using 32bit version of VS and plugins?

I am on macOS Sierra 10.12.3.
How can I verify the answer to your question ?

[feisty2]

r3:
new parameter "freq_margin"

[poisondeathray]

Some initial observations:


1) It's &^&%$*#* slow.

2) It crashes on large image sets (probably GPU memory issue, I tested on a 4GB card, but I wasn't about to switch to "CPU" mode yet for the KNLMeans part, because of possible different results. GPU-z does spike to 100% GPU/memory at times, but even on "smaller" images which complete ) .

3) It does mess up some test patterns. Partially because I didn't play with the settings enough - "you dummy you're supposed to use the 'right' settings"... But because it's so slow I can't try out different settings quickly on multiple sources at one go. Hard to give proper feedback or get a "feeling" for how it reacts. Then when I come back a few days later and if I didn't make notes I have to start over again.

You might ask who the &*^* watches test patterns?... Those low level tests have high predictive value where you're going to get problems manifesting in real life situations . If you can tune an algorithm to "pass" or at least look good on various patterns, without making a mess of others, then you've essentially got a "perfect" filter. (But you'll never get a perfect storm of "ideal" inputs, there's almost always some problems with source.) Or at the very least you can help to identify what areas you might improve on.

If I had time I'd try to give proper , more specific feedback, but this thing is so slow. The warp drive is down, and so are the impulse engines...

[feisty2]

Some initial observations:


1) It's &^&%$*#* slow.

2) It crashes on large image sets (probably GPU memory issue, I tested on a 4GB card, but I wasn't about to switch to "CPU" mode yet for the KNLMeans part, because of possible different results. GPU-z does spike to 100% GPU/memory at times, but even on "smaller" images which complete ) .

3) It does mess up some test patterns. Partially because I didn't play with the settings enough - "you dummy you're supposed to use the 'right' settings"... But because it's so slow I can't try out different settings quickly on multiple sources at one go. Hard to give proper feedback or get a "feeling" for how it reacts. Then when I come back a few days later and if I didn't make notes I have to start over again.

You might ask who the &*^* watches test patterns?... Those low level tests have high predictive value where you're going to get problems manifesting in real life situations . If you can tune an algorithm to "pass" or at least look good on various patterns, without making a mess of others, then you've essentially got a "perfect" filter. (But you'll never get a perfect storm of "ideal" inputs, there's almost always some problems with source.) Or at the very least you can help to identify what areas you might improve on.

If I had time I'd try to give proper , more specific feedback, but this thing is so slow. The warp drive is down, and so are the impulse engines...

can you post the test patterns that it messed up?
I could use them as references when I'm trying to improve this thing, if you have read the script by now, you might have noticed there're a few "magic constants" which were calculated by mathematica as the result of curve fitting on the test set I got, but they could be kind of overfitting cuz I don't have a set that covers everything

[poisondeathray]

Actually, it seems to do worse then some simpler sharpeners on any static test pattern so far. I realize there is a temporal component to this filter, but I never even got to motion tests yet. Might not be using the correct settings or cutoff. Or maybe you actually can't have your cake and eat it too.

Just download a random free one, like a zone plate / wedges/ trumpets - and if you crash, maybe crop to a region of interest to test

eg.
http://www.bealecorner.org/red/test-patterns/

[feisty2]

Actually, it seems to do worse then some simpler sharpeners on any static test pattern so far. I realize there is a temporal component to this filter, but I never even got to motion tests yet. Might not be using the correct settings or cutoff. Or maybe you actually can't have your cake and eat it too.

Just download a random free one, like a zone plate / wedges/ trumpets - and if you crash, maybe crop to a region of interest to test

eg.
http://www.bealecorner.org/red/test-patterns/

yeah, the final estimation would be rendered probably 70% useless if you're testing single frame samples

[poisondeathray]

yeah, the final estimation would be rendered probably 70% useless if you're testing single frame samples

It's easy to animate charts to induce motion. You can even impart real camera motion , but motion tracking a source and applyin that to the test chart. You can add motion blur to simulate shutter blur if you wanted to (or not.) Do you think it will do any better ? My guess is probably not. My preliminary impression on very limited testing is high contrast lines tend to be a problem here.

That's just a small part of the testing. Sure it might look ok on some more organic sources, but there will be parts that will look bad if it fails on a motion zone plate

[feisty2]

It's easy to animate charts to induce motion. You can even impart real camera motion , but motion tracking a source and applyin that to the test chart. You can add motion blur to simulate shutter blur if you wanted to (or not.) Do you think it will do any better ? My guess is probably not. My preliminary impression on very limited testing is high contrast lines tend to be a problem here.

That's just a small part of the testing. Sure it might look ok on some more organic sources, but there will be parts that will look bad if it fails on a motion zone plate

wow, these test patterns are real LARGE!!
it crashes the source filter (lsmash) even before I get started!

[poisondeathray]

You can crop to region of interest... and animate that



This 4 frame test chart sample is from real camera (ie. non synthetic , with real noise, real camera issues like aliasing)

UT Video ~2.3MB
https://www.mediafire.com/?d3xbclg2xb5tad5

[feisty2]

It's easy to animate charts to induce motion. You can even impart real camera motion , but motion tracking a source and applyin that to the test chart. You can add motion blur to simulate shutter blur if you wanted to (or not.) Do you think it will do any better ? My guess is probably not. My preliminary impression on very limited testing is high contrast lines tend to be a problem here.

That's just a small part of the testing. Sure it might look ok on some more organic sources, but there will be parts that will look bad if it fails on a motion zone plate

I tested it on "Zone720-hardedge-B.png" and I'm not seeing anything messed up,,,

BEFORE:
http://i.imgur.com/qknUq4X.png
http://i.imgur.com/BK1J3Jc.png
http://i.imgur.com/TDDcJbu.png
http://i.imgur.com/YeAmKIh.png

AFTER:

clp = core.lsmas.LWLibavSource('blah.png')
clp = core.fmtc.bitdepth(clp,bits=32,fulls=True,fulld=True)
clp = core.std.ShufflePlanes(clp,0,vs.GRAY)

ref = Plum.Basic(clp)
clp = Plum.Final([clp, ref], freq_margin=0) #freq_margin=0 means the anti-bloat filter is off, kind of cheating here cuz normally it should be something greater than 0, but it doesn't make sense on test patterns

http://i.imgur.com/ZVp61NG.png
http://i.imgur.com/aueH9FM.png
http://i.imgur.com/mhp9UVK.png
http://i.imgur.com/ifPXVtO.png


the results are exactly what I expected, "just don't do nothing at all if the source is already good enough"

can you post your test script?

[poisondeathray]

Go look at the fine patterns .

I was intrigued earlier about some claims about "sharpening" without haloing artifacts. But I don't think it's possible unless it's very weak strength, or unless you do some sort of masking or differential application. You're going to mess up the fine patterns eventually when you turn up the strength. There's got to be some give and take. If you enhance the coarse details, the fine details are going to be oversharpened eventually. That will show up on the test pattern

I used only default settings so far, should I have been using something different ?

[feisty2]

Go look at the fine patterns .

I was intrigued earlier about some claims about "sharpening" without haloing artifacts. But I don't think it's possible unless it's very weak. You're going to mess up the fine patterns eventually when you turn up the strength.

I used only default settings so far, should I have been using something different ?

the first image I posted above is the "finest" part I could find in that test image.

make "freq_margin" 0 if you're doing tests on test patterns

[feisty2]

another one, with very "fine" features (from "star-chart-bars-full-600dpi.png")
same script as in #57

BEFORE:
http://i.imgur.com/NMjAbUC.png

AFTER:
http://i.imgur.com/YDKmeGc.png

still nothing messed up

[poisondeathray]

so freq_margin=0 is the tweaked setting for high con / test patterns ?

so it takes an a$$ long time to do nothing :devil: ? Well, why not just do nothing ? :D . Hell of a lot faster

Try it on the "video" test pattern in 56 , both with default, and freq_margin=0 "cheat" settings . Both are worse than original or some weak simple sharpening

[feisty2]

Go look at the fine patterns .

I was intrigued earlier about some claims about "sharpening" without haloing artifacts. But I don't think it's possible unless it's very weak strength, or unless you do some sort of masking or differential application.

Non-Local Error is a self-adaptive algorithm, it automatically ignores the very sharp edges cuz it weights on "value" instead of "euclidean distance", which means it's a non-linear filter and it does not have a "nice" frequency domain representation like normal USM filters

so it does not produce ringings/halos


so it takes an a$$ long time to do nothing ? Well, why not just do nothing ? . Hell of a lot faster.
it does certainly "something" if the algorithm thinks the image should get sharpened..:)

[feisty2]

You can crop to region of interest... and animate that



This 4 frame test chart sample is from real camera (ie. non synthetic , with real noise, real camera issues like aliasing)

UT Video ~2.3MB
https://www.mediafire.com/?d3xbclg2xb5tad5

can you try with another codec and another container...
maybe something less,, alternative?
I tried like 5 versions of lsmash and all crashed trying to read this thing..

say, maybe, h264 lossless + mkv?

[poisondeathray]

Image is "already sharp" so do nothing - ok that makes sense. But you're using freq_margin=0 - is that what would be used for "normal" situations ? Because it messes up pattern when you don't use that setting .

Crap I thought UT was pretty standard. I think it might be because this is the VFW version

You can convert with ffmpeg

ffmpeg -i input.ext -c:v libx264rgb -crf 0 -an output.ext

[poisondeathray]

Why can ffms2 open it (the original ULRG sample), but lsmash can't ? AVISource works too, but relies on your VFW system

[feisty2]

Why can ffms2 open it (the original ULRG sample), but lsmash can't ?

who knows, I've downloaded ffms and now doing my test with it.

[feisty2]

Why can ffms2 open it (the original ULRG sample), but lsmash can't ? AVISource works too, but relies on your VFW system

I've spotted the problem for your test video
open Plum.py and go to line 125, change

clip = core.Shrink(limited)

to

clip = limited #core.Shrink(limited)


"Shrink" is a helper function and it tries to shrink the image down to make it look "finer" and more "delicate"
it's a nice feature and generally works for most videos but fails with your test clip cuz your test clip has some severe aliasing and it messed the "aliasing" up, so, it made the bad even worse and that's it...


But you're using freq_margin=0 - is that what would be used for "normal" situations ? Because it messes up pattern when you don't use that setting .

your test clip is screwed not because of "freq_margin", it's just too aliased to get shrunk, comment out the "Shrink" function and it's gonna be fine (but you might want the "Shrink" function back when you're doing tests on real life videos)

[poisondeathray]

ok thanks , I'll eventually keep on doing more tests, but I'll keep those 2 things in mind with different sources

The aliasing might look "severe" but that's actually pretty typical pattern for consumer point and shoot. For modern HD and UHD cameras, you 'll commonly see aliasing above a certain resolution (higher than the one shown there, but the pattern will be similar). For common DSLR, it's usually much worse, with ugly color moire patterns on top



your test clip is screwed not because of "freq_margin", it's just too aliased to get shrunk, comment out the "Shrink" function and it's gonna be fine (but you might want the "Shrink" function back when you're doing tests on real life videos)

But "clean" test patterns get screwed up (or look less than ideal) because of not adjusting it to zero - so there is a direct relationship independent of shrink

So what values of freq_margin are you typically using? Do you make adjustments on the fly ? Or was that only for synthetic clean test chart and you typically leave it ? I see in the py as 20 , and must be : freq_margin < 0 or freq_margin > 100-cutoff

[feisty2]

ok thanks , I'll eventually keep on doing more tests, but I'll keep those 2 things in mind with different sources

The aliasing might look "severe" but that's actually pretty typical pattern for consumer point and shoot. For modern HD and UHD cameras, you 'll commonly see aliasing above a certain resolution (higher than the one shown there, but the pattern will be similar). For common DSLR, it's usually much worse, with ugly color moire patterns on top



But "clean" test patterns get screwed up (or look less than ideal) because of not adjusting it to zero - so there is a direct relationship independent of shrink

So what values of freq_margin are you typically using? Do you make adjustments on the fly ? Or was that only for synthetic clean test chart and you typically leave it ? I see in the py as 20 , and must be : freq_margin < 0 or freq_margin > 100-cutoff

I did my tests on a lot of music videos which were shot on 35mm film and then scanned to digital format (some were shot in digital format, but shot with fancy cameras like RED), and I compared the DVD release and the master tape of the same video and tried to match them, I didn't do any test on videos shot by cheap cameras

[poisondeathray]

Actually "expensive" cameras like RED , Alexa, high end Sony's show the SAME pattern of aliasing and moire, the difference being at the higher resolutions and finer patterns. Just download some sample test footage, there's plenty around. The reason I uploaded that "low resolution" example was because of the crashing. You can't process footage from the other cameras at native dimensions with this, at least not currently . It might not be GPU memory/usage, because even "small" dimension tests max it out this 4GB card without crashing, so not sure about why

[feisty2]

Actually "expensive" cameras like RED , Alexa, high end Sony's show the SAME pattern of aliasing and moire, the difference being at the higher resolutions and finer patterns. Just download some sample test footage, there's plenty around. The reason I uploaded that "low resolution" example was because of the crashing. You can't process footage from the other cameras at native resolution with your filter

That aliasing in ur test clip is not temporally stable, yes you can kill the spatial aliasing by downscaling but that's not gonna work in temporal dimension, you still have that "shimmering" thing and I've never seen that kind of temporal aliasing in my test videos

If the aliasing was temporally stable, the "Shrink" function might have just worked cuz the motion compensation in the final estimation could have swept that crap away,,,

[poisondeathray]

That aliasing in ur test clip is not temporally stable, yes you can kill the spatial aliasing by downscaling but that's not gonna work in temporal dimension, you still have that "shimmering" thing and I've never seen that kind of temporal aliasing in my test videos

If the aliasing was temporally stable, the "Shrink" function might have just worked cuz the motion compensation in the final estimation could have swept that crap away,,,


The same thing happens with >$50K cameras (at native dimensions) . The fine details are aliased . This is how "resolution" is defined - the limit at which you can distinguish "clean" patterns. Yes, when you downsample to HD (or in your case SD), you lose the high freq. details, thus you lose the moire/aliasing/shimmering . That's one reason why oversampling is fantastic. Your SD test cases are downsampled and "clean" in that respect

ALL cameras have aliasing with finer details, higher "resolutions" . For example, the Alexa is clean until about 800-900 lines then you get the same shimmering if you move the camera when shooting something similar . No, you don't typically shoot a test pattern, but things like that occur in real life, e.g. a brick wall, tiles on a building in the distance etc... That's why in some respects, a "cheap" consumer UHD camera can outperform a $50K "HD" camera

Synthetic test pattern can be completely clean ,even in motion if you wanted it to be, but that doesn't necessarily reflect real world acquisition formats - but it's important include those types of low level tests




Traditional linear /simple sharpener will create problems on the finer patterns (when there was none to begin with) because of the artifacting and haloing, but still sharpen the coarse patterns. If you set low level it might be acceptable (because your footage might not have those fine details to begin with). But when set freq_margin to zero, it does nothing here, even on coarse patterns - I guess I'm trying to get a feeling for how to tweak/adjust this filter on different sources, but it's so hard because it's so slow

(another thing I miss is tabs in avspmod, it's easy to compare different settings in different tabs, and you can "learn" a filter a lot faster. I made request for vapoursynth editor before, but got shot down)

[madshi]

FWIW, I have some photos (converted from RAW to TIF) shot with RED Helium here, but I don't see aliasing in those. I see color fringing, but overall it's rather soft (when 1:1 pixel peeping).

[feisty2]

The same thing happens with >$50K cameras (at native dimensions) . The fine details are aliased . This is how "resolution" is defined - the limit at which you can distinguish "clean" patterns. Yes, when you downsample to HD (or in your case SD), you lose the high freq. details, thus you lose the moire/aliasing/shimmering . That's one reason why oversampling is fantastic. Your SD test cases are downsampled and "clean" in that respect

ALL cameras have aliasing with finer details, higher "resolutions" . For example, the Alexa is clean until about 800-900 lines then you get the same shimmering if you move the camera when shooting something similar . No, you don't typically shoot a test pattern, but things like that occur in real life, e.g. a brick wall, tiles on a building in the distance etc... That's why in some respects, a "cheap" consumer UHD camera can outperform a $50K "HD" camera

Synthetic test pattern can be completely clean ,even in motion if you wanted it to be, but that doesn't necessarily reflect real world acquisition formats - but it's important include those types of low level tests




Traditional linear /simple sharpener will create problems on the finer patterns (when there was none to begin with) because of the artifacting and haloing, but still sharpen the coarse patterns. If you set low level it might be acceptable (because your footage might not have those fine details to begin with). But when set freq_margin to zero, it does nothing here, even on coarse patterns - I guess I'm trying to get a feeling for how to tweak/adjust this filter on different sources, but it's so hard because it's so slow

(another thing I miss is tabs in avspmod, it's easy to compare different settings in different tabs, and you can "learn" a filter a lot faster. I made request for vapoursynth editor before, but got shot down)

ok, new information noted
issue: because aliasing is rarely seen in SD and HD videos shot by professional cameras, I never considered if that would be a problem, my main focus was on ringing cuz that's what happens with regular USM filters, the default settings might fail if the source suffers from obvious aliasing..
solution: turn off the "Shrink" filter


you seem to be very interested in the "freq_margin" parameter, I should probably explain how it works..

freq_margin was designed to act as an anti-bloat filter, it saves the image from "xylographing" as already explained here (https://forum.doom9.org/showthread.php?p=1776542#post1776542), it works as an extension to the "cutoff" parameter

first we decompose the image into MANY frequency-wise components, and rank them.
say,
0 -> lowest frequency component,
100-> highest frequency component,
Plum being a high pass filter does not affect frequencies labeled from 0 to "cutoff", frequencies labeled from "cutoff" to 100 will be processed.

now because high frequencies in the sharpened clip have more "energy", if we merge source[0, cutoff] with sharpened[cutoff, 100], there will be more energy than the source clip and the result will look "xylographed" (energy(sharpened[cutoff, 100]) > energy(source[cutoff, 100]))

to keep the overall energy amount approximately unaffected, we have to "sacrifice" certain frequencies in the source clip, it can't be the low frequencies, cuz that's the base of the entire image, it can't be the high frequencies, cuz that's what we want, fine details and stuff..

so certain medium frequencies have to die for the greater good (or more of, they are already shifted to high frequencies in the sharpened clip anyways), and "freq_margin" gets to decide, how many medium frequency components should die, basically, source[cutoff, cutoff+freq_margin] will be removed

so the final result is actually:
source[0, cutoff] + sharpened[cutoff+freq_margin, 100]

[poisondeathray]

FWIW, I have some photos (converted from RAW to TIF) shot with RED Helium here, but I don't see aliasing in those. I see color fringing, but overall it's rather soft (when 1:1 pixel peeping).

Anytime you have footage that is "soft" without aliasing, very likely it's been shot with a strong OLPF - i.e the fine details and high freq details have already been cut off



you seem to be very interested in the "freq_margin" parameter, I should probably explain how it works..

freq_margin was designed to act as an anti-bloat filter, it saves the image from "xylographing" as already explained here, it works as an extension to the "cutoff" parameter

first we decompose the image into MANY frequency-wise components, and rank them.
say,
0 -> lowest frequency component,
100-> highest frequency component,
Plum being a high pass filter does not affect frequencies labeled from 0 to "cutoff", frequencies labeled from "cutoff" to 100 will be processed.

now because high frequencies in the sharpened clip have more "energy", if we merge source[0, cutoff] with sharpened[cutoff, 100], there will be more energy than the source clip and the result will look "xylographed" (energy(sharpened[cutoff, 100]) > energy(source[cutoff, 100]))

to keep the overall energy amount approximately unaffected, we have to "sacrifice" certain frequencies in the source clip, it can't be the low frequencies, cuz that's the base of the entire image, it can't be the high frequencies, cuz that's what we want, fine details and stuff..

so certain medium frequencies have to die for the greater good (or more of, they are already shifted to high frequencies in the sharpened clip anyways), and "freq_margin" gets to decide, how many medium frequency components should die, basically, source[cutoff, cutoff+freq_margin] will be removed

so the final result is actually:
source[0, cutoff] + sharpened[cutoff+freq_margin, 100]


Ok thanks, it will take some time to digest...

I am interested, because I think this is the "key" to being useful in actual usage - being able to accurate categorize and filter certain ranges . I need to play with it more

[poisondeathray]

Is there a way to code a helper function or visualization aid (preferably quick processing preview, minus all the processing) for what "it" thinks is a frequency range ? For example , maybe shade or overlay the values (0-30) (or whatever you enter) red or something ?

[feisty2]

Is there a way to code a helper function or visualization aid (preferably quick processing preview, minus all the processing) for what "it" thinks is a frequency range ? For example , maybe shade or overlay the values (0-30) (or whatever you enter) red or something ?

What frequency range? Are you saying like calculate "cutoff" and "freq_merge" automatically?

[poisondeathray]

What frequency range? Are you saying like calculate "cutoff" and "freq_merge" automatically?



I mean as a debugging option. You have normalized to the range 0-100, so let's say your cutoff is 20. This means 0-20 will not be affected, right? I want to "see" what it has categorized as 0-20 . Is this the correct cutoff to use for that particular source, or your particular goal ? Maybe the value should have be 36.5 for this specific goal. Or maybe to visualize the modulation effect of a different freq_margin value. I supposed you could just use crazy strenght settings and see where the effect is applied , but remember this is a slow filter . The visualization of exclusion/inclusion areas without actually applying the effect won't go through all the steps/filters and should be faster shouldn't it ?

[feisty2]

I mean as a debugging option. You have normalized to the range 0-100, so let's say your cutoff is 20. This means 0-20 will not be affected, right? I want to "see" what it has categorized as 0-20 . Is this the correct cutoff to use for that particular source, or your particular goal ? Maybe the value should have be 36.5 for this specific goal. Or maybe to visualize the modulation effect of a different freq_margin value. I supposed you could just use crazy strenght settings and see where the effect is applied , but remember this is a slow filter . The visualization of exclusion/inclusion areas without actually applying the effect won't go through all the steps/filters and should be faster shouldn't it ?

"cutoff" is a frequency-wise threshold, you simply can't map it to a mask kind of thing, cuz every pixel is processed in the spatial domain, but you can indeed check the low frequency components and it's fast

call the "lowpass" function here,

def lowpass(src, p):
upsmp = core.fmtc.resample(src, src.width*2, src.height*2, kernel="gauss", a1=100, fulls=True, fulld=True)
clip = core.fmtc.resample(upsmp, src.width, src.height, kernel="gauss", a1=p, fulls=True, fulld=True)
return clip

e.g.
lowpass(clp, 20) -> clp[0, 20]
core.std.MakeDiff(clp, lowpass(clp, 45)) -> clp[55, 100]

[feisty2]

I mean as a debugging option. You have normalized to the range 0-100, so let's say your cutoff is 20. This means 0-20 will not be affected, right? I want to "see" what it has categorized as 0-20 . Is this the correct cutoff to use for that particular source, or your particular goal ? Maybe the value should have be 36.5 for this specific goal. Or maybe to visualize the modulation effect of a different freq_margin value. I supposed you could just use crazy strenght settings and see where the effect is applied , but remember this is a slow filter . The visualization of exclusion/inclusion areas without actually applying the effect won't go through all the steps/filters and should be faster shouldn't it ?

a few more things you should probably be careful with:

the final estimation does not work on videos with sudden scene changes, you might wanna cut each individual scene out and process them separately, either manually or with some batch processing program

if your test videos are very large and you don't have a supercomputer, either downscale them or crop them, I've never tested this thing on any video larger than 720p on my i7-4790k/GTX 970/16G RAM

pel = 2 is faster and does not affect the quality much

use vspipe + rawsource and save the intermediate results (like Super or Basic) as uncompressed raw binaries on your hard drive, it's gonna be A LOT faster if you plan to actually apply this thing to your videos

[feisty2]

I uploaded the alpha version (prototype) of the next Plum release: https://github.com/IFeelBloated/Plum/blob/master/PlumAlpha.py
it features 2 major changes:
the motion estimation part was mostly redesigned
the basic estimation now produces much less aliasing with NLMeans rectifying

you could replace the old version with this alpha test version now and then continue with your tests

[poisondeathray]

thanks feisty ,

I'll have to get back to it later this week, but this deserves some more testing on varied sources



Have you tried on "typical" DVD source yet, where there is a bit of grain / noise ? I noticed many of your tests are from music DVD's kylie & such , which might be "cleaner" than say some action movie DVD . What are your general impressions in that scenario

[feisty2]

thanks feisty ,

I'll have to get back to it later this week, but this deserves some more testing on varied sources



Have you tried on "typical" DVD source yet, where there is a bit of grain / noise ? I noticed many of your tests are from music DVD's kylie & such , which might be "cleaner" than say some action movie DVD . What are your general impressions in that scenario

I don't really have any DVD movies, all movie disks I got are bluray release, cuz unlike early music videos which were shot on 35mm film but mastered at an NTSC/PAL resolution, movies were at least mastered at 2k or even higher resolution, so I will have a native HD copy with bluray release, and DVD releases are just garbage. (relatively speaking)

[poisondeathray]

Ok or cropped BD (or full if you don't crash :) ) ; I'm interested in the context of underlying grain/noise . I'll eventually test these but I'm wondering if you have to prefilter or use special treatment . For example, large grain film stock vs. fine grain pattern

[feisty2]

thanks feisty ,

I'll have to get back to it later this week, but this deserves some more testing on varied sources



Have you tried on "typical" DVD source yet, where there is a bit of grain / noise ? I noticed many of your tests are from music DVD's kylie & such , which might be "cleaner" than say some action movie DVD . What are your general impressions in that scenario

if you're doing the new alpha version,
there's a value that controls the aliasing-fine structure tradeoff and ain't getting parameterized yet..

line 143

dif = core.NLMeans(dif, a, 4, 3.2, src)

"3.2" is the tradeoff value here, larger = smoother result(less aliasing) but weaker enhancement on fine structures

[feisty2]

I removed some useless legacy stuff in the alpha version, please update your PlumAlpha.py
and I did some tests on a high quality (mastertape, ProRes 422HQ, zero compression artifacts) 1080p clip (Ariana Grande - Love Me Harder)
HD videos (even the master tapes) are generally much softer than (master tape quality) SD videos, so it makes sense to sharpen them if you prefer a sharper image..

source (cropped)
http://i.imgur.com/eKsp84M.png

USM (naïve)

clp = core.std.MergeDiff(clp, core.std.MakeDiff(clp, core.std.Convolution(clp,[1,2,1,2,4,2,1,2,1])))

http://i.imgur.com/NhwG52e.png

smarter USM (LSFmod)

clp = havsfunc.LSFmod(clp,defaults="slow",strength=164)

http://i.imgur.com/KDYABhC.png

even smarter USM (finesharp)

clp = finesharp.sharpen(clp, sstr=1.2)

http://i.imgur.com/3ZG6TTq.png

plum (alpha ver)

ref = PlumAlpha.Basic(clp)
clp = PlumAlpha.Final([clp, ref], [PlumAlpha.Super(clp), PlumAlpha.Super(ref)], cutoff=20, freq_margin=12)

http://i.imgur.com/k8zzrUd.png

[feisty2]

conclusion:
the Plum result looks more like a native 1080p picture than the source image, some people might think the Plum result should be the ground truth and the source image should be a 720p upscale in a blind test...
other algorithms enhanced the overall sharpness (while Plum didn't), but failed to make the image more "delicate"

[madshi]

Personally, I'm a big fan of the "delicate" look, but I can identify some problems in the Plum screenshot:

1) There are some "distortions". E.g. the 3rd candle from the right gets fatter/distorted.
2) Some parts of the image get sharpened *a lot* (e.g. the arm chair lattice), other parts not at all (e.g. the face or the candle lamp).
3) There are some ringing artifacts (e.g. above the "bow" at the top center of the image).
4) There are some weird artifacts (e.g. left of the arm chair some of the bright window pixels "streak" into the dark drapes (or whatever that is)).
5) Sometimes some things/lines get broader instead of thinner, e.g. the sharpening of the wood plate at the very left side of the arm chair looks kinda weird to me.

I've been working on an "ideal deconvolution" filter, which almost perfectly reverts a gaussian blur. At relatively low sharpening strengths like in this screenshot comparison my wip filter looks very similar to FineSharp. Which makes me think that maybe Plum is going a bit overboard with just sharpening high frequencies to be even higher/sharper, but completely ignoring the rest of the image? I think I'd prefer Plum to sharpen e.g. the face or the candle lamp or the bra or [...], too, similar to FineSharp. Although, granted, where Plum does get active, it looks nice, probably nicer than FineSharp. The artifacts do worry me, though.

Just my 2 cents, of course. What do you think?

[poisondeathray]

conclusion:
the Plum result looks more like a native 1080p picture than the source image, some people might think the Plum result should be the ground truth and the source image should be a 720p upscale in a blind test...
other algorithms enhanced the overall sharpness (while Plum didn't), but failed to make the image more "delicate"


Keep it coming . I knew you would pop out another version, so I didn't even test the last update :)

One problem is what is "native 1080p" picture supposed to be ? Often it is supposed to be "soft" , as it's shot with diffusion filters and soft dof focus on purpose. Other times it's supposed to be razor sharp, eg. maybe reality TV or documentary.



Some pros/cons to each of them . Depends what you're going for

If you nearest neighbor 2x to look more closely, the plum version has problems with haloing around the light, edge of skirt by her knee

Some look semi-unnatural like the USM, the singer is supposed to be plane in focus, some sharpen the BG too much for this type of shot I would argue

[feisty2]

after several tests on different kinds of videos, I decided to remove NLMeans rectifying, it generally has more cons than pros..
and I changed the default value for a few parameters, you may update PlumAlpha.py if you want to.

new test result of the same video at #86, it now sharpens more low frequency details and hopefully solved some of the mentioned problems..
there still might be some problems, but the tradeoff looks good to me

source (1080p full frame)
https://s3.postimg.org/ifbci86cx/src.png
plum

ref = PlumAlpha.Basic(clp)
clp = PlumAlpha.Final([clp, ref], [PlumAlpha.Super(clp), PlumAlpha.Super(ref)], cutoff=20, freq_margin=12)

https://s18.postimg.org/5dfsdtajb/plum.png

[feisty2]

Personally, I'm a big fan of the "delicate" look, but I can identify some problems in the Plum screenshot:

1) There are some "distortions". E.g. the 3rd candle from the right gets fatter/distorted.
2) Some parts of the image get sharpened *a lot* (e.g. the arm chair lattice), other parts not at all (e.g. the face or the candle lamp).
3) There are some ringing artifacts (e.g. above the "bow" at the top center of the image).
4) There are some weird artifacts (e.g. left of the arm chair some of the bright window pixels "streak" into the dark drapes (or whatever that is)).
5) Sometimes some things/lines get broader instead of thinner, e.g. the sharpening of the wood plate at the very left side of the arm chair looks kinda weird to me.

I've been working on an "ideal deconvolution" filter, which almost perfectly reverts a gaussian blur. At relatively low sharpening strengths like in this screenshot comparison my wip filter looks very similar to FineSharp. Which makes me think that maybe Plum is going a bit overboard with just sharpening high frequencies to be even higher/sharper, but completely ignoring the rest of the image? I think I'd prefer Plum to sharpen e.g. the face or the candle lamp or the bra or [...], too, similar to FineSharp. Although, granted, where Plum does get active, it looks nice, probably nicer than FineSharp. The artifacts do worry me, though.

Just my 2 cents, of course. What do you think?

I did a new test, what do you think about the new result?

[feisty2]

One problem is what is "native 1080p" picture supposed to be ? Often it is supposed to be "soft" , as it's shot with diffusion filters and soft dof focus on purpose. Other times it's supposed to be razor sharp, eg. maybe reality TV or documentary.



Some pros/cons to each of them . Depends what you're going for

but even the sharpest (non-CG) 1080p video I've ever seen is much softer than the master tape image I posted at #1...
this particular video might have some diffusion filter applied to generate the dreamy haze, but nothing really, nothing 1080p has the sharpness even close to the master tape image I posted at #1




If you nearest neighbor 2x to look more closely, the plum version has problems with haloing around the light, edge of skirt by her knee

Some look semi-unnatural like the USM, the singer is supposed to be plane in focus, some sharpen the BG too much for this type of shot I would argue


some of the problems might have been solved in the new test result, I guess...

the arm chair is getting a lot more sharpening than Ariana because there's simply "nothing" to start with for the Ariana part in the image, it's detail-less, while the arm chair has plenty of complex but soft patterns that are prefect for sharpening, I mean Ariana's bra is getting some sharpening in the new result..

[feisty2]

any more feedbacks?
I'm gonna move on and test another video if y'all happy with the result of this Ariana video..?

[madshi]

I did a new test, what do you think about the new result?
Looks like a solid improvement to me. Still not "perfect", though (but that might impossible to achieve, I don't know). Here's the plum image with a few image locations marked where I think plum shows some artifacts:

http://madshi.net/plum.png

And generally, I still think lower frequencies could use (even) more sharpening. But then, I've only looked at this one image. It's never good to judge an algo by just looking at one single image. So please take my comments with a big pinch of salt.

[feisty2]

Looks like a solid improvement to me. Still not "perfect", though (but that might impossible to achieve, I don't know). Here's the plum image with a few image locations marked where I think plum shows some artifacts:

http://madshi.net/plum.png


I think those tiny pixel-wise artifacts might be gone with a different set of "wn" and "scale" values,,, I haven't tested thoroughly, I just thought, these pixel-wise artifacts should be "invisible" in motion anyways so guess I'm gonna move on to another test sample now :devil:


And generally, I still think lower frequencies could use (even) more sharpening. But then, I've only looked at this one image. It's never good to judge an algo by just looking at one single image. So please take my comments with a big pinch of salt.

that's more of a personal taste kinda thing, I just love the high micro contrast sort of "delicate" image (the high frequencies should be, real HIGH, but it shouldn't change the "nature" of the original image in general), you could easily get more low frequency sharpening with a lower "cutoff"

[MonoS]

Probably the default Plum parameters aren't the best when applied to anime material.
The source is just detelecined and cropped.

In order: Source, Plum, PlumAlpha
https://abload.de/img/clipboard01ydsee.png

[feisty2]

Probably the default Plum parameters aren't the best when applied to anime material.
The source is just detelecined and cropped.

In order: Source, Plum, PlumAlpha
https://abload.de/img/clipboard01ydsee.png

no, I don't make filters for CG stuff,,, it's just way too different from photo-like stuff...

you want things to be extra "delicate" with photographic materials (high micro contrast), but not with CG graphics, vectorization is the ultimate answer to all CG graphics out there

photographic materials are just, more "fragile" in every single way than CG graphics, one mistake and all that precious photographic texture is gone permanently, it degrades to some semi-CG picture, which is probably the reason why stuff like waifu2x fails miserably on photographic images

[poisondeathray]

no, I don't make filters for CG stuff,,, it's just way too different from photo-like stuff...

you want things to be extra "delicate" with photographic materials (high micro contrast), but not with CG graphics, vectorization is the ultimate answer to all CG graphics out there

photographic materials are just, more "fragile" in every single way than CG graphics, one mistake and all that precious photographic texture is gone permanently, it degrades to some semi-CG picture, which is probably the reason why stuff like waifu2x fails miserably on photographic images


Be careful how you use the term "CG" so loosely. There is realistic CG that is indistinguishable from real life photos. You're referring to a specific subset of CG

[feisty2]

Be careful how you use the term "CG" so loosely. There is realistic CG that is indistinguishable from real life photos. You're referring to a specific subset of CG

you know what I was trying to say, the cheesy basic CG, not the fancy Hollywood stuff, and that's good enough.

[poisondeathray]

you know what I was trying to say, the cheesy basic CG, not the fancy Hollywood stuff, and that's good enough.

yes, I knew what you were saying, but it's better to be clear about intended usage or applicable scenarios (maybe in the documentation or readme). You're going to save yourself some headaches when someone doing a quick search , or doing some quick tests , asks questions or how come so and so doesn't work . And you will save users the headache of having to test the slow filter on stuff it wasn't meant to be used on

[feisty2]

yes, I knew what you were saying, but it's better to be clear about intended usage or applicable scenarios (maybe in the documentation or readme). You're going to save yourself some headaches when someone doing a quick search , or doing some quick tests , asks questions or how come so and so doesn't work . And you will save users the headache of having to test the slow filter on stuff it wasn't meant to be used on

okay, please tell me an alternative word/phrase for "real life genuine photos + fancy CGs like that wormhole and blackhole in the interstellar movie that look real enough to fool your eyes", and a word/phrase for "images that are not in the aforementioned category"

[poisondeathray]

okay, please tell me an alternative word/phrase for "real life genuine photos + fancy CGs like that wormhole and blackhole in the interstellar movie that look real enough to fool your eyes", and a word/phrase for "images that are not in the aforementioned category"

You can word it however you like - you're the author. And you don't have to do anything if you don't want to . I'm not going to fly to LA and twist your arm :D

I'm just suggesting it's better to be clear and concise.

When you write something like "vectorization is the ultimate answer to all CG graphics out there" , or "photographic materials are just, more "fragile" in every single way than CG graphics" - it's just plain wrong. I know what you meant to say - but you're going to confuse some people

And there's a continuum - it doesn't have to be ultra-realistic-indistinguishable-from-real-life-photos, even semi-realistic CG renders can benefit from this filter or similar approaches

But the category it's defintely not suitable for is "simple cartoons" . That's probably not a good descriptive term but you can work on it. Even your "cheesy basic CG" term is more precise wording than "all CG"

[lansing]

I want to try this out on my bluray but it crashed on load