Oyster, placebo compression artifacts buster

[feisty2]

readme and more
https://github.com/IFeelBloated/Oyster

How this thing works:
certain component of the signal (something other than noise or blankness) would require either high resolution in time domain or frequency domain, but not both (uncertainty principle).

videos are non-stationary signals obviously, so ideal Fourier transform (something like DFTTest(sbsize=10000000000000001, tbsize=10000000000000001)) won't work, transform like that has a high frequency resolution but very poor time resolution, and leads to catastrophic results on components carrying lots of time features (sharp edges, delicate details, etc.), ringings, edges blurred out, and shit like that.

so, short time Fourier transform (something like DFTTest(sbsize=3, tbsize=3)), well, this kind of transform has a relatively higher time resolution but poor frequency resolution, and yeah, you know what kind of shit is about to happen next (washed-out, posterized, etc.), and lim(window size -> 0) gives the completely untransformed data! (frequency resolution -> 0)

anyways, it all comes down to one word, "resolution", you need higher resolution to get better quality, and BM3D, state of the art denoiser enhances frequency resolution by matching similar macroblocks. once a group of similar blocks have been matched, the group of signals go (mostly) stationary as similar blocks share similar copies of frequency distribution, and that's good news, and now go large-window-sized Fourier transform and get a high frequency resolution. and no worries about all that time resolution stuff cuz stationary signal simply has no time feature.

so why do we need more, I mean, BM3D sounds definitely awesome, the thing is, BM3D maximizes frequency resolution by re-positioning (block matching) the video signal, and after all these smart and well designed procedures, signal goes MOSTLY stationary but apparently not COMPLETELY stationary so there's still loss of time feature, BM3D manages to minimize that, but couldn't cancel that out completely, and then you notice that loss of sharpness on edges and delicate details and that's why NLMeans, a time domain filter should step in and clean the mess, and about MDeGrain? well, BM3D simply isn't aggressive enough to compression artifacts, you need something brutal and fierce to wipe the shit away first regardless of detail loss, and then recover the lost details and that's why MDeGrain stays around.

MDeGrain -> BM3D -> NLMeans, it just gets more and more delicate each time and details are restored gradually, and artifacts gets its ass kicked

log:
r4:
strict sanity check in every single function, a lot more user friendly and should be ok to work like any typical module now
added some temporal padding to KNLMeansCL
re-licensed to LGPL v3.0
trivial modifications to the doc
.gitignore

r3
algorithmic update, so quality boost

r2.1
removed global core cuz Myrsloik said it was a horrible idea, simple "import Oyster" works now
new function Search
rewrote Basic
trivial modifications to Final
removed SATD, SAD works better on compression artifacts
removed TrueMotion

[jackoneill]

Who is that?

[feisty2]

Originally Posted by jackoneill

Who is that?

what?
you mean the sample clip?
Kylie Minogue - Come Into My World
one of the best music videos like ever

[AzraelNewtype]

Originally Posted by feisty2

Waifu2x(dunno why but it looks kinda cartoonized somehow)

Because waifu2x is a neural network scaler trained to upscale cartoons, not live action? Its ability to fight compression artifacts (typically jpeg artifacts) is in service of that goal, not general purpose cleaning.

[feisty2]

Originally Posted by AzraelNewtype

Because waifu2x is a neural network scaler trained to upscale cartoons, not live action? Its ability to fight compression artifacts (typically jpeg artifacts) is in service of that goal, not general purpose cleaning.

it has a photo mode indeed, but still...
no luck with that

[BakaProxy]

#3DWaifusExist

Verstuurd vanaf mijn SM-A500FU met Tapatalk

[feisty2]

Originally Posted by BakaProxy

#3DWaifusExist

Verstuurd vanaf mijn SM-A500FU met Tapatalk

Please do tell, where

[BakaProxy]

Originally Posted by feisty2

Please do tell, where

They're called real people

Verstuurd vanaf mijn SM-A500FU met Tapatalk

[feisty2]

Originally Posted by BakaProxy

They're called real people

Verstuurd vanaf mijn SM-A500FU met Tapatalk

okay, not funny.
thought an inter-waifu2x was out. (waifu2x is intra)

[feisty2]

r2.1
removed global core cuz Myrsloik said it was a horrible idea, simple "import Oyster" works now
new function Search
rewrote Basic
trivial modifications to Final
removed SATD, SAD works better on compression artifacts
removed TrueMotion

[RazorBurn]

Got error on r2.1

Failed to evaluate the script:
Python exception: Crop: cropped area extends beyond frame dimensions
Traceback (most recent call last):
  File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26897)
  File "C:/Temp/StaxRip Temp/SampleClip_temp/SampleClip.vpy", line 11, in <module>
    ref = Oyster.Basic(y,vec)
  File "C:\Apps\Python\lib\site-packages\Oyster.py", line 171, in Basic
    _mask           = helpers.genblockmask (ShufflePlanes (src, 0, vs.GRAY))
  File "C:\Apps\Python\lib\site-packages\Oyster.py", line 70, in genblockmask
    clip            = CropAbs (clip, src.width, src.height, 0, 0)
  File "src\cython\vapoursynth.pyx", line 1383, in vapoursynth.Function.__call__ (src\cython\vapoursynth.c:25204)
vapoursynth.Error: Crop: cropped area extends beyond frame dimensions

My Code

import vapoursynth as vs
core = vs.get_core()
import mvsfunc 
import edi_rpow2 as edi
import mvmulti
import Oyster
clip = core.lsmas.LibavSMASHSource(r"D:\Installers\Multimedia\Video Test Samples\SampleClip.mp4")
clp = mvsfunc.ToRGB(input = clip, depth=32)
y = core.std.ShufflePlanes(clp,0,vs.GRAY)
vec = Oyster.Search(y)
ref = Oyster.Basic(y,vec)
y = Oyster.Final(y,ref,vec)
clip =  core.std.ShufflePlanes([y,clp], [0,1,2], vs.YUV)
clip.set_output()

[feisty2]

Originally Posted by RazorBurn

Got error on r2.1

clp = mvsfunc.ToRGB(input = clip, depth=32)
y = core.std.ShufflePlanes(clp,0,vs.GRAY)
...

???
and what's the resolution of your clip?

[feisty2]

r3
algorithmic update, so quality boost

[feisty2]

so, bad news here.
been hours and I tried like a million sets of parameters to investigate if there's any chance of quality-speed tradeoff, and, failed to get rid of the abysmal performance even tho had reached the bottom line of almost, everything.
I lowered the precision down to uint8_t, and picked jackoneill's heavily asm optimized mvtools
I used fast profile on V-BM3D
I ditched Oyster.Super, made pel 1, lowest value possible, switched vector searching algorithm from exhaustive to hex
I disabled the varying search window of NLMeans, and fixed that to 4, constant
and, the performance went from 0.00fps up to 0.11fps on 720x480 GRAY, but, I mean, that's still outrageously SLOOOW..
guys, I really tried, but... yeah, and I'm out of ideas of performance boost..

[Mystery Keeper]

feisty2, 'tis okay. There was a time when I had to ditch simple anime anti-aliasing because it was too slow on my ancient PC. Eventually we'll have enough power to put your script into good use.

[feisty2]

Originally Posted by Mystery Keeper

feisty2, 'tis okay. There was a time when I had to ditch simple anime anti-aliasing because it was too slow on my ancient PC. Eventually we'll have enough power to put your script into good use.

I actually think there will be a huge performance boost if someone could make a GPU version of BM3D...

and I wrote the underlying mechanism at #1, hopefully not some kind of "utterly bizarre misunderstanding"

[lansing]

Is this 64bit only? The required mvtool plugin only has a 64bit version

[feisty2]

Originally Posted by lansing

Is this 64bit only? The required mvtool plugin only has a 64bit version

uhmm, I could compile an x86 binary if you need that, this thing consumes A LOT OF ram, so that probably won't work anyway, you got that 4GB upper limit on x86 programs

[Are_]

Tell me about that, I was testing if I had all the dependencies right with a 1080p source and ran OOM on my 16GB Ram machine.

[feisty2]

Originally Posted by Are_

Tell me about that, I was testing if I had all the dependencies right with a 1080p source and ran OOM on my 16GB Ram machine.

I got 16G also and didn't do any 1080p test, that's just way too heavy to my crappy i7-4790k, guess you'll need Xeon E5 or maybe Xeon E7 to go for a 1080p test, the highest resolution test I did is 720p, and it ran for an hour or so to just reconstruct one single frame

and also you will have to vspipe intermediate clips (basic estimation, etc) to your hard drive to avoid OOM on 720p
basically, something like this

ref = Oyster.Basic (clp, Oyster.Super (clp))
clp = Oyster.Destaircase (clp, ref, block_step=2)
clp = Oyster.Deringing (clp, ref, block_step=2)

goes like

ref = Oyster.Basic (clp, Oyster.Super (clp))
ref.set_output()
#vspipe xxx.vpy ref.rgb

ref = core.raws.Source ("ref.rgb")
clp = Oyster.Destaircase (clp, ref, block_step=2)
clp.set_output()
#vspipe xxx.vpy clp.rgb

ref = core.raws.Source ("ref.rgb")
clp = core.raws.Source ("clp.rgb")
clp = Oyster.Deringing (clp, ref, block_step=2)
clp.set_output()
#vspipe xxx.vpy whatever.rgb