Oyster, placebo compression artifacts buster

[feisty2]

readme and more
https://github.com/IFeelBloated/Oyster

How this thing works:
certain component of the signal (something other than noise or blankness) would require either high resolution in time domain or frequency domain, but not both (uncertainty principle).

videos are non-stationary signals obviously, so ideal Fourier transform (something like DFTTest(sbsize=10000000000000001, tbsize=10000000000000001)) won't work, transform like that has a high frequency resolution but very poor time resolution, and leads to catastrophic results on components carrying lots of time features (sharp edges, delicate details, etc.), ringings, edges blurred out, and shit like that.

so, short time Fourier transform (something like DFTTest(sbsize=3, tbsize=3)), well, this kind of transform has a relatively higher time resolution but poor frequency resolution, and yeah, you know what kind of shit is about to happen next (washed-out, posterized, etc.), and lim(window size -> 0) gives the completely untransformed data! (frequency resolution -> 0)

anyways, it all comes down to one word, "resolution", you need higher resolution to get better quality, and BM3D, state of the art denoiser enhances frequency resolution by matching similar macroblocks. once a group of similar blocks have been matched, the group of signals go (mostly) stationary as similar blocks share similar copies of frequency distribution, and that's good news, and now go large-window-sized Fourier transform and get a high frequency resolution. and no worries about all that time resolution stuff cuz stationary signal simply has no time feature.

so why do we need more, I mean, BM3D sounds definitely awesome, the thing is, BM3D maximizes frequency resolution by re-positioning (block matching) the video signal, and after all these smart and well designed procedures, signal goes MOSTLY stationary but apparently not COMPLETELY stationary so there's still loss of time feature, BM3D manages to minimize that, but couldn't cancel that out completely, and then you notice that loss of sharpness on edges and delicate details and that's why NLMeans, a time domain filter should step in and clean the mess, and about MDeGrain? well, BM3D simply isn't aggressive enough to compression artifacts, you need something brutal and fierce to wipe the shit away first regardless of detail loss, and then recover the lost details and that's why MDeGrain stays around.

MDeGrain -> BM3D -> NLMeans, it just gets more and more delicate each time and details are restored gradually, and artifacts gets its ass kicked

log:
r4:
strict sanity check in every single function, a lot more user friendly and should be ok to work like any typical module now
added some temporal padding to KNLMeansCL
re-licensed to LGPL v3.0
trivial modifications to the doc
.gitignore

r3
algorithmic update, so quality boost

r2.1
removed global core cuz Myrsloik said it was a horrible idea, simple "import Oyster" works now
new function Search
rewrote Basic
trivial modifications to Final
removed SATD, SAD works better on compression artifacts
removed TrueMotion

[jackoneill]

Who is that?

[feisty2]

Originally Posted by jackoneill

Who is that?

what?
you mean the sample clip?
Kylie Minogue - Come Into My World
one of the best music videos like ever

[AzraelNewtype]

Originally Posted by feisty2

Waifu2x(dunno why but it looks kinda cartoonized somehow)

Because waifu2x is a neural network scaler trained to upscale cartoons, not live action? Its ability to fight compression artifacts (typically jpeg artifacts) is in service of that goal, not general purpose cleaning.

[feisty2]

Originally Posted by AzraelNewtype

Because waifu2x is a neural network scaler trained to upscale cartoons, not live action? Its ability to fight compression artifacts (typically jpeg artifacts) is in service of that goal, not general purpose cleaning.

it has a photo mode indeed, but still...
no luck with that

[BakaProxy]

#3DWaifusExist

Verstuurd vanaf mijn SM-A500FU met Tapatalk

[feisty2]

Originally Posted by BakaProxy

#3DWaifusExist

Verstuurd vanaf mijn SM-A500FU met Tapatalk

Please do tell, where

[BakaProxy]

Originally Posted by feisty2

Please do tell, where

They're called real people

Verstuurd vanaf mijn SM-A500FU met Tapatalk

[feisty2]

Originally Posted by BakaProxy

They're called real people

Verstuurd vanaf mijn SM-A500FU met Tapatalk

okay, not funny.
thought an inter-waifu2x was out. (waifu2x is intra)

[feisty2]

r2.1
removed global core cuz Myrsloik said it was a horrible idea, simple "import Oyster" works now
new function Search
rewrote Basic
trivial modifications to Final
removed SATD, SAD works better on compression artifacts
removed TrueMotion

[RazorBurn]

Got error on r2.1

Failed to evaluate the script:
Python exception: Crop: cropped area extends beyond frame dimensions
Traceback (most recent call last):
  File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26897)
  File "C:/Temp/StaxRip Temp/SampleClip_temp/SampleClip.vpy", line 11, in <module>
    ref = Oyster.Basic(y,vec)
  File "C:\Apps\Python\lib\site-packages\Oyster.py", line 171, in Basic
    _mask           = helpers.genblockmask (ShufflePlanes (src, 0, vs.GRAY))
  File "C:\Apps\Python\lib\site-packages\Oyster.py", line 70, in genblockmask
    clip            = CropAbs (clip, src.width, src.height, 0, 0)
  File "src\cython\vapoursynth.pyx", line 1383, in vapoursynth.Function.__call__ (src\cython\vapoursynth.c:25204)
vapoursynth.Error: Crop: cropped area extends beyond frame dimensions

My Code

import vapoursynth as vs
core = vs.get_core()
import mvsfunc 
import edi_rpow2 as edi
import mvmulti
import Oyster
clip = core.lsmas.LibavSMASHSource(r"D:\Installers\Multimedia\Video Test Samples\SampleClip.mp4")
clp = mvsfunc.ToRGB(input = clip, depth=32)
y = core.std.ShufflePlanes(clp,0,vs.GRAY)
vec = Oyster.Search(y)
ref = Oyster.Basic(y,vec)
y = Oyster.Final(y,ref,vec)
clip =  core.std.ShufflePlanes([y,clp], [0,1,2], vs.YUV)
clip.set_output()

[feisty2]

Originally Posted by RazorBurn

Got error on r2.1

clp = mvsfunc.ToRGB(input = clip, depth=32)
y = core.std.ShufflePlanes(clp,0,vs.GRAY)
...

???
and what's the resolution of your clip?

[feisty2]

r3
algorithmic update, so quality boost

[feisty2]

so, bad news here.
been hours and I tried like a million sets of parameters to investigate if there's any chance of quality-speed tradeoff, and, failed to get rid of the abysmal performance even tho had reached the bottom line of almost, everything.
I lowered the precision down to uint8_t, and picked jackoneill's heavily asm optimized mvtools
I used fast profile on V-BM3D
I ditched Oyster.Super, made pel 1, lowest value possible, switched vector searching algorithm from exhaustive to hex
I disabled the varying search window of NLMeans, and fixed that to 4, constant
and, the performance went from 0.00fps up to 0.11fps on 720x480 GRAY, but, I mean, that's still outrageously SLOOOW..
guys, I really tried, but... yeah, and I'm out of ideas of performance boost..

[zorr]

Originally Posted by feisty2

been hours and I tried like a million sets of parameters to investigate if there's any chance of quality-speed tradeoff, and, failed to get rid of the abysmal performance
...
I'm out of ideas of performance boost..

Originally Posted by feisty2

and also you HAVE TO create intermediate raw files for your process chain.
it will probably be hundreds times faster unless you have 1TB of RAM or so

I sped up one of my Avisynth processes by creating intermediate files for each phase (QTGMC -> SMDegrain -> other filters). It speeds up processing tremendously especially when using temporal filters. In theory the frames could be cached but perhaps there just isn't enough memory or maybe the caches are just too small.

Just wondering if doing that for the sub-processes inside Oyster functions would also give speedup. So for example Deblocking could make files between core.NLMeans and core.BMBasic etc. It would of course make calling the filters convoluted, you'd have to pass in a phase number and some extra input files and do the processing in multiple scripts. But I think it might be worth investigating if any speedup is possible that way.

[asarian]

Originally Posted by zorr

I sped up one of my Avisynth processes by creating intermediate files for each phase (QTGMC -> SMDegrain -> other filters).

What kind of intermediate file-size are we talking about here? I tried a 'Super' intermediate pass, but ran out of hard disk space (Sic!) at several hundred G (for a 720x480p source).

[zorr]

Originally Posted by asarian

What kind of intermediate file-size are we talking about here? I tried a 'Super' intermediate pass, but ran out of hard disk space (Sic!) at several hundred G (for a 720x480p source).

Im my case it was all 8bit and the clips were not that long. Super will need much more because it's 4x the resolution (assuming pel=4) and in 32bits per component. If you run out of hd space then maybe try to do the processing in segments. Also using a compressed (but still lossless) intermediate file such as FFV1 will help.

[asarian]

Originally Posted by zorr

Im my case it was all 8bit and the clips were not that long. Super will need much more because it's 4x the resolution (assuming pel=4) and in 32bits per component. If you run out of hd space then maybe try to do the processing in segments. Also using a compressed (but still lossless) intermediate file such as FFV1 will help.

My 720x480p source was a full-length movie. So, a compressed output is good idea.

[Mystery Keeper]

feisty2, 'tis okay. There was a time when I had to ditch simple anime anti-aliasing because it was too slow on my ancient PC. Eventually we'll have enough power to put your script into good use.

[feisty2]

Originally Posted by Mystery Keeper

feisty2, 'tis okay. There was a time when I had to ditch simple anime anti-aliasing because it was too slow on my ancient PC. Eventually we'll have enough power to put your script into good use.

I actually think there will be a huge performance boost if someone could make a GPU version of BM3D...

and I wrote the underlying mechanism at #1, hopefully not some kind of "utterly bizarre misunderstanding"