Vine, cutting edge quality de-halo filter and morphology stuff

[Tormaid]

Originally Posted by feisty2

there's no edge detection with "masking=False"
you do a halo/edge mask for your specific video on your own

^This

Only sane way to do it.

[brucethemoose]

Originally Posted by Tormaid

^This

Only sane way to do it.

Well it's not a mask problem really. That's an area I definitely need to dehalo (and debug the mask is showing it), but the dehalo process itself isn't working.


I can't automatically filter it out and just live with the halos either. It's an edge just like everything else the mask sees, AFAIK there's no way to get a mask to detect edges, but NOT detect edges where there's a white halo in the inner perimeter of an object with a black outline.

EDIT: MadVR's anti ringing seems to have the same idea:



It has a seperate setting for working with light edges.

[feisty2]

@brucethemoose

clp0 = clp
mask = Vine.Dehalo(clp,sigma=0.001,radius=[0,0],alpha=0.76,h=12,show=True)
for i in range(6):
    mask = core.std.Maximum(mask)
clp = Vine.Dehalo(clp,h=4.8,masking=False)    
clp = Vine.Dehalo(clp,h=2,masking=False,cutoff=1)
clp = Vine.Dehalo(clp,h=2,masking=False,cutoff=1)
clp = Vine.Dehalo(clp,h=2,masking=False,cutoff=1)
clp = core.std.MaskedMerge(clp0,clp,mask)

[feisty2]

I don't really get "edge detection" and whatever other stuff you were talking about
but I got my own life so please stop expecting to be spoon fed

[brucethemoose]

I know, I wasn't asking for an update after your other reply, just speculating. I don't fully understand how DeHalo itself works, which means I should look at the code itself.

That looks much better, thanks. The range mask is a good idea.

[JackCY]

Originally Posted by feisty2

readme and more
https://github.com/IFeelBloated/Vine

Vine does not turn your video to a piece of cartoonized shit like other dehalo filters do.
it's slow but the performance is kind of practical still.

I've tried this script and... well it works, kind of... the parameters have to be pushed out the wazoo so thick halos are detected, such as example given on GitHub. But, the performance is awful, outright awful. Even with KNLMeans set to a=1 via Dehalo(a=1). Even with KNLMeans commented out in the Vine.py code so there should be no other crazy expensive operation... well... there is and I don't have another day to find which it is but it certainly is CPU limited not GPU limited in OpenCL.

No matter what Vine runs < 1fps on 1920x816px clip. Around 0.60fps, maybe.

Do I have some bad conversion copied from this thread to get that format Vine wants or is Vine so slow it's practically impractical? < 1fps certainly isn't practical, I've expected something like 10ms delay from processing per full HD frame not 1600ms.

At this speed the script is only good for processing a few still frames, that's about it.

What speeds is OP/creator getting when processing full HD?
From what I see the RAM keeps going +3GB what I would say is per frame, so each frame RAM consumption goes up by 3GB and then drops back down. WTH is eating 3GB.

The moment I comment out Vine out of my script... 80fps. Hell I can do 300fps, doesn't help anyway.

If this script is running 1000ms per frame, you should really put it in the OP or readme of the script as this is not usable for videos one bit in current state of the script and hardware available.

Are there any other dehalo VapourSynth options? Ones that do run fast for video processing? Don't care what visual quality as long as it's fast and usable.

[Mystery Keeper]

If you don't care about visual quality - you should just go with the halos.

[JackCY]

Originally Posted by Mystery Keeper

If you don't care about visual quality - you should just go with the halos.

I do but not to the point of 1+s processing per frame. MadVR has no problem detecting halos at very fast speed, any similar algorithm in VapourSynth that can be used to process videos before encoding and adjust the strength?

[feisty2]

Originally Posted by JackCY

I do but not to the point of 1+s processing per frame. MadVR has no problem detecting halos at very fast speed, any similar algorithm in VapourSynth that can be used to process videos before encoding and adjust the strength?

Madvr works on gpu
nnedi3 works on cpu

[JackCY]

Originally Posted by feisty2

Madvr works on gpu
nnedi3 works on cpu

nnedi3 shouldn't run on CPU, none image processing plugins should. It's insane to run that alg. on CPU.

Output 2000 frames in 88.44 seconds (22.62 fps)

# for i in range(2):
    # clean       = Transpose(NNEDI(Transpose(NNEDI(clean, **nnedi_args)), **nnedi_args))
# clean           = Resample(clean, src.width, src.height, sx=-1.25, sy=-1.25, kernel="cubic", a1=-sharp, a2=0)

vs

Output 20 frames in 47.16 seconds (0.42 fps)

for i in range(2):
    clean       = Transpose(NNEDI(Transpose(NNEDI(clean, **nnedi_args)), **nnedi_args))
clean           = Resample(clean, src.width, src.height, sx=-1.25, sy=-1.25, kernel="cubic", a1=-sharp, a2=0)

Difference? Usable vs unusable, noisier edges (close to original, only slightly cleaned) vs cleaner edges (even more cleaner than original). The edges might be changed by doing some other cheaper operation, such as convolution blur to reduce sharpness or the opposite since the sharpness argument is now defunct, no problem to emulate 0.5 sharpness which helped to remove more of the halos.

Adding convolution blur changed the performance to...

Output 2000 frames in 89.86 seconds (22.26 fps)

Probably a margin of error. But say -0.50fps, 0.984x.

Commenting out 3 lines = 53.85x faster.
This is with "a=3" which is enough for KNLMeansCL. Radius 3.

The next one to go seems to be fmtc:

Resample        = core.fmtc.resample
MakeDiff        = core.std.MakeDiff
MergeDiff       = core.std.MergeDiff
def inline(src):
    upsmp       = Resample(src, src.width * 2, src.height * 2, kernel="gauss", a1=100, **fmtc_args)
    clip        = Resample(upsmp, src.width, src.height, kernel="gauss", a1=p, **fmtc_args)
    return clip

If there is a VS native replacement that's faster.

original cutoff
Output 2000 frames in 90.55 seconds (22.09 fps)
Output 1000 frames in 46.26 seconds (21.62 fps)
convolution 5x5
Output 2000 frames in 85.34 seconds (23.43 fps)
Output 1000 frames in 43.60 seconds (22.93 fps)
convolution 3x3
Output 2000 frames in 74.06 seconds (27.01 fps)
Output 1000 frames in 37.46 seconds (26.69 fps)

Convolution 5x5 not really worth it, looks similar too. 3x3 limits the dehalo ability but it's faster by 5fps = 1.23x, not bad at all if one doesn't mind the reduced halo reduction.

Canny, no difference.
Can't get rid of the necessary format conversion, dunno why format changes somewhere and then causes an error, what operation is changing the format in Vine?
This is what was failing without converting to 32bit float:

          clean           = helpers.cutoff(src, clean, cutoff)
          dif             = MakeDiff(src, clean)

Right between the nlmeans in dehalo.

---

In the end this script can run not 0.4fps but 27fps, 67x faster and usable, though still slow from my POV.

[ChaosKing]

Originally Posted by JackCY

Are there any other dehalo VapourSynth options? Ones that do run fast for video processing? Don't care what visual quality as long as it's fast and usable.

DeHalo_alpha & YAHR:
http://forum.doom9.org/showthread.php?t=166582

Originally Posted by JackCY

nnedi3 shouldn't run on CPU, none image processing plugins should. It's insane to run that alg. on CPU.

So you're volunteering to rewrite all Avisynth/Vapoursynth plugins to run on the GPU?

[JackCY]

Originally Posted by ChaosKing

DeHalo_alpha & YAHR:
http://forum.doom9.org/showthread.php?t=166582

Thanks will try them if I can hunt down all the dependencies.

Result:

DeHalo_alpha
# rx=3., ss = 1.0		# Output 1000 frames in 9.56 seconds (104.62 fps) # disables resampling and uses repair instead
# rx=3., ss = 1.5		# Output 1000 frames in 14.41 seconds (69.38 fps)
YAHR
# blur=2, depth=32		# Output 1000 frames in 33.02 seconds (30.28 fps)
Vine.Dehalo (no nnedi3, cutoff original)
# [3, None], a=3, sigma=1.3, alpha=0.27, h=24, cutoff=4		# Output 1000 frames in 46.09 seconds (21.70 fps)

DeHalo_alpha - general dehalo, easily strong, relatively easy to tweak it's strength up and down
YAHR - only affect 1st halo ring not 2nd, such as black + white halos only white gets removed, black stays, for me impossible to make stronger or weaker I suppose too
Vine.Dehalo - nnedi3 needs to go for it to be usable, might use the DeHalo_alpha resampling and repair instead but simply none is fine too and keeps edges original instead of smoothed via nnedi3 resampling

I even doubt any resamplings in these scripts are done with a mask, so that only used pixels (filtered edges) are resampled say 5% of the image and not all pixels 100% of the image, which would alone provide a big speed up. Resampler scripts probably weren't made with that in mind.

On my test sample with denoise etc.:

DeHalo_alpha
Output 1000 frames in 16.35 seconds (61.16 fps)
encoded 1000 frames, 68.57 fps, 4505.42 kbps, 22.40 MB
encode time 0:00:14 / CPU Usage: 8.68%

No dehalo
Output 1000 frames in 14.83 seconds (67.42 fps)
encoded 1000 frames, 80.33 fps, 5468.65 kbps, 27.19 MB
encode time 0:00:12 / CPU Usage: 11.23%

[feisty2]

Originally Posted by JackCY

I even doubt any resamplings in these scripts are done with a mask, so that only used pixels (filtered edges) are resampled say 5% of the image and not all pixels 100% of the image, which would alone provide a big speed up. Resampler scripts probably weren't made with that in mind.

"pscrn" is the internal masking for nnedi3, set it 0 and you'll know how slow nnedi3 could be when its working on the entire image..

Originally Posted by JackCY

The edges might be changed by doing some other cheaper operation, such as convolution blur to reduce sharpness or the opposite since the sharpness argument is now defunct, no problem to emulate 0.5 sharpness which helped to remove more of the halos.

Originally Posted by JackCY

This is with "a=3" which is enough for KNLMeansCL. Radius 3.

supersampling(nnedi3) is supposed to remove aliasing, there's gonna be the aliasing-ringing tradeoff like all convolution filters, supersampling makes sure you get the best result from both sides and simply avoid the tradeoff.

a=3 is not "non-local"

Vine is a quality solution rather than realtime solution, but go ahead and remove things you think are pointless, I'm not stopping you...

[Motenai Yoda]

Originally Posted by feisty2

"pscrn" is the internal masking for nnedi3, set it 0 and you'll know how slow nnedi3 could be when its working on the entire image...

actually nnedi3 pscrn mask where nnedi is applied but all the other pixels are rescaled with a bicubic resample, so a resample is still applied to the entire frame

[feisty2]

Originally Posted by Motenai Yoda

actually nnedi3 pscrn mask where nnedi is applied but all the other pixels are rescaled with a bicubic resample, so a resample is still applied to the entire frame

Bicubic is negligible performance-wise

[xekon]

I am trying to use Vine, as soon as I hit F5 to generate a preview in VSedit, it crashes. After reading though this thread I am assuming its because of one of my prerequisites. I seen the post about using Myrsloik's build of fmtconv. Where could I find his current build? I am currently using R20 of cretindesalpes' build. I am not sure where to look for a crash log. I dont know if Vapoursynth or VSedit has one when its a Fatal Crash.

here is my code:

original_format = ret.format
float_format = core.register_format(original_format.color_family, vs.FLOAT, 32, original_format.subsampling_w, original_format.subsampling_h)
ret = core.resize.Bicubic(ret, format=float_format.id)
ret = Vine.Dehalo(ret, [3, None], sigma=1.6, alpha=0.18, h=12, sharp=0.5, cutoff=2)
ret = Vine.Dehalo(ret, [3, None], sigma=2.2, alpha=0.18, h=24, sharp=0.5, cutoff=2)
ret = core.resize.Bicubic(ret, format=original_format.id)
ret.set_output()

EDIT: I found and tried jackoneill's fmtconv: https://github.com/dubhater/fmtconv/...ter/build/unix
Vine still crashes immediately every time I hit F5, not sure how to go about finding the culprit.

[feisty2]

r5:
a blending operation is now appended after super-sampling, weight determined by the "sharp" parameter
mathematical expression: weight = constant * sharp * ln(1 + 1 / (constant * sharp)) (constant determines how fast weight gets near 1.0 as sharp gets larger)
the mathematical limit for weight is 0 (simply returns the resampled result) as sharp goes infinitely close to 0, or 1 (simply returns the clip before resampling) as sharp goes towards infinity
now if you feel like to barf any time math is somewhere around you, and would like not to analyze that weight expression with differential calculus, all you gotta know is a larger "sharp" value also results in less percentage of super-sampling

[bebs]

It seems that KNLMeansCL breaks Vine, i keep having an error KNLMeansCL: 'rclip' does not match the source clip! when using KNLMeansCL 1.0.0.
KNLMeansCL .77 is working fine.

[Jindadil007]

Originally Posted by bebs

It seems that KNLMeansCL breaks Vine, i keep having an error KNLMeansCL: 'rclip' does not match the source clip! when using KNLMeansCL 1.0.0.
KNLMeansCL .77 is working fine.

Correctly Reported...i am also using .77 for the same reason...