Vine, cutting edge quality de-halo filter and morphology stuff

[feisty2]

readme and more
https://github.com/IFeelBloated/Vine

Vine does not turn your video to a piece of cartoonized shit like other dehalo filters do.
it's slow but the performance is kind of practical still.

[feisty2]

r2:
replaced hacky floatfilters with the ones in core functions
now requires vaporsynth r33 test4 or later

[feisty2]

r3:
1. strict sanity check in every single function, a lot more user friendly and should be ok to work like any typical module now
2. trivial modifications to the mask
3. removed the morphology class and exposed all morphological functions with Vine.Dehalo
4. re-licensed to LGPL v3.0
5. trivial modifications to the doc (color images instead of gray stuff, etc..)
6. gitignore

[hydra3333]

Trying out Vine and Plum but receiving an error message I'm not sure what to do with:
"C:\SOFTWARE\Vapoursynth-x64\VSPipe.exe" "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.08.24.14.00.23.61.vpy" - --y4m | "C:\SOFTWARE\ffmpeg\0-homebuilt-x64\x264-mp4.exe" - --stdin y4m --thread-input --frames 45810 --profile high --level 4.1 --preset slow --interlaced --tff --no-cabac --crf 16 --sar 64:45 --colormatrix bt470bg -o "s:\HDTV\WDTVlive\MP4-VS\x64-1\test-temp.2016.08.24.14.00.23.61.h264"
Script evaluation failed:
Python exception: Vine.Dehalo: the sample type of src has to be single precision!
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26905)
File "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.08.24.14.00.23.61.vpy", line 487, in <module>
main()
File "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.08.24.14.00.23.61.vpy", line 141, in main
video = Vine.Dehalo(video, radius=[2, None])
File "C:\SOFTWARE\Vapoursynth-x64\Vine.py", line 113, in Dehalo
raise TypeError("Vine.Dehalo: the sample type of src has to be single precision!")
TypeError: Vine.Dehalo: the sample type of src has to be single precision!
Script:
import vapoursynth as vs
import havsfuncTS as haf # this version uses vanilla TemporalSoften instead of TemporalSoften2, as it will be "better" over time
import mvsfunc as mvs # http://forum.doom9.org/showthread.php?t=172564
import finesharp as finesharp # http://forum.doom9.org/showthread.php?p=1777815#post1777815 http://avisynth.nl/index.php/FineSharp
import Plum # http://forum.doom9.org/showthread.php?t=173775 https://github.com/IFeelBloated/Plum
import Vine # http://forum.doom9.org/showthread.php?t=173703 https://github.com/IFeelBloated/Vine
core = vs.get_core(accept_lowercase=True) # leave off threads=8 so it auto-detects threads
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\AddGrain.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\BM3D.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\d2vsource.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\Deblock.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\DFTTest.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\fmtconv.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\KNLMeansCL.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libawarpsharp2.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libmvtools.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libmvtools_sf_emt64.dll') # for Plum, the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libnnedi3.dll') # the r'' indicates do not treat special characters and accept backslashes
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\scenechange.dll') # http://forum.doom9.org/showthread.php?t=166769
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\TCanny_AVX.dll') # http://forum.doom9.org/showthread.php?t=166769
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\vcfreq.dll') # http://forum.doom9.org/showthread.php?t=171413 http://www.avisynth.nl/users/vcmohan/index.html ... for Plum
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\vsdctfilter.dll') # http://vfrmaniac.fushizen.eu/works
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\vsfft3dfilter.dll') # http://vfrmaniac.fushizen.eu/works
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\Yadifmod.dll') # the r'' indicates do not treat special characters and accept backslashes
core.avs.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\DGIndex\DGDecodeNV.dll')
# NOTE:: INDENTING IS CRITICAL AND MUST BE PRECISELY THE SAME WITHIN EACH FUNCTION
# (functions removed in this example)
def main():
video = core.d2v.Source(r'T:\HDTV\WDTVlive\MP4-VS\x64-1\test.d2v')
video = core.std.SeparateFields(video, tff=True)
video = haf.Deblock_QED(video)
video = haf.Weave(video, tff=True)
video = mvs.AssumeTFF(video)
video = haf.QTGMC(video, TFF=True, Preset="Very Slow", Sharpness=1.2, SLMode=2, EZKeepGrain=1.2, NoiseProcess=2) # result is double framerate progressive, so re-interlate it later
video = Vine.Dehalo(video, radius=[2, None])
deconv = Plum.Basic(video)
conv = Plum.Basic(video,mode="convolution")
sup = Plum.Super([video, None])
supdeconv = Plum.Super([video, deconv])
supconv = Plum.Super([conv, None])
video = Plum.Final([video, deconv, conv], [sup, supdeconv, supconv], strength=2.1, cutoff=16) # middlin
video = core.std.SeparateFields(video, tff=True)
video = core.std.SelectEvery(video, cycle=4, offsets=[0, 3]) # looks like avisynth
video = haf.Weave(video, tff=True)
video = mvs.AssumeTFF(video)
video.set_output()
return True
main()

Any and all suggestions welcomed.
... also about "best" positioning of Vine for interlaced material (eg after QTGMC or within the prior separatefields/deblock)

[hydra3333]

Using "video = core.text.ClipInfo(video)" shows that the video sample type incoming into Vine = Integer.
How do I turn it into single precision for use in Vine, and back again for use in other functions ? Would there be quality losses associated with the conversions ?

[feisty2]

video = haf.QTGMC(video, TFF=True, Preset="Very Slow", Sharpness=1.2, SLMode=2, EZKeepGrain=1.2, NoiseProcess=2) # result is double framerate progressive, so re-interlate it later

video = core.fmtc.bitdepth(video,bits=32,fulls=False,fulld=True)

video = Vine.Dehalo(video, radius=[2, None])
deconv = Plum.Basic(video)
conv = Plum.Basic(video,mode="convolution")
sup = Plum.Super([video, None])
supdeconv = Plum.Super([video, deconv])
supconv = Plum.Super([conv, None])
video = Plum.Final([video, deconv, conv], [sup, supdeconv, supconv], strength=2.1, cutoff=16) # middlin
video = core.std.SeparateFields(video, tff=True)

[Mystery Keeper]

Also, here's a snippet for plugins autoloading. It also adds a path from which other scripts can be imported easily.
from __future__ import print_function

import vapoursynth as vs
core = vs.get_core(threads=8)
core.set_max_cache_size(12000)

import sys
sys.path.append('D:\\vapoursynth-plugins\\py\\')
import platform
architecture = platform.architecture()
if architecture[0] == '64bit':
vapoursynth_plugins_path = 'D:\\vapoursynth-plugins\\64bit\\'
else:
vapoursynth_plugins_path = 'D:\\vapoursynth-plugins\\32bit\\'
print('Plugins folder: ', vapoursynth_plugins_path, end='\n', file=sys.stderr)
import os
for filename in os.listdir(vapoursynth_plugins_path):
if filename[-4:] != '.dll':
continue
try:
core.std.LoadPlugin(vapoursynth_plugins_path + filename)
except Exception as e:
print('Error: ', e, end='\n', file=sys.stderr)

[hydra3333]

Thank you.

[feisty2]

r4:
algorithmic upgrade, better quality and a bit slower

[hydra3333]

Hello. With the latest versions of VS plugins and VS R33.1, the sdcript below either
(i) causes vspipe to crash straight away (the windows dialogue popup box type of crash, without any python traceback) or
(ii) the process encodes at 0.35 fps

What am I doing wrong ? ... suggestions very much welcomed.

Input is PAL 576i mpeg2.

"C:\SOFTWARE\Vapoursynth-x64\VSPipe.exe" "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.09.13.19.17.25.42.vpy" - --y4m | "C:\SOFTWARE\ffmpeg\0-homebuilt-x64\x264-mp4.exe" - --stdin y4m --thread-input --profile high --level 4.1 --preset slow --interlaced --tff --no-cabac --crf 16 --sar 64:45 --colormatrix bt470bg -o "s:\HDTV\WDTVlive\MP4-VS\x64-1\test-temp.2016.09.13.19.17.25.42.h264"

import vapoursynth as vs
import havsfuncTS as haf # this version uses vanilla TemporalSoften instead of TemporalSoften2, as it will be "better" over time
import mvsfunc as mvs # http://forum.doom9.org/showthread.php?t=172564
import finesharp as finesharp # http://forum.doom9.org/showthread.php?p=1777815#post1777815 http://avisynth.nl/index.php/FineSharp
import Plum # http://forum.doom9.org/showthread.php?t=173775 https://github.com/IFeelBloated/Plum
import Vine # http://forum.doom9.org/showthread.php?t=173703 https://github.com/IFeelBloated/Vine
import Oyster # http://forum.doom9.org/showthread.php?t=173470 https://github.com/IFeelBloated/Oyster
import mvmulti # http://forum.doom9.org/showthread.php?p=1777891#post1777891
core = vs.get_core(accept_lowercase=True) # leave off threads=8 so it auto-detects threads
# snip - loaded plugins and stuff
def main():
video = core.d2v.Source(r'T:\HDTV\WDTVlive\MP4-VS\x64-1\test.d2v')
video = core.std.Trim(video,first=0,length=1000)
video = core.std.SeparateFields(video, tff=True)
# Vine only accepts 32bit float input
video = core.fmtc.bitdepth(video,bits=32,fulls=False,fulld=True) #convert to 32bit float. Specify if the clip is full-range (True) or TV-range (False). fulls is for input, fulld for output.
# vine middlin
video = Vine.Dehalo(video, radius=[2, None])
video = core.fmtc.bitdepth(video,bits=8,fulls=True,fulld=False) #convert to 8bit Integer. Specify if the clip is full-range (True) or TV-range (False). fulls is for input, fulld for output.
video = haf.Weave(video, tff=True)
video = mvs.AssumeTFF(video)
video.set_output()
return True
main()

x64 vapoursynth, PC is an i3820 4-core, with 16Gb mem, Nvidia 750Ti with latest nvidia driver.
Most of the memory appears to be free during the encode at least for the first 10 mins, 8 hyperthrerads were running at about 20% cpu each.
Other non-Vine encodes run ok.

[feisty2]

try Myrsloik's build of fmtconv, https://dl.dropboxusercontent.com/u/73468194/fmtconvwithdebugsymbols.7z
it ain't crashed since the day I got it....
cretindesalpes' build always crashes randomly like probably at any moment for some mysterious reason(s)...

[Myrsloik]

try Myrsloik's build of fmtconv, https://dl.dropboxusercontent.com/u/73468194/fmtconvwithdebugsymbols.7z
it ain't crashed since the day I got it....
cretindesalpes' build always crashes randomly like probably at any moment for some mysterious reason(s)...

You could also try your luck with the internal resizer, that should work just as well for this. But if you do find a crash try to figure out where it happens so it can be fied.

[hydra3333]

You could also try your luck with the internal resizer, that should work just as well for this.
If I interpret that correctly ...
http://www.vapoursynth.com/doc/functions/resize.html
OK, not sure how to use the resizer to convert to 32bit float and then back to 8bit ... advice welcome.

try Myrsloik's build of fmtconv, https://dl.dropboxusercontent.com/u/73468194/fmtconvwithdebugsymbols.7z
it ain't crashed since the day I got it....
Thank you. Although, I admit to preferring not using software from dropboxes ... how quaint of me.

[jackoneill]

To convert to 32 bit float and back:

original_format = video.format

float_format = core.register_format(original_format.color_family, vs.FLOAT, 32, original_format.subsampling_w, original_format.subsampling_h)

video = core.resize.Bicubic(video, format=float_format.id)

...

video = core.resize.Bicubic(video, format=original_format.id)

[TheFluff]

I gotta say it's pretty cool that the resizer actually supports resizing to a format programmatically registered at runtime

I mean, it's not a scenario that's bound to come up often but when it does it's pretty awesome to have that functionality

[hydra3333]

Yes cool.
core = vs.get_core(accept_lowercase=True) # leave off threads=8 so it auto-detects threads
def main():
video = core.d2v.Source(r'T:\HDTV\WDTVlive\MP4-VS\x64-1\test.d2v')
video = core.std.Trim(video,first=0,length=1000)
video = core.std.SeparateFields(video, tff=True)
original_format = video.format # http://forum.doom9.org/showthread.php?p=1780587#post1780587
float_format = core.register_format(original_format.color_family, vs.FLOAT, 32, original_format.subsampling_w, original_format.subsampling_h) # http://forum.doom9.org/showthread.php?p=1780587#post1780587
video = core.resize.Bicubic(video, format=float_format.id) # http://forum.doom9.org/showthread.php?p=1780587#post1780587
#video = Vine.Dehalo(video, radius=[2, None])
video = video = core.resize.Bicubic(video, format=original_format.id) # http://forum.doom9.org/showthread.php?p=1780587#post1780587
video = haf.Weave(video, tff=True)
video = mvs.AssumeTFF(video)
video.set_output()
return True
main()
Gives me only 49.5 fps x264 encoding speed, without any other processing like deblocking and qtgmc and sharpening etc :) The "cost of doing business" I suppose.

On the other matter of the previsouly reported crash, [hang on, ignore this while I re-try] if I un-comment the Vine.Dehalo line then vspipe crashes with th Windows dialogue popup box and no python trace, which suggests that maybe Vine.Dehalo could be a tad less than stable from time to time.[/hang on, ignore this while I re-try]
Windows event log says error event 1000 :-
Faulting application name: VSPipe.exe, version: 0.0.0.0, time stamp: 0x57bd7828
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000000
Faulting process ID: 0xb64
Faulting application start time: 0x01d20e7b2fbaa584
Faulting application path: C:\SOFTWARE\Vapoursynth-x64\VSPipe.exe
Faulting module path: unknown
Report ID: a8707734-60ee-4dd5-8b45-35ed7814a9e2
Faulting package full name:
Faulting package-relative application ID:
Scrub that, I can't reproduce the issue. edit: yes I can, see over at KNLmeansCL http://forum.doom9.org/showthread.php?p=1780660#post1780660
The only thing I can say is that other scripts are very reliable, but not this one containing these conversions and vine. When I add the vine line it sits there and takes about 3 to 5 minutes before actual x264 encoding commences by displaying frame number and fps etc ... then it goes at about 0.35 fps.

[feisty2]

Vine is stable, cretindesalpes' build of fmtconv isn't so
And main() is redundant, Python is a scripting language

[hydra3333]

OK agreed, thank you. It just runs a bit glacially for me even with the internal resizers changing the format back and forth. I wonder if it's related in any way to use of KNLmeansCL for me.

The main() approach is used so I can position that statement last in the script, and declare a number of home-grown functions just underneath the "main code at the top".
Python apparently doesn't seem to like calling functions which are declared later in the script.

[Elegant]

Haven't had the chance to setup my encoding box again so I thought I'd ask: Is this appropriate for anime/cartoons? Overkill? I have a few series on DVD that have poor transfers and suffer from halos. Thanks!

[feisty2]

Haven't had the chance to setup my encoding box again so I thought I'd ask: Is this appropriate for anime/cartoons? Overkill? I have a few series on DVD that have poor transfers and suffer from halos. Thanks!

it should work on cartoon stuff also I guess, but probably not worth it, Vine is pretty slow, and other lower quality but levels faster dehalo filters might be good enough for cartoons since getting cartoonized is one of the last things to worry about on literally cartoon vids

[brucethemoose]

it should work on cartoon stuff also I guess, but probably not worth it, Vine is pretty slow, and other lower quality but levels faster dehalo filters might be good enough for cartoons since getting cartoonized is one of the last things to worry about on literally cartoon vids

Some toons have more detailed, painted backgrounds that you have to worry about, but that's still true.

In fact, my problem is that Vine isn't strong enough... I've tried tweaking parameters, but it barely puts a dent in big sharpening halos like this:

http://i.cubeupload.com/YAZVwX.png

[Mystery Keeper]

Some toons have more detailed, painted backgrounds that you have to worry about, but that's still true.

In fact, my problem is that Vine isn't strong enough... I've tried tweaking parameters, but it barely puts a dent in big sharpening halos like this:

http://i.cubeupload.com/YAZVwX.png

You need to learn that miracles don't happen. And be skeptic about halo removing in general. AFAIR, there has been no method to remove them that wasn't destructive to other parts of video. What you've got is a horrible multi-peaks ringing. No hope to fight it. Better to not cause it.

[Nevilne]

"you can't expect this halo remover to remove these halos"

[brucethemoose]

No hope to fight it. Better to not cause it.

Unfortunately, that's the raw NTSC DVD rip. Yeah, it's that bad :(.

And yeah, I know... but there is a compromise somewhere between the Vine settings I've tried and DeHalo_Alpha, I just haven't found it yet.

[Mystery Keeper]

"you can't expect this halo remover to remove these halos"These multi-peaks? No, you can't. Narrow one peak halos - sure, you can. But expect it to screw up something else.

[feisty2]

Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"

[brucethemoose]

Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"

Not yet, thanks.


But holy moly... now I'm running this:

Vine.Dehalo(clip, radius=[8, 4], a=32, h=24, sharp=1.0, sigma=0.6, alpha=0.36, beta=32, cutoff=4, show=False)

And its taking like 8 seconds per 720x480 frame to encode. I'll get back to you when I get results.

[brucethemoose]

Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"

24 is way too strong, this is 20 with radius 4,2 (with a straight spline36 scale for reference):

http://www.screenshotcomparison.com/comparison/185123

Look at the guy's eyes, in particular.

I'm messing with it now, and 12-16 is looking better... the higher order halos aren't completely gone, but I might be able to mask them with another filter now.


This is some great code though, thanks feisty2.

[feisty2]

Now I think a large "h" is not it, something like "h=8" might be enough already
You need to disable all mask stuff in Vine cuz the halos are roaming VERY far from the exact edges, pointless tryna "mask" them out

[feisty2]

I'll add such option(disable mask stuff) after college

[brucethemoose]

Now I think a large "h" is not it, something like "h=8" might be enough already
You need to disable all mask stuff in Vine cuz the halos are roaming VERY far from the exact edges, pointless tryna "mask" them out

Sounds good.

However, if I'm looking at it right, I think the mask is actually reaching WAY out to the halos as it should.

http://u.cubeupload.com/brucethemoose/Disc1Alphavpymask.png

I don't think disabling it would help. There are other scenes with much more intricate painted backgrounds, and it looks like the mask will help preserve those.

EDIT: I could be wrong though, maybe I'm misunderstanding what the mask does...

So for now, what parameters should I adjust for the mask?

[feisty2]

radius=[10, None], sigma=1.6, alpha=0.48 works on your first screenshot
I'm on cellphone right now so...

[brucethemoose]

radius=[10, None], sigma=1.6, alpha=0.48 works on your first screenshot
I'm on cellphone right now so...

Sorry for not responding earlier, I went down the deinterlacing/IVTC rabbit hole.

What h value did you use?

[feisty2]

r5:
new parameter "masking"

[feisty2]

@brucethemoose

clp = Vine.Dehalo(clp, h=6, masking=False, cutoff=1, sharp=0.4)
clp = Vine.Dehalo(clp, h=4, masking=False, cutoff=1, sharp=0.6)
clp = Vine.Dehalo(clp, h=2, masking=False, cutoff=1, sharp=0.8)

http://i.imgur.com/VaQQ8Ud.png

[feisty2]

and of course you can go for a custom mask for this specific vid and protect soft features a bit more

[brucethemoose]

Fantastic, thanks! I'm playing with this now, might upload some before/after screenshots/clips later to show Vine off, as those Avatar DVDs are a DeHalo torture test.

[brucethemoose]

@brucethemoose

clp = Vine.Dehalo(clp, h=6, masking=False, cutoff=1, sharp=0.4)
clp = Vine.Dehalo(clp, h=4, masking=False, cutoff=1, sharp=0.6)
clp = Vine.Dehalo(clp, h=2, masking=False, cutoff=1, sharp=0.8)



Out of curiosity, why call Dehalo 3 times instead of just increasing h? Does it help with higher order halos or something?

[brucethemoose]

In certain material, Vine has trouble correctly detecting the actual edges:

Look at the edges of the cards here:

http://www.screenshotcomparison.com/comparison/185510

In that crack between the 2 black lines, it's so bad that the halos themselves are detected as edges, and it ends up blurring the actual edges. The mask and strength settings don't seem to affect that.



Feisty2, I know this is a live-action filter, but could you add an option that gives priority to black lines during edge detection? This would help oversharpened animation dramatically.

[feisty2]

there's no edge detection with "masking=False"
you do a halo/edge mask for your specific video on your own

[Tormaid]

there's no edge detection with "masking=False"
you do a halo/edge mask for your specific video on your own
^This

Only sane way to do it.

[brucethemoose]

^This

Only sane way to do it.

Well it's not a mask problem really. That's an area I definitely need to dehalo (and debug the mask is showing it), but the dehalo process itself isn't working.


I can't automatically filter it out and just live with the halos either. It's an edge just like everything else the mask sees, AFAIK there's no way to get a mask to detect edges, but NOT detect edges where there's a white halo in the inner perimeter of an object with a black outline.

EDIT: MadVR's anti ringing seems to have the same idea:

https://i.imgur.com/LtIULrD.png

It has a seperate setting for working with light edges.

[feisty2]

@brucethemoose

clp0 = clp
mask = Vine.Dehalo(clp,sigma=0.001,radius=[0,0],alpha=0.76,h=12,show=True)
for i in range(6):
mask = core.std.Maximum(mask)
clp = Vine.Dehalo(clp,h=4.8,masking=False)
clp = Vine.Dehalo(clp,h=2,masking=False,cutoff=1)
clp = Vine.Dehalo(clp,h=2,masking=False,cutoff=1)
clp = Vine.Dehalo(clp,h=2,masking=False,cutoff=1)
clp = core.std.MaskedMerge(clp0,clp,mask)

http://i.imgur.com/Xk8l8d1.png

[feisty2]

I don't really get "edge detection" and whatever other stuff you were talking about
but I got my own life so please stop expecting to be spoon fed

[brucethemoose]

I know, I wasn't asking for an update after your other reply, just speculating. I don't fully understand how DeHalo itself works, which means I should look at the code itself.

That looks much better, thanks. The range mask is a good idea.

[JackCY]

readme and more
https://github.com/IFeelBloated/Vine

Vine does not turn your video to a piece of cartoonized shit like other dehalo filters do.
it's slow but the performance is kind of practical still.

I've tried this script and... well it works, kind of... the parameters have to be pushed out the wazoo so thick halos are detected, such as example given on GitHub. But, the performance is awful, outright awful. Even with KNLMeans set to a=1 via Dehalo(a=1). Even with KNLMeans commented out in the Vine.py code so there should be no other crazy expensive operation... well... there is and I don't have another day to find which it is but it certainly is CPU limited not GPU limited in OpenCL.

No matter what Vine runs < 1fps on 1920x816px clip. Around 0.60fps, maybe.

Do I have some bad conversion copied from this thread to get that format Vine wants or is Vine so slow it's practically impractical? < 1fps certainly isn't practical, I've expected something like 10ms delay from processing per full HD frame not 1600ms.

At this speed the script is only good for processing a few still frames, that's about it.

What speeds is OP/creator getting when processing full HD?
From what I see the RAM keeps going +3GB what I would say is per frame, so each frame RAM consumption goes up by 3GB and then drops back down. WTH is eating 3GB.

The moment I comment out Vine out of my script... 80fps. Hell I can do 300fps, doesn't help anyway.

If this script is running 1000ms per frame, you should really put it in the OP or readme of the script as this is not usable for videos one bit in current state of the script and hardware available.

Are there any other dehalo VapourSynth options? Ones that do run fast for video processing? Don't care what visual quality as long as it's fast and usable.

[Mystery Keeper]

If you don't care about visual quality - you should just go with the halos.

[JackCY]

If you don't care about visual quality - you should just go with the halos.

I do but not to the point of 1+s processing per frame. MadVR has no problem detecting halos at very fast speed, any similar algorithm in VapourSynth that can be used to process videos before encoding and adjust the strength?

[feisty2]

I do but not to the point of 1+s processing per frame. MadVR has no problem detecting halos at very fast speed, any similar algorithm in VapourSynth that can be used to process videos before encoding and adjust the strength?

Madvr works on gpu
nnedi3 works on cpu

[JackCY]

Madvr works on gpu
nnedi3 works on cpu
nnedi3 shouldn't run on CPU, none image processing plugins should. It's insane to run that alg. on CPU.

Output 2000 frames in 88.44 seconds (22.62 fps)
# for i in range(2):
# clean = Transpose(NNEDI(Transpose(NNEDI(clean, **nnedi_args)), **nnedi_args))
# clean = Resample(clean, src.width, src.height, sx=-1.25, sy=-1.25, kernel="cubic", a1=-sharp, a2=0)
vs
Output 20 frames in 47.16 seconds (0.42 fps)
for i in range(2):
clean = Transpose(NNEDI(Transpose(NNEDI(clean, **nnedi_args)), **nnedi_args))
clean = Resample(clean, src.width, src.height, sx=-1.25, sy=-1.25, kernel="cubic", a1=-sharp, a2=0)

Difference? Usable vs unusable, noisier edges (close to original, only slightly cleaned) vs cleaner edges (even more cleaner than original). The edges might be changed by doing some other cheaper operation, such as convolution blur to reduce sharpness or the opposite since the sharpness argument is now defunct, no problem to emulate 0.5 sharpness which helped to remove more of the halos.

Adding convolution blur changed the performance to...
Output 2000 frames in 89.86 seconds (22.26 fps)
Probably a margin of error. But say -0.50fps, 0.984x.

Commenting out 3 lines = 53.85x faster.
This is with "a=3" which is enough for KNLMeansCL. Radius 3.

The next one to go seems to be fmtc:
Resample = core.fmtc.resample
MakeDiff = core.std.MakeDiff
MergeDiff = core.std.MergeDiff
def inline(src):
upsmp = Resample(src, src.width * 2, src.height * 2, kernel="gauss", a1=100, **fmtc_args)
clip = Resample(upsmp, src.width, src.height, kernel="gauss", a1=p, **fmtc_args)
return clip
If there is a VS native replacement that's faster.

original cutoff
Output 2000 frames in 90.55 seconds (22.09 fps)
Output 1000 frames in 46.26 seconds (21.62 fps)
convolution 5x5
Output 2000 frames in 85.34 seconds (23.43 fps)
Output 1000 frames in 43.60 seconds (22.93 fps)
convolution 3x3
Output 2000 frames in 74.06 seconds (27.01 fps)
Output 1000 frames in 37.46 seconds (26.69 fps)

Convolution 5x5 not really worth it, looks similar too. 3x3 limits the dehalo ability but it's faster by 5fps = 1.23x, not bad at all if one doesn't mind the reduced halo reduction.

Canny, no difference.
Can't get rid of the necessary format conversion, dunno why format changes somewhere and then causes an error, what operation is changing the format in Vine?
This is what was failing without converting to 32bit float:
clean = helpers.cutoff(src, clean, cutoff)
dif = MakeDiff(src, clean)
Right between the nlmeans in dehalo.

---

In the end this script can run not 0.4fps but 27fps, 67x faster and usable, though still slow from my POV.