Vine, cutting edge quality de-halo filter and morphology stuff

[feisty2]

readme and more
https://github.com/IFeelBloated/Vine

Vine does not turn your video to a piece of cartoonized shit like other dehalo filters do.
it's slow but the performance is kind of practical still.

[feisty2]

r2:
replaced hacky floatfilters with the ones in core functions
now requires vaporsynth r33 test4 or later

[feisty2]

r3:
1. strict sanity check in every single function, a lot more user friendly and should be ok to work like any typical module now
2. trivial modifications to the mask
3. removed the morphology class and exposed all morphological functions with Vine.Dehalo
4. re-licensed to LGPL v3.0
5. trivial modifications to the doc (color images instead of gray stuff, etc..)
6. gitignore

[hydra3333]

Trying out Vine and Plum but receiving an error message I'm not sure what to do with:

"C:\SOFTWARE\Vapoursynth-x64\VSPipe.exe" "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.08.24.14.00.23.61.vpy" - --y4m   | "C:\SOFTWARE\ffmpeg\0-homebuilt-x64\x264-mp4.exe" - --stdin y4m --thread-input --frames 45810 --profile high --level 4.1 --preset slow --interlaced --tff --no-cabac --crf 16 --sar 64:45 --colormatrix bt470bg -o "s:\HDTV\WDTVlive\MP4-VS\x64-1\test-temp.2016.08.24.14.00.23.61.h264"
Script evaluation failed:
Python exception: Vine.Dehalo: the sample type of src has to be single precision!
Traceback (most recent call last):
  File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26905)
  File "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.08.24.14.00.23.61.vpy", line 487, in <module>
    main()
  File "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.08.24.14.00.23.61.vpy", line 141, in main
    video = Vine.Dehalo(video, radius=[2, None])
  File "C:\SOFTWARE\Vapoursynth-x64\Vine.py", line 113, in Dehalo
    raise TypeError("Vine.Dehalo: the sample type of src has to be single precision!")
TypeError: Vine.Dehalo: the sample type of src has to be single precision!

Script:

import vapoursynth as vs 
import havsfuncTS as haf # this version uses vanilla TemporalSoften instead of TemporalSoften2, as it will be "better" over time 
import mvsfunc as mvs  # http://forum.doom9.org/showthread.php?t=172564 
import finesharp as finesharp # http://forum.doom9.org/showthread.php?p=1777815#post1777815 http://avisynth.nl/index.php/FineSharp 
import Plum # http://forum.doom9.org/showthread.php?t=173775 https://github.com/IFeelBloated/Plum 
import Vine # http://forum.doom9.org/showthread.php?t=173703 https://github.com/IFeelBloated/Vine 
core = vs.get_core(accept_lowercase=True) # leave off threads=8 so it auto-detects threads 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\AddGrain.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\BM3D.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\d2vsource.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\Deblock.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\DFTTest.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\fmtconv.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\KNLMeansCL.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libawarpsharp2.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libmvtools.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libmvtools_sf_emt64.dll') # for Plum, the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\libnnedi3.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\scenechange.dll') # http://forum.doom9.org/showthread.php?t=166769 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\TCanny_AVX.dll') # http://forum.doom9.org/showthread.php?t=166769 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\vcfreq.dll') # http://forum.doom9.org/showthread.php?t=171413 http://www.avisynth.nl/users/vcmohan/index.html ... for Plum 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\vsdctfilter.dll')  # http://vfrmaniac.fushizen.eu/works 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\vsfft3dfilter.dll')  # http://vfrmaniac.fushizen.eu/works 
core.std.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\vapoursynth64\plugins\dll-to-choose-from\Yadifmod.dll') # the r'' indicates do not treat special characters and accept backslashes 
core.avs.LoadPlugin(r'C:\SOFTWARE\Vapoursynth-x64\DGIndex\DGDecodeNV.dll')  
# NOTE:: INDENTING IS CRITICAL AND MUST BE PRECISELY THE SAME WITHIN EACH FUNCTION  
# (functions removed in this example)
def main(): 
    video = core.d2v.Source(r'T:\HDTV\WDTVlive\MP4-VS\x64-1\test.d2v') 
    video = core.std.SeparateFields(video, tff=True) 
    video = haf.Deblock_QED(video) 
    video = haf.Weave(video, tff=True)  
    video = mvs.AssumeTFF(video) 
    video = haf.QTGMC(video, TFF=True, Preset="Very Slow", Sharpness=1.2, SLMode=2, EZKeepGrain=1.2, NoiseProcess=2)     # result is double framerate progressive, so re-interlate it later 
    video = Vine.Dehalo(video, radius=[2, None]) 
    deconv = Plum.Basic(video) 
    conv = Plum.Basic(video,mode="convolution") 
    sup = Plum.Super([video, None]) 
    supdeconv = Plum.Super([video, deconv]) 
    supconv = Plum.Super([conv, None]) 
    video = Plum.Final([video, deconv, conv], [sup, supdeconv, supconv], strength=2.1, cutoff=16) # middlin 
    video = core.std.SeparateFields(video, tff=True) 
    video = core.std.SelectEvery(video, cycle=4, offsets=[0, 3])     # looks like avisynth  
    video = haf.Weave(video, tff=True)  
    video = mvs.AssumeTFF(video) 
    video.set_output() 
    return True 
main()

Any and all suggestions welcomed.
... also about "best" positioning of Vine for interlaced material (eg after QTGMC or within the prior separatefields/deblock)

[hydra3333]

Using "video = core.text.ClipInfo(video)" shows that the video sample type incoming into Vine = Integer.
How do I turn it into single precision for use in Vine, and back again for use in other functions ? Would there be quality losses associated with the conversions ?

[feisty2]

video = haf.QTGMC(video, TFF=True, Preset="Very Slow", Sharpness=1.2, SLMode=2, EZKeepGrain=1.2, NoiseProcess=2) # result is double framerate progressive, so re-interlate it later

video = core.fmtc.bitdepth(video,bits=32,fulls=False,fulld=True)

video = Vine.Dehalo(video, radius=[2, None])
deconv = Plum.Basic(video)
conv = Plum.Basic(video,mode="convolution")
sup = Plum.Super([video, None])
supdeconv = Plum.Super([video, deconv])
supconv = Plum.Super([conv, None])
video = Plum.Final([video, deconv, conv], [sup, supdeconv, supconv], strength=2.1, cutoff=16) # middlin
video = core.std.SeparateFields(video, tff=True)

[Mystery Keeper]

Also, here's a snippet for plugins autoloading. It also adds a path from which other scripts can be imported easily.

from __future__ import print_function

import vapoursynth as vs
core = vs.get_core(threads=8)
core.set_max_cache_size(12000)

import sys
sys.path.append('D:\\vapoursynth-plugins\\py\\')
import platform
architecture = platform.architecture()
if architecture[0] == '64bit':
	vapoursynth_plugins_path = 'D:\\vapoursynth-plugins\\64bit\\'
else:
	vapoursynth_plugins_path = 'D:\\vapoursynth-plugins\\32bit\\'
print('Plugins folder: ', vapoursynth_plugins_path, end='\n', file=sys.stderr)
import os
for filename in os.listdir(vapoursynth_plugins_path):
	if filename[-4:] != '.dll':
		continue
	try:
		core.std.LoadPlugin(vapoursynth_plugins_path + filename)
	except Exception as e:
		print('Error: ', e, end='\n', file=sys.stderr)

[hydra3333]

Thank you.

[feisty2]

r4:
algorithmic upgrade, better quality and a bit slower

[hydra3333]

Hello. With the latest versions of VS plugins and VS R33.1, the sdcript below either
(i) causes vspipe to crash straight away (the windows dialogue popup box type of crash, without any python traceback) or
(ii) the process encodes at 0.35 fps

What am I doing wrong ? ... suggestions very much welcomed.

Input is PAL 576i mpeg2.

"C:\SOFTWARE\Vapoursynth-x64\VSPipe.exe" "T:\HDTV\WDTVlive\MP4-VS\x64-1\test.mpg.2016.09.13.19.17.25.42.vpy" - --y4m   | "C:\SOFTWARE\ffmpeg\0-homebuilt-x64\x264-mp4.exe" - --stdin y4m --thread-input --profile high --level 4.1 --preset slow --interlaced --tff --no-cabac --crf 16 --sar 64:45 --colormatrix bt470bg -o "s:\HDTV\WDTVlive\MP4-VS\x64-1\test-temp.2016.09.13.19.17.25.42.h264"

import vapoursynth as vs 
import havsfuncTS as haf # this version uses vanilla TemporalSoften instead of TemporalSoften2, as it will be "better" over time 
import mvsfunc as mvs  # http://forum.doom9.org/showthread.php?t=172564 
import finesharp as finesharp # http://forum.doom9.org/showthread.php?p=1777815#post1777815 http://avisynth.nl/index.php/FineSharp 
import Plum # http://forum.doom9.org/showthread.php?t=173775 https://github.com/IFeelBloated/Plum 
import Vine # http://forum.doom9.org/showthread.php?t=173703 https://github.com/IFeelBloated/Vine 
import Oyster # http://forum.doom9.org/showthread.php?t=173470 https://github.com/IFeelBloated/Oyster 
import mvmulti  # http://forum.doom9.org/showthread.php?p=1777891#post1777891 
core = vs.get_core(accept_lowercase=True) # leave off threads=8 so it auto-detects threads 
# snip - loaded plugins and stuff
def main(): 
    video = core.d2v.Source(r'T:\HDTV\WDTVlive\MP4-VS\x64-1\test.d2v') 
    video = core.std.Trim(video,first=0,length=1000) 
    video = core.std.SeparateFields(video, tff=True) 
    # Vine only accepts 32bit float input 
    video = core.fmtc.bitdepth(video,bits=32,fulls=False,fulld=True) #convert to 32bit float. Specify if the clip is full-range (True) or TV-range (False). fulls is for input, fulld for output.  
    # vine middlin 
    video = Vine.Dehalo(video, radius=[2, None]) 
    video = core.fmtc.bitdepth(video,bits=8,fulls=True,fulld=False) #convert to 8bit Integer. Specify if the clip is full-range (True) or TV-range (False). fulls is for input, fulld for output.  
    video = haf.Weave(video, tff=True)  
    video = mvs.AssumeTFF(video) 
    video.set_output() 
    return True 
main()

x64 vapoursynth, PC is an i3820 4-core, with 16Gb mem, Nvidia 750Ti with latest nvidia driver.
Most of the memory appears to be free during the encode at least for the first 10 mins, 8 hyperthrerads were running at about 20% cpu each.
Other non-Vine encodes run ok.

[feisty2]

try Myrsloik's build of fmtconv, https://dl.dropboxusercontent.com/u/...ebugsymbols.7z
it ain't crashed since the day I got it....
cretindesalpes' build always crashes randomly like probably at any moment for some mysterious reason(s)...

[Myrsloik]

Originally Posted by feisty2

try Myrsloik's build of fmtconv, https://dl.dropboxusercontent.com/u/...ebugsymbols.7z
it ain't crashed since the day I got it....
cretindesalpes' build always crashes randomly like probably at any moment for some mysterious reason(s)...

You could also try your luck with the internal resizer, that should work just as well for this. But if you do find a crash try to figure out where it happens so it can be fied.

[hydra3333]

Originally Posted by Myrsloik

You could also try your luck with the internal resizer, that should work just as well for this.

If I interpret that correctly ...
http://www.vapoursynth.com/doc/functions/resize.html
OK, not sure how to use the resizer to convert to 32bit float and then back to 8bit ... advice welcome.

Originally Posted by feisty2

try Myrsloik's build of fmtconv, https://dl.dropboxusercontent.com/u/...ebugsymbols.7z
it ain't crashed since the day I got it....

Thank you. Although, I admit to preferring not using software from dropboxes ... how quaint of me.

[jackoneill]

To convert to 32 bit float and back:

original_format = video.format

float_format = core.register_format(original_format.color_family, vs.FLOAT, 32, original_format.subsampling_w, original_format.subsampling_h)

video = core.resize.Bicubic(video, format=float_format.id)

...

video = core.resize.Bicubic(video, format=original_format.id)

[TheFluff]

I gotta say it's pretty cool that the resizer actually supports resizing to a format programmatically registered at runtime

I mean, it's not a scenario that's bound to come up often but when it does it's pretty awesome to have that functionality

[hydra3333]

Yes cool.

core = vs.get_core(accept_lowercase=True) # leave off threads=8 so it auto-detects threads 
def main(): 
    video = core.d2v.Source(r'T:\HDTV\WDTVlive\MP4-VS\x64-1\test.d2v') 
    video = core.std.Trim(video,first=0,length=1000) 
    video = core.std.SeparateFields(video, tff=True) 
    original_format = video.format # http://forum.doom9.org/showthread.ph...87#post1780587 
    float_format = core.register_format(original_format.color_family, vs.FLOAT, 32, original_format.subsampling_w, original_format.subsampling_h) # http://forum.doom9.org/showthread.ph...87#post1780587 
    video = core.resize.Bicubic(video, format=float_format.id) # http://forum.doom9.org/showthread.ph...87#post1780587 
    #video = Vine.Dehalo(video, radius=[2, None]) 
    video = video = core.resize.Bicubic(video, format=original_format.id) # http://forum.doom9.org/showthread.ph...87#post1780587 
    video = haf.Weave(video, tff=True)  
    video = mvs.AssumeTFF(video) 
    video.set_output() 
    return True 
main()

Gives me only 49.5 fps x264 encoding speed, without any other processing like deblocking and qtgmc and sharpening etc The "cost of doing business" I suppose.

On the other matter of the previsouly reported crash, [hang on, ignore this while I re-try] if I un-comment the Vine.Dehalo line then vspipe crashes with th Windows dialogue popup box and no python trace, which suggests that maybe Vine.Dehalo could be a tad less than stable from time to time.[/hang on, ignore this while I re-try]
Windows event log says error event 1000 :-

Faulting application name: VSPipe.exe, version: 0.0.0.0, time stamp: 0x57bd7828
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000000
Faulting process ID: 0xb64
Faulting application start time: 0x01d20e7b2fbaa584
Faulting application path: C:\SOFTWARE\Vapoursynth-x64\VSPipe.exe
Faulting module path: unknown
Report ID: a8707734-60ee-4dd5-8b45-35ed7814a9e2
Faulting package full name:
Faulting package-relative application ID:

Scrub that, I can't reproduce the issue. edit: yes I can, see over at KNLmeansCL http://forum.doom9.org/showthread.ph...60#post1780660
The only thing I can say is that other scripts are very reliable, but not this one containing these conversions and vine. When I add the vine line it sits there and takes about 3 to 5 minutes before actual x264 encoding commences by displaying frame number and fps etc ... then it goes at about 0.35 fps.

[feisty2]

Vine is stable, cretindesalpes' build of fmtconv isn't so
And main() is redundant, Python is a scripting language

[hydra3333]

OK agreed, thank you. It just runs a bit glacially for me even with the internal resizers changing the format back and forth. I wonder if it's related in any way to use of KNLmeansCL for me.

The main() approach is used so I can position that statement last in the script, and declare a number of home-grown functions just underneath the "main code at the top".
Python apparently doesn't seem to like calling functions which are declared later in the script.

[Elegant]

Haven't had the chance to setup my encoding box again so I thought I'd ask: Is this appropriate for anime/cartoons? Overkill? I have a few series on DVD that have poor transfers and suffer from halos. Thanks!

[feisty2]

Originally Posted by Elegant

Haven't had the chance to setup my encoding box again so I thought I'd ask: Is this appropriate for anime/cartoons? Overkill? I have a few series on DVD that have poor transfers and suffer from halos. Thanks!

it should work on cartoon stuff also I guess, but probably not worth it, Vine is pretty slow, and other lower quality but levels faster dehalo filters might be good enough for cartoons since getting cartoonized is one of the last things to worry about on literally cartoon vids