HAvsFunc

[HolyWu]

I'm on Netrunner 14 x64 (Ubuntu 14.04 derivative) and I use Python 3.4.0 and VapourSynth 27.

I use MPV, I also tried with Vspipe, and I only have this error with LSFMod : Segfault

I fear I can't help you with so few information. Can your script output successfully with only

clip = core.std.MakeDiff(clip, core.rgvs.RemoveGrain(clip, 11))

or

clip = core.generic.Minimum(clip)

?

[jeremy33]

This works :

clip = core.std.MakeDiff(clip, core.rgvs.RemoveGrain(clip, 11))


But this give me a segfault :

clip = core.generic.Minimum(clip)


If you need more infos ask me what to try ;)

[djcj]

What license is HAvsFunc?
No license, just like most of the AVS scripts.
No license = public domain?
Sorry for that question but I've added it to a collection of VS plugins and scripts and I just want some clarity on the copyright.

[captainadamo]

No license = public domain?

No. I don't think any such jurisdiction has such a policy when it comes to copyright. No license = all rights reserved. But in this case, HolyWu is clearly allowing everyone to use it and do what they want with it.

Sorry for that question but I've added it to a collection of VS plugins and scripts and I just want some clarity on the copyright.

HolyWu is putting it out there for all to freely use like all other scripts shared on here. That was their point.

[HolyWu]

But this give me a segfault :

clip = core.generic.Minimum(clip)


If you need more infos ask me what to try ;)

Was your GenericFilters compiled from https://github.com/myrsloik/GenericFilters? Anyway, you can just replace all core.generic. with core.std. in the script, except core.generic.Blur and core.generic.Hysteresis.

No license = public domain?
Sorry for that question but I've added it to a collection of VS plugins and scripts and I just want some clarity on the copyright.

captainadamo has answered for me nicely.

[jeremy33]

Was your GenericFilters compiled from https://github.com/myrsloik/GenericFilters? Anyway, you can just replace all core.generic. with core.std. in the script, except core.generic.Blur and core.generic.Hysteresis
I don't know, I use this PPA (https://launchpad.net/~djcj/+archive/ubuntu/vapoursynth) that djcj have created. By chance djcj is here to answer ;)
By the way djcj, it's with me that you talk about nnedi3 and other things by mail ;)

[foxyshadis]

It'd still be a good idea to at least add CC0 (public domain) or CC-BY (which just says your name has to stay attached somehow) to the top. License questions have caused some quarrels in the past.

[djcj]

This is the collection I was talking about: https://github.com/darealshinji/vapoursynth-plugins
I've created it because I didn't want to add like 40+ Ubuntu packages to the PPA.

Like foxyshadis pointed out, mentioning a license is still a good idea. Other examples for short and liberal licenses would be: MIT, X11, WTFPL, BSD (2-clause), Expat, ISC or zlib. You can also release it under multiple licenses.

[jeremy33]

Was your GenericFilters compiled from https://github.com/myrsloik/GenericFilters?
What is recommended, use GenericFilters from myrsloik or chikuzen ? In the djcj ppa it's the one from chikuzen.

I have tested a new build from myrsloik and chikuzen and they now work with
clip = core.generic.Minimum(clip)

But I still have segfault with LSFmod. Do you have an idea of what the problem can be ?

[captainadamo]

This is the collection I was talking about: https://github.com/darealshinji/vapoursynth-plugins
I've created it because I didn't want to add like 40+ Ubuntu packages to the PPA.

Like foxyshadis pointed out, mentioning a license is still a good idea. Other examples for short and liberal licenses would be: MIT, X11, WTFPL, BSD (2-clause), Expat, ISC or zlib. You can also release it under multiple licenses.

Sure, but HolyWu has clearly affirmed that you are allowed to use it, modify it and do whatever you want with it. Plus, considering many of these are ported scripts that other people wrote previously, it's likely HolyWu can't just slap their own license on what is a derivative of someone else's work. Ultimately, you're likely to not get any such clear licensing since most scripts are thrown up here with no license.

[jackoneill]

What is recommended, use GenericFilters from myrsloik or chikuzen ? In the djcj ppa it's the one from chikuzen.

I have tested a new build from myrsloik and chikuzen and they now work with
clip = core.generic.Minimum(clip)

But I still have segfault with LSFmod. Do you have an idea of what the problem can be ?

Chikuzen's repository has several bugs that were fixed in Myrsloik's fork. If the filter you want to use exists in the "std" namespace, use that. Otherwise, use Myrsloik's fork. At this point, there is no reason to use Chikuzen's repository, and several reasons to avoid it.

[jeremy33]

Ok thank you jackoneill ;)

I still don't understand why I have a segfault when I try to use LSFmod even with GenericFilters from myrsloik.

[jackoneill]

Ok thank you jackoneill ;)

I still don't understand why I have a segfault when I try to use LSFmod even with GenericFilters from myrsloik.

Post the script, please.

[jeremy33]

This is my test script :

import vapoursynth as vs
import havsfunc as haf
core = vs.get_core()
clip = video_in
clip = haf.LSFmod(clip, defaults="slow")
clip.set_output()

[jackoneill]

This is my test script :

import vapoursynth as vs
import havsfunc as haf
core = vs.get_core()
clip = video_in
clip = haf.LSFmod(clip, defaults="slow")
clip.set_output()

Does it still crash, now that you built fmtconv yourself?

[jeremy33]

Yes it still crash

[jackoneill]

Yes it still crash

Run it in gdb to see at least what plugin is responsible:

gdb --args mpv file.mov etc


Then type "run" and when it crashes, type "bt". Paste the output from that.

[Myrsloik]

You may also want to compile vapoursynth with --enable-guard-pattern which may be able to detect when the corruption happens.

[jeremy33]

This is the output of gdb :

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff940a2700 (LWP 22339)]
0x00007fff967dcac2 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libgenericfilters.so
(gdb) bt
#0 0x00007fff967dcac2 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libgenericfilters.so
#1 0x00007fff967e7699 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libgenericfilters.so
#2 0x00007ffff18c2edd in VSNode::getFrameInternal(int, int, VSFrameContext&) ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#3 0x00007ffff18d1607 in VSThreadPool::runTasks(VSThreadPool*, std::atomic<bool>&) ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#4 0x00007fffee1b9bf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff5d4c182 in start_thread (arg=0x7fff940a2700) at pthread_create.c:312
#6 0x00007fffee70c47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[HolyWu]

Update r17.


Use the equivalent functions in the std namespace instead of GenericFilters', except Hysteresis.

[jackoneill]

This is the output of gdb :

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff940a2700 (LWP 22339)]
0x00007fff967dcac2 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libgenericfilters.so
(gdb) bt
#0 0x00007fff967dcac2 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libgenericfilters.so
#1 0x00007fff967e7699 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libgenericfilters.so
#2 0x00007ffff18c2edd in VSNode::getFrameInternal(int, int, VSFrameContext&) ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#3 0x00007ffff18d1607 in VSThreadPool::runTasks(VSThreadPool*, std::atomic<bool>&) ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#4 0x00007fffee1b9bf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff5d4c182 in start_thread (arg=0x7fff940a2700) at pthread_create.c:312
#6 0x00007fffee70c47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

It's GenericFilters... I think HAvsFunc r17 will be better.

[jeremy33]

It's not better :(

*** Error in `/usr/bin/mpv': double free or corruption (out): 0x00007fff88003060 ***

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff8ffff700 (LWP 26886)]
0x00007fffee648cc9 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: Aucun fichier ou dossier de ce type.
(gdb) bt
#0 0x00007fffee648cc9 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fffee64c0d8 in __GI_abort () at abort.c:89
#2 0x00007fffee685394 in __libc_message (do_abort=do_abort@entry=1,
fmt=fmt@entry=0x7fffee793b28 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007fffee69166e in malloc_printerr (ptr=<optimized out>,
str=0x7fffee793c58 "double free or corruption (out)", action=1) at malloc.c:4996
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x00007ffff18c1eae in VSVariant::~VSVariant() ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#6 0x00007ffff18be519 in ?? () from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#7 0x00007fff983a3fd7 in ?? () from /usr/lib/x86_64-linux-gnu/vapoursynth/libscenechange.so
#8 0x00007ffff18c2edd in VSNode::getFrameInternal(int, int, VSFrameContext&) ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#9 0x00007ffff18d1607 in VSThreadPool::runTasks(VSThreadPool*, std::atomic<bool>&) ()
from /usr/lib/x86_64-linux-gnu/libvapoursynth.so
#10 0x00007fffee1b9bf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007ffff5d4c182 in start_thread (arg=0x7fff8ffff700) at pthread_create.c:312
#12 0x00007fffee70c47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


Edit 1: Maybe that can help if I use LSFmod with "defaults=old" it works (clip = haf.LSFmod(clip, defaults="old"))

Edit 2: I find that if I replace diff2 = TemporalSoften(diff, 1, 255<<shift, 0, 32<<shift, 2) by diff2 = diff it works so the problem can be here
def TemporalSoften(clip, radius=4, luma_threshold=4, chroma_threshold=8, scenechange=15, mode=2):
core = vs.get_core()

if scenechange:
clip = set_scenechange(clip, scenechange)

return core.focus2.TemporalSoften2(clip, radius, luma_threshold, chroma_threshold, scenechange)

Edit 3: I found the bug, the call TemporalSoften(diff, 1, 255<<shift, 0, 32<<shift, 2) have to be TemporalSoften(diff, 1, 255<<shift, 32<<shift, 0, 2)
Parameters chroma_threshold and scenechange are swapped!

[jackoneill]

Edit 1: Maybe that can help if I use LSFmod with "defaults=old" it works (clip = haf.LSFmod(clip, defaults="old"))

Edit 2: I find that if I replace diff2 = TemporalSoften(diff, 1, 255<<shift, 0, 32<<shift, 2) by diff2 = diff it works so the problem can be here
def TemporalSoften(clip, radius=4, luma_threshold=4, chroma_threshold=8, scenechange=15, mode=2):
core = vs.get_core()

if scenechange:
clip = set_scenechange(clip, scenechange)

return core.focus2.TemporalSoften2(clip, radius, luma_threshold, chroma_threshold, scenechange)

Edit 3: I found the bug, the call TemporalSoften(diff, 1, 255<<shift, 0, 32<<shift, 2) have to be TemporalSoften(diff, 1, 255<<shift, 32<<shift, 0, 2)
Parameters chroma_threshold and scenechange are swapped!

That does look like a bug, but I'm not convinced it's the source of your crashes. Neither scd.Detect nor focus2.TemporalSoften2 do anything stupid when those two parameters are swapped.

Please switch them back, confirm that it crashes again, then run it in valgrind and upload the resulting text file somewhere (maybe compressed):

valgrind --log-file=lsfmod-crash-valgrind.txt mpv file.mov etc

[jeremy33]

Ok I just did it. That's weird (but maybe it's normal I don't know valgrind) because with valgrind mpv start playing the video.

http://www.mediafire.com/view/8ddnzxua3400fbz/lsfmod-crash-valgrind.txt

[jackoneill]

Ok I just did it. That's weird (but maybe it's normal I don't know valgrind) because with valgrind mpv start playing the video.

http://www.mediafire.com/view/8ddnzxua3400fbz/lsfmod-crash-valgrind.txt

It doesn't find anything. This is inconvenient. I guess the last thing you can try is compile VapourSynth with --enable-guard-pattern and see if that uncovers anything.

[HolyWu]

Edit 3: I found the bug, the call TemporalSoften(diff, 1, 255<<shift, 0, 32<<shift, 2) have to be TemporalSoften(diff, 1, 255<<shift, 32<<shift, 0, 2)
Parameters chroma_threshold and scenechange are swapped!

Um...no, they are not swapped by accident. It does intend to be used like that in the original version.

[jeremy33]

Too bad :(
I will try to compile VapourSynth with --enable-guard-pattern

[jeremy33]

Ok I built VapourSynth with --enable-guard-pattern. How I use it ?

[Myrsloik]

Ok I built VapourSynth with --enable-guard-pattern. How I use it ?

It's always active so run the same things again and see what error you get.

[jeremy33]

How can I be sure that I built vapoursynth right with --enable-guard-pattern ?
Because it seems that I don't have any new info :

*** Error in `mpv': malloc(): memory corruption (fast): 0x00007fb57800df20 ***
Abandon

Erreur de segmentation (Segfault)

*** Error in `Erreur de segmentation (Segfault)

*** Error in `mpv': double free or corruption (fasttop): 0x00007f47ce291080 ***
Abandon

This is the errors I get with the same script and the same video lanched 6 times

[Myrsloik]

How can I be sure that I built vapoursynth right with --enable-guard-pattern ?
Because it seems that I don't have any new info :

*** Error in `mpv': malloc(): memory corruption (fast): 0x00007fb57800df20 ***
Abandon

Erreur de segmentation (Segfault)

*** Error in `Erreur de segmentation (Segfault)

*** Error in `mpv': double free or corruption (fasttop): 0x00007f47ce291080 ***
Abandon

This is the errors I get with the same script and the same video lanched 6 times

You don't know unless you trigger that specific kind of error. So I guess it didn't.

[jeremy33]

OK.
This is here that the problem occur I think :
def TemporalSoften(clip, radius=4, luma_threshold=4, chroma_threshold=8, scenechange=15, mode=2):
core = vs.get_core()
if scenechange:
clip = set_scenechange(clip, scenechange)
return core.focus2.TemporalSoften2(clip, radius, luma_threshold, chroma_threshold, scenechange)
If I set scenechange to 0 it works

If I comment this
if scenechange:
clip = set_scenechange(clip, scenechange)
I get this error :
Property read unsuccessful but no error output: _SceneChangePrev
Abandon

If I comment this line (and add return clip) I still have segfault
return core.focus2.TemporalSoften2(clip, radius, luma_threshold, chroma_threshold, scenechange)


It seems to be this line that trigger the segfault
if scenechange:
clip = set_scenechange(clip, scenechange)

Can it help ?

[HolyWu]

Try clip = core.scd.Detect(clip, 32) and see what happens. Have you built SceneChange yourself as well?

[jeremy33]

I have also a segfault with clip = core.scd.Detect(clip, 32)

I don't think I built SceneChange, I don't even know what it is.

[jeremy33]

Ok I think I make it works. Thank you everybody :)
I build SceneChange myself from http://forum.doom9.org/showthread.php?t=166769 and it works!

Is it the right SceneChange version ?

[jackoneill]

Ok I think I make it works. Thank you everybody :)
I build SceneChange myself from http://forum.doom9.org/showthread.php?t=166769 and it works!

Is it the right SceneChange version ?

That's the one. I'm beginning to think the binaries in that PPA of yours are somehow broken...

[jeremy33]

Yes there is some problem with the PPA but I'm in touch with the owner, djcj, to improve it.
This PPA is the only easy way to install VapourSynth and the needed plugins on Ubuntu and djcj is kind enough to maintain it so I'm glad to help to make it works.

[Tarutaru]

I got 3.3fps when converting a 1080i Blu-ray to FFV1.
I compiled ffmpeg, all QTGMC plugins with "-march=native -O3"
Is it normal?

My script -
import vapoursynth as vs
import havsfunc as haf
core = vs.get_core()
ret = core.ffms2.Source(source='foo', fpsnum=30000, fpsden=1001)
ret = haf.QTGMC(ret, Preset='Slow', TFF=True)
ret = ret[::2]
ret.set_output()

Command -
vspipe --y4m $input.vpy - | ffmpeg -y -f yuv4mpegpipe -i - -vcodec ffv1 -level 3 -threads 8 -slices 24 -an $output

My computer -
CPU - Intel Xeon E3-1230 v3
RAM - 20GB DDR3
OS - Linuxmint 17.1

[HolyWu]

I think it's normal with a 1920x1080 clip and Slow preset. I get approximately the same speed as yours.

[HolyWu]

Update r18.


SMDegrain: Fix limit and limitc arguments are not scaled to the bit depth of the input.
Bob: Fix the result is always outputted in 16-bit integer mistakenly, affecting QTGMC and ivtc_txt60mc for 8-15 bits input.

[stax76]

Is necessary to put havsfunc.py to Lib\site-packages or is there a way to load it from the script?

[HolyWu]

You can import modules put in arbitary places like this:


import importlib
haf = importlib.machinery.SourceFileLoader('havsfunc', r'C:\Foo & Bar\havsfunc.py').load_module()
haf.QTGMC(...)

[stax76]

:thanks:

[stax76]

I would like to add some presets to StaxRip and could use some help, so far I have only this:

QTGMC Slow in category 'Field'

clip = havsfunc.QTGMC(Input = clip, TFF = True, Preset = 'Slow')


SMDegrain in category 'Noise'

clip = havsfunc.SMDegrain(input = clip)

maybe somebody can suggest some useful presets, I have not really experience with anything included.

[MonoS]

I suggest you to make two version for SD contents and HD contents, QTGMC and degrain are very heavy function and on bd can slow down a lot the encode.

I personally use the settings in this script
https://github.com/MonoS/MonoS-VS-Func/blob/master/MFunc.py

Denoise2 works pretty well with both SD and HD content, be sure to use fast=True when dealing with BD

MQTGMC is a bit to extreme for BD and i'm tweaking its parameter a bit for both SD and HD contents
Here an except from my personal copy

# Check MatchEnhance and/or Sharpness because with SourceMatch the result seems to be a bit less sharper

# has.QTGMC(src16, Preset="Very Slow", SourceMatch=3, MatchPreset="Slow", MatchPreset2="Ultra Fast", Lossless=2, NoisePreset="Slow", TFF=TFF, EZDenoise=EZDenoise, FPSDivisor=FPSDivisor) # SD
# has.QTGMC(src16, Preset="Medium", SourceMatch=3, MatchPreset="Fast", MatchPreset2="Ultra Fast", Lossless=2, NoisePreset="Medium", TFF=TFF, EZDenoise=EZDenoise, FPSDivisor=FPSDivisor) #HD


But maybe a simple
has.QTGMC(src16, Preset="Medium", Denoiser="dfttest", TFF=TFF, EZDenoise=EZDenoise, FPSDivisor=FPSDivisor)

Should be enough for BD [i'll also suggest a downscale after deinterlacing cause usually interlaced bd don't have so much details imho].

MQTGMC is extremely slow, like 3fps on SD contents, the same for the speedupped version on the previous line on HD contents

[foxyshadis]

The SMDegrain wiki page (http://avisynth.nl/index.php/SMDegrain#Ready-to-run_Samples) has a bunch of sample ideas. I personally use:
df = core.dfttest.DFTTest(grainy_clip, sstring="0.0:4.0 0.2:9.0 1.0:15.0", tbsize=1)
smooth = havs.SMDegrain(up,tr=2,contrasharp=True,RefineMotion=True,pel=2,prefilter=df,mfilter=df)
but it is SLOW.

[Boulder]

EDIT: Nevermind, got it working. Note to self: use the 64-bit version of the fftw library :P

[tormento]

Could you please update SMDegrain up to 3.1.2d? It supports KNLMeansCL and I'd like to do some tests against AviSynth and AviSynth+.

[HolyWu]

Update r19.


Add functions santiag, STPresso.
Delete functions ediaa, maa, SharpAAMCmod, GammaToLinear, LinearToGamma.
HQDeringmod: The Y, U, V parameters are replaced with planes parameter.
QTGMC: Delete the option 'FFT3dGPU' from Denoiser parameter, and add KNLMeansCL as another option.
GSMC: The Y, U, V parameters are replaced with planes parameter.
SMDegrain: Update to v3.1.2d.

[Boulder]

Could you consider porting SRestore to Vapoursynth? It's a monster of a function but quite slow in Avisynth as it doesn't like multithreading much.