Vapoursynth

[Myrsloik]

Here's RC1 for R20 (https://dl.dropboxusercontent.com/u/73468194/vapoursynth_r20_rc1.exe).

Report any bugs you find here. I'll post some FrameEval examples later.

Changes:
r20:
rewrote the ugliest threading code, the locking is now a bit more fine grained in general
added frameeval, a function to allow per frame filter evaluation
blankclip now generates a new frame on every request to keep memory usage down, added the keep argument to optionally always return the same frame
the message handler now takes an extra user data pointer, the handler and data pointer is still per process instead of per core though
added plugin autoloading for linux (jackoneill)
added plugin autoloading for windows, autoloaded plugins for all users go into <installdir>\plugins and per user autoloaded plugins go into <appdata>\vapoursynth\plugins
include the python backtrace in errors when available
fixed a bug in the propagation of filter errors, requesting a frame with an error twice no longer makes vapoursynth hang
fixed the error that happens when avisource is used to open a vs script inside a vs script
added override support to vdecimate (nodame)
added -version option to vspipe and other small fixes
addborders now properly rejects clips where the colorspace can change
addborders and blankclip now properly default to black for all colorspaces
added tdeint propery to the prefetch list_functions
fixed a cache bug that would make caches adapt too slowly/not at all
only output the first 200 slow warnings per avisynth filter
don't prefetch any frames by default for avisynth filters

[Tima]

Anyone else has RemoveGrain crash when trying to use QTGMC? :)

[the same was with r19 too]

[Are_]

Long time no testing due to various things and now non of my scripts works :/

I can easily make VS crash with a simple Repari call (0.9 and 1.0).

This script for instance:
import vapoursynth as vs
core = vs.get_core()

core.avs.LoadPlugin(path=r'DGDecodeNV.dll')
core.avs.LoadPlugin(path=r'RepairSSE2.dll')
core.avs.LoadPlugin(path=r'RemoveGrainSSE2.dll')

src = core.avs.DGSource('01.dgi')

main = core.avs.RemoveGrain(src, 1)
main = core.avs.Repair(main, src, 17)

main.set_output()

[Reel.Deel]

Just tested with the script Are_ posted, I can confirm the crash with R20 RC1.
Here's the log (https://www.dropbox.com/s/2c7zswoxrob79d0/crashinfo.txt) from VDub.

[Myrsloik]

Just tested with the script Are_ posted, I can confirm the crash with R20 RC1.
Here's the log (https://www.dropbox.com/s/2c7zswoxrob79d0/crashinfo.txt) from VDub.

Time to look into it. Let's see what evils these plugins do...

[Myrsloik]

Here's R20 RC2 (https://dl.dropboxusercontent.com/u/73468194/vapoursynth_r20_rc2.exe).

It fixes the avisynth compatibility for filters without prefetching. I also added removegrain and repair to the prefetch list since apparently I forgot them.

[Tima]

RC2 still crashes with RemoveGrain.

VirtualDub log (http://pastebin.com/08RYUaNU)

[Myrsloik]

RC2 still crashes with RemoveGrain.

VirtualDub log (http://pastebin.com/08RYUaNU)

I used the same script Are_ posted (but with blankclip as the source). Does that one crash for you as well now?

If it still crashes, can you send me the exact repair and removegrain dlls you are using?

There are simply too many variations of them for me to find and try all. It's also interesting because it now crashes inside removegrain code and not in vapoursynth like like in reel.deel's log.

[Tima]

I used the same script Are_ posted (but with blankclip as the source). Does that one crash for you as well now?

If it still crashes, can you send me the exact repair and removegrain dlls you are using?


I used my own script, similar to Are_'s. It crashes for me with any RemoveGrain version I tried.

src=r'video-raw\\n01-01.avi'
plugins_avs_custom=r'C:\\Bin\\avisynth_plugins_custom\\'

import vapoursynth as vs
import sys
core = vs.get_core(threads = 1)

core.avs.LoadPlugin(plugins_avs_custom + r'RemoveGrainSSE2.dll')

clip = core.ffms2.Source(src)
clip = core.avs.RemoveGrain(clip, 1)

clip.set_output()

RemoveGrain DLL (correct link) (http://rghost.net/49213483)

[Myrsloik]

I used my own script, similar to Are_'s. It crashes for me with any RemoveGrain version I tried.

src=r'video-raw\\n01-01.avi'
plugins_avs_custom=r'C:\\Bin\\avisynth_plugins_custom\\'

import vapoursynth as vs
import sys
core = vs.get_core(threads = 1)

core.avs.LoadPlugin(plugins_avs_custom + r'RemoveGrainSSE2.dll')

clip = core.ffms2.Source(src)
clip = core.avs.RemoveGrain(clip, 1)

clip.set_output()

RemoveGrain DLL (http://rghost.net/49202817)

You uploaded the wrong dll. Try again.

[Tima]

You uploaded the wrong dll. Try again.

Oops, fixed :)

[Myrsloik]

Oops, fixed :)

Odd. That file works on my computer. Head scratching time.

Your cpu does have sse2 support, right?

[Tima]

Odd. That file works on my computer. Head scratching time.

Your cpu does have sse2 support, right?

Yes (BTW it crashes with any RemoveGrain version, including non-SSE).

(relevant system specs: Core i7 940, Win8 x64)

[Myrsloik]

Yes (BTW it crashes with any RemoveGrain version, including non-SSE).

(relevant system specs: Core i7 940, Win8 x64)

Can you run the script with vspipe and see if it prints any error?

Do you have the source code for that version of removegrain or know where i can find it?

[Tima]

Can you run the script with vspipe and see if it prints any error?
No errors in console (just an std warning there is no user-specific plugin auto-loading folder), vspipe just crashes :)

I ran it 10 times and got these errors:
Unhandled exception at 0x0288F6C0 in vspipe.exe: 0xC0000005: Access violation executing location 0x0288F6C0.
Unhandled exception at 0x726F486B in vspipe.exe: 0xC0000005: Access violation executing location 0x726F486B.
Unhandled exception at 0x726F486B in vspipe.exe: 0xC0000005: Access violation executing location 0x726F486B.
Unhandled exception at 0x726F486B in vspipe.exe: 0xC0000005: Access violation executing location 0x726F486B.
Unhandled exception at 0x029A2FB8 in vspipe.exe: 0xC0000005: Access violation executing location 0x029A2FB8.
Unhandled exception at 0x55446D56 (RemoveGrainSSE2.dll) in vspipe.exe: 0xC0000005: Access violation reading location 0x00000014.
Unhandled exception at 0x726F486B in vspipe.exe: 0xC0000005: Access violation executing location 0x726F486B.
Unhandled exception at 0x696C436B in vspipe.exe: 0xC0000005: Access violation executing location 0x696C436B.
Unhandled exception at 0x55446D56 (RemoveGrainSSE2.dll) in vspipe.exe: 0xC0000005: Access violation reading location 0x00000014.
Unhandled exception at 0x726F486B in vspipe.exe: 0xC0000005: Access violation executing location 0x726F486B.

Stack trace hints that RemoveGrainSSE2.dll is involved in every case.

Could you build debug version (with pdb) for me to test?

The version Tima uses is Vit's Modded Plugins (http://forum.doom9.org/showthread.php?t=156028)(source code included in the package).
I wonder whether he tried the original version instead of the modded version.
Correct. I tried all dlls from all three Vit's packages -- each of them crashes.
Stable v0.9 from http://avisynth.nl/index.php/RemoveGrain crashes too.

[Myrsloik]

Here are two compiles of the vit removegrain version (https://dl.dropboxusercontent.com/u/73468194/rg.7z).

Try the release one first and see if it crashes. There's a pdb included for the debug version.

[Tima]

Try the release one first and see if it crashes. There's a pdb included for the debug version.

Both r20-rc1 and r20-rc2 with both builds of RG crash in similar ways:
Unhandled exception at 0x726F486B in vspipe.exe: 0xC0000005: Access violation executing location 0x726F486B.
Unhandled exception at 0x6BEC8DFB (RemoveGrain.dll) in vspipe.exe: 0xC0000005: Access violation reading location 0x00000014.

Stack trace is the same in all cases:
> 726f486b() Unknown
[Frames below may be incorrect and/or missing]
RemoveGrain.dll!RemoveGrain::RemoveGrain(PClip clip, int * mode, bool planar) Line 7565 C++
RemoveGrain.dll!CreateRemoveGrain(AVSValue args, void * user_data, IScriptEnvironment * env) Line 7628 C++
vapoursynth.dll!5bc57a91() Unknown
vapoursynth.dll!5bc44553() Unknown
vapoursynth.dll!5bc47dd3() Unknown
vapoursynth.pyd!5cdef90c() Unknown
vapoursynth.pyd!5cdf24ff() Unknown
python33.dll!1e07bd5b() Unknown
python33.dll!1e0f887c() Unknown
python33.dll!1e0f8ec1() Unknown
python33.dll!1e0faf2d() Unknown
python33.dll!1e0fbf7d() Unknown
python33.dll!1e0fc0e4() Unknown
vapoursynth.pyd!5cdd576a() Unknown
vapoursynth.pyd!5cde039d() Unknown
vapoursynth.pyd!5cde6931() Unknown
vsscript.dll!64fc141b() Unknown
vspipe.exe!000a2360() Unknown
ntdll.dll!77b6dd84() Unknown
ntdll.dll!77b973bc() Unknown
msvcr100.dll!6caf0949() Unknown
vspipe.exe!000a2896() Unknown
vspipe.exe!000a2919() Unknown
vspipe.exe!000a3377() Unknown
msvcr100.dll!6caf263d() Unknown
vspipe.exe!000a2a90() Unknown
kernel32.dll!7528850d() Unknown
ntdll.dll!77b9bf39() Unknown
ntdll.dll!77b9bf0c() Unknown

[Myrsloik]

The removegrain/repair issue has now been debugged. Basically removegrain/repair are horribly broken and will never work well.

I'll try to get a proper port of removegrain done quickly so QTGMC remains usable.

[Myrsloik]

Here's R20 RC3 (https://dl.dropboxusercontent.com/u/73468194/vapoursynth_r20_rc3.exe). This will probably be the final released version.

Changes from RC2:
Checks the input colorspace to avisynth filters, prevents odd crashes under certain conditions
Doesn't keep copies of all filter arguments around

Note that RemoveGrain/Repair violate the avisynth api so they'll never be supported. It's even explicitly stated in the avisynth documentation to not store a pointer to the IScriptEnvironment in a global variable but apparently some people don't read much.

[Mystery Keeper]

I've got WinPython 3.3 installed and its internal python directory in my PATH variable. And installer still says I haven't got python installed.

[the_weirdo]

I've got WinPython 3.3 installed and its internal python directory in my PATH variable. And installer still says I haven't got python installed.

Is your WinPython 32-bit or 64-bit? Vapoursynth currently need 32-bit Python, I think.

[Mystery Keeper]

Solved. Had to open WinPython control panel and "Register distribution".

[Myrsloik]

I've released R20 and have made the usual blog post summary (http://www.vapoursynth.com/2013/10/r20-closer-to-done/). I also ditched the annoying adfly advertisement stuff. Full changelog in the first post in this thread. The release is identical to RC3.

In case anyone's wondering I made like $3.5 in a year...

[Myrsloik]

https://dl.dropboxusercontent.com/u/73468194/error.png
This is in no way inspired by avisynth. Just saying.

[sl1pkn07]

Myrsloik, removegrain/repair port is ready to use/test? not added in a waf build script (i add by myself, but get a lots of error http://sl1pkn07.no-ip.com/paste/view/21e988df (i use this wscript patch: http://sl1pkn07.no-ip.com/paste/view/4e3ff441. but i not sure if is correct))

greetings

edit: the auto [url.][/url.] is broken?

[Myrsloik]

Myrsloik, removegrain/repair port is ready to use/test? not added in a waf build script (i add by myself, but get a lots of error http://sl1pkn07.no-ip.com/paste/view/21e988df (i use this wscript patch: http://sl1pkn07.no-ip.com/paste/view/4e3ff441. but i not sure if is correct))

greetings

It's not added yet because there are some very small non-windows issues. I'll probably fix the last lines tomorrow.

[sl1pkn07]

oks, thanks

[LaTo]

I've released R20 and have made the usual blog post summary (http://www.vapoursynth.com/2013/10/r20-closer-to-done/). I also ditched the annoying adfly advertisement stuff. Full changelog in the first post in this thread. The release is identical to RC3.

In case anyone's wondering I made like $3.5 in a year...

I just read your blog post...
And I'm using a build-in font in my AVS plugins, which is more or less the same principle as vps-scrawl but with alpha-blending.
It is slower to draw but it is nice looking (more modern style).
Let me know if you're interested or not, if so I will post it here...

[Myrsloik]

I just read your blog post...
And I'm using a build-in font in my AVS plugins, which is more or less the same principle as vps-scrawl but with alpha-blending.
It is slower to draw but it is nice looking (more modern style).
Let me know if you're interested or not, if so I will post it here...

Why not make it a plugin so the users can decide? And is the font you're using under a reasonable license?

In my opinion the scrawl features are enough since it's only for debug printing/simple labeling and so on. If you want something better you mas as well go libass or some other proper subtitle renderer.

[jackoneill]

I just read your blog post...
And I'm using a build-in font in my AVS plugins, which is more or less the same principle as vps-scrawl but with alpha-blending.
It is slower to draw but it is nice looking (more modern style).
Let me know if you're interested or not, if so I will post it here...

What does it look like?

[Reel.Deel]

Maybe like the font used in AutoGain's (http://forum.doom9.org/showthread.php?t=167573) or SmoothAdjust's (http://forum.doom9.org/showthread.php?t=154971) elegant debug view mode?

http://latoninf.free.fr/d9/AG/AutoGain_DebugView_thumb.png (http://latoninf.free.fr/d9/AG/AutoGain_DebugView.png)
* Click the image to enlarge *

http://latoninf.free.fr/d9/SA/SCshow_mini.png (http://latoninf.free.fr/d9/SA/SCshow.png)
* SmoothCurve - Click the image to enlarge *

[jackoneill]

Maybe like the font used in AutoGain's (http://forum.doom9.org/showthread.php?t=167573) or SmoothAdjust's (http://forum.doom9.org/showthread.php?t=154971) elegant debug view mode?


Heh, that looks like DejaVu Sans Mono (which isn't a bitmap font, I think).

[LaTo]

AvsFont.7z (http://latoninf.free.fr/d9/AvsFont.7z)

The .7z file contains only a header with the raw data and a small example (pseudo-code, not tested).

Unfortunately I do not have time to do a more complete package,
so you will have to create your classes and functions yourself.

[sneaker_ger]

I got a strange issue running R20 and Chikuzen's ImageReader 0.2.1. I'm trying to open this image (http://www.parabolaresearch.com/mona_lisa_original.jpg). When using the VfW interface the application simply closes immediately, no error message or anything (tested VirtualDub and MPC-HC). vspipe shows the correct info but if you try to pipe it shows:
No frame returned at the end of processing by
QWaitCondition: Destroyed while threads are still waiting

[Myrsloik]

I got a strange issue running R20 and Chikuzen's ImageReader 0.2.1. I'm trying to open this image (http://www.parabolaresearch.com/mona_lisa_original.jpg). When using the VfW interface the application simply closes immediately, no error message or anything (tested VirtualDub and MPC-HC). vspipe shows the correct info but if you try to pipe it shows:
No frame returned at the end of processing by
QWaitCondition: Destroyed while threads are still waiting

That's odd. Will test it tomorrow. It should at least print the name of the offending filter.

[Myrsloik]

I got a strange issue running R20 and Chikuzen's ImageReader 0.2.1. I'm trying to open this image (http://www.parabolaresearch.com/mona_lisa_original.jpg). When using the VfW interface the application simply closes immediately, no error message or anything (tested VirtualDub and MPC-HC). vspipe shows the correct info but if you try to pipe it shows:
No frame returned at the end of processing by
QWaitCondition: Destroyed while threads are still waiting

The issue is that vsimagereader accesses a NULL pointer. I think Chikuzen will post an updated version later today.

[Myrsloik]

Here's a test version (https://dl.dropboxusercontent.com/u/73468194/vapoursynth_r21_test1.exe) with many changes and improvements.

Notable changes since R20:

now properly flips compatrgb32 when converting, previously conversions to/from would make the picture upside down
optimized 8 bit merge and maskedmerge
fixed crash bugs in planeaverage/difference/cliptoprop/proptoclip introduced in r20
now flattens frame properties as well, this is the proper documented behavior
merged all selectclip functionality into frameeval, selectclip was removed as a separate function
vsvfw now returns a clip with the error message if there's an error
added a basic text printing filter to the core, code written by nodame
it's now possible to use dir() on frame.props and core.namespace in python to list interesting properties and attributes
added a compiled version of assvapour, it's probably a bit broken when it comes to font selection at the moment
ported removegrain/repair based on dither tools' code, support 8-16 bit input
fixed a cache issues where it would shrink to far too small sizes
fixed several bugs in shuffleplanes, now it will correctly deduce the dimensions of the output in more cases and properly reject variable size/format input
fixed a rare hang that could happen in avisynth compatibility when only using one thread
adjusted avisynth compat prefetching lists, specifically MCompensate should be faster now

Help test the changes. This build should be better than R20 in every way.

[gpower2]

Will we see any speed difference with this version, compared to a traditional AviSynth script?

Especially in true multicore processors...

[aegisofrime]

Hi,

I seem to be getting strange artifacts with QTGMC on r21 that wasn't present on r19 (never tried r20):

Good (r19):

http://i.imgur.com/5bbXG3wl.jpg (http://i.imgur.com/5bbXG3w.jpg)

Bad (r21):

http://i.imgur.com/ZKK1JPLl.jpg (http://i.imgur.com/ZKK1JPL.jpg)

Note: The artifacts aren't as obvious on the thumbnail, click on thumbnail for full size image in which the artifacts are more obvious.

Here's my script that I used for r21, just in case it's a screwup on my end (which is probably very likely)


import vapoursynth as vs
import havsfunc
import sys
core = vs.get_core(threads=4)
haf = havsfunc.HAvsFunc(core)
core.std.LoadPlugin(r'C:\Program Files (x86)\VapourSynth\filters\temporalsoften.dll')
core.std.LoadPlugin(r'C:\Program Files (x86)\VapourSynth\filters\libnnedi3.dll')
core.std.LoadPlugin(r'C:\Program Files (x86)\VapourSynth\filters\fmtconv.dll')
core.std.LoadPlugin(r'C:\Program Files (x86)\VapourSynth\filters\scenechange.dll')
core.std.LoadPlugin(r'C:\Program Files (x86)\VapourSynth\filters\temporalsoften2.dll')
core.std.LoadPlugin(r'C:\Program Files (x86)\VapourSynth\filters\genericfilters.dll')
core.avs.LoadPlugin(r"C:\Program Files (x86)\AviSynth 2.5\plugins\DGDecode.dll")
core.avs.LoadPlugin(path=r'C:\Program Files (x86)\VapourSynth\filters\avisynthfilters.dll')
core.avs.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools-25.dll')
core.avs.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\mvtools2.dll')
core.avs.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\RemoveGrainSSE3.dll')
core.avs.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\FFT3DFilter.dll')
core.avs.LoadPlugin(path=r'C:\DGAVCIndex\DGAVCDecode.dll')
clip = core.avs.AVCSource(dga =r"I:\Test\Test.dga")
clip = haf.QTGMC(clip, Preset='Slow', SubPel=2,TFF=True)
clip.set_output(index = 0)


For r21 I commented out the two temporalsoften filters, moving the temporalfilter2.dll file to the autoload folder, since somehow if I load them manually it throws an error.

For r19 I am able to load them successfully without an issue.

Am I doing something wrong?

P.S: Apologies for the large image size as it was what I could get quickest!

[Myrsloik]

1. There's a bug in merge() in the test version. That's where the artifacts come from.
2. You get the error because you're probably loading the same thing several times.
3. Use the native vs removegrain/repair. The avisynth one is too broken to work.

I'll probably post a new test build tonight after I test all the new as properly.

[Myrsloik]

Here's a fixed test build (https://dl.dropboxusercontent.com/u/73468194/vapoursynth-r21-test2.exe). No other changes.

[Myrsloik]

http://i.imgur.com/a1nWNYM.png
Is cache size not allowed to be set >= 2048MB?

It is allowed. But there may be bugs related to it. I'll check and fix the code.

Note that vspipe.exe is flagged to support 3GB address space but even then you need to leave a bit as margin. VS only counts allocated frames towards the memory use and ignores the rest.

[Myrsloik]

Here's R21 RC1 (https://dl.dropboxusercontent.com/u/73468194/vapoursynth-r21-RC1.exe).

Changes since test 2:
fixed setting maximum memory use over 2GB from python
added an additional output format argument to modifyframe, also added checks to prevent crashes
now completely rejects adding known to be hopelessly broken functions from avisynth plugins (removegrain/repair/iscombed/colormatrix)

I will be going C++11/C99 for R22. There are simply too many useful things in there to skip it. It's also a long term (veeeeeery long term) path to not relying on Qt at all.

Don't forget to take a look at the bug tracker (https://github.com/vapoursynth/vapoursynth/issues?state=open) as well. There are some things like documentation and not too difficult programming tasks I need help with. Such improving the Expr filter (https://github.com/vapoursynth/vapoursynth/issues/3).

[Wilbert]

What's broken in ColorMatrix?

[Myrsloik]

What's broken in ColorMatrix?

It's only broken in VS. If I recall correctly it uses invoke internally to do fancy things and I'm not going to bother implementing that part of avisynth.

Likewise IsCombed is conceptually broken as well. You can run it just fine but you'll never be able to use its output.

[Mystery Keeper]

Thank you for a great work, Mysrloik. Could you please make an example of plugin working with different bit depths and both integer and real color component values?

[Myrsloik]

Thank you for a great work, Mysrloik. Could you please make an example of plugin working with different bit depths and both integer and real color component values?

Would extending the invert example (https://github.com/vapoursynth/vapoursynth/blob/master/sdk/invert_example.c) be enough for you or do you have a particular simple filtering operation in mind? The other place to look is at the existing internal filters. For example MaskedMerge (https://github.com/vapoursynth/vapoursynth/blob/master/src/core/simplefilters.c#L3491) is a very straightforward filter that support 8-16 bit int and 32 bit float.

For integer types there's really not much difference. Everything just has more significant bits.

Floating point is slightly different. There RGB and Y channels are in the 0 to 1 range. And UV/CoCg are -0.5 to 0.5 range.

[Mystery Keeper]

Ugh. Thank you for the hint. But IMO that source really needs to be broken into several smaller ones.

[Mystery Keeper]

Trying to port TempLinearApproximate to VapourSynth. So far it works well on Y plane, but produces weird result on U and V (plane=6). What am I doing wrong?
http://paste.org.ru/?tgifsl
Also, for some reason it is much slower than in AvySynth.

[Myrsloik]

Trying to port TempLinearApproximate to VapourSynth. So far it works well on Y plane, but produces weird result on U and V (plane=6). What am I doing wrong?
http://paste.org.ru/?tgifsl
Also, for some reason it is much slower than in AvySynth.

You're using the R21 RC, right? R20 has a cache bug that can make it very slow.

I'll compile and take a look at what exactly the code does. A bit later today...